Small sample learning and out-of-distribution detection method and system based on global and local image-text alignment

Abstract

The invention discloses a small sample learning and out-of-distribution detection method and system based on global and local image-text alignment, and the method comprises the steps: collecting image data and corresponding class labels, obtaining a data set, processing the data set, and dividing the data set into a training set, an in-distribution test set, and an out-of-distribution test set; an out-of-distribution detection model is constructed and trained, the training comprises two stages, in the pre-training stage, text description is carried out on various image data of a training set, text description is obtained, in the training stage, global and local image-text features are extracted, and related and unrelated local image features are screened out; performing local supervised contrast learning on the fine-tuned text and related local image features, calculating global and local matching scores by using the image features and the fine-tuned text features, and completing model training; and performing classification prediction probability detection on the trained model. According to the invention, the performance of small sample image classification and small sample target detection is improved at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and specifically relates to a small sample learning and out-of-distribution detection method and system based on global and local image-text alignment. Background Art

[0002] With the rapid development of deep learning technology, image classification tasks have made significant progress. However, image classification models generally rely on large-scale training data. In practical scenarios, collecting high-quality large-scale training data is often costly or difficult to achieve; in addition, the collected training data is usually difficult to cover all categories. This may not only lead to model overfitting, but may also cause the model to incorrectly classify data from categories not covered by the training set (out-of-distribution data) as known categories, causing serious security issues. Therefore, small-sample image classification and small-sample out-of-distribution detection are of great significance. The core goal of small-sample image classification is to reduce overfitting and achieve higher classification performance under the condition of only a small number of training samples; small-sample out-of-distribution detection aims to achieve effective detection of out-of-distribution data under the condition of limited samples, thereby ensuring the security and reliability of the system.

[0003] Existing methods for small-shot image classification and small-shot out-of-distribution detection based on visual language models (VLMs) have made significant progress. However, these methods still have problems and defects that cannot be ignored. Small-shot image classification methods based on visual language models often rely on the assumption that images and texts are aligned, which is often difficult to hold in real scenarios, resulting in low accuracy. Research on small-shot out-of-distribution detection methods based on visual language mainly focuses on prompt tuning. When the similarity between out-of-distribution samples and in-distribution samples is high, this method cannot capture the subtle differences between the two, resulting in poor out-of-distribution detection performance. In addition, in existing research, small-shot image classification and small-shot out-of-distribution detection are still in two relatively separate fields. Having excellent classification performance and out-of-distribution detection performance is important for practical applications, but no method has yet explored ways to improve the performance of both at the same time. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a small sample learning and out-of-distribution detection method and system based on global and local image-text alignment. By enriching semantic prior knowledge, constructing global and local image-text alignment, and improving image-text similarity measurement, the problems of low image-text alignment and inability to distinguish subtle differences between out-of-distribution and in-distribution data in the existing technology are alleviated, thereby achieving the simultaneous improvement of the performance of small sample image classification and small sample target detection.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for small-sample learning and out-of-distribution detection based on global and local image-text alignment, comprising the following steps:

[0007] Collecting image data and corresponding category labels to obtain a dataset, processing the dataset and dividing the dataset into a training set, an in-distribution test set, and an out-of-distribution test set, wherein the categories of the training set and the in-distribution test set are consistent, and each category in the training set includes a small number of training images;

[0008] Build an out-of-distribution detection model. Model training includes pre-training and training stages.

[0009] In the pre-training stage, various types of image data in the training set are described in text to obtain text descriptions; in the training stage, image features of various types of image data and text features of text descriptions are extracted, wherein the image features and text features both include global features and local features, image features are screened according to correlation, relevant local image features and irrelevant local image features are obtained, and text features are fine-tuned using the relevant local image features to obtain fine-tuned text features; positive and negative sample pairs are constructed using the relevant local image features and irrelevant local image features, and local supervised contrastive learning is performed; global and local matching scores are calculated using the image features and the fine-tuned text features, the category with the highest matching score is selected as the predicted category, and the cross entropy loss is calculated;

[0010] The predicted category gradient is passed back to update the out-of-distribution detection model, and multiple rounds of training are performed until the model converges to obtain a trained out-of-distribution detection model; the in-distribution test set and the out-of-distribution test set are input into the trained out-of-distribution detection model to obtain the classification prediction probability, and the out-of-distribution samples are detected and analyzed based on the classification prediction probability.

[0011] As an optimal technical solution, the out-of-distribution detection model includes: a text enhancement module, an image feature extractor, a text feature extractor, a text feature fine-tuning module, a local supervised contrastive learning module and a global local image-text matching module; the text feature fine-tuning module includes a self-attention layer, a cross-attention layer and a two-layer feedforward network with layer normalization.

[0012] As a preferred technical solution, the model performs a text description on various types of image data in the training set before training, including: using the text enhancement module to describe each category based on the brief information provided by the training set and the input text prompts to obtain a text description; the brief information includes information on the number of category labels and the name of the category label.

[0013] As a preferred technical solution, the extraction of image features of various types of image data and text features of text descriptions includes:

[0014] Use the text feature extractor to extract text features and obtain global text features and local text features;

[0015] The image feature extractor is used to extract image features to obtain global image features and local image features.

[0016] As a preferred technical solution, the method of screening image features based on correlation includes:

[0017] Remove visual information irrelevant to the category from image features;

[0018] The average value of the text features is calculated to obtain the average text features. The relevant local image features are selected according to the cosine similarity between each image feature and the average text features, and the irrelevant local image features are selected according to the cosine similarity between each image feature and the average text features.

[0019] As a preferred technical solution, the method of fine-tuning text features using relevant local image features includes:

[0020] Input the relevant local image features and text features into the text feature fine-tuning module, use the text features as the query, and the relevant local image features as the key and value, and fuse the output of the text feature fine-tuning module with the corresponding text features to obtain the fine-tuned text features;

[0021] The output of the text feature fine-tuning module is fused with the corresponding text features as follows: where t′ c，j is the output of the text feature fine-tuning module, tc j is the corresponding text feature, and α is the fusion weight.

[0022] As a preferred technical solution, for performing local supervised contrastive learning using a local supervised contrastive learning module, the local supervised contrastive loss is calculated as follows:

[0023]

[0024] where P(s) is a set of visual embedding indices whose corresponding visual embeddings share the same class label as the visual embedding indexed by s, the set of uncorrelated local image features from all images in the batch share the out-of-distribution class label, g(·,·) is the cosine similarity function, τ is the temperature scaling factor, exp is the exponential function with base e, g(,) is the cosine similarity function, and z s is the sth local image feature contained in the current batch, z p for and z s The pth local image feature with the same label, z a is the ath local image feature of the current batch.

[0025] As a preferred technical solution, the calculation of global and local matching scores using image features and fine-tuned text features includes:

[0026] The global local image-text matching module is used to select image features of several categories, and the selected image features are adjusted according to the importance weights, as shown in the following formula:

[0027] Importance Weight:

[0028] Calculate the global and local matching scores of the selected image features and each fine-tuned text feature as follows:

[0029] Global and local matching scores:

[0030] Represents the relevant local image technical features, represents the relevant local image features, z0 represents the global image features, Represents local text features, Represents local text features, represents the global text feature based on class name, h(·,·) is the cosine similarity, and M is the number of categories of the fine-tuned text feature;

[0031] Select the category with the highest matching score as the predicted category and calculate the cross entropy loss function

[0032] The total loss for training is calculated by applying constraints on the updated visual and textual embeddings to ensure that they are consistent with the corresponding original CLIP embeddings to reduce overfitting. The total loss is in and is the L1 regularization term;

[0033] Gradient backpropagation updates the parameters of the last layer of the text fine-tuning module and the image encoder, while other parameters remain frozen.

[0034] As a preferred technical solution, the calculation of the classification prediction probability includes:

[0035]

[0036] If the score is lower than the threshold, it is determined that the sample does not belong to any category in the training set and is out-of-distribution data; if the score is higher than the threshold, the corresponding category with the highest predicted probability is selected as the classification result output.

[0037] In a second aspect, the present invention further provides a small sample learning and out-of-distribution detection system based on global and local image-text alignment, which is applied to the small sample learning and out-of-distribution detection method based on global and local image-text alignment, including a data acquisition module, a model training module, and a detection and classification module;

[0038] A data acquisition module is used to collect image data and corresponding category labels to obtain a dataset, preprocess the dataset, and divide the dataset into a training set, an in-distribution test set, and an out-of-distribution test set. The categories of the training set and the in-distribution test set are consistent, and each category in the training set includes a small number of training images;

[0039] The model training module is used to build an out-of-distribution detection model. Before model training, various types of image data in the training set are described in text to obtain text descriptions. During model training, image features of various types of image data and text features of text descriptions are extracted, and the image features and text features include global features and local features. Image features are screened according to relevance to obtain relevant image features and irrelevant image features. The text features are fine-tuned using the relevant image features to obtain fine-tuned text features. Positive and negative sample pairs are constructed using relevant local image features and irrelevant local image features, and local supervised contrastive learning is performed. Global and local matching scores are calculated using image features and fine-tuned text features, and the category with the highest matching score is selected as the predicted category, and the cross-entropy loss is calculated.

[0040] The out-of-distribution detection model includes a text enhancement module, an image feature extractor, a text feature extractor, a text feature fine-tuning module, a local supervised contrastive learning module, and a global local image-text matching module; the text feature fine-tuning module includes a self-attention layer, a cross-attention layer, and a two-layer feedforward network with layer normalization;

[0041] The detection and classification module is used to pass the predicted category gradient back to update the out-of-distribution detection model, perform multiple rounds of training until the model converges, and obtain a trained out-of-distribution detection model; input the in-distribution test set and the out-of-distribution test set into the trained out-of-distribution detection model to obtain the classification prediction probability, and detect and analyze the out-of-distribution samples based on the classification prediction probability.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] The small-sample learning and out-of-distribution detection method based on global and local image-text alignment proposed in this paper solves the problem of poor alignment of visual language models on challenging downstream tasks. For the first time, a unified framework is used to simultaneously improve the performance of small-sample image classification and out-of-distribution detection, and it exceeds the current optimal method in both tasks, significantly improving the model's classification performance and out-of-distribution detection capabilities, and enhancing the reliability of deep learning references. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 This is a flowchart of a method for small-sample learning and out-of-distribution detection based on global and local image-text alignment according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a text feature enhancement module according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of local supervised contrastive learning according to an embodiment of the present invention;

[0048] Figure 4 Schematic diagram of global and local image-text matching according to an embodiment of the present invention;

[0049] Figure 5 Schematic diagram of the structure of a small sample learning and out-of-distribution detection system based on global and local image-text alignment in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0052] See also Figure 1 This embodiment provides a small sample learning and out-of-distribution detection method based on global and local image-text alignment, including the following steps:

[0053] S1. Collect image data and corresponding category labels from public datasets. After data preprocessing, data cleaning and organization, construct a training set for training, an in-distribution test set for testing classification performance, and an out-of-distribution test set for testing out-of-distribution detection.

[0054] The training set and the in-distribution test set have the same categories, while the out-of-distribution test set and the training set have no overlapping categories. The training set contains only a small number of training images for each category, while the test set contains all the images used for testing for each category.

[0055] S2. Build and train an out-of-distribution detection model.

[0056] In this embodiment, the out-of-distribution detection model includes a text enhancement module, an image feature extractor, a text feature extractor, a text feature fine-tuning module, a local supervised contrastive learning module and a global local image-text matching module; the text feature fine-tuning module includes a self-attention layer, a cross-attention layer and a two-layer feedforward network with layer normalization.

[0057] In order to better implement the present invention, this embodiment adopts two stages of training, including a pre-training stage and a training stage:

[0058] like Figure 2 As shown, in the pre-training phase, the text enhancement module generates detailed text descriptions for each training category. These text descriptions are fed into the text feature extractor, generating a set of global and local text features for each category. Image data is fed into the image feature extractor to generate a set of global and local image features. The text features are optimized using the text feature fine-tuning module, and the image encoder is optimized using local supervised contrastive learning. The local and global image text features are fed into the image-text matching module to generate predicted probabilities.

[0059] During the training phase, the process includes the following steps:

[0060] S21. Use a large language model to obtain detailed category descriptions to represent the visual characteristics of each category.

[0061] Specifically, we first provided a brief description of the training dataset, primarily including the number and names of the categories. Then, by entering the prompt "Generate questions to classify images that are from {Dataset Description}.", we guided the large language model to generate targeted questions describing each category. These generated questions then served as input to the large language model to generate detailed category descriptions. For each category, the large language model generated a total of 50 descriptions, including the general "A photo of {Class Name}", for a total of 51 text descriptions.

[0062] S22, using a text feature extractor to extract text features, where t c，0 It is the text feature corresponding to “A photo of {Class Name}”, representing the global text information.

[0063] S23, using the image encoder to obtain image features, Z = [z0, ..., z N ], where z0 corresponds to the global image feature and the rest correspond to the local image features.

[0064] S24. Fine-tune text features using visual features.

[0065] In order to optimize text features using category-related visual information from the input image, irrelevant visual information appearing in background (i.e., category-related) regions should be discarded. Here, we simply i (i=1,...,N) and average text features The cosine similarity between the two is used to select a subset of 50 local visual features that are more likely to come from category-related regions. The selected 50 local visual features and 51 text features are fed into the text feature fine-tuning module, as shown in Figure 1 (3). For the cross-attention layer, the text embedding is used as the query and the visual embedding is used as the key and value. The output t′ from the text feature fine-tuning module c，j and the corresponding text embedding tc j by The final text embedding of the j-th text description of the c-th category is fused in the form of , where α is a balancing hyperparameter and the optimal value is 0.99.

[0066] S25. Locally supervised contrastive learning.

[0067] like Figure 3 As shown, for each training image, in addition to the category-related visual embeddings selected as described in step S24 above, another subset of visual embeddings (containing 50 visual embeddings) that are least similar to the average text embedding is selected in a similar way. Given a mini-batch training set, assuming S is the total number of visual embeddings selected from all images in the mini-batch set, calculate the local supervision contrast loss:

[0068]

[0069] Where P(s) is a set of visual embedding indices where the corresponding visual embedding shares the same class label as the visual embedding at index s. The set of most dissimilar visual embeddings from all images in the batch share a special new class label (i.e., out-of-distribution class). g(·,·) is the similarity metric function (cosine similarity by default), τ is the temperature scaling factor, exp is the exponential function with base e, g(,) is the cosine similarity function, and z s is the sth local image feature contained in the current batch, z p for and z s The pth local image feature with the same label, z a is the ath local image feature of the current batch.

[0070] S26: Perform global and local image-text matching.

[0071] like Figure 4 As shown, for the visual embedding set Z obtained in step S23, Z = [z0, ..., z N ] and fine-tuned text features First, as described in step S24 above, 50 class-related visual embeddings are selected Then adjust the importance of each selected visual embedding and each text embedding, the importance weight is calculated as

[0072]

[0073] Then, the global and local image-text matching scores are calculated as follows:

[0074]

[0075] in, Represents the relevant local image technical features, represents the relevant local image features, z0 represents the global image features, Represents local text features, Represents local text features, represents the global text feature based on class name, h(·,·) is the cosine similarity, and M is the number of categories of the fine-tuned text feature;

[0076] Select the category with the highest matching score as the predicted category and calculate the cross entropy loss function

[0077] The total loss for training is calculated by applying constraints on the updated visual and textual embeddings to ensure that they are consistent with the corresponding original CLIP embeddings to reduce overfitting. The total loss is in and is an L1 regularization term used to impose constraints on the updated visual and text embeddings, ensuring that they are consistent with the corresponding original CLIP embeddings to reduce overfitting and backpropagate updated parameters.

[0078] Repeat the above steps S23-S26 until the model training reaches the expected stable state.

[0079] S3. Image category prediction and out-of-distribution detection.

[0080] The images to be predicted are fed into the model trained in step S2 in sequence to obtain the predicted probability of classification. The out-of-distribution score is obtained based on the predicted probability:

[0081]

[0082] If the score is lower than the threshold, it is determined that the sample does not belong to any category in the training set and is out-of-distribution data; if the score is higher than the threshold, the corresponding category with the highest predicted probability is selected as the classification result output.

[0083] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0084] Based on the same idea as the small sample learning and out-of-distribution detection method based on global and local image-text alignment in the above-mentioned embodiment, the present invention also provides a small sample learning and out-of-distribution detection system based on global and local image-text alignment, which can be used to execute the above-mentioned small sample learning and out-of-distribution detection method based on global and local image-text alignment. For ease of explanation, the structural diagram of the embodiment of the small sample learning and out-of-distribution detection system based on global and local image-text alignment only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0085] See also Figure 5 In another embodiment of the present application, a small sample learning and out-of-distribution detection system 10 based on global and local image-text alignment is provided, which includes a data acquisition module 11, a model training module 12, and a detection and classification module 13;

[0086] A data acquisition module 11 is configured to acquire image data and corresponding category labels to obtain a dataset, preprocess the dataset, and divide the dataset into a training set, an in-distribution test set, and an out-of-distribution test set. The training set and the in-distribution test set have the same categories, and each category in the training set includes a small number of training images.

[0087] The model training module 12 is used to build an out-of-distribution detection model. Before model training, various types of image data in the training set are described in text to obtain text descriptions. During model training, image features of various types of image data and text features of text descriptions are extracted, wherein the image features and text features include global features and local features. Image features are screened according to relevance to obtain relevant image features and irrelevant image features. The text features are fine-tuned using the relevant image features to obtain fine-tuned text features. Positive and negative sample pairs are constructed using relevant local image features and irrelevant local image features, and local supervised contrastive learning is performed. Global and local matching scores are calculated using image features and fine-tuned text features, the category with the highest matching score is selected as the predicted category, and cross-entropy loss is calculated.

[0088] The out-of-distribution detection model includes a text enhancement module, an image feature extractor, a text feature extractor, a text feature fine-tuning module, a local supervised contrastive learning module, and a global local image-text matching module; the text feature fine-tuning module includes a self-attention layer, a cross-attention layer, and a two-layer feedforward network with layer normalization;

[0089] The detection and classification module 13 is used to pass the predicted category gradient back to update the out-of-distribution detection model, perform multiple rounds of training until the model converges, and obtain a trained out-of-distribution detection model; input the in-distribution test set and the out-of-distribution test set into the trained out-of-distribution detection model to obtain the classification prediction probability, and detect and analyze the out-of-distribution samples based on the classification prediction probability.

[0090] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A small sample learning and out-of-distribution detection method based on global and local image-text alignment, characterized by: The steps include: Collecting image data and corresponding category labels to obtain a dataset, processing the dataset and dividing the dataset into a training set, an in-distribution test set, and an out-of-distribution test set, wherein the categories of the training set and the in-distribution test set are consistent, and each category in the training set includes a small number of training images; Build an out-of-distribution detection model. Model training includes pre-training and training stages. In the pre-training stage, various types of image data in the training set are described in text to obtain text descriptions; in the training stage, image features of various types of image data and text features of text descriptions are extracted, wherein the image features and text features both include global features and local features, image features are screened according to correlation, relevant local image features and irrelevant local image features are obtained, and text features are fine-tuned using the relevant local image features to obtain fine-tuned text features; positive and negative sample pairs are constructed using the relevant local image features and irrelevant local image features, and local supervised contrastive learning is performed; global and local matching scores are calculated using the image features and the fine-tuned text features, the category with the highest matching score is selected as the predicted category, and the cross entropy loss is calculated; The predicted category gradient is passed back to update the out-of-distribution detection model, and multiple rounds of training are performed until the model converges to obtain a trained out-of-distribution detection model; the in-distribution test set and the out-of-distribution test set are input into the trained out-of-distribution detection model to obtain the classification prediction probability, and the out-of-distribution samples are detected and analyzed based on the classification prediction probability.

2. The method for small-sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 1, characterized in that: The out-of-distribution detection model includes: a text enhancement module, an image feature extractor, a text feature extractor, a text feature fine-tuning module, a local supervised contrastive learning module and a global local image-text matching module; the text feature fine-tuning module includes a self-attention layer, a cross-attention layer and a two-layer feedforward network with layer normalization.

3. The method for small-sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 2, characterized in that: Before the model is trained, a text description is performed on various types of image data in the training set, including: using the text enhancement module to describe each category based on the brief information provided by the training set and the input text prompts to obtain a text description; the brief information includes information on the number of category labels and the name of the category labels.

4. The method for small sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 2, characterized in that: The extraction of image features of various types of image data and text features of text descriptions includes: Use the text feature extractor to extract text features and obtain global text features and local text features; The image feature extractor is used to extract image features to obtain global image features and local image features.

5. The method for small sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 2, characterized in that: The screening of image features according to correlation includes: Remove visual information irrelevant to the category from image features; The average value of the text features is calculated to obtain the average text features. The relevant local image features are selected according to the cosine similarity between each image feature and the average text features, and the irrelevant local image features are selected according to the cosine similarity between each image feature and the average text features.

6. The method for small-sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 2, characterized in that: The fine-tuning of text features using relevant local image features includes: Input the relevant local image features and text features into the text feature fine-tuning module, use the text features as the query, and the relevant local image features as the key and value, and fuse the output of the text feature fine-tuning module with the corresponding text features to obtain the fine-tuned text features; The output of the text feature fine-tuning module is fused with the corresponding text features as follows: where t′ c，j is the output of the text feature fine-tuning module, tc j is the corresponding text feature, and α is the fusion weight.

7. The method for small sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 2, characterized in that: For local supervised contrastive learning using the local supervised contrastive learning module, including calculating the local supervised contrastive loss, it is as follows: where P(s) is a set of visual embedding indices whose corresponding visual embeddings share the same class label as the visual embedding indexed by s, the set of uncorrelated local image features from all images in the batch share the out-of-distribution class label, g(·,·) is the cosine similarity function, τ is the temperature scaling factor, exp is the exponential function with base e, g(,) is the cosine similarity function, and z s is the sth local image feature contained in the current batch, z p for and z s The pth local image feature with the same label, z a is the ath local image feature of the current batch.

8. The method for small-sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 2, characterized in that: The global and local matching scores are calculated using the image features and the fine-tuned text features, including: The global local image-text matching module is used to select image features of several categories, and the selected image features are adjusted according to the importance weights, as shown in the following formula: Importance Weight: Calculate the global and local matching scores of the selected image features and each fine-tuned text feature as follows: Global and local matching scores: Represents the relevant local image technical features, represents the relevant local image features, z0 represents the global image features, Represents local text features, Represents local text features, represents the global text feature based on class name, h(·,·) is the cosine similarity, and M is the number of categories of the fine-tuned text feature; Select the category with the highest matching score as the predicted category and calculate the cross entropy loss function The total loss for training is calculated by applying constraints on the updated visual and textual embeddings to ensure that they are consistent with the corresponding original CLIP embeddings to reduce overfitting. The total loss is in and is the L1 regularization term; Gradient backpropagation updates the parameters of the last layer of the text fine-tuning module and the image encoder, while other parameters are frozen.

9. The method for small sample learning and out-of-distribution detection based on global and local image-text alignment according to claim 1, characterized in that: The calculation of the classification prediction probability includes: If the score is lower than the threshold, it is determined that the sample does not belong to any category in the training set and is out-of-distribution data; if the score is higher than the threshold, the corresponding category with the highest predicted probability is selected as the classification result output.

10. A small sample learning and out-of-distribution detection system based on global and local image-text alignment, characterized by: A small sample learning and out-of-distribution detection method based on global and local image-text alignment applied to any one of claims 1-9, comprising a data acquisition module, a model training module, and a detection and classification module; A data acquisition module is used to collect image data and corresponding category labels to obtain a dataset, preprocess the dataset, and divide the dataset into a training set, an in-distribution test set, and an out-of-distribution test set. The categories of the training set and the in-distribution test set are consistent, and each category in the training set includes a small number of training images; The model training module is used to build an out-of-distribution detection model. Before model training, various types of image data in the training set are described in text to obtain text descriptions. During model training, image features of various types of image data and text features of text descriptions are extracted, and the image features and text features include global features and local features. Image features are screened according to relevance to obtain relevant image features and irrelevant image features. The text features are fine-tuned using the relevant image features to obtain fine-tuned text features. Positive and negative sample pairs are constructed using relevant local image features and irrelevant local image features, and local supervised contrastive learning is performed. Global and local matching scores are calculated using image features and fine-tuned text features, and the category with the highest matching score is selected as the predicted category, and the cross-entropy loss is calculated. The out-of-distribution detection model includes a text enhancement module, an image feature extractor, a text feature extractor, a text feature fine-tuning module, a local supervised contrastive learning module, and a global local image-text matching module; the text feature fine-tuning module includes a self-attention layer, a cross-attention layer, and a two-layer feedforward network with layer normalization; The detection and classification module is used to pass the predicted category gradient back to update the out-of-distribution detection model, perform multiple rounds of training until the model converges, and obtain a trained out-of-distribution detection model; input the in-distribution test set and the out-of-distribution test set into the trained out-of-distribution detection model to obtain the classification prediction probability, and detect and analyze the out-of-distribution samples based on the classification prediction probability.

Images