Visual language model distribution external detection method and device based on forced cue word learning

Abstract

The invention discloses a visual language model out-of-distribution detection method and device based on forced cue word learning. The method comprises the steps that an image data set in downstream task distribution is acquired and divided into a small sample training set and an in-out-of-distribution test set; constructing a visual language model distribution external detection framework, respectively obtaining a forced text feature and an initial text feature by using a forced cue word and an initial cue word, and calculating cross entropy loss in combination with image features; initializing parameters of an image and text encoder end by using a pre-trained visual language model weight, and performing end-to-end iterative training on a training set in constructed small sample distribution to optimize loss; and inputting the in-distribution test set and the out-distribution test set into the trained visual language type, combining the image-text features output by the model with the out-distribution detection score, and outputting an out-distribution detection result. According to the method, the ability of detecting out-of-distribution samples by the model is effectively improved while the classification performance of the visual language model is not sacrificed.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and in particular relates to a method and device for out-of-distribution detection of a visual language model based on forced cue word learning. Background Art

[0002] The ability to detect out-of-distribution data is crucial for the reliability of AI systems. However, solving this problem becomes more difficult in areas such as intelligent diagnosis where only a small amount of labeled data is available, or in areas such as edge computing where computing resources are limited. This has driven research on the problem of out-of-distribution detection in small samples.

[0003] To address the out-of-distribution detection problem, previous out-of-distribution detection methods, such as those by Dan Hendrycks et al. in "A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks" and "Scaling Out-of-Distribution Detection for Real-World Settings," have achieved significant success by analyzing the model's output or internal state to identify out-of-distribution samples, given sufficient training data. However, these methods also have limitations, such as requiring significant computational resources and annotation costs for training, which has hindered the advancement of small-sample out-of-distribution detection.

[0004] With the development of large pre-trained visual language models (such as the CLIP model proposed by Alec Radford et al. in "Learning Transferable Visual Models From Natural Language Supervision"), small-shot OOD detection has achieved remarkable performance. Existing CLIP-based OOD detection methods mainly focus on improving the performance of the model by learning out-of-distribution knowledge, but such methods have many drawbacks. For example, Wang et al. in "CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No" improved OOD detection by relying on training on a large-scale external auxiliary dataset, which requires a lot of resource overhead. To eliminate reliance on external datasets, prompt learning methods such as "LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning" by Atsuyuki Miyai et al., "ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection" by Bai et al., and "Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection" by Yu et al. propose learning out-of-distribution-related knowledge based solely on exposed outliers in the in-distribution training data (such as background regions in in-distribution training images). However, these out-of-distribution-related features extracted from specific regions are difficult to match with the real out-of-distribution data encountered in practice. In addition, "Learning Transferable Negative Prompts for Out-of-Distribution Detection" proposed by Li et al. aims to learn negative prompts that are semantically opposite to the in-distribution class labels. However, these limited negative prompts are often insufficient to capture the differences between the diverse out-of-distribution and in-distribution data. In summary, improving out-of-distribution detection capabilities by learning out-of-distribution related knowledge has its inherent limitations or requires a lot of computing resources and manpower costs. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and device for out-of-distribution detection of visual language models based on forced cue word learning. From the perspective of fully exploring the knowledge within the distribution, there is no need to pay attention to how to obtain complex and diverse out-of-distribution related knowledge. Without sacrificing the classification performance of the visual language model, the model's ability to detect out-of-distribution samples is effectively improved.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for out-of-distribution detection of a visual language model based on forced cue word learning, comprising the following steps:

[0008] Obtain an image dataset within the downstream task distribution and divide the dataset into a small sample within-distribution training set, an within-distribution test set, and an out-of-distribution test set.

[0009] Constructing a visual language model out-of-distribution detection framework, including an image encoder and an improved text encoder, wherein the improved text encoder includes a forced prompt word input and an initial prompt word input;

[0010] The forced prompt word and the initial prompt word are input into the text encoder respectively to obtain the forced text features and the initial text features, and the cross entropy loss is calculated by combining the image features obtained by the image encoder;

[0011] The pre-trained visual language model weights are used to initialize the visual language model out-of-distribution detection framework. End-to-end iterative training is performed using a small sample in-distribution training set. After calculating the optimization loss, a fine-tuned visual language model is obtained.

[0012] The in-distribution test set and the out-of-distribution test set are input into the visual language model to obtain the test image and text features, and the test image and text features and the out-of-distribution detection score function are calculated to obtain the out-of-distribution detection results.

[0013] As an optimal technical solution, the method of obtaining an image dataset within the distribution of downstream tasks includes: a small sample within-distribution training set under different settings, randomly sampling 1 to 16 images for each category in the within-distribution training set, and randomly sampling seeds of 1, 3, 5 and 7 in sequence to obtain the average results under the four seeds.

[0014] As a preferred technical solution, the forced prompt word input end includes learnable parameters, and the initial prompt word input end includes non-learnable parameters.

[0015] As a preferred technical solution, the cross entropy loss is calculated by combining the image features obtained by the image encoder, as follows:

[0016]

[0017] Among them, the training set is within the small sample distribution Includes the image and the corresponding category label pair (x, y c ), and Represents the forced text features of the jth category and initial text features The cosine similarity with the image feature z, C is the number of all categories, K is the forcing coefficient, and τ represents the temperature scaling hyperparameter.

[0018] As a preferred technical solution, the value of K is [1, 6].

[0019] As a preferred technical solution, the pre-trained model on the image encoder side is the VIT-B / 16 model, which performs end-to-end iterative training on the constructed small sample distribution training set, and uses the stochastic gradient descent method to optimize the cross entropy loss to obtain a fine-tuned visual language model.

[0020] As a preferred technical solution, the calculation of the test image and text features and the out-of-distribution detection score function includes the MCM score S MCM (x) and GL-MCM score S GL-MCM (x).

[0021] As a preferred technical solution, the MCM score S MCM (x), as follows:

[0022]

[0023] The GL-MCM score is S GL-MCM (x), as follows:

[0024]

[0025] Among them, z g is the global image feature, represents N local image features extracted by CLIP’s image encoder, is the forced text feature of the jth class, is the initial text feature K is the forcing coefficient, C is the number of all categories, and τ0 is the temperature coefficient during inference.

[0026] In a second aspect, the present invention further provides a visual language model out-of-distribution detection system based on forced cue word learning, which is applied to the above-mentioned visual language model out-of-distribution detection method based on forced cue word learning, including a data processing module, a detection framework construction module, a feature extraction module, a model training module and a sample detection module;

[0027] The data processing module is used to obtain the image dataset within the downstream task distribution and divide the dataset into a small sample within-distribution training set, an within-distribution test set, and an out-of-distribution test set;

[0028] A detection framework construction module, for constructing a visual language model out-of-distribution detection framework, including an image encoder end and an improved text encoder end, wherein the improved text encoder end includes a forced prompt word input end and an initial prompt word input end;

[0029] The feature extraction module is used to input the forced prompt word and the initial prompt word into the text encoder respectively to obtain the forced text features and the initial text features, and calculate the cross entropy loss by combining the image features obtained by the image encoder;

[0030] The model training module is used to initialize the out-of-distribution detection framework of the visual language model using the pre-trained visual language model weights, perform end-to-end iterative training using the small sample in-distribution training set, and calculate the optimization loss to obtain a fine-tuned visual language model;

[0031] The sample detection module is used to input the in-distribution test set and the out-of-distribution test set into the visual language model, obtain the test image and text features, calculate the test image and text features and the out-of-distribution detection score function, and obtain the out-of-distribution detection results.

[0032] In a third aspect, the present invention provides an electronic device, comprising:

[0033] at least one processor; and,

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the visual language model distribution out-of-distribution detection method based on forced cue word learning.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] (1) The present invention uses the cross-entropy loss function on the text features generated by the forced prompt words and the initial prompt words and the image features generated by the training images, forcing the model to learn richer in-distribution knowledge beyond the semantics of the class label text. The obtained out-of-distribution detection model has high accuracy and low misjudgment rate.

[0038] (2) The present invention can improve the model's ability to detect out-of-distribution data by only utilizing in-distribution knowledge, and uses non-learnable prompt words as a reference instead of setting more learnable parameters, making the model architecture more lightweight and well adapted to actual scenarios with small sample data. It has good applicability and does not require reliance on learning out-of-distribution related knowledge and large external auxiliary data sets or background areas of in-distribution data compared to existing out-of-distribution detection methods based on visual language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 This is a flow chart of a method for out-of-distribution detection of a visual language model based on forced cue word learning according to an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of an out-of-distribution detection framework for a visual language model according to an embodiment of the present invention;

[0042] Figure 3 Schematic diagram of the structure of the out-of-distribution detection system of the visual language model based on forced cue word learning according to an embodiment of the present invention;

[0043] Figure 4 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0046] Forcing prompts are carefully designed input prompts that force the model to produce outputs that meet specific requirements or expectations. Such prompts may contain explicit instructions, constraints, or contextual information to guide the model's behavior.

[0047] See also Figure 1 This embodiment provides a method for out-of-distribution detection of a visual language model based on forced cue word learning, comprising the following steps:

[0048] S1. Obtain an image dataset within the downstream task distribution and divide the dataset into a small sample within-distribution training set, an within-distribution test set, and an out-of-distribution test set.

[0049] S2. Constructing a visual language model out-of-distribution detection framework, including an image encoder and an improved text encoder, wherein the improved text encoder includes a forced prompt word input and an initial prompt word input;

[0050] S3. Input the forced prompt word and the initial prompt word into the text encoder respectively to obtain the forced text features and the initial text features, and calculate the cross entropy loss by combining the image features obtained by the image encoder;

[0051] S4. Initialize the out-of-distribution detection framework using the pre-trained visual language model weights, perform end-to-end iterative training using the small sample in-distribution training set, calculate the optimization loss, and obtain a fine-tuned visual language model.

[0052] S5. Input the in-distribution test set and the out-of-distribution test set into the visual language model to obtain the test image and text features, calculate the test image and text features and the out-of-distribution detection score function, and obtain the out-of-distribution detection results.

[0053] In particular, in step S1, the small sample in-distribution training set under different settings consists of 1 / 4 / 16 images randomly sampled from each category in the in-distribution training set, and the randomly sampled seeds are 1 / 3 / 5 / 7 respectively. The final experimental result is the average result under the 4 seeds.

[0054] Specifically, in step S2, the constructed out-of-distribution detection framework for the visual language model is modified based on the CLIP model, with the modifications specifically addressing the text encoder input. As a cue word learning method, this method's text encoder input consists of a forced cue word and an initial cue word, unlike previous text encoders that only had a single learnable cue word input. Both cue words are initialized using the encoding features corresponding to the template "a photo of a{classname}." (classname represents the class name), but only the forced cue word parameters are learnable, while the initial cue word parameters are not.

[0055] It is worth explaining that the initialization of the "forced prompt word" and the "original prompt word" involved in this embodiment is consistent. For the class label, both are frozen. For the prefix template, the forced prompt word is set to be learnable, and the original prompt word is set to be non-learnable. Figure 1 , and then combine it with the defined formula for training.

[0056] In addition, the visual language model can be constructed using a network with the same function, and is not limited to the CLIP model of this application.

[0057] In particular, in step S3, the cross entropy loss calculated between the text and image features is:

[0058]

[0059] The training set is within the small sample distribution Contains the image and the corresponding category label pair (x, y c ),Then and Represents the forced text features of the jth category and initial text features The cosine similarity with the image feature z, C is the number of all categories, τ represents the temperature scaling hyperparameter, τ is 1 during training, and the forcing coefficient K represents the cosine similarity between the text feature of the initial prompt word and the image feature iterated K times. The visual language model is not very sensitive to the value of K in the range of [1,6], usually 3 or 4. When K is 0, it is equivalent to the conventional prompt word learning method.

[0060] It can be explained that the model is not sensitive to the hyperparameter K, which means that within a given range, users do not need to spend a lot of effort to find a suitable hyperparameter. They only need to take any number within the given range to achieve good results.

[0061] In particular, in step S4, the parameters of the image and text encoders are initialized using the weights of the pre-trained CLIP visual language model, where the pre-trained model of the image encoder is the VIT-B / 16 model. Then, end-to-end iterative training is performed on the constructed small sample distribution training set, and the cross-entropy loss is optimized using the stochastic gradient descent method to obtain a fine-tuned visual language model.

[0062] It should be noted that the image encoder side can use a pre-trained model with the same function for parameter initialization, and is not limited to the VIT-B / 16 model of this application.

[0063] In particular, in step S5, the present method can flexibly combine different out-of-distribution detection score functions, such as the MCM score S MCM(x) and GL-MCM score S GL-MCM (x):

[0064]

[0065] The global image feature z g That is z, represents N local image features extracted by the CLIP image encoder, τ0 is the temperature coefficient during inference, and τ0 is set to 1. The in-distribution and out-of-distribution test sets are input into the trained visual language model, and the image and text features output by the model are combined with the out-of-distribution detection score function to output the out-of-distribution detection results.

[0066] It is worth noting that this embodiment uses a defined cross-entropy loss on the text features generated by the forced prompt words and the initial prompt words, and the image features generated by the training images, forcing the model to learn richer in-distribution knowledge beyond the semantics of the class label text (class label), resulting in a lower error rate for the out-of-distribution detection model. This is because the two types of prompt words are initialized consistently and have the same semantic information. Therefore, based on the given cross-entropy loss formula, gradient descent updates are performed, and the in-distribution image features will produce a higher semantic similarity with the learnable forced prompt words than with the original prompt words. Then, thanks to the richer semantic information specific to the in-distribution data learned by these models, in the test phase, the similarity between the out-of-distribution test images and the two types of prompt words will show a smaller difference, making the out-of-distribution test image scores significantly lower than the in-distribution test image scores, thereby effectively reducing the model error rate.

[0067] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0068] Based on the same idea as the visual language model out-of-distribution detection method based on forced cue word learning in the above-mentioned embodiment, the present invention also provides a visual language model out-of-distribution detection system based on forced cue word learning, which can be used to execute the above-mentioned visual language model out-of-distribution detection method based on forced cue word learning. For ease of explanation, the structural diagram of the embodiment of the visual language model out-of-distribution detection system based on forced cue word learning only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0069] See also Figure 3In another embodiment of the present application, a visual language model out-of-distribution detection system 10 based on forced cue word learning is provided, the system comprising a data processing module 11, a detection framework construction module 12, a feature extraction module 13, a model training module 14, and a sample detection module 15;

[0070] The data processing module 11 is used to obtain the image dataset within the downstream task distribution and divide the dataset into a small sample within-distribution training set, an within-distribution test set, and an out-of-distribution test set;

[0071] A detection framework construction module 12 is used to construct a visual language model out-of-distribution detection framework, including an image encoder end and an improved text encoder end, wherein the improved text encoder end includes a forced prompt word input end and an initial prompt word input end;

[0072] A feature extraction module 13 is used to input the forced prompt word and the initial prompt word into the text encoder respectively to obtain forced text features and initial text features, and calculate the cross entropy loss by combining the image features obtained by the image encoder;

[0073] The model training module 14 is used to initialize the out-of-distribution detection framework of the visual language model using the pre-trained visual language model weights, perform end-to-end iterative training using the small sample in-distribution training set, calculate the optimization loss, and obtain a fine-tuned visual language model;

[0074] The sample detection module 15 is used to input the in-distribution test set and the out-of-distribution test set into the visual language model, obtain the test image and text features, calculate the test image and text features and the out-of-distribution detection score function, and obtain the out-of-distribution detection result.

[0075] It should be noted that the visual language model out-of-distribution detection system based on forced cue word learning of the present invention corresponds one-to-one to the visual language model out-of-distribution detection method based on forced cue word learning of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the visual language model out-of-distribution detection method based on forced cue word learning are all applicable to the embodiment of the visual language model out-of-distribution detection method based on forced cue word learning. For specific details, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0076] In addition, in the implementation of the out-of-distribution detection system of the visual language model based on forced cue word learning in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the out-of-distribution detection system of the visual language model based on forced cue word learning is divided into different program modules to complete all or part of the functions described above.

[0077] See also Figure 4 In one embodiment, an electronic device is provided for implementing a visual language model out-of-distribution detection method based on forced cue word learning. The electronic device 20 may include a first processor 21, a first memory 22, and a bus. It may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a visual language model out-of-distribution detection program 23 based on forced cue word learning.

[0078] The first memory 22 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as a mobile hard disk of the electronic device 20. In other embodiments, the first memory 22 can also be an external storage device of the electronic device 20, such as a plug-in mobile hard disk, smart memory card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device 20. Furthermore, the first memory 22 can also include both an internal storage unit of the electronic device 20 and an external storage device. The first memory 22 can not only be used to store application software and various types of data installed in the electronic device 20, such as the code of the visual language model distribution out-of-distribution detection program 23 based on forced prompt word learning, but can also be used to temporarily store data that has been output or is about to be output.

[0079] In some embodiments, the first processor 21 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The first processor 21 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 22, as well as calling data stored in the first memory 22, to perform various functions of the electronic device 20 and process data.

[0080] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4The structure shown does not constitute a limitation on the electronic device 20 , and the electronic device 20 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0081] The visual language model out-of-distribution detection program 23 based on forced cue word learning stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions. When running in the first processor 21, it can achieve the following:

[0082] Obtain an image dataset within the downstream task distribution and divide the dataset into a small sample within-distribution training set, an within-distribution test set, and an out-of-distribution test set.

[0083] Constructing a visual language model out-of-distribution detection framework, including an image encoder and an improved text encoder, wherein the improved text encoder includes a forced prompt word input and an initial prompt word input;

[0084] The forced prompt word and the initial prompt word are input into the text encoder respectively to obtain the forced text features and the initial text features, and the cross entropy loss is calculated by combining the image features obtained by the image encoder;

[0085] The pre-trained visual language model weights are used to initialize the visual language model out-of-distribution detection framework. End-to-end iterative training is performed using a small sample in-distribution training set. After calculating the optimization loss, a fine-tuned visual language model is obtained.

[0086] The in-distribution test set and the out-of-distribution test set are input into the visual language model to obtain the test image and text features, and the test image and text features and the out-of-distribution detection score function are calculated to obtain the out-of-distribution detection results.

[0087] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0088] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0089] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for out-of-distribution detection of visual language models based on forced cue word learning, characterized in that: The steps include: Obtain an image dataset within the downstream task distribution and divide the dataset into a small sample within-distribution training set, an within-distribution test set, and an out-of-distribution test set. Constructing a visual language model out-of-distribution detection framework, including an image encoder and an improved text encoder, wherein the improved text encoder includes a forced prompt word input and an initial prompt word input; The forced prompt word and the initial prompt word are input into the text encoder respectively to obtain the forced text features and the initial text features, and the cross entropy loss is calculated by combining the image features obtained by the image encoder; The pre-trained visual language model weights are used to initialize the visual language model out-of-distribution detection framework. End-to-end iterative training is performed using a small sample in-distribution training set. After calculating the optimization loss, a fine-tuned visual language model is obtained. The in-distribution test set and the out-of-distribution test set are input into the visual language model to obtain the test image and text features, and the test image and text features and the out-of-distribution detection score function are calculated to obtain the out-of-distribution detection results.

2. The out-of-distribution detection method for a visual language model based on forced cue word learning according to claim 1, characterized in that: The method for obtaining an in-distribution image dataset for a downstream task includes: a small sample in-distribution training set under different settings, randomly sampling 1 to 16 images from each category in the in-distribution training set, randomly sampling seeds 1, 3, 5, and 7 in sequence, and obtaining the average results under the four seeds.

3. The out-of-distribution detection method for a visual language model based on forced cue word learning according to claim 1, characterized in that: The forced prompt word input end includes learnable parameters, and the initial prompt word input end includes non-learnable parameters.

4. The out-of-distribution detection method for a visual language model based on forced cue word learning according to claim 1, characterized in that: The cross entropy loss is calculated by combining the image features obtained by the image encoder, as follows: Among them, the training set is within the small sample distribution Includes the image and the corresponding category label pair (x, y c ), and Represents the forced text features of the jth category and initial text features is the cosine similarity with the image feature z, C is the number of all categories, K is the forcing coefficient, and τ represents the temperature scaling hyperparameter.

5. The out-of-distribution detection method for a visual language model based on forced cue word learning according to claim 4, characterized in that: The value of K is [1, 6].

6. The out-of-distribution detection method for a visual language model based on forced cue word learning according to claim 1, characterized in that: The pre-trained model on the image encoder side is the VIT-B / 16 model, which performs end-to-end iterative training on the constructed small sample distribution training set, and uses the stochastic gradient descent method to optimize the cross entropy loss to obtain a fine-tuned visual language model.

7. The out-of-distribution detection method for a visual language model based on forced cue word learning according to claim 1, characterized in that: The calculation of the test image and text features and the out-of-distribution detection score function includes the MCM score S MCM (x) and GL-MCM score S GL-MCM (x).

8. The out-of-distribution detection method for a visual language model based on forced cue word learning according to claim 1, characterized in that: The MCM score is S MCM (x), as follows: The GL-MCM score is S GL-MCM (x), as follows: Among them, z g is the global image feature, represents N local image features extracted by CLIP’s image encoder, is the forced text feature of the jth class, is the initial text feature K is the forcing coefficient, C is the number of all categories, and τ0 is the temperature coefficient during inference.

9. A visual language model out-of-distribution detection system based on forced cue word learning, characterized by: The out-of-distribution detection method of a visual language model based on forced cue word learning, applied to any one of claims 1-8, comprises a data processing module, a detection framework construction module, a feature extraction module, a model training module, and a sample detection module; The data processing module is used to obtain the image dataset within the downstream task distribution and divide the dataset into a small sample within-distribution training set, an within-distribution test set, and an out-of-distribution test set; A detection framework construction module, for constructing a visual language model out-of-distribution detection framework, including an image encoder end and an improved text encoder end, wherein the improved text encoder end includes a forced prompt word input end and an initial prompt word input end; The feature extraction module is used to input the forced prompt word and the initial prompt word into the text encoder respectively to obtain the forced text features and the initial text features, and calculate the cross entropy loss by combining the image features obtained by the image encoder; The model training module is used to initialize the out-of-distribution detection framework of the visual language model using the pre-trained visual language model weights, perform end-to-end iterative training using the small sample in-distribution training set, and calculate the optimization loss to obtain a fine-tuned visual language model; The sample detection module is used to input the in-distribution test set and the out-of-distribution test set into the visual language model, obtain the test image and text features, calculate the test image and text features and the out-of-distribution detection score function, and obtain the out-of-distribution detection results.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the visual language model out-of-distribution detection method based on forced cue word learning as described in any one of claims 1-8.

Images