Prompt learning method based on modal consistency and computer equipment

Abstract

The invention discloses a prompt learning method based on modal consistency and computer equipment, and belongs to the technical field of computers. According to the technical scheme provided by the embodiment of the invention, the target visual language pre-training model is utilized, and the plurality of text embedding features are obtained based on the text prompt and the plurality of sample category texts. A plurality of visual embedding features are obtained by using a visual prompt and a plurality of sample images, and both the text prompt and the visual prompt belong to the target field. Determining a visual feature distribution of the plurality of visual embedded features and a text feature distribution of the plurality of text embedded features, and determining a prediction transmission matrix by using the visual feature distribution and the text feature distribution, the prediction transmission matrix being used for representing a mode of converting the plurality of visual embedded features into the plurality of text embedded features. The text prompt and the visual prompt are adjusted based on the prediction transmission matrix, the multiple visual embedded features and the annotation information, so that prompt learning is performed by using the features and the feature distribution, and the prompt learning effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a prompt learning method based on modality consistency and a computer device. Background Art

[0002] Visual and Language Pretraining (VLP) models have demonstrated strong performance in areas such as image classification and object localization. In order to apply the upstream visual and language pretraining models to different downstream tasks, prompt learning is usually used to adapt the visual and language pretraining models to downstream tasks with limited data. Specifically, the goal of prompt learning is to find suitable textual or visual prompts for specific tasks, thereby effectively transferring the knowledge of the visual and language pretraining model to specific downstream applications.

[0003] In the related art, prompt learning methods generally focus on improving the structure or quantity of text prompts or visual prompts. However, such prompt learning methods are not effective. Summary of the invention

[0004] The embodiment of the present application provides a prompt learning method and a computer device based on modality consistency, which can improve the effect of prompt learning. The technical solution is as follows:

[0005] In one aspect, a prompt learning method based on modality consistency is provided, the method comprising:

[0006] Obtaining multiple text embedding features based on text prompts and multiple sample category texts through a target visual language pre-training model; obtaining multiple visual embedding features based on visual prompts and multiple sample images, wherein the text prompts and the visual prompts both belong to the target domain;

[0007] Determining a visual feature distribution of the plurality of visual embedding features and a text feature distribution of the plurality of text embedding features;

[0008] Determining a prediction transfer matrix based on the visual feature distribution and the text feature distribution, wherein the prediction transfer matrix is ​​used to represent a way of transforming the plurality of visual embedding features into the plurality of text embedding features;

[0009] The text prompt and the visual prompt are adjusted based on the predicted transfer matrix, the multiple visual embedding features and the annotation information, and the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0010] In a possible implementation, the target visual language pre-training model is used to obtain multiple text embedding features based on text prompts and multiple sample category texts, including:

[0011] For any sample category text among the multiple sample category texts, combining the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination; performing semantic feature extraction on the sample text combination through the target visual language pre-training model to obtain a text embedding feature of the sample text combination;

[0012] Based on the visual prompts and the multiple sample images, multiple visual embedding features are obtained, including:

[0013] For any sample image among the multiple sample images, the sample image and the visual prompt are combined through the target visual language pre-training model to obtain a sample image combination; image features are extracted from the sample image combination through the target visual language pre-training model to obtain visual embedding features of the sample image combination.

[0014] In a possible implementation, determining the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features includes:

[0015] Substituting each of the visual embedding features into the first relationship data to obtain a first distribution feature of each of the visual embedding features; dividing the first distribution feature of each of the visual embedding features by the number of the visual embedding features to obtain the visual feature distribution;

[0016] Substitute each of the text embedding features into the second relational data to obtain a second distribution feature of each of the text embedding features; divide the second distribution feature of each of the text embedding features by the number of the text embedding features to obtain the text feature distribution.

[0017] In a possible implementation, determining a predicted transfer matrix based on the visual feature distribution and the text feature distribution includes:

[0018] Determining an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine a cost of transforming the plurality of visual embedding features into the plurality of text embedding features;

[0019] The initial transmission matrix is ​​optimized based on the cost function to obtain the predicted transmission matrix, and the cost corresponding to the predicted transmission matrix is ​​the minimum.

[0020] In a possible implementation, the adjusting the text prompt and the visual prompt based on the predicted transfer matrix, the multiple visual embedding features, and the annotation information includes:

[0021] Determining a first distribution probability of each of the visual embedding features on the multiple text embedding features based on the predicted transfer matrix and the multiple visual embedding features;

[0022] Based on the labeled information, determining a target transmission matrix;

[0023] The text prompt and the visual prompt are adjusted based on a first distribution probability of each of the visual embedding features on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

[0024] In a possible implementation, determining, based on the predicted transfer matrix and the plurality of visual embedding features, a first distribution probability of each of the visual embedding features on the plurality of text embedding features includes:

[0025] For any visual embedding feature among the plurality of visual embedding features, determining a first transmission vector corresponding to the visual embedding feature in the prediction transmission matrix;

[0026] Based on a first transmission vector corresponding to the visual embedding feature and a plurality of first transmission vectors in the predicted transmission matrix, a first distribution probability of the visual embedding feature on the plurality of text embedding features is determined.

[0027] In a possible implementation manner, determining a target transmission matrix based on the labeling information includes:

[0028] The labeling information is converted into a matrix form to obtain the target transmission matrix.

[0029] In a possible implementation, the adjusting the text prompt and the visual prompt based on the first distribution probability of each of the visual embedding features on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information includes:

[0030] Adjusting the text prompt and the visual prompt based on first difference information between a first distribution probability of each of the visual embedding features on a plurality of text embedding features and the annotated category text in the annotated information, and second difference information between the predicted transfer matrix and the target transfer matrix;

[0031] The annotated category text is the sample category text corresponding to the sample image corresponding to each of the visual embedding features in the annotation information.

[0032] In a possible implementation, the adjusting the text prompt and the visual prompt based on the first distribution probability of each of the visual embedding features on the multiple text embedding features and the first difference information between the labeled category text in the labeling information, and the second difference information between the predicted transfer matrix and the target transfer matrix includes:

[0033] Constructing a first loss function based on the first difference information;

[0034] Constructing a second loss function based on the second difference information;

[0035] The first loss function, the second loss function and the third loss function are combined to obtain a target loss function, wherein the third loss function is determined based on third difference information, wherein the third difference information is difference information between a second distribution probability of each of the visual embedding features on a plurality of text embedding features and the annotated category text in the annotation information, and the second distribution probability is a distribution probability determined directly based on the visual embedding feature;

[0036] Based on the objective loss function, the textual prompt and the visual prompt are adjusted.

[0037] In one aspect, a prompt learning device based on modality consistency is provided, the device comprising:

[0038] An embedding module is used to obtain a plurality of text embedding features based on text cues and a plurality of sample category texts through a target visual language pre-training model; and obtain a plurality of visual embedding features based on visual cues and a plurality of sample images, wherein the text cues and the visual cues both belong to the target domain;

[0039] a distribution determination module, configured to determine a visual feature distribution of the plurality of visual embedding features and a text feature distribution of the plurality of text embedding features;

[0040] A transfer matrix determination module, configured to determine a predicted transfer matrix based on the visual feature distribution and the text feature distribution, wherein the predicted transfer matrix is ​​used to represent a manner of transforming the plurality of visual embedding features into the plurality of text embedding features;

[0041] An adjustment module is used to adjust the text prompt and the visual prompt based on the predicted transfer matrix, the multiple visual embedding features and the annotation information, and the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0042] In a possible implementation, the embedding module is used to, for any sample category text among the multiple sample category texts, combine the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination; perform semantic feature extraction on the sample text combination through the target visual language pre-training model to obtain a text embedding feature of the sample text combination; for any sample image among the multiple sample images, combine the sample image and the visual prompt through the target visual language pre-training model to obtain a sample image combination; perform image feature extraction on the sample image combination through the target visual language pre-training model to obtain a visual embedding feature of the sample image combination.

[0043] In a possible implementation, the distribution determination module is used to bring each of the visual embedding features into the first relational data to obtain a first distribution feature of each of the visual embedding features; divide the first distribution feature of each of the visual embedding features by the number of the visual embedding features to obtain the visual feature distribution; substitute each of the text embedding features into the second relational data to obtain a second distribution feature of each of the text embedding features; and divide the second distribution feature of each of the text embedding features by the number of the text embedding features to obtain the text feature distribution.

[0044] In one possible implementation, the transfer matrix determination module is used to determine an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine the cost of transforming the multiple visual embedding features into the multiple text embedding features; based on the cost function, the initial transfer matrix is ​​optimized to obtain the predicted transfer matrix, and the cost corresponding to the predicted transfer matrix is ​​the smallest.

[0045] In one possible implementation, the adjustment module is used to determine a first distribution probability of each of the visual embedding features on the multiple text embedding features based on the predicted transfer matrix and the multiple visual embedding features; determine a target transfer matrix based on the annotation information; and adjust the text prompts and the visual prompts based on the first distribution probability of each of the visual embedding features on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

[0046] In a possible implementation, the adjustment module is used to determine, for any visual embedding feature among the multiple visual embedding features, a first transmission vector corresponding to the visual embedding feature in the prediction transmission matrix;

[0047] Based on a first transmission vector corresponding to the visual embedding feature and a plurality of first transmission vectors in the predicted transmission matrix, a first distribution probability of the visual embedding feature on the plurality of text embedding features is determined.

[0048] In a possible implementation, the adjustment module is used to convert the labeling information into a matrix form to obtain the target transmission matrix.

[0049] In one possible implementation, the adjustment module is used to adjust the text prompts and the visual prompts based on first difference information between a first distribution probability of each of the visual embedding features on multiple text embedding features and annotated category text in the annotation information, and second difference information between the predicted transfer matrix and the target transfer matrix; wherein the annotated category text is a sample category text corresponding to a sample image corresponding to each of the visual embedding features in the annotation information.

[0050] In one possible implementation, the adjustment module is used to construct a first loss function based on the first difference information; construct a second loss function based on the second difference information; combine the first loss function, the second loss function and the third loss function to obtain a target loss function, wherein the third loss function is determined based on the third difference information, wherein the third difference information is the difference information between the second distribution probability of each of the visual embedding features on multiple text embedding features and the annotated category text in the annotation information, wherein the second distribution probability is the distribution probability determined directly based on the visual embedding feature; and based on the target loss function, the text prompt and the visual prompt are adjusted.

[0051] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the prompt learning method based on modality consistency.

[0052] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the prompt learning method based on modality consistency.

[0053] On the one hand, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device performs the above-mentioned prompt learning method based on modal consistency.

[0054] Through the technical solution provided in the embodiment of the present application, a target visual language pre-training model is used to obtain multiple text embedding features based on text prompts and multiple sample category texts. Multiple visual embedding features are obtained using visual prompts and multiple sample images, and both text prompts and visual prompts belong to the target field. The visual feature distribution of multiple visual embedding features and the text feature distribution of multiple text embedding features are determined, and the visual feature distribution and the text feature distribution are used to determine a prediction transfer matrix, which is used to represent the way in which multiple visual embedding features are transformed into multiple text embedding features. Based on the prediction transfer matrix, the multiple visual embedding features and the annotation information, the text prompts and visual prompts are adjusted, so that the features and feature distributions are used to perform prompt learning, thereby improving the effect of prompt learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a schematic diagram of an implementation environment of a prompt learning method based on modality consistency provided in an embodiment of the present application;

[0057] Figure 2 is a flowchart of a prompt learning method based on modality consistency provided in an embodiment of the present application;

[0058] Figure 3 is a flowchart of another prompt learning method based on modality consistency provided in an embodiment of the present application;

[0059] Figure 4 is a flowchart of another prompt learning method based on modality consistency provided in an embodiment of the present application;

[0060] Figure 5 This is a schematic diagram of an effect provided by an embodiment of the present application;

[0061] Figure 6 This is another effect schematic diagram provided by an embodiment of the present application;

[0062] Figure 7 It is a schematic diagram of the structure of a prompt learning device based on modal consistency provided in an embodiment of the present application;

[0063] Figure 8 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0064] Fig. 9 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0066] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0067] In order to more clearly illustrate the technical solution provided by the embodiments of the present application, some terms involved in the embodiments of the present application are introduced below.

[0068] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain better results.

[0069] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It specializes in studying how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are in all areas of artificial intelligence.

[0070] Prompt Learning: is a new learning paradigm that has emerged in the field of natural language processing. It aims to better utilize the knowledge of pre-trained language models. The core idea of ​​prompt learning is to transform downstream tasks to align them with the form of pre-trained models, so as to achieve good task results in zero-shot or few-shot scenarios. This method reduces the dependence on a large amount of labeled data, allowing the model to achieve good performance even with limited resources.

[0071] Vision-language pre-training model: Vision-language pre-training model is an important technology in the field of artificial intelligence, which aims to enable computers to understand and generate content that combines visual and language information. These models are usually pre-trained on large-scale multimodal datasets to learn the deep semantic connection between vision and language. Pre-trained models can achieve transfer learning in different downstream tasks, such as image captioning, visual question answering, visual reasoning, etc.

[0072] Optimal Transport (OT) is an active field in modern mathematics that provides a powerful tool to understand and manipulate the relationship between probability distributions. In OT theory, the core problem is to find the optimal way to transform one probability distribution into another probability distribution, which aims to minimize the total cost of the transformation process. The cost function can be physical distance, information loss in information theory, energy consumption, or other metrics relevant to the problem.

[0073] Cross-Entropy (CE): It is a concept in information theory used to measure the difference between two probability distributions. In machine learning and deep learning, cross entropy is often used to measure the difference between the model's predicted distribution and the true label distribution.

[0074] Normalization: Mapping sequences with different value ranges to the interval (0, 1) facilitates data processing. In some cases, the normalized values ​​can be directly implemented as probabilities.

[0075] Embedded Coding: Embedded coding mathematically represents a correspondence, that is, mapping data in X space to Y space through a function F, where the function F is an injective function, and the mapping result is structural preservation. The injective function indicates that the data after mapping is uniquely corresponding to the data before mapping, and structural preservation indicates that the size relationship of the data before mapping is the same as the size relationship of the data after mapping. For example, there are data X1 and X2 before mapping, and after mapping, Y1 corresponding to X1 and Y2 corresponding to X2 are obtained. If the data X1 before mapping is greater than X2, then correspondingly, the data Y1 after mapping is greater than Y2. For words, it means mapping the words to another space to facilitate subsequent machine learning and processing.

[0076] After introducing some terms involved in the embodiments of the present application, the implementation environment of the embodiments of the present application is introduced below.

[0077] Figure 1 This is a schematic diagram of an implementation environment of a prompt learning method based on modality consistency provided in an embodiment of the present application, see Figure 1 , the implementation environment may include a terminal 110 and a server 140.

[0078] The terminal 110 is connected to the server 140 via a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 110 installs and runs an application with a prompt learning.

[0079] The server 140 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The server 140 provides background services for applications running on the terminal 110.

[0080] Optionally, terminal 110 generally refers to one of multiple terminals, and the embodiment of the present application only takes terminal 110 as an example.

[0081] After introducing the implementation environment of the embodiment of the present application, the application scenario of the embodiment of the present application is introduced below. The technical solution provided by the embodiment of the present application can be applied to a visual language pre-training model of any architecture. After adopting the technical solution provided by the embodiment of the present application, text prompts and visual prompts in a specific field can be learned, thereby improving the performance of the visual language pre-training model in a specific field.

[0082] After introducing the application scenarios of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below. Figure 2 is a flowchart of a prompt learning method based on modality consistency provided in an embodiment of the present application, see Figure 2 Taking the execution subject as a server as an example, the method includes the following steps.

[0083] 201. The server obtains multiple text embedding features based on text prompts and multiple sample category texts through the target visual language pre-training model.

[0084] Among them, the target visual language pre-training model is a pre-trained visual language model with the ability of visual understanding and language understanding. The text prompt belongs to the target field, and the target field is related to the downstream task that needs to use the target visual language pre-training model. For example, when the downstream task is a picture-text mutual inspection, the text prompt is used to indicate the retrieval of the corresponding image based on the input text. The text prompt is set by a technician or determined by a prompt word engineering, and the embodiment of the present application is not limited to this. In some embodiments, the text prompt is in the form of a vector. The sample category text is used to describe the category of the image, and the multiple sample category texts are sample category texts of a batch (Batch). In the embodiment of the present application, prompt learning needs to use multiple batches of samples. A sample category text corresponds to a text embedding feature, and a text embedding feature is generated based on a text prompt and a sample category text. When generating a text embedding feature, the text prompts corresponding to different sample category texts are the same.

[0085] 202. The server obtains a plurality of visual embedding features based on the visual prompt and the plurality of sample images, wherein the text prompt and the visual prompt both belong to the target domain.

[0086] Among them, the visual cue belongs to the target domain, and the target domain is related to the downstream task that needs to use the target visual language pre-training model. For example, when the downstream task is a picture-text mutual inspection, the visual cue is used to indicate the retrieval of the corresponding category text based on the input image. The visual cue is set by a technician or determined by a prompt word project. The embodiment of the present application does not limit this. The multiple sample images are sample images of a batch (Batch). In the embodiment of the present application, prompt learning requires the use of samples from multiple batches. A sample image corresponds to a visual embedding feature, and a visual embedding feature is generated based on a visual cue and a sample image. When generating a visual embedding feature, the visual cues corresponding to different sample images are the same.

[0087] 203. The server determines a visual feature distribution of the multiple visual embedding features and a text feature distribution of the multiple text embedding features.

[0088] Among them, the visual feature distribution of multiple visual embedding features is used to represent the distribution of multiple visual embedding features in the visual feature space; the text feature distribution of multiple text embedding features is used to represent the distribution of multiple text embedding features in the text feature space.

[0089] 204. The server determines a prediction transfer matrix based on the visual feature distribution and the text feature distribution, where the prediction transfer matrix is ​​used to represent a way of transforming the multiple visual embedding features into the multiple text embedding features.

[0090] Among them, the predicted transfer matrix is ​​a transfer matrix predicted based on the visual feature distribution and the text feature distribution. The transfer matrix corresponds to a cost function, which is used to represent the cost of transforming the multiple visual embedding features into the multiple text embedding features. The cost function is used as a constraint when determining the predicted transfer matrix.

[0091] 205. The server adjusts the text prompt and the visual prompt based on the predicted transfer matrix, the multiple visual embedding features, and the annotation information, where the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0092] Among them, adjusting the text prompt and the visual prompt is the task of prompt learning. In other words, prompt learning aims to learn appropriate text prompts and visual prompts so that downstream tasks can use the learned text prompts and visual prompts to achieve better results. There is a corresponding relationship between multiple sample images and multiple sample category texts. Taking a sample image as an example, the sample category text corresponding to the sample image is the category annotation of the sample image, that is, the ground truth of the category of the sample image. The annotation information is set by the technician according to the actual situation, and the embodiments of the present application do not limit this.

[0093] Through the technical solution provided in the embodiment of the present application, a target visual language pre-training model is used to obtain multiple text embedding features based on text prompts and multiple sample category texts. Multiple visual embedding features are obtained using visual prompts and multiple sample images, and both text prompts and visual prompts belong to the target field. The visual feature distribution of multiple visual embedding features and the text feature distribution of multiple text embedding features are determined, and the visual feature distribution and the text feature distribution are used to determine a prediction transfer matrix, which is used to represent the way in which multiple visual embedding features are transformed into multiple text embedding features. Based on the prediction transfer matrix, the multiple visual embedding features and the annotation information, the text prompts and visual prompts are adjusted, so that the features and feature distributions are used to perform prompt learning, thereby improving the effect of prompt learning.

[0094] The above steps 201-205 are a brief introduction to the technical solution provided in the embodiment of the present application. The technical solution provided in the embodiment of the present application will be more clearly explained below with reference to some examples. Figure 3 Taking the execution subject as a server as an example, the method includes the following steps.

[0095] 301. The server obtains multiple text embedding features based on text prompts and multiple sample category texts through the target visual language pre-training model.

[0096] Among them, the target visual language pre-training model is a pre-trained visual language model with the ability of visual understanding and language understanding. The text prompt belongs to the target domain, and the target domain is related to the downstream task that needs to use the target visual language pre-training model. The text prompt is used to instruct the execution of the downstream task. For example, when the downstream task is image-text mutual inspection, the text prompt is used to instruct the retrieval of the corresponding image based on the input text. The text prompt is set by a technician or determined by a prompt word project. The embodiment of the present application does not limit this. In some embodiments, the text prompt is in the form of a vector, and the text prompt is also referred to as a text prompt latent vector. The sample category text is used to describe the category of the image. The multiple sample category texts are a batch of sample category texts. In the embodiment of the present application, prompt learning requires the use of multiple batches of samples. A sample category text corresponds to a text embedding feature, and a text embedding feature is generated based on a text prompt and a sample category text. When generating a text embedding feature, the text prompts corresponding to different sample category texts are the same. In the embodiment of the present application, the text embedding feature is represented as g k (t), multiple text embedding features are represented as K is the number of text embedding features, that is, the number of sample category texts, and K is a positive integer.

[0097] In a possible implementation, for any sample category text among the multiple sample category texts, the server combines the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination. The server extracts semantic features from the sample text combination through the target visual language pre-training model to obtain text embedding features of the sample text combination.

[0098] Among them, by combining the sample category text and the text prompt, a sample text combination for executing downstream tasks can be obtained. In an embodiment of the present application, the sample text combination is used to indicate the image corresponding to the sample category text. The text embedding feature is used to represent the semantics of the sample text combination. Since the sample text combination is used to indicate the corresponding image, the text embedding feature can reflect the image characteristics of the corresponding image to a certain extent. It should be noted that the above implementation is described by taking the processing of a sample category text as an example. The way in which the server processes other sample category texts belongs to the same inventive concept as the above description, and the implementation process will not be repeated.

[0099] In this implementation, the target visual language pre-training model is used to combine the sample category text with the text prompt to obtain a sample text combination, and semantic features are extracted from the sample text combination to obtain a text embedding feature of the sample text combination, which can represent the semantics of the text embedding feature.

[0100] In order to explain the above implementation more clearly, the above implementation will be explained in several parts below.

[0101] In the first part, the server combines the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination.

[0102] In a possible implementation, the server concatenates the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination.

[0103] Among them, splicing refers to putting the sample category text and text prompt together without any processing.

[0104] For example, the text prompt is in the form of a vector, and the server embeds and encodes the sample category text through the target visual language pre-training model to obtain a text embedding vector of the sample category text. The server concatenates the text embedding vector with the text prompt through the target visual language pre-training model to obtain the sample text combination.

[0105] In a possible implementation, the server fuses the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination.

[0106] Fusion refers to the transformation of sample category text and text prompt into the same form and then superimposing them.

[0107] For example, the text prompt is in the form of a vector, and the server embeds and encodes the sample category text through the target visual language pre-training model to obtain a text embedding vector of the sample category text. The server adds the text embedding vector to the text prompt through the target visual language pre-training model to obtain the sample text combination.

[0108] In the second part, the server extracts semantic features of the sample text combination through the target visual language pre-training model to obtain text embedding features of the sample text combination.

[0109] In a possible implementation, the server encodes the sample text combination based on the attention mechanism through the target visual language pre-training model to obtain a text embedding feature of the sample text combination.

[0110] Among them, the target visual language pre-training model includes a text encoder, which is used to encode text into text embedding features. In the above embodiment, the text encoder encodes the text based on an attention mechanism. In some embodiments, the text encoder is an encoder of a BERT (Bidirectional Encoder Representations from Transformers) model.

[0111] 302. The server obtains a plurality of visual embedding features based on the visual prompt and the plurality of sample images, wherein the text prompt and the visual prompt both belong to the target domain.

[0112] Among them, the visual cue belongs to the target domain, and the target domain is related to the downstream tasks that need to use the target visual language pre-training model. The visual cue is used to instruct the execution of the downstream task. For example, when the downstream task is image-text mutual inspection, the visual cue is used to instruct the retrieval of the corresponding category text based on the input image. The visual cue is set by a technician or determined by a cue word project, which is not limited in this embodiment of the present application. The multiple sample images are sample images of a batch. In the embodiment of the present application, prompt learning requires the use of multiple batches of samples. A sample image corresponds to a visual embedding feature, and a visual embedding feature is generated based on the visual cue and a sample image. When generating the visual embedding feature, the visual cues corresponding to different sample images are the same. In the embodiment of the present application, the visual embedding feature is represented as f b (v), multiple visual embedding features are represented as B is the batch size, which indicates the number of visual embedding features. In addition, in the embodiment of the present application, both the single text embedding feature and the single visual embedding feature are independent and identically distributed.

[0113] In a possible implementation, for any sample image among the multiple sample images, the server combines the sample image and the visual prompt through the target visual language pre-training model to obtain a sample image combination. The server extracts image features from the sample image combination through the target visual language pre-training model to obtain visual embedding features of the sample image combination.

[0114] Among them, by combining the sample image and the visual prompt, a sample image combination for performing downstream tasks can be obtained. In an embodiment of the present application, the sample image combination is used to indicate the category text corresponding to the sample image. The visual embedding feature is used to represent the semantics of the sample image combination. Since the sample image combination is used to indicate the corresponding category text, the visual embedding feature can reflect the text characteristics of the corresponding category text to a certain extent. It should be noted that the above implementation is described by taking the processing of a sample image as an example. The way the server processes other sample images belongs to the same inventive concept as the above description, and the implementation process will not be repeated.

[0115] In order to explain the above implementation more clearly, the above implementation will be explained in several parts below.

[0116] In the first part, the server combines the sample image and the visual prompt through the target visual language pre-training model to obtain a sample image combination.

[0117] In a possible implementation, the server splices the sample image and the visual prompt through the target visual language pre-training model to obtain a sample image combination.

[0118] Among them, stitching refers to putting the sample images and visual cues together without any processing.

[0119] For example, the visual cue is in the form of a vector, and the server convolves the sample image through the target visual language pre-trained model to obtain an image feature vector of the sample image. The server concatenates the image feature vector with the visual cue through the target visual language pre-trained model to obtain the sample image combination.

[0120] In a possible implementation, the server fuses the sample image and the visual prompt through the target visual language pre-training model to obtain a sample image combination.

[0121] Fusion refers to the superposition of sample images and visual cues after they are transformed into the same form.

[0122] For example, the visual cue is in the form of a vector, and the server convolves the sample image through the target visual language pre-trained model to obtain an image feature vector of the sample image. The server fuses the image feature vector with the visual cue through the target visual language pre-trained model to obtain the sample image combination.

[0123] In the second part, the server extracts image features of the sample image combination through the target visual language pre-training model to obtain the visual embedding features of the sample image combination.

[0124] In a possible implementation, the server performs multiple convolutions on the sample image combination through the target visual language pre-training model to obtain visual embedding features of the sample image combination.

[0125] Among them, the target visual language pre-training model includes an image encoder, which is used to encode the image into visual embedding features. In the above embodiment, the image encoder is an encoder based on ResNET.

[0126] 303. The server determines a visual feature distribution of the multiple visual embedding features and a text feature distribution of the multiple text embedding features.

[0127] The visual feature distribution of the multiple visual embedding features is used to represent the distribution of the multiple visual embedding features in the visual feature space. The text feature distribution of the multiple text embedding features is used to represent the distribution of the multiple text embedding features in the text feature space.

[0128] In a possible implementation, the server brings each visual embedding feature into the first relational data to obtain a first distribution feature of each visual embedding feature. The server divides the first distribution feature of each visual embedding feature by the number of the visual embedding feature to obtain the visual feature distribution. The server substitutes each text embedding feature into the second relational data to obtain a second distribution feature of each text embedding feature. The server divides the second distribution feature of each text embedding feature by the number of the text embedding feature to obtain the text feature distribution.

[0129] The first relationship data and the second relationship are functions, and the first relationship data and the second relationship are set by the technician according to the actual situation, and the embodiment of the present application does not limit this. For each sample (text embedding feature and visual embedding feature), a uniformly distributed probability is used, and accordingly, the probability of the text embedding feature is 1 / B, and the probability of the visual embedding feature is 1 / K.

[0130] For example, the server determines the visual feature distribution by the following formula (1). The server determines the text feature distribution by the following formula (2).

[0131]

[0132] Among them, P(v) is the visual feature distribution, v represents the visual cue, b represents the sequence number of the visual embedding feature, is the first relation data, B is the number of visual embedding features, Q(t) is the text feature distribution, t represents the text prompt, k represents the sequence number of the text embedding feature, is the second relation data, and K is the number of text embedding features.

[0133] In some embodiments, the first relationship data and the second relation data are all Dirac functions.

[0134] 304. The server determines a prediction transfer matrix based on the visual feature distribution and the text feature distribution, where the prediction transfer matrix is ​​used to represent a way of transforming the multiple visual embedding features into the multiple text embedding features.

[0135] Among them, the predicted transfer matrix is ​​a transfer matrix predicted based on the visual feature distribution and the text feature distribution. The transfer matrix corresponds to a cost function, which is used to represent the cost of transforming the multiple visual embedding features into the multiple text embedding features. The cost function is used as a constraint when determining the predicted transfer matrix.

[0136] In a possible implementation, the server determines an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine a cost of transforming the multiple visual embedding features into the multiple text embedding features. The server optimizes the initial transfer matrix based on the cost function to obtain the predicted transfer matrix, and the predicted transfer matrix corresponds to the minimum cost.

[0137] The cost function is expressed as To represent the cost function The element C in bk Measure the distance between the sample image b and the sample category text k, which is also the cost. The above optimization to minimize the cost corresponding to the predicted transfer matrix refers to optimizing the OT distance between the visual feature distribution P(v) and the text feature distribution Q(t). By minimizing the OT distance between the visual feature distribution P(v) and the text feature distribution Q(t), the visual feature distribution P(v) and the text feature distribution Q(t) can be aligned in an unsupervised manner according to the similarity (i.e., cost function) between the visual feature distribution P(v) and the text feature distribution Q(t). In some embodiments, the distance is represented by cosine similarity, for example, C bk =1-cos(f b (v),g k (t)). In addition, according to OT theory, the transmission matrix The following conditions should be met: The transfer matrix T is a probability matrix that describes how to transfer B visual embedding features to K text embedding features in a small batch sample distribution matching process.

[0138] For example, the server uses the following formula (3) to represent the process of minimizing the OT distance, and uses the following formula (4) to represent the method of determining the predicted transmission matrix.

[0139] min t,v OT(P(v),Q(t))=min t,v min T∈∏(P(v),Q(t)) <T,C> (3)

[0140] Among them, OT(P(v), Q(t)) represents the OT distance between the visual feature distribution P(v) and the text feature distribution Q(t), T is the transfer matrix, C is the cost function, ∏(P(v), Q(t)) represents the set of all joint distributions that make the marginal distribution P(v) and Q(t), the transfer matrix belongs to the set of joint distributions, and the initial transfer matrix and the predicted transfer matrix in the above implementation both belong to the set of joint distributions.

[0141] T * = arg min T∈∏(P(v),Q(t)) <T,C> -λh(T) (4)

[0142] Among them, T * is the predicted transmission matrix, λ≥0, is the hyperparameter of entropy regularization, h(T)=-∑ ij T ij oeLh ij Represents the entropy regularization term to reduce computational cost.

[0143] 305. The server determines a first distribution probability of each visual embedding feature on the multiple text embedding features based on the predicted transfer matrix and the multiple visual embedding features.

[0144] In one possible implementation, for any visual embedding feature among the multiple visual embedding features, the server determines a first transmission vector corresponding to the visual embedding feature in the predicted transmission matrix. The service determines a first distribution probability of the visual embedding feature on the multiple text embedding features based on the first transmission vector corresponding to the visual embedding feature and the multiple first transmission vectors in the predicted transmission matrix.

[0145] Wherein, the prediction transfer matrix includes multiple rows, where a row corresponds to a visual embedding feature, in other words, a row corresponds to a sample image; a column in the prediction transfer matrix corresponds to a text embedding feature, in other words, a column corresponds to a sample category text. Correspondingly, where a column corresponds to a visual embedding feature, in other words, a column corresponds to a sample image; a row in the prediction transfer matrix corresponds to a text embedding feature, in other words, a row corresponds to a sample category text. Where a row corresponds to a visual embedding feature, the first transfer vector corresponding to the visual embedding feature is a row in the prediction transfer matrix.

[0146] In the above implementation, the predicted transfer matrix is ​​used to determine the first distribution probability of the visual embedding feature on multiple text embedding features, that is, the optimized transfer matrix is ​​used to predict the probability that the sample image corresponds to multiple sample category texts, thereby enhancing the subsequent learning of visual cues and text cues.

[0147] For example, the server determines the first distribution probability of the visual embedding feature on multiple text embedding features by using the following formula (5).

[0148]

[0149] Among them, T * is the predicted transmission matrix, that is, the optimized transmission matrix, x b is the visual embedding feature, y represents the text embedding feature, p(y|x b ,T * ) represents the first distribution probability of the visual embedding feature on multiple text embedding features, τ is a constant, is the first transmission vector corresponding to the visual embedding feature, The predicted transmission matrix T * The transmission vector in .

[0150] It should be noted that p(y|x b ,T * ) is different from the probability calculation based on point-to-point cross entropy loss in related technologies, where p(y|x b ,T * ) in the transmission vector (which can be considered as similarity) is optimized from the perspective of distribution matching, while the related technology is calculated in a point-to-point manner. In other words, since the transfer matrix T needs to satisfy the constraints of the two marginal distributions of the visual feature distribution P(v) and the text feature distribution Q(t) when solving the optimal transfer problem, the elements of T in the transfer matrix It will be affected by all K text embedding features and B visual embedding features in the current batch, not just the relationship between a single text embedding feature and a single visual embedding feature.

[0151] 306. The server determines a target transfer matrix based on the annotation information, where the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0152] Among them, there is a corresponding relationship between multiple sample images and multiple sample category texts. Taking a sample image as an example, the sample category text corresponding to the sample image is the category annotation of the sample image, that is, the ground truth of the category of the sample image. The annotation information is set by the technician according to the actual situation, and the embodiment of the present application does not limit this.

[0153] In a possible implementation manner, the server converts the annotation information into a matrix form to obtain the target transmission matrix.

[0154] Among them, the annotation information records the correspondence between multiple sample images and multiple sample category texts. In the process of converting the annotation information into a matrix form, the rows of the matrix are defined as corresponding to the sample images, and the columns of the matrix are defined as corresponding to the sample category texts. The method for determining the values ​​in the matrix is ​​that if a sample image in the annotation information corresponds to a sample category text, then the values ​​corresponding to the sample image and the sample category text in the matrix are determined to be 1; if a sample image in the annotation information does not correspond to a sample category text, then the values ​​corresponding to the sample image and the sample category text in the matrix are determined to be 0, thereby obtaining the target transmission matrix.

[0155] 307. The server adjusts the text prompt and the visual prompt based on the first distribution probability of each visual embedding feature on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

[0156] Among them, adjusting the text prompt and the visual prompt is the task of prompt learning. In other words, prompt learning aims to learn appropriate text prompts and visual prompts so that downstream tasks can use the learned text prompts and visual prompts to achieve better results.

[0157] In one possible implementation, the server adjusts the text prompt and the visual prompt based on first difference information between a first distribution probability of each visual embedding feature on multiple text embedding features and the labeled category text in the labeling information, and second difference information between the predicted transfer matrix and the target transfer matrix.

[0158] Among them, the annotated category text is the sample category text corresponding to the sample image corresponding to each visual embedding feature in the annotation information. The first difference information is the category text difference determined from the distribution perspective, and the second difference information is the distribution difference of the transmission matrix. In some embodiments, the first difference information is the difference information between the first distribution probability and the annotated distribution probability, and the annotated distribution probability is the distribution probability corresponding to the annotated category text. For example, there is an annotated distribution probability (1, 0, 0, 0, 0), 1 represents the probability of the annotated category text, and 0 represents the probability of other category texts.

[0159] For example, the server constructs a first loss function based on the first difference information. The server constructs a second loss function based on the second difference information. The server combines the first loss function, the second loss function, and the third loss function to obtain a target loss function, wherein the third loss function is determined based on the third difference information, wherein the third difference information is the difference information between the second distribution probability of each visual embedding feature on multiple text embedding features and the annotated category text in the annotation information, and the second distribution probability is a distribution probability determined directly based on the visual embedding feature. The server adjusts the text prompt and the visual prompt based on the target loss function.

[0160] Among them, the first loss function is also called the cross entropy loss function, the second loss function is also called the distribution matching loss function, and the third loss function is the unimodal matching loss function.

[0161] For example, the form of the objective loss function is shown in the following formula (6).

[0162]

[0163] in, is the target loss function, is the first loss function, is the second loss function, is the third loss function, β c is the weight of the first loss function, β m is the weight of the second loss function, β m and β c The configuration is performed by technicians based on actual conditions, and the embodiments of the present application do not limit this.

[0164] The first loss function is expressed as follows by the following formula (7):

[0165]

[0166] in, Expressing expectation, y bis the distribution probability corresponding to the labeled category text. Using the first loss function The purpose of prompt learning is to make the sample image and the corresponding sample category text closer from the perspective of distribution matching.

[0167] The second loss function is expressed by the following formula (9):

[0168]

[0169] Where Ts is the target transmission matrix.

[0170] In some embodiments, the server uses a gradient descent method to adjust the text prompt and the visual prompt based on the target loss function.

[0171] The purpose of adjusting the text prompt and the visual prompt is to reduce the loss value of the target loss function. Since the text prompt and the visual prompt are both in the form of vectors, adjusting the text prompt and the visual prompt is to adjust the corresponding vectors.

[0172] The following will be combined Figure 4 The technical solution provided in the embodiments of the present application is described.

[0173] See also Figure 4 , combine multiple sample images and visual cues and input them into the image encoder of the target visual language pre-trained model to obtain multiple visual embedding features. Combine multiple sample category texts and text cues and input them into the text encoder to obtain multiple text embedding features. Determine the visual feature distribution P(v) of multiple visual embedding features, and determine the text feature distribution Q(t) of multiple text embedding features. Minimize the OT distance between the visual feature distribution P(v) and the text feature distribution Q(t) to obtain a prediction transfer matrix, where one row of the prediction transfer matrix corresponds to a sample image and one column corresponds to a sample category text. Use the matrix corresponding to the annotation information to adjust the text cues and visual cues, thereby realizing cue learning.

[0174] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0175] Through the technical solution provided in the embodiment of the present application, a target visual language pre-training model is used to obtain multiple text embedding features based on text prompts and multiple sample category texts. Multiple visual embedding features are obtained using visual prompts and multiple sample images, and both text prompts and visual prompts belong to the target field. The visual feature distribution of multiple visual embedding features and the text feature distribution of multiple text embedding features are determined, and the visual feature distribution and the text feature distribution are used to determine a prediction transfer matrix, which is used to represent the way in which multiple visual embedding features are transformed into multiple text embedding features. Based on the prediction transfer matrix, the multiple visual embedding features and the annotation information, the text prompts and visual prompts are adjusted, so that the features and feature distributions are used to perform prompt learning, thereby improving the effect of prompt learning.

[0176] See also Figure 5 During the experiment, the technical solution provided by the embodiment of the present application was applied to different basic models and data sets, which can improve the performance of the basic model. Figure 5 The figure shows the average performance on 11 datasets.

[0177] See also Figure 6 ,The visualization experiment confirms that the technical solution provided by the embodiment of the present application can indeed improve the matching degree of the benchmark model in the text space and the image space, and the distance between the features of the two spaces is closer. In addition, the learned transfer matrix is ​​indeed more accurate for sample prediction, proving that it is very meaningful to improve the prompt learning framework through the distribution alignment method of modal matching.

[0178] Figure 7 is a schematic diagram of a structure of a prompt learning device based on modality consistency provided in an embodiment of the present application, see Figure 7 , the device comprises:

[0179] The embedding module 701 is used to obtain multiple text embedding features based on text prompts and multiple sample category texts through a target visual language pre-training model. Based on visual prompts and multiple sample images, multiple visual embedding features are obtained, and the text prompts and the visual prompts both belong to the target domain.

[0180] The distribution determination module 702 is used to determine the visual feature distribution of the multiple visual embedding features and the text feature distribution of the multiple text embedding features.

[0181] The transfer matrix determination module 703 is used to determine a predicted transfer matrix based on the visual feature distribution and the text feature distribution, where the predicted transfer matrix is ​​used to represent a way of transforming the multiple visual embedding features into the multiple text embedding features.

[0182] The adjustment module 704 is used to adjust the text prompt and the visual prompt based on the predicted transfer matrix, the multiple visual embedding features and the annotation information, and the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0183] In a possible implementation, the embedding module 701 is used to combine, for any sample category text among the multiple sample category texts, the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination. The target visual language pre-training model is used to perform semantic feature extraction on the sample text combination to obtain a text embedding feature of the sample text combination. For any sample image among the multiple sample images, the target visual language pre-training model is used to combine the sample image and the visual prompt to obtain a sample image combination. The target visual language pre-training model is used to perform image feature extraction on the sample image combination to obtain a visual embedding feature of the sample image combination.

[0184] In a possible implementation, the distribution determination module 702 is used to substitute each visual embedding feature into the first relational data to obtain a first distribution feature of each visual embedding feature. The first distribution feature of each visual embedding feature is divided by the number of the visual embedding feature to obtain the visual feature distribution. Each text embedding feature is brought into the second relational data to obtain a second distribution feature of each text embedding feature. The second distribution feature of each text embedding feature is divided by the number of the text embedding feature to obtain the text feature distribution.

[0185] In a possible implementation, the transfer matrix determination module 703 is used to determine an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine the cost of transforming the multiple visual embedding features into the multiple text embedding features. The initial transfer matrix is ​​optimized based on the cost function to obtain the predicted transfer matrix, and the predicted transfer matrix corresponds to the minimum cost.

[0186] In a possible implementation, the adjustment module 704 is used to determine a first distribution probability of each visual embedding feature on the multiple text embedding features based on the predicted transfer matrix and the multiple visual embedding features. Based on the annotation information, a target transfer matrix is ​​determined. Based on the first distribution probability of each visual embedding feature on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information, the text prompt and the visual prompt are adjusted.

[0187] In a possible implementation, the adjustment module 704 is configured to determine, for any visual embedding feature among the multiple visual embedding features, a first transmission vector corresponding to the visual embedding feature in the prediction transmission matrix.

[0188] Based on a first transmission vector corresponding to the visual embedding feature and a plurality of first transmission vectors in the predicted transmission matrix, a first distribution probability of the visual embedding feature on the plurality of text embedding features is determined.

[0189] In a possible implementation, the adjustment module 704 is configured to convert the labeling information into a matrix form to obtain the target transmission matrix.

[0190] In a possible implementation, the adjustment module 704 is used to adjust the text prompt and the visual prompt based on the first difference information between the first distribution probability of each visual embedding feature on the multiple text embedding features and the labeled category text in the annotation information, and the second difference information between the predicted transfer matrix and the target transfer matrix. The labeled category text is the sample category text corresponding to the sample image corresponding to each visual embedding feature in the annotation information.

[0191] In a possible implementation, the adjustment module 704 is used to construct a first loss function based on the first difference information. A second loss function is constructed based on the second difference information. The first loss function, the second loss function and the third loss function are combined to obtain a target loss function, wherein the third loss function is determined based on the third difference information, wherein the third difference information is the difference information between the second distribution probability of each visual embedding feature on multiple text embedding features and the annotated category text in the annotation information, and the second distribution probability is the distribution probability determined directly based on the visual embedding feature. Based on the target loss function, the text prompt and the visual prompt are adjusted.

[0192] It should be noted that: the prompt learning device based on modal consistency provided in the above embodiment only uses the division of the above functional modules as an example when performing prompt learning based on modal consistency. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the prompt learning device based on modal consistency provided in the above embodiment and the prompt learning method embodiment based on modal consistency belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0193] Through the technical solution provided in the embodiment of the present application, a target visual language pre-training model is used to obtain multiple text embedding features based on text prompts and multiple sample category texts. Multiple visual embedding features are obtained using visual prompts and multiple sample images, and both text prompts and visual prompts belong to the target field. The visual feature distribution of multiple visual embedding features and the text feature distribution of multiple text embedding features are determined, and the visual feature distribution and the text feature distribution are used to determine a prediction transfer matrix, which is used to represent the way in which multiple visual embedding features are transformed into multiple text embedding features. Based on the prediction transfer matrix, the multiple visual embedding features and the annotation information, the text prompts and visual prompts are adjusted, so that the features and feature distributions are used to perform prompt learning, thereby improving the effect of prompt learning.

[0194] The embodiment of the present application provides a computer device for executing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal is first introduced below:

[0195] Figure 8 800 is a schematic diagram of a terminal provided in an embodiment of the present application. The terminal 800 may be a smart phone, a tablet computer, a laptop computer or a desktop computer. The terminal 800 may also be called a user equipment, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0196] Typically, the terminal 800 includes: one or more processors 801 and one or more memories 802 .

[0197] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0198] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one computer program, which is used to be executed by the processor 801 to implement the prompt learning method based on modality consistency provided in the method embodiment of the present application.

[0199] In some embodiments, the terminal 800 may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807 and a power supply 808.

[0200] The peripheral device interface 803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0201] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like.

[0202] The display screen 805 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to collect touch signals on the surface or above the surface of the display screen 805. The touch signal may be input to the processor 801 as a control signal for processing. At this time, the display screen 805 may also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0203] The camera assembly 806 is used to collect images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal.

[0204] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them to the processor 801 for processing, or input them to the radio frequency circuit 804 to achieve voice communication.

[0205] The power supply 808 is used to supply power to various components in the terminal 800. The power supply 808 may be an alternating current, a direct current, a disposable battery, or a rechargeable battery.

[0206] In some embodiments, the terminal 800 further includes one or more sensors 809 , including but not limited to: an acceleration sensor 810 , a gyroscope sensor 811 , a pressure sensor 812 , an optical sensor 813 , and a proximity sensor 814 .

[0207] The acceleration sensor 810 can detect the magnitude of acceleration on three coordinate axes of the coordinate system established by the terminal 800 .

[0208] The gyro sensor 811 can detect the body direction and rotation angle of the terminal 800 . The gyro sensor 811 can cooperate with the acceleration sensor 810 to collect the user's 3D actions on the terminal 800 .

[0209] The pressure sensor 812 may be provided at the side frame of the terminal 800 and / or the lower layer of the display screen 805. When the pressure sensor 812 is provided at the side frame of the terminal 800, it may detect the user's holding signal of the terminal 800, and the processor 801 may perform left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 812. When the pressure sensor 812 is provided at the lower layer of the display screen 805, the processor 801 may control the operability controls on the UI interface according to the user's pressure operation on the display screen 805.

[0210] The optical sensor 813 is used to collect the ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 according to the ambient light intensity collected by the optical sensor 813.

[0211] The proximity sensor 814 is used to collect the distance between the user and the front of the terminal 800 .

[0212] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0213] The above-mentioned computer device can also be implemented as a server. The structure of the server is introduced below:

[0214] Fig. 9It is a structural diagram of a server provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein at least one computer program is stored in the one or more memories 902, and the at least one computer program is loaded and executed by the one or more processors 901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 900 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 900 may also include other components for implementing device functions, which will not be described in detail here.

[0215] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program, and the computer program can be executed by a processor to complete the prompt learning method based on modality consistency in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0216] In an exemplary embodiment, a computer program product or a computer program is also provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device performs the above-mentioned prompt learning method based on modal consistency.

[0217] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0218] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0219] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A prompt learning method based on modality consistency, characterized in that: The method comprises: Obtaining multiple text embedding features based on text prompts and multiple sample category texts through a target visual language pre-training model; obtaining multiple visual embedding features based on visual prompts and multiple sample images, wherein the text prompts and the visual prompts both belong to the target domain; Determining a visual feature distribution of the plurality of visual embedding features and a text feature distribution of the plurality of text embedding features; Determining a prediction transfer matrix based on the visual feature distribution and the text feature distribution, wherein the prediction transfer matrix is ​​used to represent a way of transforming the plurality of visual embedding features into the plurality of text embedding features; The text prompt and the visual prompt are adjusted based on the predicted transfer matrix, the multiple visual embedding features and the annotation information, and the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

2. The method according to claim 1, characterized in that The target visual language pre-training model is used to obtain multiple text embedding features based on text prompts and multiple sample category texts, including: For any sample category text among the multiple sample category texts, combining the sample category text and the text prompt through the target visual language pre-training model to obtain a sample text combination; performing semantic feature extraction on the sample text combination through the target visual language pre-training model to obtain a text embedding feature of the sample text combination; Based on the visual prompts and the multiple sample images, multiple visual embedding features are obtained, including: For any sample image among the multiple sample images, the sample image and the visual prompt are combined through the target visual language pre-training model to obtain a sample image combination; image features are extracted from the sample image combination through the target visual language pre-training model to obtain visual embedding features of the sample image combination.

3. The method according to claim 1, characterized in that The determining of the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features comprises: Substituting each of the visual embedding features into the first relationship data to obtain a first distribution feature of each of the visual embedding features; dividing the first distribution feature of each of the visual embedding features by the number of the visual embedding features to obtain the visual feature distribution; Substitute each of the text embedding features into the second relational data to obtain a second distribution feature of each of the text embedding features; divide the second distribution feature of each of the text embedding features by the number of the text embedding features to obtain the text feature distribution.

4. The method according to claim 1, characterized in that: The step of determining a predicted transfer matrix based on the visual feature distribution and the text feature distribution includes: Determining an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine a cost of transforming the plurality of visual embedding features into the plurality of text embedding features; The initial transmission matrix is ​​optimized based on the cost function to obtain the predicted transmission matrix, and the cost corresponding to the predicted transmission matrix is ​​the minimum.

5. The method according to claim 1, characterized in that: The adjusting the text prompt and the visual prompt based on the predicted transfer matrix, the multiple visual embedding features and the annotation information includes: Determine, based on the predicted transfer matrix and the plurality of visual embedding features, a first distribution probability of each of the visual embedding features on the plurality of text embedding features; Based on the labeled information, determining a target transmission matrix; The text prompt and the visual prompt are adjusted based on a first distribution probability of each of the visual embedding features on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

6. The method according to claim 5, characterized in that The step of determining, based on the predicted transfer matrix and the plurality of visual embedding features, a first distribution probability of each of the visual embedding features on the plurality of text embedding features comprises: For any visual embedding feature among the plurality of visual embedding features, determining a first transmission vector corresponding to the visual embedding feature in the prediction transmission matrix; Based on a first transmission vector corresponding to the visual embedding feature and a plurality of first transmission vectors in the predicted transmission matrix, a first distribution probability of the visual embedding feature on the plurality of text embedding features is determined.

7. The method according to claim 5, characterized in that The determining a target transmission matrix based on the labeling information includes: The labeling information is converted into a matrix form to obtain the target transmission matrix.

8. The method according to claim 5, characterized in that The adjusting the text prompt and the visual prompt based on the first distribution probability of each of the visual embedding features on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information includes: Adjusting the text prompt and the visual prompt based on first difference information between a first distribution probability of each of the visual embedding features on a plurality of text embedding features and the annotated category text in the annotated information, and second difference information between the predicted transfer matrix and the target transfer matrix; The annotated category text is the sample category text corresponding to the sample image corresponding to each of the visual embedding features in the annotation information.

9. The method according to claim 8, characterized in that The adjusting the text prompt and the visual prompt based on the first distribution probability of each of the visual embedding features on the multiple text embedding features and the first difference information between the labeled category text in the labeling information, and the second difference information between the predicted transfer matrix and the target transfer matrix, includes: Constructing a first loss function based on the first difference information; Constructing a second loss function based on the second difference information; The first loss function, the second loss function and the third loss function are combined to obtain a target loss function, wherein the third loss function is determined based on third difference information, wherein the third difference information is difference information between a second distribution probability of each of the visual embedding features on a plurality of text embedding features and the annotated category text in the annotation information, and the second distribution probability is a distribution probability determined directly based on the visual embedding feature; Based on the objective loss function, the textual prompt and the visual prompt are adjusted.

10. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the prompt learning method based on modality consistency as described in any one of claims 1 to 9.

Images