Modality consistency-based prompting learning methods and computer devices

Abstract

This application discloses a cue learning method and computer device based on modality consistency, belonging to the field of computer technology. The technical solution provided in this application utilizes a target visual language pre-trained model to obtain multiple text embedding features based on text cues and multiple sample text categories. Multiple visual embedding features are obtained using visual cues and multiple sample images, where both text cues and visual cues belong to the target domain. The visual feature distribution of the multiple visual embedding features and the text feature distribution of the multiple text embedding features are determined. A prediction transfer matrix is ​​determined using the visual feature distribution and the text feature distribution, representing the transformation of the multiple visual embedding features into multiple text embedding features. The text cues and visual cues are adjusted based on the prediction transfer matrix, the multiple visual embedding features, and annotation information, thereby utilizing features and feature distributions for cue learning and improving the effectiveness of cue learning.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a cue learning method and computer device based on modal consistency. Background Technology

[0002] Visual and Language Pretraining (VLP) models have demonstrated powerful performance in areas such as image classification and object localization. To enable upstream VLP pretraining models to be applied to different downstream tasks, cue learning is typically employed to adapt them to downstream tasks with limited data. Specifically, the goal of cue learning is to find appropriate textual or visual cues for a specific task, thereby effectively transferring the knowledge of the VLP pretraining model to specific downstream applications.

[0003] In related technologies, cue learning methods generally focus on improving the structure or quantity of textual or visual cues. However, such cue learning methods are not very effective. Summary of the Invention

[0004] This application provides a cue learning method and computer device based on modal consistency, which can improve the effect of cue learning. The technical solution is as follows:

[0005] On the one hand, a cue learning method based on modality consistency is provided, the method comprising:

[0006] By using a target visual language pre-trained model, multiple text embedding features are obtained based on text prompts and multiple sample text categories; multiple visual embedding features are obtained based on visual prompts and multiple sample images, where both the text prompts and the visual prompts belong to the target domain.

[0007] Determine the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features;

[0008] Based on the visual feature distribution and the text feature distribution, a prediction transfer matrix is ​​determined, which is used to represent the way the plurality of visual embedding features are transformed into the plurality of text embedding features;

[0009] Based on the predicted transfer matrix, the multiple visual embedding features, and the annotation information, the text prompts and the visual prompts are adjusted. The annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0010] In one possible implementation, the step involves obtaining multiple text embedding features based on text prompts and multiple sample text categories using a target visual language pre-trained model, including:

[0011] For any sample category text among the multiple sample category texts, the sample category text and the text prompt are combined using the target visual language pre-training model to obtain a sample text combination; semantic features are extracted from the sample text combination using the target visual language pre-training model to obtain the text embedding features of the sample text combination;

[0012] Based on visual cues and multiple sample images, multiple visual embedding features are obtained, including:

[0013] For any one of the plurality of sample images, the sample image and the visual cue are combined using the target visual language pre-training model to obtain a sample image combination; image features are extracted from the sample image combination using the target visual language pre-training model to obtain the visual embedding features of the sample image combination.

[0014] In one possible implementation, determining the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features includes:

[0015] Substitute each of the visual embedding features into the first relation data to obtain the first distribution feature of each of the visual embedding features; divide the first distribution feature of each of the visual embedding features by the number of visual embedding features to obtain the visual feature distribution;

[0016] Substitute each of the text embedding features into the second relation data to obtain the second distribution features of each of the text embedding features; divide the second distribution features of each of the text embedding features by the number of text embedding features to obtain the text feature distribution.

[0017] In one possible implementation, determining the prediction transfer matrix based on the visual feature distribution and the text feature distribution includes:

[0018] Determine an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine the cost of transforming the plurality of visual embedding features into the plurality of text embedding features;

[0019] The initial transmission matrix is ​​optimized based on the cost function to obtain the predicted transmission matrix, which has the minimum cost.

[0020] In one possible implementation, adjusting the text prompt and the visual prompt based on the predicted transfer matrix, the plurality of visual embedding features, and annotation information includes:

[0021] Based on the predicted transfer matrix and the plurality of visual embedding features, a first distribution probability of each of the visual embedding features on the plurality of text embedding features is determined;

[0022] Based on the annotation information, the target transmission matrix is ​​determined;

[0023] The text prompts and the visual prompts are adjusted based on the first distribution probability of each of the visual embedding features on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

[0024] In one possible implementation, determining the first distribution probability of each of the visual embedding features on the plurality of text embedding features based on the predicted transfer matrix and the plurality of visual embedding features includes:

[0025] For any visual embedding feature among the plurality of visual embedding features, determine the first transmission vector corresponding to the visual embedding feature in the prediction transmission matrix;

[0026] Based on the first transmission vector corresponding to the visual embedding feature and multiple first transmission vectors in the predicted transmission matrix, the first distribution probability of the visual embedding feature on the multiple text embedding features is determined.

[0027] In one possible implementation, determining the target transmission matrix based on the annotation information includes:

[0028] The annotation information is converted into matrix form to obtain the target transmission matrix.

[0029] In one possible implementation, adjusting the text prompts and visual prompts based on the first distribution probability of each of the visual embedding features over the plurality of text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information includes:

[0030] Based on the first distribution probability of each of the visual embedding features on multiple text embedding features and the first difference information between the labeled category text in the annotation information, and the second difference information between the predicted transfer matrix and the target transfer matrix, the text prompts and the visual prompts are adjusted.

[0031] Wherein, the labeled category text is the sample category text corresponding to the sample image corresponding to each of the visual embedding features in the labeled information.

[0032] In one possible implementation, adjusting the text prompts and the visual prompts based on a first difference information between the first distribution probability of each of the visual embedding features on multiple text embedding features and the labeled category text in the annotation information, and a second difference information between the predicted transfer matrix and the target transfer matrix, includes:

[0033] A first loss function is constructed based on the first difference information;

[0034] A second loss function is constructed based on the second difference information;

[0035] The first loss function, the second loss function, and the third loss function are combined to obtain the target loss function. The third loss function is determined based on the third difference information, which is the difference information between the second distribution probability of each visual embedding feature on multiple text embedding features and the labeled category text in the annotation information. The second distribution probability is the distribution probability directly determined based on the visual embedding features.

[0036] The text prompts and visual prompts are adjusted based on the target loss function.

[0037] On the one hand, a cue learning device based on modal consistency is provided, the device comprising:

[0038] The embedding module is used to obtain multiple text embedding features based on text prompts and multiple sample text categories through a target visual language pre-trained model; and to obtain multiple visual embedding features based on visual prompts and multiple sample images, wherein both the text prompts and the visual prompts belong to the target domain;

[0039] The distribution determination module is used to determine the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features;

[0040] A transfer matrix determination module is used to determine a predicted transfer matrix based on the visual feature distribution and the text feature distribution, wherein the predicted transfer matrix is ​​used to represent the way the plurality of visual embedding features are transformed into the plurality of text embedding features;

[0041] The adjustment module is used to adjust the text prompts and the visual prompts based on the predicted transfer matrix, the multiple visual embedding features, and the annotation information, wherein the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0042] In one possible implementation, the embedding module is configured to, for any sample category text among the plurality of sample category texts, combine the sample category text and the text prompt using the target visual language pre-training model to obtain a sample text combination; extract semantic features from the sample text combination using the target visual language pre-training model to obtain the text embedding features of the sample text combination; and, for any sample image among the plurality of sample images, combine the sample image and the visual prompt using the target visual language pre-training model to obtain a sample image combination; extract image features from the sample image combination using the target visual language pre-training model to obtain the visual embedding features of the sample image combination.

[0043] In one possible implementation, the distribution determination module is configured to: input each of the visual embedding features into first relational data to obtain a first distribution feature of each of the visual embedding features; divide the first distribution feature of each of the visual embedding features by the number of visual embedding features to obtain the visual feature distribution; input each of the text embedding features into second relational data to obtain a second distribution feature of each of the text embedding features; and divide the second distribution feature of each of the text embedding features by the number of text embedding features to obtain the text feature distribution.

[0044] In one possible implementation, the transfer matrix determination module is used to determine an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine the cost of transforming the plurality of visual embedding features into the plurality of text embedding features; and to optimize the initial transfer matrix based on the cost function to obtain the predicted transfer matrix, wherein the predicted transfer matrix corresponds to the minimum cost.

[0045] In one possible implementation, the adjustment module is configured to determine a first distribution probability of each visual embedding feature on the plurality of text embedding features based on the predicted transfer matrix and the plurality of visual embedding features; determine a target transfer matrix based on the annotation information; and adjust the text prompt and the visual prompt based on the first distribution probability of each visual embedding feature on the plurality of text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

[0046] In one possible implementation, the adjustment module is configured to determine, for any visual embedding feature among the plurality of visual embedding features, a first transmission vector corresponding to the visual embedding feature in the prediction transmission matrix;

[0047] Based on the first transmission vector corresponding to the visual embedding feature and multiple first transmission vectors in the predicted transmission matrix, the first distribution probability of the visual embedding feature on the multiple text embedding features is determined.

[0048] In one possible implementation, the adjustment module is used to convert the annotation information into a matrix form to obtain the target transmission matrix.

[0049] In one possible implementation, the adjustment module is used to adjust the text prompt and the visual prompt based on a first difference information between the first distribution probability of each visual embedding feature on multiple text embedding features and the labeled category text in the annotation information, and a second difference information between the predicted transfer matrix and the target transfer matrix; wherein the labeled category text is the sample category text corresponding to the sample image corresponding to each visual embedding feature in the annotation information.

[0050] In one possible implementation, the adjustment module is configured to construct a first loss function based on the first difference information; construct a second loss function based on the second difference information; combine the first loss function, the second loss function, and the third loss function to obtain a target loss function, wherein the third loss function is determined based on third difference information, which is the difference between the second distribution probability of each visual embedding feature on multiple text embedding features and the labeled category text in the annotation information, wherein the second distribution probability is the distribution probability directly determined based on the visual embedding features; and adjust the text prompts and the visual prompts based on the target loss function.

[0051] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the modality consistency-based cueing learning method.

[0052] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer program, which is loaded and executed by a processor to implement the modality consistency-based cueing learning method.

[0053] On the one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the above-described modality consistency-based cueing learning method.

[0054] The technical solution provided in this application utilizes a target visual language pre-trained model to obtain multiple text embedding features based on text prompts and multiple sample text categories. Multiple visual embedding features are obtained using visual prompts and multiple sample images, where both text and visual prompts belong to the target domain. The visual feature distributions of the multiple visual embedding features and the text feature distributions of the multiple text embedding features are determined. A prediction transfer matrix is ​​then used to represent the transformation of the multiple visual embedding features into multiple text embedding features. Based on this prediction transfer matrix, the multiple visual embedding features, and annotation information, the text and visual prompts are adjusted, thereby utilizing features and feature distributions for prompt learning and improving the effectiveness of prompt learning. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the implementation environment of a prompting learning method based on modal consistency provided in an embodiment of this application;

[0057] Figure 2 This is a flowchart of a cue learning method based on modal consistency provided in an embodiment of this application;

[0058] Figure 3 This is a flowchart of another cue learning method based on modal consistency provided in an embodiment of this application;

[0059] Figure 4 This is a flowchart of another prompting learning method based on modal consistency provided in the embodiments of this application;

[0060] Figure 5 This is a schematic diagram illustrating the effect provided in an embodiment of this application;

[0061] Figure 6 This is another schematic diagram illustrating the effect provided in the embodiment of this application;

[0062] Figure 7 This is a schematic diagram of a prompting learning device based on modal consistency provided in an embodiment of this application;

[0063] Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0064] Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0066] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0067] To provide a clearer explanation of the technical solutions provided in the embodiments of this application, some terms involved in the embodiments of this application will be introduced below.

[0068] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain better results.

[0069] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence.

[0070] Prompt learning is a novel learning paradigm in natural language processing that aims to better utilize the knowledge of pre-trained language models. The core idea of ​​prompt learning is to modify downstream tasks to align them with the form of the pre-trained model, thereby achieving good performance in zero-shot or few-shot scenarios. This approach reduces the dependence on large amounts of labeled data, allowing the model to achieve good performance even with limited resources.

[0071] Visual language pre-trained models: Visual language pre-trained models are an important technique in the field of artificial intelligence, designed to enable computers to understand and generate content that combines visual and linguistic information. These models typically learn deep semantic connections between vision and language through pre-training on large-scale multimodal datasets. Pre-trained models can then be used for transfer learning in various downstream tasks, such as image captioning, visual question answering, and visual reasoning.

[0072] Optimal Transport (OT) is an active area of ​​modern mathematics that provides a powerful tool for understanding and manipulating relationships between probability distributions. The core problem in OT theory is finding the optimal way to transform one probability distribution into another, aiming to minimize the total cost of the transformation process. The cost function can be physical distance, information loss in information theory, energy consumption, or other problem-related metrics.

[0073] Cross-entropy (CE) is a concept in information theory used to measure the difference between two probability distributions. In machine learning and deep learning, cross-entropy is often used to measure the difference between the model's predicted distribution and the true label distribution.

[0074] Normalization: Mapping sequences of values ​​with different ranges to the interval (0, 1) to facilitate data processing. In some cases, normalized values ​​can be directly expressed as probabilities.

[0075] Embedded coding, mathematically speaking, represents a correspondence, that is, mapping data in space X to space Y using a function F. This function F is injective, and the mapping result preserves the structure. An injective function means that the mapped data uniquely corresponds to the original data, and preserving the structure means that the size relationship between the original and mapped data is the same. For example, if there are data X1 and X2 before mapping, after mapping we get Y1 corresponding to X1 and Y2 corresponding to X2. If the original data X1 > X2, then correspondingly, the mapped data Y1 > Y2. For words, this means mapping words to another space to facilitate subsequent machine learning and processing.

[0076] After introducing some terms involved in the embodiments of this application, the implementation environment of the embodiments of this application will be described below.

[0077] Figure 1 This is a schematic diagram illustrating the implementation environment of a modality consistency-based prompting learning method provided in this application embodiment. See also... Figure 1 The implementation environment may include terminal 110 and server 140.

[0078] Terminal 110 is connected to server 140 via a wireless or wired network. Optionally, terminal 110 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 110 has an application with prompts for learning installed and running.

[0079] Server 140 is a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. Server 140 provides background services for applications running on terminal 110.

[0080] Optionally, terminal 110 refers to one of a plurality of terminals. This application embodiment only uses terminal 110 as an example for illustration.

[0081] After introducing the implementation environment of the embodiments of this application, the application scenarios of the embodiments of this application are described below. The technical solutions provided by the embodiments of this application can be applied to visual language pre-training models of any architecture. After adopting the technical solutions provided by the embodiments of this application, text prompts and visual prompts in a specific domain can be learned, thereby improving the performance of the visual language pre-training model in that specific domain.

[0082] After introducing the application scenarios of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below. Figure 2 This is a flowchart of a cue learning method based on modal consistency provided in an embodiment of this application. See also... Figure 2 Taking the server as the executing entity as an example, the method includes the following steps.

[0083] 201. The server obtains multiple text embedding features based on text prompts and multiple sample text categories through a target visual language pre-trained model.

[0084] The target visual language pre-trained model is a pre-trained visual language model with visual and language understanding capabilities. The text prompt belongs to the target domain, which is related to the downstream task requiring the use of the target visual language pre-trained model. For example, in the case of image-text mutual retrieval, the text prompt is used to instruct the retrieval of the corresponding image based on the input text. This text prompt is set by a technician or determined through prompt word engineering; this embodiment does not limit this. In some embodiments, the text prompt is in vector form. Sample category text describes the image category; these multiple sample category texts form a batch. In this embodiment, prompt learning requires the use of multiple batches of samples. One sample category text corresponds to one text embedding feature, which is generated based on the text prompt and one sample category text. When generating text embedding features, the text prompts corresponding to different sample category texts are the same.

[0085] 202. Based on visual cues and multiple sample images, the server obtains multiple visual embedding features. Both the text cues and the visual cues belong to the target domain.

[0086] The visual cues belong to the target domain, which is related to the downstream task that requires the use of the target visual language pre-trained model. For example, in the case of image-text mutual retrieval, the visual cues are used to instruct the retrieval of the corresponding category text based on the input image. These visual cues are set by technical personnel or determined through cue word engineering; this embodiment does not limit this. The multiple sample images constitute a batch of sample images. In this embodiment, cue learning requires the use of multiple batches of samples. One sample image corresponds to one visual embedding feature, and one visual embedding feature is generated based on a visual cue and a sample image. When generating visual embedding features, the visual cues corresponding to different sample images are the same.

[0087] 203. The server determines the visual feature distribution of the multiple visual embedding features and the text feature distribution of the multiple text embedding features.

[0088] Among them, the visual feature distribution of multiple visual embedding features is used to represent the distribution of multiple visual embedding features in the visual feature space; the text feature distribution of multiple text embedding features is used to represent the distribution of multiple text embedding features in the text feature space.

[0089] 204. Based on the visual feature distribution and the text feature distribution, the server determines the prediction transfer matrix, which is used to represent the way the multiple visual embedding features are transformed into the multiple text embedding features.

[0090] The predicted transfer matrix is ​​a transfer matrix predicted based on the visual feature distribution and the text feature distribution. The transfer matrix corresponds to a cost function, which represents the cost of transforming the multiple visual embedding features into the multiple text embedding features. The determination of the predicted transfer matrix is ​​constrained by the cost function.

[0091] 205. Based on the predicted transfer matrix, the multiple visual embedding features, and the annotation information, the server adjusts the text prompt and the visual prompt. The annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0092] The task of cue learning involves adjusting the text and visual cues. In other words, cue learning aims to learn appropriate text and visual cues so that downstream tasks can utilize them to achieve better results. There is a correspondence between multiple sample images and multiple sample category texts. Taking a single sample image as an example, the corresponding sample category text is the category label of that sample image, which is also the ground truth of its category. This labeling information is set by technicians according to the actual situation, and this application embodiment does not limit this.

[0093] The technical solution provided in this application utilizes a target visual language pre-trained model to obtain multiple text embedding features based on text prompts and multiple sample text categories. Multiple visual embedding features are obtained using visual prompts and multiple sample images, where both text and visual prompts belong to the target domain. The visual feature distributions of the multiple visual embedding features and the text feature distributions of the multiple text embedding features are determined. A prediction transfer matrix is ​​then used to represent the transformation of the multiple visual embedding features into multiple text embedding features. Based on this prediction transfer matrix, the multiple visual embedding features, and annotation information, the text and visual prompts are adjusted, thereby utilizing features and feature distributions for prompt learning and improving the effectiveness of prompt learning.

[0094] Steps 201-205 above are a brief introduction to the technical solutions provided in the embodiments of this application. The technical solutions provided in the embodiments of this application will be explained more clearly below with reference to some examples. See [link to relevant documentation]. Figure 3 Taking the server as the executing entity as an example, the method includes the following steps.

[0095] 301. The server obtains multiple text embedding features based on text prompts and multiple sample text categories through a target visual language pre-trained model.

[0096] The target visual language pre-trained model is a pre-trained visual language model with visual and language understanding capabilities. Text prompts belong to the target domain, which is related to downstream tasks requiring the use of the target visual language pre-trained model. Text prompts instruct the execution of downstream tasks; for example, in the case of image-text cross-referencing, the text prompt instructs the retrieval of the corresponding image based on the input text. These text prompts are set by technical personnel or determined through prompt word engineering; this application does not limit this. In some embodiments, the text prompt is in the form of a vector, also referred to as a text prompt latent vector. Sample category text describes the image category; these multiple sample category texts form a batch. In this application embodiment, prompt learning requires the use of multiple batches of samples. One sample category text corresponds to one text embedding feature, which is generated based on the text prompt and one sample category text. When generating text embedding features, the text prompts corresponding to different sample category texts are the same. In this application embodiment, the text embedding feature is represented as g. k (t), multiple text embedding features are represented as K represents the number of text embedding features, which is also the number of sample category texts. K is a positive integer.

[0097] In one possible implementation, for any sample category text among the multiple sample category texts, the server combines the sample category text and the text prompt using the target visual language pre-trained model to obtain a sample text combination. The server then uses the target visual language pre-trained model to extract semantic features from the sample text combination to obtain the text embedding features of the sample text combination.

[0098] In this embodiment, combining sample category text and text prompts yields a sample text combination used to perform downstream tasks. This sample text combination is used to indicate the image corresponding to the sample category text. The text embedding feature represents the semantics of the sample text combination. Since the sample text combination is used to indicate the corresponding image, the text embedding feature can reflect the image characteristics of the corresponding image to a certain extent. It should be noted that the above implementation describes processing one sample category text as an example. The server's processing of other sample category texts belongs to the same inventive concept as described above, and the implementation process will not be repeated.

[0099] In this implementation, a target visual language pre-trained model is used to combine sample category text with text prompts to obtain sample text combinations. Semantic features are extracted from the sample text combinations to obtain text embedding features of the sample text combinations. These text embedding features can represent the semantics of the text embedding features.

[0100] To provide a clearer explanation of the above embodiments, the following description will be divided into several parts.

[0101] The first part involves the server combining the sample category text and the text prompt using the target visual language pre-trained model to obtain the sample text combination.

[0102] In one possible implementation, the server uses the target visual language pre-trained model to concatenate the sample category text and the text prompt to obtain a sample text combination.

[0103] Among them, splicing refers to putting the sample category text and text prompt together directly without processing.

[0104] For example, the text prompt is in vector form. The server uses the target visual language pre-trained model to embed and encode the text of the sample category, obtaining the text embedding vector of the sample category. The server then uses the target visual language pre-trained model to concatenate the text embedding vector with the text prompt to obtain the combined sample text.

[0105] In one possible implementation, the server uses the target visual language pre-trained model to fuse the sample category text and the text prompt to obtain a sample text combination.

[0106] Fusion refers to the process of overlaying sample category text and text prompts after they have been transformed into the same form.

[0107] For example, the text prompt is in vector form. The server uses the target visual language pre-trained model to embed the text of the sample category, obtaining the text embedding vector of that sample category. The server then uses the target visual language pre-trained model to add the text embedding vector to the text prompt, obtaining the combined sample text.

[0108] In the second part, the server uses the target visual language pre-trained model to extract semantic features from the sample text combination, thereby obtaining the text embedding features of the sample text combination.

[0109] In one possible implementation, the server encodes the sample text combination using the target visual language pre-trained model based on an attention mechanism to obtain the text embedding features of the sample text combination.

[0110] The target visual language pre-trained model includes a text encoder, which encodes text into text embedding features. In the above implementation, the text encoder encodes text based on an attention mechanism. In some embodiments, the text encoder is an encoder of a BERT (Bidirectional Encoder Representations from Transformers) model.

[0111] 302. The server obtains multiple visual embedding features based on visual cues and multiple sample images. Both the text cues and the visual cues belong to the target domain.

[0112] In this application, visual cues belong to the target domain, which is related to the downstream task that requires the use of the target visual language pre-trained model. The visual cues are used to instruct the execution of the downstream task. For example, in the case of image-text cross-referencing, the visual cues instruct the retrieval of the corresponding category text based on the input image. These visual cues are set by technical personnel or determined through cue word engineering; this application does not limit this. The multiple sample images constitute a batch of sample images. In this application embodiment, cue learning requires the use of multiple batches of samples. One sample image corresponds to one visual embedding feature, and one visual embedding feature is generated based on a visual cue and a sample image. When generating visual embedding features, the visual cues corresponding to different sample images are the same. In this application embodiment, the visual embedding feature is represented as f. b (v), multiple visual embedding features are represented as B represents the batch size, indicating the number of visual embedding features. Furthermore, in this embodiment, both individual text embedding features and individual visual embedding features are independently and identically distributed.

[0113] In one possible implementation, for any one of the plurality of sample images, the server combines the sample image and the visual cue using the target visual language pre-trained model to obtain a sample image combination. The server then uses the target visual language pre-trained model to extract image features from the sample image combination to obtain the visual embedding features of the sample image combination.

[0114] In this embodiment, combining sample images and visual cues yields a sample image combination used to perform downstream tasks. This sample image combination is used to indicate the category text corresponding to the sample image. The visual embedding feature represents the semantics of the sample image combination. Since the sample image combination is used to indicate the corresponding category text, the visual embedding feature can reflect the textual characteristics of the corresponding category text to a certain extent. It should be noted that the above implementation describes processing a single sample image; the server's processing of other sample images follows the same inventive concept, and the implementation process will not be elaborated further.

[0115] To provide a clearer explanation of the above embodiments, the following description will be divided into several parts.

[0116] The first part involves the server combining the sample image and the visual cue using the target visual language pre-trained model to obtain a sample image combination.

[0117] In one possible implementation, the server uses the target visual language pre-trained model to stitch the sample image and the visual cue together to obtain a sample image combination.

[0118] Among them, splicing refers to putting sample images and visual cues together directly without processing.

[0119] For example, the visual cue is in the form of a vector. The server uses the target visual language pre-trained model to convolve the sample image to obtain the image feature vector of the sample image. The server then uses the target visual language pre-trained model to concatenate the image feature vector with the visual cue to obtain the combined sample image.

[0120] In one possible implementation, the server fuses the sample image and the visual cue using the target visual language pre-trained model to obtain a sample image combination.

[0121] Fusion refers to the process of overlaying sample images and visual cues after they have been transformed into the same form.

[0122] For example, the visual cue is in the form of a vector. The server uses the target visual language pre-trained model to convolve the sample image to obtain the image feature vector of the sample image. The server then uses the target visual language pre-trained model to fuse the image feature vector with the visual cue to obtain the sample image combination.

[0123] In the second part, the server uses the target visual language pre-trained model to extract image features from the sample image combination, thereby obtaining the visual embedding features of the sample image combination.

[0124] In one possible implementation, the server performs multiple convolutions on the sample image combination using the target visual language pre-trained model to obtain the visual embedding features of the sample image combination.

[0125] The target visual language pre-trained model includes an image encoder, which is used to encode images into visual embedding features. In the above implementation, the image encoder is a ResNET-based encoder.

[0126] 303. The server determines the visual feature distribution of the multiple visual embedding features and the text feature distribution of the multiple text embedding features.

[0127] The visual feature distribution of multiple visual embedding features represents the distribution of multiple visual embedding features in the visual feature space. Similarly, the text feature distribution of multiple text embedding features represents the distribution of multiple text embedding features in the text feature space.

[0128] In one possible implementation, the server inputs each visual embedding feature into first relational data to obtain a first distribution feature for each visual embedding feature. The server divides the first distribution feature of each visual embedding feature by the number of visual embedding features to obtain the visual feature distribution. The server inputs each text embedding feature into second relational data to obtain a second distribution feature for each text embedding feature. The server divides the second distribution feature of each text embedding feature by the number of text embedding features to obtain the text feature distribution.

[0129] In this embodiment, the first relation data and the second relation are functions, and the first relation data and the second relation are set by technicians according to the actual situation. This application does not limit this setting. For each sample (text embedding feature and visual embedding feature), a uniformly distributed probability is used. Accordingly, the probability of text embedding feature is 1 / B, and the probability of visual embedding feature is 1 / K.

[0130] For example, the server determines the visual feature distribution using the following formula (1). The server determines the text feature distribution using the following formula (2).

[0131]

[0132] Where P(v) represents the visual feature distribution, v represents the visual cue, and b represents the index of the visual embedding feature. Let B be the first relational data, B be the number of visual embedding features, Q(t) be the text feature distribution, t represent the text prompt, and k represent the index of the text embedding feature. For the second relational data, K is the number of text embedding features.

[0133] In some embodiments, the first relational data Second relation data Both are Dirac functions.

[0134] 304. Based on the visual feature distribution and the text feature distribution, the server determines a prediction transfer matrix, which is used to represent the way the multiple visual embedding features are transformed into the multiple text embedding features.

[0135] The predicted transfer matrix is ​​a transfer matrix predicted based on the visual feature distribution and the text feature distribution. The transfer matrix corresponds to a cost function, which represents the cost of transforming the multiple visual embedding features into the multiple text embedding features. The determination of the predicted transfer matrix is ​​constrained by the cost function.

[0136] In one possible implementation, the server determines an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution. This cost function determines the cost of transforming the multiple visual embedding features into the multiple text embedding features. The server then optimizes the initial transfer matrix based on the cost function to obtain a predicted transfer matrix with the minimum cost.

[0137] Wherein, the cost function is obtained through To represent, the cost function element C bk The distance between sample image b and sample category text k is measured, and this distance is also the cost. The optimization described above, which minimizes the cost corresponding to the predicted transfer matrix, refers to optimizing the OT distance between the visual feature distribution P(v) and the text feature distribution Q(t). By minimizing the OT distance between the visual feature distribution P(v) and the text feature distribution Q(t), it is possible to align the visual feature distribution P(v) and the text feature distribution Q(t) based on their similarity (i.e., the cost function) in an unsupervised manner. In some embodiments, this distance is represented by cosine similarity, for example, C... bk =1-cos(f b (v),g k (t)). Furthermore, according to OT theory, the transfer matrix... The following conditions must be met: The transfer matrix T is a probability matrix that describes how B visual embedding features are transferred to K text embedding features during a mini-batch sample distribution matching process.

[0138] For example, the server uses the following formula (3) to represent the process of minimizing the OT distance and the following formula (4) to represent the method of determining the predicted transmission matrix.

[0139] min t,v OT(P(v),Q(t))=min t,v min T∈∏(P(v),Q(t)) <T,C> (3)

[0140] Wherein, OT(P(v),Q(t)) represents the OT distance between the visual feature distribution P(v) and the text feature distribution Q(t), T is the transfer matrix, C is the cost function, and ∏(P(v),Q(t)) represents the set of all joint distributions that make the marginal distributions P(v) and Q(t). The transfer matrix belongs to the set of this joint distribution. In the above implementation, both the initial transfer matrix and the predicted transfer matrix belong to the set of this joint distribution.

[0141] T * =arg min T∈∏(P(v),Q(t)) <T,C> -λh(T) (4)

[0142] Among them, T * To predict the transfer matrix, λ≥0, where is the hyperparameter for entropy regularization, h(T)=-∑ ij T ij lnT ij The regularization term represents entropy and is used to reduce computational costs.

[0143] 305. Based on the predicted transfer matrix and the multiple visual embedding features, the server determines the first distribution probability of each visual embedding feature on the multiple text embedding features.

[0144] In one possible implementation, for any one of the plurality of visual embedding features, the server determines a first transmission vector corresponding to that visual embedding feature in the prediction transmission matrix. Based on the first transmission vector corresponding to the visual embedding feature and the plurality of first transmission vectors in the prediction transmission matrix, the service determines a first distribution probability of the visual embedding feature across the plurality of text embedding features.

[0145] The prediction transfer matrix comprises multiple rows. In the case where one row corresponds to a visual embedding feature (i.e., one sample image), one column corresponds to a text embedding feature (i.e., one sample text category). Similarly, in the case where one column corresponds to a visual embedding feature (i.e., one sample image), and one row corresponds to a text embedding feature (i.e., one sample text category), the first transfer vector corresponding to that visual embedding feature is one row in the prediction transfer matrix.

[0146] In the above implementation, the prediction transfer matrix is ​​used to determine the first distribution probability of visual embedding features on multiple text embedding features. That is, the optimized transfer matrix is ​​used to predict the probability of a sample image corresponding to multiple sample category texts, thereby enhancing the learning of subsequent visual and text cues.

[0147] For example, the server determines the first distribution probability of the visual embedding feature over multiple text embedding features using the following formula (5).

[0148]

[0149] Among them, T * To predict the transfer matrix, that is, the optimized transfer matrix, x b Let y represent the visual embedding feature, and p(y|x) represent the text embedding feature. b ,T * ) represents the first probability distribution of the visual embedding feature across multiple text embedding features, where τ is a constant. This is the first transfer vector corresponding to the visual embedding feature. For the predicted transfer matrix T * The transfer vector in.

[0150] It should be noted that p(y|x) b ,T * Unlike the probability calculation based on point-to-point cross-entropy loss in related technologies, p(y|x) b ,T * The transfer vector in ) (Similarity can be considered) optimization from the perspective of distribution matching, while related technologies calculate it in a point-to-point manner. In other words, since the transfer matrix T needs to satisfy the constraints of two marginal distributions, visual feature distribution P(v) and text feature distribution Q(t), when solving the optimal transfer problem, the elements of the transfer matrix T... It will be affected by all K text embedding features and B visual embedding features in the current batch, rather than just the relationship between a single text embedding feature and a single visual embedding feature.

[0151] 306. The server determines the target transfer matrix based on the annotation information, which is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0152] There is a correspondence between multiple sample images and multiple sample category texts. Taking a sample image as an example, the sample category text corresponding to the sample image is the category label of the sample image, which is the ground truth of the category of the sample image. This label information is set by technicians according to the actual situation, and this application embodiment does not limit it.

[0153] In one possible implementation, the server converts the annotation information into a matrix form to obtain the target transmission matrix.

[0154] The annotation information records the correspondence between multiple sample images and multiple sample category texts. In the process of converting the annotation information into matrix form, the rows of the matrix are defined as corresponding to sample images, and the columns of the matrix are defined as corresponding to sample category texts. The method for determining the values ​​in the matrix is ​​as follows: if a sample image in the annotation information corresponds to a sample category text, then the value corresponding to that sample image and sample category text in the matrix is ​​set to 1; if a sample image in the annotation information does not correspond to a sample category text, then the value corresponding to that sample image and sample category text in the matrix is ​​set to 0, thus obtaining the target transmission matrix.

[0155] 307. The server adjusts the text prompt and the visual prompt based on the first distribution probability of each visual embedding feature on the multiple text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

[0156] The task of cue learning is to adjust the text and visual cues. In other words, cue learning aims to learn appropriate text and visual cues so that downstream tasks can use the learned text and visual cues to achieve better results.

[0157] In one possible implementation, the server adjusts the text prompt and the visual prompt based on a first distribution probability of each visual embedding feature on multiple text embedding features and a first difference information between the labeled category text in the annotation information, and a second difference information between the predicted transfer matrix and the target transfer matrix.

[0158] Here, the labeled category text refers to the sample category text corresponding to the sample images corresponding to each visual embedding feature in the labeled information. The first difference information is the category text difference determined from a distribution perspective, and the second difference information is the distribution difference of the transfer matrix. In some embodiments, the first difference information is the difference information between the first distribution probability and the labeled distribution probability, where the labeled distribution probability is the distribution probability corresponding to the labeled category text. For example, there exists a labeled distribution probability (1,0,0,0,0), where 1 represents the probability of labeled category text and 0 represents the probability of other category text.

[0159] For example, the server constructs a first loss function based on the first difference information. The server constructs a second loss function based on the second difference information. The server combines the first, second, and third loss functions to obtain a target loss function. This third loss function is determined based on third difference information, which is the difference between the second distribution probabilities of each visual embedding feature across multiple text embedding features and the labeled category text in the annotation information. The second distribution probabilities are directly determined based on the visual embedding features. The server adjusts the text prompt and the visual prompt based on this target loss function.

[0160] Among them, the first loss function is also known as the cross-entropy loss function, the second loss function is also known as the distribution matching loss function, and the third loss function is the single-mode matching loss function.

[0161] For example, the form of the target loss function can be seen in the following formula (6).

[0162]

[0163] in, Let be the target loss function. For the first loss function, For the second loss function, For the third loss function, β c β is the weight of the first loss function. m β is the weight of the second loss function. m and β c The settings should be made by technicians according to the actual situation, and this application embodiment does not limit this.

[0164] The first loss function is expressed by the following formula (7).

[0165]

[0166] in, To express expectation, y bThe probability distribution corresponding to the labeled category text is determined using the first loss function. The purpose of using cue learning is to make the sample images and corresponding sample category texts closer from the perspective of distribution matching.

[0167] The second loss function is expressed by the following formula (9).

[0168]

[0169] Where Ts is the target transmission matrix.

[0170] In some embodiments, the server employs gradient descent to adjust the text prompt and the visual prompt based on the target loss function.

[0171] The purpose of adjusting the text and visual cues is to reduce the loss value of the target loss function. Since both text and visual cues are in vector form, adjusting them is equivalent to adjusting the corresponding vectors.

[0172] The following will combine Figure 4 The technical solutions provided in the embodiments of this application will be described.

[0173] See Figure 4 Multiple sample images and visual cues are combined and input into the image encoder of the target visual language pre-trained model to obtain multiple visual embedding features. Multiple sample text categories and text cues are combined and input into the text encoder to obtain multiple text embedding features. The visual feature distribution P(v) of the multiple visual embedding features is determined, and the text feature distribution Q(t) of the multiple text embedding features is determined. The OT distance between the visual feature distribution P(v) and the text feature distribution Q(t) is minimized to obtain the prediction transfer matrix, where one row of the prediction transfer matrix corresponds to one sample image, and one column corresponds to one sample text category. The text cues and visual cues are adjusted using the matrix corresponding to the annotation information to achieve cue learning.

[0174] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0175] The technical solution provided in this application utilizes a target visual language pre-trained model to obtain multiple text embedding features based on text prompts and multiple sample text categories. Multiple visual embedding features are obtained using visual prompts and multiple sample images, where both text and visual prompts belong to the target domain. The visual feature distributions of the multiple visual embedding features and the text feature distributions of the multiple text embedding features are determined. A prediction transfer matrix is ​​then used to represent the transformation of the multiple visual embedding features into multiple text embedding features. Based on this prediction transfer matrix, the multiple visual embedding features, and annotation information, the text and visual prompts are adjusted, thereby utilizing features and feature distributions for prompt learning and improving the effectiveness of prompt learning.

[0176] See Figure 5 During the experiment, the technical solutions provided in the embodiments of this application were applied to different base models and datasets, and all of them were able to improve the performance of the base models. Figure 5 The middle value represents the average performance across 11 datasets.

[0177] See Figure 6 Visual experiments confirm that the technical solution provided in this application can indeed improve the matching degree of the benchmark model in the text space and image space, and the distance between the features in the two spaces is closer. In addition, the learned transfer matrix does have a higher accuracy in sample prediction, proving that it is very meaningful to improve the cue learning framework through the distribution alignment method of modality matching.

[0178] Figure 7 This is a schematic diagram of the structure of a prompting learning device based on modal consistency provided in an embodiment of this application. See also... Figure 7 The device includes:

[0179] Embedding module 701 is used to obtain multiple text embedding features based on text prompts and multiple sample text categories using a target visual language pre-trained model. It also obtains multiple visual embedding features based on visual prompts and multiple sample images, where both the text prompts and visual prompts belong to the target domain.

[0180] The distribution determination module 702 is used to determine the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features.

[0181] The transfer matrix determination module 703 is used to determine a predicted transfer matrix based on the visual feature distribution and the text feature distribution. The predicted transfer matrix is ​​used to represent the way the multiple visual embedding features are transformed into the multiple text embedding features.

[0182] The adjustment module 704 is used to adjust the text prompt and the visual prompt based on the predicted transfer matrix, the multiple visual embedding features and the annotation information, wherein the annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images.

[0183] In one possible implementation, the embedding module 701 is used to combine any sample category text from the plurality of sample category texts with the text prompt using the target visual language pre-training model to obtain a sample text combination. Semantic features are extracted from the sample text combination using the target visual language pre-training model to obtain the text embedding features of the sample text combination. Similarly, for any sample image from the plurality of sample images, the sample image is combined with the visual prompt using the target visual language pre-training model to obtain a sample image combination. Image features are extracted from the sample image combination using the target visual language pre-training model to obtain the visual embedding features of the sample image combination.

[0184] In one possible implementation, the distribution determination module 702 is used to substitute each visual embedding feature into first relation data to obtain a first distribution feature for each visual embedding feature. The first distribution feature of each visual embedding feature is divided by the number of visual embedding features to obtain the visual feature distribution. Each text embedding feature is then substituted into second relation data to obtain a second distribution feature for each text embedding feature. The second distribution feature of each text embedding feature is divided by the number of text embedding features to obtain the text feature distribution.

[0185] In one possible implementation, the transfer matrix determination module 703 is used to determine an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution. The cost function is used to determine the cost of transforming the plurality of visual embedding features into the plurality of text embedding features. Based on the cost function, the initial transfer matrix is ​​optimized to obtain the predicted transfer matrix, which corresponds to the minimum cost.

[0186] In one possible implementation, the adjustment module 704 is configured to determine a first distribution probability of each visual embedding feature on the plurality of text embedding features based on the predicted transfer matrix and the plurality of visual embedding features. A target transfer matrix is ​​determined based on the annotation information. The text prompt and the visual prompt are adjusted based on the first distribution probability of each visual embedding feature on the plurality of text embedding features, the predicted transfer matrix, the target transfer matrix, and the annotation information.

[0187] In one possible implementation, the adjustment module 704 is used to determine, for any one of the plurality of visual embedding features, a first transmission vector in the prediction transmission matrix corresponding to the visual embedding feature.

[0188] Based on the first transfer vector corresponding to the visual embedding feature and multiple first transfer vectors in the predicted transfer matrix, the first distribution probability of the visual embedding feature on the multiple text embedding features is determined.

[0189] In one possible implementation, the adjustment module 704 is used to convert the annotation information into a matrix form to obtain the target transmission matrix.

[0190] In one possible implementation, the adjustment module 704 is used to adjust the text prompt and the visual prompt based on a first difference information between the first distribution probability of each visual embedding feature on multiple text embedding features and the labeled category text in the annotation information, and a second difference information between the predicted transfer matrix and the target transfer matrix. Here, the labeled category text refers to the sample category text corresponding to the sample images corresponding to each visual embedding feature in the annotation information.

[0191] In one possible implementation, the adjustment module 704 is used to construct a first loss function based on the first difference information. A second loss function is constructed based on the second difference information. The first loss function, the second loss function, and the third loss function are combined to obtain a target loss function. The third loss function is determined based on third difference information, which is the difference between the second distribution probability of each visual embedding feature on multiple text embedding features and the labeled category text in the annotation information. The second distribution probability is a distribution probability directly determined based on the visual embedding features. Based on the target loss function, the text prompt and the visual prompt are adjusted.

[0192] It should be noted that the modality consistency-based prompting learning device provided in the above embodiments is only illustrated by the division of the above functional modules when performing prompting learning based on modality consistency. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the modality consistency-based prompting learning device and the modality consistency-based prompting learning method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0193] The technical solution provided in this application utilizes a target visual language pre-trained model to obtain multiple text embedding features based on text prompts and multiple sample text categories. Multiple visual embedding features are obtained using visual prompts and multiple sample images, where both text and visual prompts belong to the target domain. The visual feature distributions of the multiple visual embedding features and the text feature distributions of the multiple text embedding features are determined. A prediction transfer matrix is ​​then used to represent the transformation of the multiple visual embedding features into multiple text embedding features. Based on this prediction transfer matrix, the multiple visual embedding features, and annotation information, the text and visual prompts are adjusted, thereby utilizing features and feature distributions for prompt learning and improving the effectiveness of prompt learning.

[0194] This application provides a computer device for performing the above-described method. This computer device can be implemented as a terminal or a server. The structure of the terminal will be described below:

[0195] Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal 800 can be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal 800 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0196] Typically, terminal 800 includes one or more processors 801 and one or more memories 802.

[0197] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0198] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 are used to store at least one computer program, which is executed by the processor 801 to implement the modality-consistent cue learning method provided in the method embodiments of this application.

[0199] In some embodiments, the terminal 800 may also optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 808.

[0200] Peripheral device interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 801 and memory 802. In some embodiments, processor 801, memory 802 and peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 801, memory 802 and peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0201] The radio frequency (RF) circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc.

[0202] Display screen 805 is used to display a user interface (UI). This UI may include graphics, text, icons, video, and any combination thereof. When display screen 805 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 801 for processing. In this case, display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0203] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal.

[0204] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 801 for processing, or input to the radio frequency circuit 804 to realize voice communication.

[0205] The power supply 808 is used to supply power to the various components in the terminal 800. The power supply 808 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0206] In some embodiments, the terminal 800 further includes one or more sensors 809. The one or more sensors 809 include, but are not limited to, an accelerometer 810, a gyroscope 811, a pressure sensor 812, an optical sensor 813, and a proximity sensor 814.

[0207] Accelerometer 810 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by terminal 800.

[0208] The gyroscope sensor 811 can detect the orientation and rotation angle of the terminal 800. The gyroscope sensor 811 can work in conjunction with the accelerometer sensor 810 to collect the user's 3D movements on the terminal 800.

[0209] The pressure sensor 812 can be installed on the side bezel of the terminal 800 and / or on the lower layer of the display screen 805. When the pressure sensor 812 is installed on the side bezel of the terminal 800, it can detect the user's grip signal on the terminal 800, and the processor 801 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 812. When the pressure sensor 812 is installed on the lower layer of the display screen 805, the processor 801 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 805.

[0210] An optical sensor 813 is used to collect ambient light intensity. In one embodiment, a processor 801 can control the display brightness of a display screen 805 based on the ambient light intensity collected by the optical sensor 813.

[0211] The proximity sensor 814 is used to detect the distance between the user and the front of the terminal 800.

[0212] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on terminal 800 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0213] The aforementioned computer equipment can also be implemented as a server. The structure of a server is described below:

[0214] Figure 9This is a schematic diagram of a server structure provided in an embodiment of this application. The server 900 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 901 and one or more memories 902. The one or more memories 902 store at least one computer program, which is loaded and executed by the one or more processors 901 to implement the methods provided in the various method embodiments described above. Of course, the server 900 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 900 may also include other components for implementing device functions, which will not be elaborated upon here.

[0215] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the modality-consistency-based cue learning method in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0216] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the above-described modality consistency-based cueing learning method.

[0217] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0218] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0219] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A cue learning method based on modal consistency, characterized in that, The method includes: By using a target visual language pre-trained model, multiple text embedding features are obtained based on text prompts and multiple sample text categories; multiple visual embedding features are obtained based on visual prompts and multiple sample images. The text prompts are used to indicate the retrieval of the corresponding image based on the input text, and the visual prompts are used to indicate the retrieval of the corresponding category text based on the input image. Determine the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features; Based on the visual feature distribution and the text feature distribution, a prediction transfer matrix is ​​determined, which is used to represent the way the plurality of visual embedding features are transformed into the plurality of text embedding features; Based on the predicted transfer matrix, the multiple visual embedding features, and the annotation information, the text prompts and the visual prompts are adjusted. The annotation information is used to record the correspondence between the multiple sample category texts and the multiple sample images. The adjustment of the text prompts and visual prompts based on the predicted transfer matrix, the multiple visual embedding features, and the annotation information includes: Based on the predicted transfer matrix and the plurality of visual embedding features, a first distribution probability of each of the visual embedding features on the plurality of text embedding features is determined; Based on the annotation information, the target transmission matrix is ​​determined; Based on the first distribution probability of each of the visual embedding features on multiple text embedding features and the first difference information between the labeled category text in the annotation information, and the second difference information between the predicted transfer matrix and the target transfer matrix, the text prompts and the visual prompts are adjusted. Wherein, the labeled category text is the sample category text corresponding to the sample image corresponding to each of the visual embedding features in the labeled information.

2. The method according to claim 1, characterized in that, The method utilizes a target visual language pre-trained model to obtain multiple text embedding features based on text prompts and multiple sample text categories, including: For any sample category text among the multiple sample category texts, the sample category text and the text prompt are combined using the target visual language pre-training model to obtain a sample text combination; semantic features are extracted from the sample text combination using the target visual language pre-training model to obtain the text embedding features of the sample text combination; Based on visual cues and multiple sample images, multiple visual embedding features are obtained, including: For any one of the plurality of sample images, the sample image and the visual cue are combined using the target visual language pre-training model to obtain a sample image combination; image features are extracted from the sample image combination using the target visual language pre-training model to obtain the visual embedding features of the sample image combination.

3. The method according to claim 1, characterized in that, Determining the visual feature distribution of the plurality of visual embedding features and the text feature distribution of the plurality of text embedding features includes: Substitute each of the visual embedding features into the first relation data to obtain the first distribution feature of each of the visual embedding features; divide the first distribution feature of each of the visual embedding features by the number of visual embedding features to obtain the visual feature distribution; Substitute each of the text embedding features into the second relation data to obtain the second distribution features of each of the text embedding features; divide the second distribution features of each of the text embedding features by the number of text embedding features to obtain the text feature distribution.

4. The method according to claim 1, characterized in that, The step of determining the prediction transfer matrix based on the visual feature distribution and the text feature distribution includes: Determine an initial transfer matrix and a cost function between the visual feature distribution and the text feature distribution, wherein the cost function is used to determine the cost of transforming the plurality of visual embedding features into the plurality of text embedding features; The initial transmission matrix is ​​optimized based on the cost function to obtain the predicted transmission matrix, which has the minimum cost.

5. The method according to claim 1, characterized in that, The step of determining the first distribution probability of each visual embedding feature on the plurality of text embedding features based on the predicted transfer matrix and the plurality of visual embedding features includes: For any visual embedding feature among the plurality of visual embedding features, determine the first transmission vector corresponding to the visual embedding feature in the prediction transmission matrix; Based on the first transmission vector corresponding to the visual embedding feature and multiple first transmission vectors in the predicted transmission matrix, the first distribution probability of the visual embedding feature on the multiple text embedding features is determined.

6. The method according to claim 1, characterized in that, Determining the target transmission matrix based on the annotation information includes: The annotation information is converted into matrix form to obtain the target transmission matrix.

7. The method according to claim 1, characterized in that, The adjustment of the text prompts and the visual prompts based on the first distribution probability of each of the visual embedding features on multiple text embedding features and the first difference information between the labeled category text in the annotation information, and the second difference information between the predicted transfer matrix and the target transfer matrix, includes: A first loss function is constructed based on the first difference information; A second loss function is constructed based on the second difference information; The first loss function, the second loss function, and the third loss function are combined to obtain the target loss function. The third loss function is determined based on the third difference information, which is the difference information between the second distribution probability of each visual embedding feature on multiple text embedding features and the labeled category text in the annotation information. The second distribution probability is the distribution probability directly determined based on the visual embedding features. The text prompts and visual prompts are adjusted based on the target loss function.

8. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the modality consistency-based cueing learning method as described in any one of claims 1 to 7.

Images