Visual language understanding task processing method and system

Abstract

The present disclosure relates to a visual language understanding task processing method and system. The method is based on a visual language pre-training model operation, and comprises: constructing task training data comprising a plurality of image-text pairs; feeding the training data into the pre-training model to obtain a plurality of image and text embedding vectors; fusing the embedding vectors of the same image-text pair into one fused feature; constructing a knowledge base prototype matrix based on the plurality of fused features; and performing similarity matching using the prototype matrix to obtain an inference result. The present invention re-formulates various visual language understanding tasks as an open-book similarity matching problem, uses a knowledge base prototype matrix to record the salient features of each category through a fused feature, and then uses image-text pairs to perform similarity matching with the prototype of each category. In the fine-tuning stage of the downstream task, more supervision clues can be provided from the low-resource training set by utilizing the implicit ranking information of the real label through contrastive learning.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and in particular to a method and system for processing visual language understanding tasks. Background Technology

[0002] Pre-trained visual language models, such as XVLM and CLIP (Contrastive Visual-Language Pretraining), have shown great potential in VLU (Visual Language Understanding) tasks. Since pre-trained models are not designed for specific tasks, fine-tuning is often necessary for downstream tasks. Conventional fine-tuning methods rely heavily on time-consuming and labor-intensive data annotation, which can be problematic in resource-constrained scenarios. Methods have been proposed to fine-tune some parameters while preserving the model's pre-training knowledge. Furthermore, visual object extraction and visual question answering can be reformulated as cloze tests using handcrafted prompts; lightweight adapters can also be used to preserve CLIP's knowledge.

[0003] However, existing methods require designing different task execution architectures for different VLU tasks, which consumes a lot of manpower when constructing prompts and may encounter problems such as poor performance in low-resource scenarios.

[0004] Therefore, an improved method is needed to implement downstream tasks. Summary of the Invention

[0005] The technical problem this disclosure aims to solve is to provide a method and system for visual language understanding (VLU) tasks. The method uniformly reformulates various VLU tasks as an open-book affinity-matching problem. Specifically, this invention uses a knowledge base prototype matrix to record the salient features of each category through visual-text fusion features, and then performs similarity matching between image-text pairs and the prototypes of each category. Furthermore, in the fine-tuning stage for downstream tasks, implicit ranking information from real labels can be further utilized through contrastive learning to provide more supervisory cues from the low-resource training set. At this point, all parameters of the text and visual encoders in the pre-trained visual language model are fixed, thus providing extremely high parameter training efficiency.

[0006] According to a first aspect of this disclosure, a method for processing a visual language understanding task is provided. The method operates based on a visual language pre-trained model (VL-PTM) and includes: constructing training data for the visual language understanding task, the training data including multiple image-text pairs; feeding the training data of the visual language understanding task into the VL-PTM to obtain multiple image embedding vectors and multiple text embedding vectors located in the same embedding space; fusing the image embedding vectors and text embedding vectors belonging to the same image-text pair into a fusion feature; constructing a knowledge base prototype matrix based on the multiple fusion features; and using the knowledge base prototype matrix to perform similarity matching to obtain the inference result of the visual language understanding task.

[0007] Optionally, the training data also includes labels corresponding to each image-text pair, and constructing a knowledge base prototype matrix based on multiple fusion features includes: classifying the multiple fusion features based on labels; calculating the average fusion feature of each category fusion feature; and using each average fusion feature as the prototype feature of the corresponding category to obtain the knowledge base prototype matrix.

[0008] Optionally, using the knowledge base prototype matrix to perform similarity matching to obtain the inference result of the visual language understanding task includes: in the inference stage, feeding images and text into the VL-PTM to obtain image embedding vectors and text embedding vectors to be classified; fusing the image embedding vectors and text embedding vectors to be classified into a fusion feature to be classified; performing similarity matching between the fusion feature to be classified and the prototype feature of each category in the knowledge base prototype matrix; and selecting the category corresponding to the prototype feature that best matches the fusion feature to be classified as the inference result.

[0009] Optionally, fusing the image embedding vector and the text embedding vector belonging to the same image-text pair into a fusion feature includes concatenating at least two of the following vectors: the image embedding vector belonging to the same image-text pair; the text embedding vector belonging to the same image-text pair; the sum of the image embedding vector and the text embedding vector; the difference between the image embedding vector and the text embedding vector; and the element-wise product of the image embedding vector and the text embedding vector.

[0010] Optionally, the visual language understanding task includes one of the following: a visual question answering task, wherein in the training data, the text in a plurality of image-text pairs corresponds to a non-open-ended question posed for the image, and the label corresponds to the answer option for the non-open-ended question; and a visual entailment task, wherein in the training data, the text in the plurality of image-text pairs corresponds to a hypothesis posed for the image, and the label corresponds to entailment, contradiction, or neutrality.

[0011] Optionally, the visual language understanding task is an image classification task. During the training phase of the image classification task, the text in the multiple image-text pairs corresponds to the text description of the category. During the inference phase of the image classification task, the unclassified fusion feature between the image to be classified and the text description of each category is calculated. Each unclassified fusion feature is matched with its corresponding prototype feature based on similarity, and the category with the highest matching degree is selected as the inference result.

[0012] Optionally, the method further includes: during the training phase of the visual language understanding task, updating parameters based on a first loss function that reflects the difference between the representation prediction classification and the true label.

[0013] Optionally, the method further includes: during the training phase of the visual language understanding task, treating the probability of the image text pair belonging to its category as a positive sample and the probability of the other categories as a negative sample, constructing a second loss function for contrastive learning; and updating parameters based on the first loss function and the second function.

[0014] Optionally, during the training and inference phases of the visual language understanding task, the parameters of the VL-PTM image encoder and text encoder remain unchanged.

[0015] According to a second aspect of this disclosure, a visual language understanding task processing system is provided for performing the visual language understanding task processing method as described in the first aspect, and includes: a training device for acquiring a knowledge base prototype matrix for a specific visual language understanding task; and an inference device for performing inference for the specific visual language understanding task based on the knowledge base prototype matrix.

[0016] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0017] According to a fourth aspect of this disclosure, a computer program product is provided, including executable code that, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0018] According to a fifth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0019] Therefore, this invention proposes an efficient parameter fine-tuning method for pre-trained visual-text models in low-resource visual-text understanding tasks. By unifying the VLU task into a similarity matching problem based on a knowledge base prototype matrix, the parameter scale of the method is greatly reduced. Furthermore, contrastive learning utilizes the implicit ranking information of real labels to provide more supervision signals for model training, thereby enhancing model performance. Experimental results show that the proposed method achieves better performance than existing techniques in few-sample image classification tasks, as well as various VE and VQA tasks. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0021] Figure 1 The basic training process of CLIP is shown.

[0022] Figure 2 The process of image classification using CLIP is shown.

[0023] Figure 3 Several examples of visual question answering are shown.

[0024] Figure 4 An example of a visual entailment task is shown.

[0025] Figure 5 A schematic flowchart of a visual language understanding task processing method according to an embodiment of the present invention is shown.

[0026] Figure 6 A schematic diagram of the composition of a visual language understanding task processing system according to an embodiment of the present invention is shown.

[0027] Figure 7 A schematic diagram of a computing device that can be used to implement the above-described VLU task processing method according to an embodiment of the present invention is shown. Detailed Implementation

[0028] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0029] Conventional image classification models are often trained under full supervision on labeled image datasets, which requires a large amount of data and manual annotation. This limits the model's applicability and generalization ability, making it unsuitable for task transfer. Pre-trained visual language models have been proposed, especially CLIP (Contrastive Visual-Language Pretraining), which is trained using a large amount of text-image pairing data (400 million text-image pairs) easily obtained from the Internet.

[0030] Figure 1 The basic training process of CLIP is illustrated. As shown in the figure, the CLIP model structure consists of two parts: a text encoder and an image encoder. The text encoder can be implemented using a text Transformer model; the image encoder uses ResNet based on CNN and ViT based on Transformer.

[0031] During the training phase, assume a training batch contains N text-image pairs. Each text-image pair can be represented as follows: Figure 1 The example on the left shows an image and its text description ("a man wearing a white hat"). For these N text-image pairs, the N texts are first encoded using a text encoder. Assume the text encoder encodes each text into a string of length d. t If the text data is a one-dimensional vector, then the output of this batch of text data after being processed by the text encoder is [T1, T2, ..., T]. N ], with dimensions (N, d t Similarly, N images are first encoded using an image encoder. Assume the image encoder encodes each text line into a string of length d. i If the image data in this batch is a one-dimensional vector, then the output of the image encoder is [I1, I2, ..., I...]. N ], with dimensions (N,d i ).

[0032] The obtained [T1,T2,...,T] N [I1, I2, ..., I] N In this context, there is a one-to-one correspondence between text and images. For example, T1 corresponds to I1, T2 corresponds to I2, ..., T N with I NCorrespondingly, these N correspondences are marked as positive samples; while text-image pairs that do not originally correspond are marked as negative samples, for example, T1 and I2 do not correspond. Thus, N positive samples are obtained, N... 2 -N negative samples, these positive and negative samples can be used as positive and negative labels to train text encoders and image encoders.

[0033] In the specific training process, T can be calculated. i with I j The cosine similarity T among (i,j∈[1,N]) i ·I j This is used to measure the correspondence between corresponding text and images. The higher the cosine similarity, the stronger the correlation between text and images. i with I j The stronger the correspondence, the better. By training the parameters of the text encoder and image encoder, we maximize the cosine similarity of N positive samples and minimize N. 2 - Cosine similarity of N negative samples. For example... Figure 1 As shown, the goal is to maximize the values ​​in the diagonal and minimize the values ​​in the other off-diagonal areas. Therefore, the optimization objective can be written as:

[0034]

[0035] After large-scale text-image pre-training, CLIP can first calculate the cosine similarity between the input text and the image through encoding to determine the degree of matching of the data pairs.

[0036] CLIP, trained as described above, serves as a Vision-Textual Pre-Trained Model (VT-PLM). Its parameters contain a wealth of knowledge for determining whether images and text match. CLIP utilizes the same embedding space to uniformly represent images and text, significantly improving the performance of zero-shot and few-shot vision-language understanding tasks.

[0037] Visual language understanding tasks are cross-modal tasks involving images and text. Common cross-modal tasks include image classification tasks based on image-text matching, visual entailment tasks, and visual question answering tasks.

[0038] Image classification (IC) is a task that involves classifying an input image into a specific category. CLIP, thanks to the vast amount of knowledge acquired through pre-training, is even capable of performing this task in zero-shot mode. Figure 2This demonstrates the process of image classification using CLIP. First, all categories in the transfer dataset can be converted into text. Specifically, image descriptions can be constructed such as "A photo of {object}", and the categories can be converted into input text. For example, ImageNet has 1000 categories, resulting in 1000 texts, such as "A photo of Dog", "A photo of cat", etc. These 1000 texts can then be input into a text encoder to obtain 1000 encoded vectors T. i (i=1,…,1000)(N=1000) are used as text features.

[0039] The image to be classified (e.g., a single image) is input into an image encoder to obtain the encoded vector I1. Cosine similarity is calculated between I1 and each of the 1000 text features to find the largest similarity among the 1000. Figure 2 In the example, the vector with the highest cosine similarity to I1 is T3, which is the vector obtained by inputting the text "A photo of Dog" into the text encoder. Therefore, it is determined that the image to be classified is most matched with the third text label (dog), and the image can be classified as a dog.

[0040] In a typical IC task, the training data includes images and their classification labels. During training, a loss function representing the difference between the model's output classification and the label classification is used to update the parameters. In the inference phase, the input is an image, and the model needs to output the correct classification to which the image belongs.

[0041] Visual Question Answering (VQA) tasks require a model to observe and analyze an image and then answer questions related to that image. VQA is a research area that combines image recognition and natural language understanding. In visual question answering databases, visual question answering systems need to answer binary classification questions, counting questions, or open-ended questions based on images. Figure 3 Several examples of visual question answering are shown.

[0042] exist Figure 3 The question at the top, "Is the umbrella upside down?", is a binary classification question requiring a yes or no answer. Clearly, the left image answers "yes," while the right image answers "no." Figure 3In the middle section, the question "How many children are in the bed?" is a counting problem, requiring an answer of a specific number. Clearly, the answer in the left image is "2," and the answer in the right image is "1." However,... Figure 3 The question "Who is wearing glasses?" in the lower part is an open question. For example, the answer to the left picture is "man" and the answer to the right picture is "woman".

[0043] In a typical VQA task, training data consists of image-text pairs and their labels. Each image-text pair includes an image and a question posed to the image (as text), while the label corresponds to the correct answer to the question. During training, a loss function representing the difference between the model's output answer and the correct answer indicated by the label is used for parameter updates. During inference, the input consists of an image and text (the question posed to the image), and the model is required to output the correct answer based on the image and the question.

[0044] The goal of the Visual Entailment (VE) task is to predict the logical relationship between a text and an image, requiring the model to make a prediction for each specific item in the text. In the VE task, the premise is defined by the image, rather than a natural language sentence as in the traditional Text Entailment (TE) task. The trained VE model aims to predict whether an image semantically contains text. In the VE task, a real-world image premise P is given. image And a natural language hypothesis H text The goal is to determine whether it is possible to determine based on P image The information provided leads to H text According to (P) image H text The relationship conveyed is assigned three labels: implied, neutral, or contradictory.

[0045] If P image There is sufficient evidence to conclude that H text If the conclusion is true, then the entailment holds.

[0046] If P image There is sufficient evidence to conclude that H text If it is wrong, then the contradiction holds.

[0047] Otherwise, the relationship is neutral, implying that P image The evidence in the text is insufficient to draw conclusions about H. text The conclusion.

[0048] The main difference between VE and TE tasks is that the premise in TE is a natural language sentence H. text rather than an image premise P image Compared to the previous binary classification VQA task where the inference results only included "yes-no", the presence of "neutrality" makes the VE task more challenging because "neutrality" requires the model to derive the uncertainty between "implied (yes)" and "contradictory (no)".

[0049] For ease of understanding, Figure 4 This illustrates an example of a visual entailment task. The image on the left serves as a premise for the visual entailment task, namely, P. image The middle hypothesis includes the textual description that the model needs to determine, while the right side corresponds to the model's answers to the three hypotheses. Since the answers are correct, the right-hand answers can also be seen as labels used during training with the left-hand image and the middle text. Specifically, the first hypothesis, "Two women are holding packages," is deemed "implied" (or the image-text pair consisting of this hypothesis and the premise image is labeled "implied") because it can be completely deduced from the right image. The third hypothesis, "The men are fighting outside a deli," is deemed "contradictory" because it completely contradicts the right image (or the image-text pair is labeled "contradictory"). The second hypothesis, "The sisters are hugging goodbye while holding to go packages after just eating lunch," although it does not contain any content that contradicts the right image, is deemed "neutral" (or the image-text pair is labeled "neutral") because it cannot be inferred whether the two women are sisters or whether they have just eaten lunch.

[0050] In a typical VE task, training data consists of image-text pairs and their labels. Each image-text pair includes an image and a hypothesis proposed for the image (as text), while the label represents the correct answer to the hypothesis. During training, a loss function representing the difference between the model's output answer and the correct answer indicated by the label is used for parameter updates. During the inference phase, the input is an image and text (the hypothesis proposed for the image), and the model needs to output the correct answer based on the image and the question—that is, whether the hypothesis about the image is implied, contradictory, or neutral.

[0051] The IC, VQA, and VE tasks mentioned above can all be implemented as downstream tasks of the CLIP model. It should be understood that, in addition to these three tasks, VLU can also include other proposed or yet-to-be-proposed task paradigms. For various existing VLU tasks, several methods have been proposed to implement or improve the performance of downstream VLU tasks based on CLIP, including but not limited to:

[0052] 1. Partial Parameter Fine-Tuning Method: This method fine-tunes the bias terms, LayerNorm, and classification head of the pre-trained Visual Language Model (VL-PTM) while keeping the remaining parameters of the VL-PTM fixed. This reduces the scale of parameter fine-tuning while preserving as much knowledge as possible from the VL-PTM learned during pre-training. However, due to the hundreds of millions of parameters in the VL-PTM model, the scale of parameters fine-tuned using this method is still enormous, requiring more training data to achieve good performance.

[0053] 2. Manual Templates: Manually constructed prompts are used to test visual object extraction and visual relationship extraction tasks, and to infer which colors VL-PTM is sensitive to. These colors are then used to construct masks to mask objects in the image. Based on the mask format, a manual prompt template is designed, transforming the above problem into a cloze test problem.

[0054] However, such methods require complex data preprocessing and are difficult to generalize to all tasks.

[0055] 3. Parameterized soft cues: Used for few-sample image classification, these cues replace manually designed cue templates with parameterized vectors. During training, the parameters of VL-PTM are frozen, and the parameterized cue template plus the category name is used as the VL-PTM parameter.

[0056] The PTM text encoder is used as input to mine knowledge from the VL-PTM to obtain better category representations. Finally, the image classification is determined by calculating the similarity between the image and the category representation. This type of method has limited applicability, only suitable for image classification tasks based on vector similarity calculations, and has a long training time, resulting in less than ideal performance. Furthermore, setting the parameters of the parameterized cue template requires extensive grid searches to achieve optimal results.

[0057] 4. Heuristic rule-based data preprocessing methods: These methods utilize language rules or pre-trained sequence-to-sequence language models to transform questions into statements, thereby converting the VQA task into a cloze test. However, this approach requires complex heuristic rules or external language models for data preprocessing, which is very time-consuming, and its application is limited.

[0058] 5. Specific Model Structure Adapter Method: Also used for few-sample image classification. During training, the parameters of VL-PTM are frozen, and a lightweight adapter structure is used to learn image-text features for downstream tasks from low-resource training data. Then, the image-text features of the downstream tasks are weighted and summed with the image-text features of CLIP, followed by image-text vector similarity calculation. However, the parameter scale of the adapter structure is too large for few-class classification tasks.

[0059] As mentioned earlier, although the introduction of pre-trained visual language models such as CLIP has greatly facilitated the implementation of downstream VLU tasks, existing CLIP-based fine-tuning methods are still too complex, require a lot of manual labor, and different VLU tasks require different architectures, resulting in low efficiency in parameter fine-tuning.

[0060] In view of this, this invention proposes a prototype matrix similarity matching scheme for constructing a knowledge base as a general operational method for visual language understanding tasks. This invention uniformly reformulates a series of VLU tasks into an open-book affinity-matching problem. It obtains image-text fusion representations through visual-text interaction, thereby uniformly representing multimodal information of images and text. A knowledge base prototype matrix is ​​constructed based on low-resource training data (e.g., small-sample fine-tuning data), and various visual language understanding tasks are then uniformly formalized into matching problems based on the knowledge base prototype matrix. Furthermore, to provide more supervision signals during the training phase, the model utilizes implicit ranking information of real labels through contrastive learning to provide more supervision information during training. Experiments on public benchmark datasets show that this model effectively improves the performance of visual language understanding tasks (e.g., image-text understanding tasks) under low training resources.

[0061] Figure 5A schematic flowchart illustrating a visual language understanding task processing method according to an embodiment of the present invention is shown. This method operates based on a visual language pre-trained model (VL-PTM). It should be clarified that the application of this invention presupposes the existence of a trained VL-PTM, such as CLIP (in the following description of the invention, CLIP will be used as an instance of VL-PTM, but it should be understood that the invention can also be applied to other visual language models that unify text and image embedding vectors into a single embedding space). Since CLIP is not specifically designed for any particular task, the trained CLIP is still a "pre-trained" model. VL-PTM can be used to complete various downstream visual language understanding tasks, and this invention is based on fine-tuning operations performed on VL-PTM (at this point, the parameters in the model already contain a large amount of image and text knowledge). Here, fine-tuning means training directly on the pre-trained model according to the requirements of the downstream task. In other words, the VLU task training phase mentioned below refers to training for the VLU task, not for the VL-PTM; and during the VLU task training phase, the parameters of the VL-PTM itself are only fine-tuned, or even most parameters are not updated.

[0062] Although the processing method of this invention also targets the training and inference phases of VLU tasks, unlike existing technologies, this invention's method is universally applicable to various downstream VLU tasks, eliminating the need to design a different architecture for each task. However, it should be noted that while this invention trains a single knowledge base prototype matrix for the VE, VQA, and IC tasks mentioned above, and completes inference through open-book similarity matching with the prototype matrix, different knowledge base prototype matrices still need to be trained for different tasks, rather than using the same knowledge base prototype matrix for all tasks. The advantage of this invention is that when implementing VLU tasks where VL-PTM is considered a downstream component, it does not require adjusting the parameters of the VL-PTM's text encoder and image encoder; only other parameters, such as the knowledge base prototype matrix, need to be adjusted, and the required training data is extremely small, thus greatly improving parameter training efficiency.

[0063] In step S510, training data for the VLU task is constructed, which includes multiple image-text pairs. In practice, the training data can be obtained from publicly available datasets corresponding to a specific VLU task or collected from open-source resources on the Internet. To construct the prototype matrix of the knowledge base of this invention, images and text need to be input simultaneously during training and inference (rather than outputting only images or inputting only text). Therefore, this invention is particularly applicable to VLU tasks that conventionally input images and text simultaneously during training and inference, such as the VQA and VE tasks described above. However, for VLU tasks that input only images or only text during training and inference, such as the IC task described above, this invention is still applicable as long as the missing text or images can be additionally constructed (for the IC task, a categorical text description is constructed based on the category label).

[0064] In step S520, the training data for the VLU task is fed into VL-PTM to obtain multiple image embedding vectors and multiple text embedding vectors located in the same embedding space. See also Figure 1 As shown, the image and text in an image-text pair can be fed into the CLIP's image sensor and text sensor, respectively, to obtain image embedding vectors and text embedding vectors. The image embedding vectors and text embedding vectors reside in the same embedding space; for example, they are both embedding vectors of dimension d. Since the training data contains multiple image-text pairs, multiple image embedding vectors and multiple text embedding vectors can be obtained through the CLIP's image sensor and text sensor, respectively.

[0065] In step S530, the image embedding vector and text embedding vector belonging to the same image-text pair are fused into a single fusion feature. Here, the image embedding vector and text embedding vector of the same image-text pair can be fused through some operation, such that the resulting fusion feature includes features from both embedding vectors. In a simple embodiment, two embedding vectors can be concatenated; in another embodiment, two embedding vectors, as well as the sum and difference of the two vectors, can be concatenated; in one embodiment, fusing the image embedding vector and text embedding vector belonging to the same image-text pair into a single fusion feature includes concatenating at least two vectors: the image embedding vector belonging to the same image-text pair; the text embedding vector belonging to the same image-text pair; the sum of the image embedding vector and the text embedding vector; the difference between the image embedding vector and the text embedding vector; and the element-wise product of the image embedding vector and the text embedding vector.

[0066] If v1 and v2 are represented as the input visual representation (i.e., image) and text representation, respectively, that is, corresponding to the image embedding vector and text embedding vector encoded by the image encoder and text encoder, respectively, then in a preferred embodiment, a fusion function that efficiently computes the interaction between visual and textual information can be used. To obtain a unified representation of image-text pairs:

[0067]

[0068] in d is the dimension of the embedding vector, and × is the element-wise operator. In the example above, the fusion function... Five vectors of length d are concatenated, corresponding to the image embedding vector, the text embedding vector, the sum of two vectors (adding corresponding elements), the difference of two vectors (subtracting corresponding elements), and the element-wise product (combining corresponding elements). In other embodiments, the fusion function... It can also take other forms.

[0069] During training, the fusion vector for each image-text pair can be obtained; that is, the fusion vector for each image-text pair can be calculated. Subsequently, in step S540, a knowledge base prototype matrix can be constructed based on the fusion vector of each image-text pair. Here, the knowledge base prototype matrix refers to the prototype matrix that serves as the knowledge base. "Knowledge base" means that the prototype matrix contains the encoded knowledge of the CLIP and the knowledge of the training data, and can be used as a knowledge base during inference. "Prototype matrix" means that the knowledge base takes matrix form and includes multiple prototype features.

[0070] In one embodiment, the prototype matrix includes multiple prototype features, and the number of prototype features is the same as the number of categories in the specific VLP task (and also the same as the value type of the labels in the training data). For example, in the VE task above, since the answer can only be "implication," "contradictory," or "neutral," the prototype matrix of the VE task includes three prototype features, corresponding to the implication prototype, contradictory prototype, and neutral prototype, respectively. In this case, the knowledge base prototype matrix is ​​a 3×5d matrix. Constructing the knowledge base prototype matrix based on multiple fusion features includes: classifying the multiple fusion features based on labels; calculating the average fusion feature of each category; and using each average fusion feature as the prototype feature of the corresponding category to obtain the knowledge base prototype matrix. In other words, prototype features can be obtained by averaging each fusion feature under the current label. In other embodiments, prototype features can also be obtained using methods other than averaging.

[0071] After obtaining the knowledge base prototype matrix, it can be directly used for reasoning. Therefore, in step S550, the knowledge base prototype matrix is ​​used for similarity matching to obtain the reasoning result of the visual language understanding task.

[0072] Here, the knowledge base prototype matrix is ​​also a type of similarity matrix, also known as an affinity matrix. Affinity is a statistical technique used to organize the mutual similarity between a set of data points. Similarity is similar to distance, but it does not satisfy the properties of a metric; two identical points have a similarity score of 1, while the result of calculating the metric is zero. Typical examples of similarity matching are cosine similarity and Jaccard similarity. These similarity measures can be interpreted as the probability that two points are related. For example, if the coordinates of two data points are close, then their cosine similarity score (also called their respective "affinity" scores) will be closer to 1 than data points that are spatially distant. Therefore, in this invention, after obtaining the knowledge base prototype matrix for a specific VLU task, inference can be performed by using the affinity scores of the fused features of the image text to be inferred and the prototype features.

[0073] Therefore, using the knowledge base prototype matrix for similarity matching to obtain the inference result of the visual language understanding task may include: in the inference stage of the VLU task, feeding images and text into the VL-PTM to obtain image embedding vectors and text embedding vectors to be classified; fusing the image embedding vectors and text embedding vectors to be classified into a fusion feature to be classified; performing similarity matching between the fusion feature to be classified and the prototype feature of each category in the knowledge base prototype matrix; and selecting the category corresponding to the prototype feature that best matches the fusion feature to be classified as the inference result.

[0074] Specifically, during the inference phase, the image-text pairs to be inferred can also be fed into CLIP's image and text encoders to obtain image embedding vectors and text embedding vectors, and the same fusion function used when constructing prototype features can be applied. We obtain fusion features as the fusion features to be inferred. Then, we calculate the similarity between the fusion features to be inferred and each prototype feature in the prototype matrix of the knowledge base, and select the category corresponding to the prototype feature with the highest similarity score as the inference result.

[0075] To deepen the understanding of the principles of this invention, the processing method of this invention will be described below in conjunction with a specific VLU task.

[0076] In one embodiment, the VLU task capable of implementing the method of the present invention can be the VE task as described above. During the training phase, the training data includes multiple image-text pairs and their corresponding labels. For example, a set of training data may include data from... Figure 4 The image shown on the right (as a premise) and the text "The men are fighting outside a deli" (as a hypothesis) form an image-text pair, along with the corresponding label (true classification) "Contradiction". The image is then fed into CLIP's image sensor to obtain embedding vector v1; the text is fed into CLIP's text sensor to obtain embedding vector v2. It should be understood that although v1 and v2 correspond to the image embedding vector and text embedding vector, respectively, representing the embedding vectors obtained from the image and text, v1 and v2 are vectors with the same dimension d and located in the same embedding space. Subsequently, v1 and v2 can be fused to obtain...

[0077] The N sets of training data constructed for this VE task can be transformed as follows to obtain N sets of training data. Since VE tasks have three categories—"implication," "contradiction," and "neutrality"—they can also be classified according to these N categories. Each corresponding label divides the system into three groups, and the average of the fused features in each group is calculated as the prototype feature for that category. This yields the implied prototype feature, contradictory prototype feature, and neutral prototype feature. Combining these three prototypes yields the knowledge base prototype matrix for that VE task.

[0078] During the inference phase, the prototype matrix of this knowledge base can be used for open-ended similarity matching. Here, "open-ended" refers to the inference operation being a retrieval operation within the scope defined by the existing prototype matrix, rather than a "closed-ended" operation within an uncertain range. In the inference phase of the VE task, it is necessary to determine whether the input text (as a hypothesis) is "implied," "contradictory," or "neutral" relative to the input image (as a premise). Due to the existence of the prototype matrix as a knowledge base, it is only necessary to convert the input image-text pair into fused features, and then perform similarity matching calculations (e.g., calculating cosine similarity) between the fused features to be inferred and the three prototype features in the prototype matrix. The most similar prototype feature is then selected; for example, the one with the highest similarity score to the neutral prototype feature yields the inference result "neutral." In other words, the existence of the prototype matrix transforms the inference process into a similarity matching process between fused features and prototype features.

[0079] Similarly, in one embodiment, the VLU task capable of implementing the processing method of the present invention can be the VQA task as described above. In the training data, the text in multiple image-text pairs corresponds to non-open-ended questions posed to the images, and the labels correspond to the answer options for the non-open-ended questions. For example, in one embodiment, the present invention is applicable to binary classification tasks in VQA tasks, where the labels only include yes and no categories, and the constructed knowledge base prototype matrix also only includes two features: yes prototype and no prototype. During inference, only binary classification questions can be answered.

[0080] In one embodiment, the present invention is applicable to counting problems in VQA tasks (but the range of counting needs to be limited). In this case, the category of the label is the same as the counting range. For example, if the labels include 1 to 5, then the constructed knowledge base prototype matrix will only include five features: prototype 1 to prototype 5. During reasoning, only questions within the counting range can be answered.

[0081] In a more specific embodiment, the VLU task capable of implementing the processing method of the present invention can be an IC task as described above, i.e., an image classification task. As mentioned above, in conventional image classification tasks, neither the training data nor the inference data includes text descriptions. However, by constructing category-based descriptive text, a knowledge base prototype matrix can still be constructed for the IC task, and inference based on similarity matching can be performed. In this case, during the training phase of the IC task, a set of training data can include the original image and labels, as well as text descriptions constructed based on the label's corresponding classification. For example, a set of training data can include an image, the label "dog," and the text description "photo of a dog" constructed according to the label's corresponding classification. Subsequently, the image is fed into the CLIP's image sensor to obtain an embedding vector v1; the text description "photo of a dog" is fed into the CLIP's text sensor to obtain an embedding vector v2. Subsequently, v1 and v2 can be fused to obtain...

[0082] The N sets of training data constructed for this IC task can be transformed as follows to obtain N sets of training data. And based on the number of IC task categories C, based on these N... Each group is divided into C groups based on its corresponding label, and the average of the fused features in each group is calculated as the prototype feature for that category, thus obtaining the knowledge base prototype matrix for this IC task. During the inference phase of the IC task, since there is also a lack of default text input, the fused features to be classified between the image to be classified and the text description of each category can be calculated. Each fused feature to be classified is then matched with its corresponding prototype feature based on similarity, and the category with the highest matching degree is selected as the inference result.

[0083] For example, in an image classification task, there are four categories: cat, dog, car, and tree. Therefore, during the training phase, a knowledge base prototype matrix containing four prototype features can be obtained. During the inference phase, the image to be classified can be fused with the text descriptions "photo of a cat," "photo of a dog," "photo of a car," and "photo of a tree," respectively, resulting in four fused features. The fused feature obtained with the text description "photo of a cat" is then matched for similarity with the cat prototype feature; the fused feature obtained with the text description "photo of a dog" is matched for similarity with the dog prototype feature; the fused feature obtained with the text description "photo of a car" is matched for similarity with the car prototype feature; and the fused feature obtained with the text description "photo of a tree" is matched for similarity with the tree prototype feature. The scores of these four similarity matches are compared, and the category with the highest similarity score is selected as the classification of the image.

[0084] It should be understood that this invention, by constructing a knowledge base prototype matrix and similarity matching, ensures that the parameters of the VL-PTM image encoder and text encoder remain unchanged during the training and inference phases of the VLU task.

[0085] Specifically, during the training phase, a first loss function (e.g., the cross-entropy loss function L) can be used based on the difference between the predicted classification and the true label. CE Parameter updates are performed. In a preferred embodiment, when training data is limited, the implicit ranking information that the similarity between an image-text pair and its class should be higher than that of other classes can guide the model to identify the correct classification class through contrastive learning. In this case, during the training phase, the probability of the image-text pair belonging to its class can be considered as a positive sample, and the probabilities of other classes as negative samples, constructing a second loss function for contrastive learning (e.g., the contrastive file loss function L). CL The parameters are updated based on both the first loss function and the second function.

[0086] The VLU task processing method according to the present invention has been proposed above. This method constructs a knowledge base prototype matrix using visual-text fusion features to record the salient features of each class. Next, open-book similarity matching is performed between image-text pairs and the prototypes of each class.

[0087] Specifically, during the prototype matrix construction phase, based on CLIP, it can be observed that all VE, VQA, and IC tasks require both text and visual content as input. For IC, the input text can be additionally crafted cues describing the image category. Given a set of N instances of image-text pairs: Among them l i Represents real tags, txti Image (img) i The corresponding text description (here, txt) i This corresponds to the assumptions in VE; the questions in VQA; or the textual descriptions of categories in IC. (Using CLIP's visual encoder.) (Also known as image encoders) and text encoders Encode image text instances and apply... This is used to obtain the fused features. Next, the knowledge base prototype matrix, W, is constructed by averaging the fused features based on the real labels. P :

[0088]

[0089] W P = [M1, ..., M C ](W P ∈R C×5d )

[0090] Where C represents the number of classes, M c Let I(·) represent the prototype of class c, where c∈1…C, I(·) represents the indicator function, and [·] represents the cascade operation.

[0091] During the open-book matching phase, different operations are performed for VE, VQA, and IC tasks.

[0092] Specifically, given an image-text pair with either VE or VQA, similarity matching is performed between its fused features and the prototype of each class. The class probability distribution P... i The calculation formula is:

[0093]

[0094] In the original IC task, only images are provided without corresponding text descriptions. To address this issue, text descriptions (hints) are constructed for all categories. Given an image (img) i and text descriptions for all image categories {t c |c=1…C},img i The predicted probability for the c-th image category (denoted as P) i,c )as follows:

[0095]

[0096] Therefore, the entire probability distribution P i For: P i =[P i,c |c=1···C].

[0097] During training, all parameters of CLIP's visual and text encoders were frozen to ensure parameter efficiency, while other parameters were adjustable (including W). P In one embodiment, using a given P i The model is trained using cross-entropy (CE) loss, where the sample-wise CE loss is defined as follows:

[0098]

[0099] Among them l i,c Let represent the true label for class c. Here, the per-sample CE loss refers to the loss function for a single sample, while the overall loss function is the sum of the losses for all samples.

[0100] However, to further improve performance, additional supervisory signals besides the CE supervisory signal can be introduced in low-resource tasks. Given that the similarity between an instance and the true class should be ranked higher than other classes, this implicit ranking information can be used to guide the model to identify the true class of an instance through contrastive learning. Therefore, in a preferred embodiment, the similarity of the true class (i.e., the prototype matching probability, denoted as P) is... i,l Define P as a positive sample. i Other similarities are defined as negative samples. Therefore, the contrastive learning (CL) loss for each sample is calculated as follows:

[0101]

[0102] At this point, the total loss function for training the VLU task in this invention, i.e., L, can be defined as: L = L CE +L CL Similarly, here, the per-sample L-loss refers to the loss function for a single sample, while the overall loss function requires summing the losses of all samples.

[0103] In practical applications, this invention can be implemented as a VLU task processing system. Figure 6A schematic diagram of a visual language understanding task processing system according to an embodiment of the present invention is shown. As shown, the VLU task processing system 600 can be used to implement the visual language understanding task processing method described above, and may include a training device 610 and an inference device 620. The training device 610 is used to obtain a knowledge base prototype matrix for a specific visual language understanding task, and the inference device 620 is used to perform inference for the specific visual language understanding task based on the knowledge base prototype matrix. Specifically, when a specific VLU task needs to be performed, the training device 610 can construct small sample training data and generate a knowledge base prototype matrix for that specific task using CLIP. It should be understood that the training device 610 can train different knowledge base prototype matrices for various types of VLU tasks, and even for VLU tasks of the same type but corresponding to different datasets. The constructed knowledge base prototype matrix can then be provided to the inference device 620 for inference based on the similarity matching of fused features and prototype features.

[0104] In practice, VLU task processing systems can be applied to various scenarios, such as classifying images to determine whether they meet publication standards.

[0105] Furthermore, although CLIP was mainly used above to describe the VL-PTM-based processing of this invention, those skilled in the art should understand that other VL-PTMs besides CLIP can also be used for the VLU task training and inference operations of this invention, as long as the pre-trained module can output image and text embedding vectors for fusion.

[0106] Figure 7 A schematic diagram of a computing device that can be used to implement the above-described VLU task processing method according to an embodiment of the present invention is shown.

[0107] See Figure 7 The computing device 700 includes a memory 710 and a processor 720.

[0108] Processor 720 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 720 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 720 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0109] Memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 710 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0110] The memory 710 stores executable code, which, when processed by the processor 720, enables the processor 720 to execute the VLU task processing method described above.

[0111] The VLU task processing method according to the present invention has been described in detail above with reference to the accompanying drawings.

[0112] This invention first proposes an efficient parameter fine-tuning method for pre-trained visual-text models in low-resource visual-text understanding tasks. By unifying the VLU task into a similarity matching problem based on a knowledge base prototype matrix, the parameter scale of the method is significantly reduced. Furthermore, contrastive learning utilizes the implicit ranking information of real labels to provide more supervision signals for model training, thereby enhancing model performance. Experimental results show that the proposed method achieves better performance than existing techniques in few-sample image classification tasks, as well as various VE and VQA tasks.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0114] Furthermore, the method according to the invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the invention.

[0115] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0116] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0118] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual language understanding task processing method, the method operating based on a visual language pre-trained model VL-PTM, and comprising: Construct training data for the visual language understanding task, the training data including multiple image-text pairs and labels corresponding to each image-text pair; The training data for the visual language understanding task is fed into the VL-PTM to obtain multiple image embedding vectors and multiple text embedding vectors located in the same embedding space. The image embedding vector and the text embedding vector belonging to the same image-text pair are fused into a single fusion feature; The multiple fused features are classified based on labels; Calculate the average fusion feature for each category; Each average fusion feature is used as the prototype feature of the corresponding category to obtain the knowledge base prototype matrix; Similarity matching is performed using the knowledge base prototype matrix to obtain the inference results of the visual language understanding task, including: During the inference phase, images and text are fed into the VL-PTM to obtain image embedding vectors and text embedding vectors to be classified. The image embedding vector and text embedding vector to be classified are fused into a fused feature to be classified; The fusion features to be classified are matched with the prototype features of each category in the knowledge base prototype matrix based on similarity; and The category corresponding to the prototype feature that best matches the fusion feature to be classified is selected as the inference result.

2. The method as described in claim 1, wherein, Fusing image embedding vectors and text embedding vectors belonging to the same image-text pair into a single fusion feature involves concatenating at least two of the following vectors: The image embedding vectors belonging to the same image-text pair; The text embedding vectors belonging to the same image-text pair; The sum of the image embedding vector and the text embedding vector; The difference between the image embedding vector and the text embedding vector; and The element-wise product of the image embedding vector and the text embedding vector.

3. The method as described in claim 1, wherein, The visual language understanding task includes one of the following: A visual question answering task, wherein in the training data, text in multiple image-text pairs corresponds to a non-open-ended question posed to an image, and labels correspond to answer options for the non-open-ended question; and The visual implication task, wherein in the training data, the text in the plurality of image-text pairs corresponds to a hypothesis proposed for the image, and the label corresponds to implication, contradiction, or neutrality.

4. The method of claim 1, wherein, The visual language understanding task is an image classification task. During the training phase of the image classification task, the text in the plurality of image-text pairs corresponds to the text description of the category; In the inference phase of the image classification task, the fusion features to be classified between the image to be classified and the text description of each category are calculated. Each fusion feature to be classified is matched with its corresponding prototype feature, and the category with the highest matching degree is selected as the inference result.

5. The method of claim 1, comprising: During the training phase of the visual language understanding task, parameters are updated using a first loss function based on the difference between the representation prediction classification and the true label.

6. The method of claim 5, further comprising: During the training phase of the visual language understanding task, the probability of the image-text pair belonging to its category is regarded as a positive sample, and the probability of the other categories is regarded as a negative sample, and a second loss function is constructed for contrastive learning. Parameters are updated based on the first loss function and the second loss function.

7. The method of claim 1, wherein, During the training and inference phases of the visual language understanding task, the parameters of the VL-PTM image encoder and text encoder remain unchanged.

8. A visual language understanding task processing system for performing the visual language understanding task processing method as described in any one of claims 1 to 7, and comprising: Training equipment is used to acquire a knowledge base prototype matrix for a specific visual language understanding task; as well as An inference device for performing inference on the specific visual language understanding task based on the knowledge base prototype matrix.

9. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 7.

10. A computer program product comprising executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 7.

11. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 7.

Images