Pre-training model fine tuning method and device, computer equipment and storage medium

Abstract

The invention discloses a pre-training model fine tuning method and device, computer equipment and a storage medium, which can be applied to various scenes such as a task understanding scene, a task classification scene and a target recognition scene of multimedia resources, and the method comprises the following steps: obtaining an initial pre-training model; inputting a local prompt word and a target text description in an input layer of the initial pre-training model; aligning the local cue word with an image in an image set of an input layer, constructing a first loss function of an initial pre-training model, and regularizing an original cue word of the input layer by using a target text description, and constructing a second loss function of the initial pre-training model; and based on the first loss function and the second loss function, determining a target pre-training model so as to execute the understanding task of the multimedia resources through the target pre-training model. Original cue words are prevented from being over-fitted to the downstream training data set, the downstream task effect is enhanced, and the generalization ability of the pre-training model is improved.

Description

Technical Field

[0001] The present application relates to the field of model processing technology, and more specifically, to a pre-training model fine-tuning method, apparatus, computer equipment, and storage medium. Background Art

[0002] Pretrained models are deep learning architectures that are trained to perform specific tasks on large amounts of data, such as identifying categories in images.

[0003] Currently, fine-tuning of pre-trained models generally uses manually designed prompts fine-tuning technology, which can effectively fine-tune pre-trained models under few-sample and full-sample settings.

[0004] However, the above method causes downstream tasks to be overly dependent on manually designed prompt words, which reduces the effectiveness of downstream tasks. Summary of the Invention

[0005] The main purpose of this application is to provide a pre-training model fine-tuning method, device, computer equipment and storage medium, which can improve the performance of downstream tasks and enhance the generalization ability of the pre-training model.

[0006] To achieve the above objectives, in a first aspect, the present application provides a pre-training model fine-tuning method, comprising:

[0007] Get the initial pre-trained model;

[0008] Input local prompt words and target text descriptions into the input layer of the initial pre-trained model, where the local prompt words are used to represent prompt words for downstream tasks, and the target text descriptions are used to represent text descriptions of images in the image collection;

[0009] Aligning the local prompt words with the images in the image set of the input layer to construct a first loss function of the initial pre-trained model, and regularizing the original prompt words of the input layer using the target text description to construct a second loss function of the initial pre-trained model;

[0010] Based on the first loss function and the second loss function, a target pre-training model is determined to perform a multimedia resource understanding task through the target pre-training model.

[0011] In one embodiment, aligning the local prompt word with an image in the image set input to the input layer to construct a first loss function of the initial pre-training model includes:

[0012] For each image in the image set, obtain the local prompt word corresponding to each image;

[0013] Calculate the similarity between each image and the local prompt word corresponding to each image to obtain the first similarity corresponding to each image;

[0014] Summarize the first similarities corresponding to each image to obtain a first similarity matrix corresponding to the image set;

[0015] A first loss function of an initial pre-trained model is constructed using the first similarity matrix and the image labels in the image set.

[0016] In one embodiment, obtaining the local prompt word corresponding to each image includes:

[0017] Get the category name of each image;

[0018] A prompt is added before the category name of each image, and the prompt is used as the local prompt word corresponding to each image.

[0019] In one embodiment, calculating the similarity between each image and the local prompt word corresponding to each image to obtain the first similarity corresponding to each image includes:

[0020] Perform text encoding on the local prompt words corresponding to each image to obtain the local prompt word feature vector;

[0021] Perform image encoding on each image to obtain the global feature vector of each image;

[0022] Based on the local prompt word feature vector and the global feature vector of each image, a first similarity corresponding to each image is calculated.

[0023] In one embodiment, calculating the first similarity corresponding to each image based on the local prompt word feature vector and the global feature vector of each image includes:

[0024] For each image block in each image, extract a feature vector of each image block from the global feature vector of each image;

[0025] Calculate the similarity between the feature vector of each image block and the feature vector of the local prompt word to obtain the similarity of each image block;

[0026] Summarize the similarity of each image block to obtain the similarity of all image blocks;

[0027] The similarities of all image blocks are averaged to obtain the first similarity corresponding to each image.

[0028] In one embodiment, the target text description is used to regularize the original prompt words of the input layer to construct a second loss function of the initial pre-training model, including:

[0029] Get the target text description of the image in the image collection;

[0030] Perform text encoding on the target text description and the original prompt word respectively to obtain the text description feature vector and the original prompt word feature vector;

[0031] Calculate the similarity between the text description feature vector and the original prompt word feature vector to obtain the similarity corresponding to the original prompt word;

[0032] The second loss function of the initial pre-training model is constructed using the similarity corresponding to the original prompt word.

[0033] In one embodiment, obtaining a target text description of an image in an image set includes:

[0034] Get the initial text description of the image in the image collection;

[0035] The initial text description is fine-grainedly rewritten using a first preset model to obtain a target text description, wherein the first preset model is obtained by training text training data.

[0036] In one embodiment, obtaining an initial text description of an image in an image collection includes:

[0037] The images in the image set are inferred using a second preset model to output an initial text description, wherein the second preset model is obtained by training the image training data.

[0038] In one embodiment, determining a target pre-trained model based on the first loss function and the second loss function includes:

[0039] Obtaining a third loss function, wherein the third loss function is determined by image labels in the image set and a similarity matrix corresponding to the image set;

[0040] Based on the first loss function, the second loss function and the third loss function, a target pre-training model is determined.

[0041] In one embodiment, obtaining the third loss function includes:

[0042] For each image in the image set, obtain the original prompt word corresponding to each image;

[0043] Calculate the similarity between each image and the original prompt word corresponding to each image to obtain the second similarity corresponding to each image;

[0044] Summarize the second similarities corresponding to each image to obtain a second similarity matrix corresponding to the image set;

[0045] A third loss function of the initial pre-trained model is constructed using the second similarity matrix and the image labels in the image collection.

[0046] In one embodiment, calculating the similarity between each image and the original prompt word corresponding to each image to obtain the second similarity corresponding to each image includes:

[0047] Perform text encoding on the original prompt word corresponding to each image to obtain the original prompt word feature vector;

[0048] Perform image encoding on each image to obtain the global feature vector of each image;

[0049] Based on the original prompt word feature vector and the global feature vector of each image, a second similarity corresponding to each image is calculated.

[0050] In a second aspect, an embodiment of the present application provides a pre-training model fine-tuning device, comprising:

[0051] Acquisition module, used to obtain the initial pre-training model;

[0052] An input module, configured to input local prompt words and target text descriptions into the input layer of the initial pre-trained model, wherein the local prompt words are used to represent prompt words for downstream tasks, and the target text descriptions are used to represent text descriptions of images in the image collection;

[0053] a loss function construction module for aligning the local prompt words with the images in the image set of the input layer to construct a first loss function of the initial pre-trained model, and regularizing the original prompt words of the input layer using the target text description to construct a second loss function of the initial pre-trained model;

[0054] The task execution module is used to determine a target pre-training model based on the first loss function and the second loss function, so as to perform the multimedia resource understanding task through the target pre-training model.

[0055] In a third aspect, an embodiment of the present application provides a device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0057] The embodiment of the present application provides a pre-training model fine-tuning method, device, computer equipment and storage medium, including: first obtaining an initial pre-training model, then inputting local prompt words and target text descriptions into the input layer of the initial pre-training model, wherein the local prompt words are used to represent the prompt words of the downstream task, and the target text descriptions are used to represent the text descriptions of the images in the image set, and then the local prompt words are aligned with the images in the image set of the input layer, and the first loss function of the initial pre-training model is constructed, and the original prompt words of the input layer are regularized using the target text descriptions to construct the second loss function of the initial pre-training model, and then based on the first loss function and the second loss function, the target pre-training model is determined to perform the multimedia resource understanding task through the target pre-training model. The present application constructs the first loss function of the initial pre-training model by aligning the local prompt words with the images in the image set of the input layer to enhance the effect of the downstream task. In addition, the original prompt words of the input layer are regularized using the target text descriptions to construct the second loss function of the initial pre-training model to avoid overfitting of the original prompt words to the downstream training data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings that constitute part of this application are used to provide a further understanding of this application and make other features, objects and advantages of this application more apparent. The illustrative embodiment drawings of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0059] Figure 1 This is a flow chart of a pre-training model fine-tuning method provided in an embodiment of the present application;

[0060] Figure 2 This is a flow chart of another method for fine-tuning a pre-trained model provided in an embodiment of the present application;

[0061] Figure 3 This is a structural diagram of a pre-training model fine-tuning device provided in an embodiment of the present application;

[0062] Figure 4 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] The terms "first," "second," "third," "fourth," and so on (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in orders other than those illustrated or described herein.

[0065] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0066] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0067] It should be understood that in this application, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0068] It should be understood that in this application, "multiple" means two or more. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0069] It should be understood that, in this application, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0070] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0071] The data involved in this application may be data authorized by the tester or fully authorized by all parties. The collection, dissemination, and use of the data shall comply with the relevant laws, regulations, and standards of the relevant countries and regions. The implementation methods / examples of this application may be combined with each other.

[0072] The embodiments of the present application can be applied to various scenarios such as task understanding scenarios, task classification scenarios, and target recognition scenarios of multimedia resources.

[0073] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0074] Pretrained models are deep learning architectures that are trained to perform specific tasks on large amounts of data, such as identifying categories in images.

[0075] In the past two years, mainstream image and text pre-training models have aligned large numbers of images and texts into a common space through comparative learning, demonstrating good generalization and transfer capabilities, as embodied in zero-shot learning and good downstream task results.

[0076] However, in downstream tasks, this type of image-text pre-training model often requires manually designed prompts during the reasoning process. The effectiveness of downstream tasks depends largely on the quality of prompts, which leads to greater manual participation.

[0077] Therefore, a prompt tuning method is proposed to improve the above problems. Prompt tuning replaces manually designed prompts with learnable tokens (prompts) at the input layer. Prompt tuning has low training cost and fast adaptation speed, making it a mainstream method for image and text pre-training and fine-tuning.

[0078] However, prompt tuning still faces certain challenges, primarily the following: 1) While the performance on downstream tasks is better than that of manually designed prompts, it is worse than fully fine-tuning the parameters; 2) After prompt tuning, overfitting may occur to a certain extent, leading to a loss of generalization in the pre-trained model. This is reflected in a decrease in the performance of zero-shot learning compared to before fine-tuning.

[0079] To solve the above problems, this application proposes a pre-training model fine-tuning method.

[0080] Next, the present application solution will be described through specific embodiments with reference to the accompanying drawings.

[0081] See also Figure 1 , Figure 1 A flow chart of a pre-training model fine-tuning method provided in an embodiment of the present application. Figure 1 As shown, the following steps are included:

[0082] Step S101: Obtain an initial pre-training model.

[0083] A pre-trained model (i.e., an initial pre-trained model) refers to a neural network model that has been pre-trained on a large-scale dataset. Usually, the features learned on general tasks can be transferred to other specific tasks. The main idea of ​​the pre-trained model is to use the information of large-scale data to initialize the model parameters, and then apply the model to the target task through fine-tuning or transfer learning.

[0084] Pre-trained models include, but are not limited to, natural speech processing models, computer vision models, recommendation system models, etc. Natural speech processing models include autoregressive language models, autoencoder language models, and sequence-to-sequence models.

[0085] Exemplarily, the pre-training model may adopt the Transformer model, which is a deep learning model based on the self-attention mechanism. Before fine-tuning the Transformer model, the encoder and decoder in the Transformer model may be pre-trained using training data to determine the parameters in the encoder and decoder, which remain fixed during the subsequent model fine-tuning process. The encoder consists of a linear mapping layer, a position encoding layer, and two Transformer modules. The linear mapping layer is used to map the input data to a high-dimensional feature control; the position encoding layer adds position information to the high-dimensional features in the high-dimensional feature space; the two Transformer modules interact with the high-dimensional features of different position information based on the self-attention mechanism.

[0086] To obtain a pre-trained model, you can directly obtain a trained model, or you can first obtain training data and then use the training data to perform multi-task training on the initial model to form a pre-trained model. The multi-task includes at least one cross-modal contrastive learning task and at least one single-modal learning task.

[0087] Among them, the training data includes unimodal corpus and multimodal corpus, wherein the multimodal corpus includes a corpus pair consisting of a first modal corpus and a second modal corpus. The unimodal corpus involved in this application refers to corpus of a single modality, such as corpus of modalities such as image, text, video or audio. In this application, only one unimodal corpus may be included, or more than one unimodal corpus may be included. Multimodal corpus refers to a corpus pair consisting of corpus of two modalities, such as a corpus pair consisting of image and text, a corpus pair consisting of image and audio, a corpus pair consisting of text and video, and so on.

[0088] In one embodiment, the multimodal corpus may include positive multimodal corpus and negative multimodal corpus, wherein the first modal corpus and the second modal corpus contained in the positive multimodal corpus express the same semantics, while the first modal corpus and the second modal corpus contained in the negative multimodal corpus express different semantics.

[0089] The pre-trained language model of the present application can use a multi-layer Transformer as the main model, and use the training data as the input of the pre-trained language model. The pre-trained language model maps all the corpora in the training data to a unified vector space, that is, outputs the vector representation of each corpus (including each corpus of the unimodal corpus and each first modal corpus and second modal corpus in the multimodal corpus).

[0090] Step S102: inputting local prompt words and target text description into the input layer of the initial pre-training model.

[0091] Among them, local prompt words are used to represent the prompt words of downstream tasks. The prompt words corresponding to the downstream tasks are hot-started based on the prompt words of similar pre-training tasks. This can further effectively inherit the knowledge of the language model fine-tuning paradigm in the pre-training stage, so as to better adapt to downstream tasks.

[0092] The local prompt words of this application are to add N learnable prompts (tokens) before the category name CLS (classname) of the category to which the object displayed in the image belongs, and use the added N learnable prompts (tokens) as local image prompt words, where N is an integer greater than or equal to 1.

[0093] Specifically, for text, tokens can be individual characters or words. For images, tokens can be target regions or pixels within an image. Video can be broken down into individual frames, and tokens can be individual frames or keyframes. For audio, tokens can be individual sound tracks, audio segments, or spectrograms derived from the frequency and intensity of the audio.

[0094] The target text description is used to represent the text description of the images in the image set.

[0095] Currently, cue learning achieves good results in zero- or few-shot scenarios by modifying downstream tasks and augmenting expert knowledge to adapt task inputs and outputs to the original speech model. However, the cue words obtained through current cue learning are typically represented using shared learnable parameters, meaning that the resulting cue words represent the characteristics of the entire training set.

[0096] The local prompt words added in this application are dynamic, that is, dynamic prompt words are generated based on the current input data. The dynamic prompt words represent the contextual features of the current data and contain instance-level information. Among them, the instance level can be understood as the level of input data. Each input data is an instance, that is, dynamic prompt words are generated for each input data. In other words, the local prompt words change as the input data changes and are dynamically generated. Compared with the non-instance-level features that use the same prompt words for all input data, the local prompt words in this application are more expressive and more specific.

[0097] like Figure 2 As shown in the example, the objects in the image in this application are a dog and a bone. Therefore, the category names before the local prompt word are "dog" and "bone." If there are multiple category names, a corresponding learnable prompt needs to be added before each category name. In other words, each category is configured with a local prompt word. The text description of the image can be "a dog holding a bone."

[0098] Taking the image-text pre-training model as an example, the initial input layer of the image-text pre-training model only inputs the image and the original prompt word, and then calculates the similarity between the input image and the original prompt word to obtain the loss function of the image-text pre-training model.

[0099] However, the original prompt words in the above-mentioned image-text pre-training model are mainly obtained through manual design and rely on manual work, resulting in downstream tasks being too dependent on the manually designed prompt words, reducing the effectiveness of downstream tasks and the generalization ability of the image-text pre-training model.

[0100] Therefore, this application inputs local prompt words and target text descriptions into the input layer of the initial pre-trained model to construct a first loss function for the initial pre-trained model by aligning the local prompt words with images in the image set input to the input layer, thereby enhancing the performance of downstream tasks. In addition, the target text description is used to regularize the original prompt words in the input layer to construct a second loss function for the initial pre-trained model to prevent the original prompt words from overfitting to the downstream training dataset.

[0101] Step S103: Align the local prompt words with the images in the image set of the input layer to construct the first loss function of the initial pre-training model, and regularize the original prompt words of the input layer using the target text description to construct the second loss function of the initial pre-training model.

[0102] In order to align the local prompt words with the images in the image set of the input layer and construct the first loss function of the initial pre-trained model, it is necessary to first obtain the local prompt words corresponding to each image in the image set, and then calculate the similarity between each image and the local prompt words corresponding to each image to obtain the first similarity corresponding to each image, and then summarize the first similarities corresponding to each image to obtain the first similarity matrix corresponding to the image set, and then use the first similarity matrix and the image labels in the image set to construct the first loss function of the initial pre-trained model.

[0103] The step of obtaining the local prompt word corresponding to each image includes: obtaining the category name of each image; adding a prompt before the category name of each image, and using the prompt as the local prompt word corresponding to each image.

[0104] Since the pre-trained model adds corresponding local prompt words to the input layer for each image, the local prompt words corresponding to each image first obtain the category name of each image, and then add N learnable prompts in front of this category name. The local prompt words can be represented by local prompt Pl.

[0105] After obtaining the local prompt words corresponding to each image, it is necessary to calculate the similarity between each image and the local prompt words corresponding to each image to obtain the first similarity corresponding to each image. Specifically, text encoding is performed on the local prompt words corresponding to each image to obtain the local prompt word feature vector; image encoding is performed on each image to obtain the global feature vector of each image; based on the local prompt word feature vector and the global feature vector of each image, the first similarity corresponding to each image is calculated.

[0106] For example, for Figure 2 The local prompt P corresponding to the image in lFirst, the text is encoded to obtain the local prompt word feature vector, that is, F l t =g(P l ), where g represents the text encoding. Figure 2 The image in the image is encoded to obtain the global feature vector of the image, namely F l v =f(x), where f represents the image code. The global feature vector of the image refers to the feature vector of the entire image. The global feature vector is composed of feature vectors of multiple image blocks. The number of image blocks obtained by dividing the image includes but is not limited to nine, sixteen, etc.

[0107] The encoding of the local prompt words and the image can be performed simultaneously or sequentially, which is not limited here.

[0108] After obtaining the local prompt word feature vector and the global feature vector of each image, it is necessary to calculate the first similarity corresponding to each image based on the local prompt word feature vector and the global feature vector of each image. Specifically, for each image block in each image, extract the feature vector of each image block from the global feature vector of each image; perform similarity calculation on the feature vector of each image block and the local prompt word feature vector to obtain the similarity of each image block; summarize the similarity of each image block to obtain the similarity of all image blocks; and calculate the mean of the similarities of all image blocks to obtain the first similarity corresponding to each image.

[0109] The similarity calculation methods include but are not limited to Euclidean distance, cosine similarity, etc.

[0110] For example, assuming that an image includes nine image blocks, the similarity between the feature vector of each of the nine image blocks and the feature vector of the local prompt word is calculated. Specifically, the feature vector of each image block can be aligned with the feature vector of the local prompt word, that is, a neural network matrix is ​​constructed, and the product of the feature vector of each image block to be aligned and the feature vector of the local prompt word is used as the numerical vector to be aligned. Then, a gradient descent algorithm is used to update the neural network matrix. Based on each updated neural network matrix, the feature vector of each aligned image block and the feature vector of the local prompt word are determined. After the update iteration is completed, the final neural network matrix is ​​obtained. Through the final neural network matrix, the final feature vector and local prompt word of each image block are obtained. At this time, by calculating the similarity between the final feature vector of each image block and the local prompt word, the obtained similarity is more accurate, which is more conducive to improving the generalization ability of the pre-trained model.

[0111] The similarity of each image block is obtained, and then the similarities of the nine image blocks are summed to obtain the total similarity of the nine image blocks. The total similarity is then divided by nine to obtain the similarity corresponding to this image, that is, the first similarity.

[0112] Since there are multiple images input to the input layer, the first similarity corresponding to each image can be aggregated to obtain a first similarity matrix corresponding to the image set. The first similarity matrix and the image labels in the image set are then used to construct the first loss function of the initial pre-training model. The image label is the actual label of the image, which can be represented by text, letters, etc.

[0113] The first loss function is used for contrastive learning. In contrastive learning, contrast is generated between different categories. This contrast can be between categories, between different perspectives of the same object, between different people, or other types of contrast. Generally, within a batch, samples of the same category are called positive samples, while samples outside of the positive category are called negative samples.

[0114] Therefore, the first loss function includes but is not limited to maximum edge contrast loss, triplet loss, N-Pair Multi-Class Loss, and Info NCE Loss (noise contrastive estimation Loss).

[0115] In order to regularize the original prompt words of the input layer using the target text description and construct the second loss function of the initial pre-trained model, it is necessary to first obtain the target text description of the image in the image set, and then respectively perform text encoding on the target text description and the original prompt word to obtain the text description feature vector and the original prompt word feature vector. Then, similarity is calculated between the text description feature vector and the original prompt word feature vector to obtain the similarity corresponding to the original prompt word, and then the similarity corresponding to the original prompt word is used to construct the second loss function of the initial pre-trained model.

[0116] Regularization, a key concept in machine learning and deep learning, is used to prevent model overfitting. Overfitting occurs when a model performs well on training data but poorly on test data or real-world data. Regularization works by imposing constraints on model parameters to reduce model complexity and prevent overfitting.

[0117] Regularization plays a key role in pre-trained models. By limiting the size of model parameters, it helps the model learn more general features, thereby improving its performance on new tasks. Regularization of target text descriptions uses regular expressions to accurately match and filter prompt words in the input layer, providing an effective means of constructing the pre-trained model's second loss function.

[0118] Obtaining target text descriptions for images in an image collection includes: obtaining initial text descriptions for the images in the image collection; and fine-grained rewriting the initial text descriptions using a first preset model to obtain target text descriptions, wherein the first preset model is obtained by training on text training data. Obtaining initial text descriptions for images in an image collection includes: inferring the images in the image collection using a second preset model to output the initial text descriptions, wherein the second preset model is obtained by training on image training data.

[0119] For example, the first preset model is a large language model, and the second preset model is a large image-text model. When each image in the image collection is trained and inferred by the large image-text model, the initial text description of each image will be obtained, such as Figure 2 "A dog with a bone" in the novel.

[0120] However, the initial text description is not fine-grained enough. It only contains coarse-grained categories (such as "dog") and does not generate fine-grained categories (such as "corgi"). Therefore, a large language model is needed to rewrite the generated initial text description in a fine-grained manner. The rewriting instructions are as follows:

[0121] Short description of an image:{caption}.

[0122] Tags of this image:{class name}.

[0123] Please rewrite a definite and brief description according to givendescription and tags.

[0124] The above instructions can be understood as: input a brief description of the image and the label of the image, and then redefine a clear and brief description of the image based on the given brief description and label.

[0125] By rewriting the large language model, the fine-grained description, that is, the target text description is "a corgi holding a bone".

[0126] Then, the target text description and the original prompt word are respectively encoded to obtain the text description feature vector and the original prompt word feature vector. The text encoding here is similar to the text encoding method of the local prompt word mentioned above and will not be repeated here.

[0127] Finally, the similarity between the text description feature vector and the original prompt word feature vector is calculated to obtain the similarity corresponding to the original prompt word. The similarity corresponding to the original prompt word is then used to construct the second loss function of the initial pre-trained model. The similarity calculation methods include but are not limited to Euclidean distance and cosine similarity.

[0128] Among them, the second loss function is the same type as the first loss function, and is also a loss function of contrastive learning, and the second loss function includes but is not limited to maximum edge contrast loss, triple loss, N-Pair Multi-Class Loss, and Info NCE Loss (noise contrastive estimation Loss).

[0129] Step S104: Based on the first loss function and the second loss function, determine a target pre-training model to perform the multimedia resource understanding task through the target pre-training model.

[0130] In order to determine the target pre-training model based on the first loss function and the second loss function, it is necessary to first obtain the third loss function, wherein the third loss function is determined by the image labels in the image set and the similarity matrix corresponding to the image set, and then the target pre-training model is determined based on the first loss function, the second loss function and the third loss function.

[0131] Among them, obtaining the third loss function includes: obtaining the original prompt word corresponding to each image in the image set; calculating the similarity between each image and the original prompt word corresponding to each image to obtain the second similarity corresponding to each image; summarizing the second similarity corresponding to each image to obtain a second similarity matrix corresponding to the image set; and using the second similarity matrix and the image labels in the image set to construct the third loss function of the initial pre-trained model.

[0132] The method of calculating the similarity between each image and the original prompt word corresponding to each image to obtain the second similarity corresponding to each image includes: performing text encoding on the original prompt word corresponding to each image to obtain the original prompt word feature vector; performing image encoding on each image to obtain the global feature vector of each image; and calculating the second similarity corresponding to each image based on the original prompt word feature vector and the global feature vector of each image.

[0133] For example, for Figure 2The original prompt word corresponding to the image in general prompt P k First, the text is encoded to obtain the original prompt word feature vector, that is, F k t =g(P k ), where g represents the text encoding. Figure 2 The image in the image is encoded to obtain the global feature vector of the image, namely F l v =f(x), where f represents the image code. The global feature vector of the image refers to the feature vector of the entire image. The global feature vector is composed of feature vectors of multiple image blocks. The number of image blocks obtained by dividing the image includes but is not limited to nine, sixteen, etc.

[0134] The encoding of the original prompt word and the image can be performed simultaneously or sequentially, which is not limited here.

[0135] After obtaining the original prompt word feature vector and the global feature vector of each image, it is necessary to calculate the second similarity corresponding to each image based on the original prompt word feature vector and the global feature vector of each image. Specifically, for each image block in each image, extract the feature vector of each image block from the global feature vector of each image; perform similarity calculation on the feature vector of each image block and the original prompt word feature vector to obtain the similarity of each image block; summarize the similarity of each image block to obtain the similarity of all image blocks; and calculate the mean of the similarities of all image blocks to obtain the second similarity corresponding to each image.

[0136] The similarity calculation methods include but are not limited to Euclidean distance, cosine similarity, etc.

[0137] Exemplarily, assuming that the image includes nine image blocks, the similarity between the feature vector of each of the nine image blocks and the original prompt word corresponding to the image is calculated to obtain the similarity of each image block, and then the similarities of the nine image blocks are summed to obtain the total similarity of the nine image blocks. The total similarity is then divided by nine to obtain the similarity corresponding to this image, that is, the second similarity.

[0138] Since there are multiple images input to the input layer, the second similarity corresponding to each image can be aggregated to obtain a second similarity matrix corresponding to the image set. The second similarity matrix and the image labels in the image set are then used to construct the third loss function of the initial pre-trained model. The image label is the actual label of the image, which can be represented by text, letters, etc.

[0139] Among them, the third loss function is the loss function of contrastive learning, and the first loss function includes but is not limited to maximum edge contrast loss, triple loss, N-Pair Multi-Class Loss, and Info NCE Loss (noise contrastive estimation Loss).

[0140] After obtaining the third loss function, the first loss function, the second loss function, and the third loss function constitute the loss function of the fine-tuned pre-trained model, that is, the loss function of the target pre-trained model. To obtain the target pre-trained model, specifically, the loss value can be calculated using the obtained loss functions, namely the first loss function, the second loss function, and the third loss function, and the parameters corresponding to the fully connected layer of the initial pre-trained model can be updated based on the calculated loss value to fine-tune the initial pre-trained model to obtain the target pre-trained model. The target pre-trained model formed at this time avoids overfitting of the original prompt words to the downstream training dataset, enhances the effect of downstream tasks, and improves the generalization ability of the pre-trained model.

[0141] After obtaining the target pre-training model, the target pre-training model can be applied, such as performing multimedia resource understanding tasks, task classification, target recognition, etc. through the target pre-training model.

[0142] When classifying tasks, we first acquire the task, such as images or text. For example, we first acquire the target image. This target image is the image for which object detection and classification are required. The target image can be acquired using an image acquisition device or directly from various data sources, which will not be discussed here. The target image format is also determined based on actual needs and can be a photograph, etc., without limitation.

[0143] After obtaining the target image that needs to be detected and classified, the target image needs to be input into the target pre-training model (i.e., the multi-task model) to obtain the target detection result and the prediction result of at least one task corresponding to the target detection result.

[0144] After the target image is input into the multi-task model, it is first encoded and then partitioned, such as into blocks, to determine its feature map. This feature map is then input into the target detection block to obtain the target detection result. The target detection result can be the location of the target to be detected, which is not detailed here.

[0145] The target detection result is mapped to the feature map of the target image to obtain the feature map of the target to be detected. The feature map of the target to be detected is input into each detection block to obtain the prediction result of at least one task corresponding to the target detection result.

[0146] Because the target to be detected in the image is first identified using target task recognition, and the feature maps of the target to be detected in the training set are then input into the target detection block, the target detection block can output the corresponding classification prediction results for the target to be detected. Compared to using multiple task heads in a multi-task model to output target detection results and classification prediction results, there is no need to determine the target detection result corresponding to each classification prediction result.

[0147] The embodiment of the present application provides a method for fine-tuning a pre-trained model, comprising: first obtaining an initial pre-trained model, then inputting local prompt words and target text descriptions into the input layer of the initial pre-trained model, wherein the local prompt words are used to represent prompt words for downstream tasks, and the target text descriptions are used to represent text descriptions of images in an image set, and then aligning the local prompt words with images in the image set of the input layer, constructing a first loss function of the initial pre-trained model, and regularizing the original prompt words of the input layer using the target text descriptions, constructing a second loss function of the initial pre-trained model, and then determining the target pre-trained model based on the first loss function and the second loss function, so as to perform the multimedia resource understanding task through the target pre-trained model. The present application constructs the first loss function of the initial pre-trained model by aligning the local prompt words with the images in the image set of the input layer to enhance the effect of the downstream task. In addition, the target text descriptions are also used to regularize the original prompt words of the input layer, and construct a second loss function of the initial pre-trained model to avoid overfitting the original prompt words to the downstream training data set.

[0148] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0149] The following are device embodiments of the present application. For details not fully described therein, please refer to the corresponding method embodiments described above.

[0150] Figure 3 A schematic diagram of the structure of a pre-training model fine-tuning device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown. A pre-training model fine-tuning device includes an acquisition module 301, an input module 302, a loss function construction module 303 and a task execution module 304, which are specifically as follows:

[0151] Acquisition module 301, used to obtain an initial pre-training model;

[0152] An input module 302 is configured to input local prompt words and target text descriptions into the input layer of the initial pre-trained model, wherein the local prompt words are used to represent prompt words for downstream tasks, and the target text descriptions are used to represent text descriptions of images in the image collection;

[0153] a loss function construction module 303 for aligning the local prompt words with the images in the image set input to the input layer to construct a first loss function of the initial pre-trained model, and regularizing the original prompt words in the input layer using the target text description to construct a second loss function of the initial pre-trained model;

[0154] The task execution module 304 is used to determine a target pre-training model based on the first loss function and the second loss function, so as to perform the multimedia resource understanding task through the target pre-training model.

[0155] In one embodiment, the loss function construction module 303 is further configured to obtain, for each image in the image set, a local prompt word corresponding to each image;

[0156] Calculate the similarity between each image and the local prompt word corresponding to each image to obtain the first similarity corresponding to each image;

[0157] Summarize the first similarities corresponding to each image to obtain a first similarity matrix corresponding to the image set;

[0158] A first loss function of an initial pre-trained model is constructed using the first similarity matrix and the image labels in the image set.

[0159] In one embodiment, the loss function construction module 303 is further used to obtain the category name of each image;

[0160] A prompt is added before the category name of each image, and the prompt is used as the local prompt word corresponding to each image.

[0161] In one embodiment, the loss function construction module 303 is further configured to perform text encoding on the local prompt word corresponding to each image to obtain a local prompt word feature vector;

[0162] Perform image encoding on each image to obtain the global feature vector of each image;

[0163] Based on the local prompt word feature vector and the global feature vector of each image, a first similarity corresponding to each image is calculated.

[0164] In one embodiment, the loss function construction module 303 is further configured to extract, for each image block in each image, a feature vector of each image block from the global feature vector of each image;

[0165] Calculate the similarity between the feature vector of each image block and the feature vector of the local prompt word to obtain the similarity of each image block;

[0166] Summarize the similarity of each image block to obtain the similarity of all image blocks;

[0167] The similarities of all image blocks are averaged to obtain the first similarity corresponding to each image.

[0168] In one embodiment, the loss function construction module 303 is further configured to obtain target text descriptions of images in the image set;

[0169] Perform text encoding on the target text description and the original prompt word respectively to obtain the text description feature vector and the original prompt word feature vector;

[0170] Calculate the similarity between the text description feature vector and the original prompt word feature vector to obtain the similarity corresponding to the original prompt word;

[0171] The second loss function of the initial pre-training model is constructed using the similarity corresponding to the original prompt word.

[0172] In one embodiment, the loss function construction module 303 is further configured to obtain initial text descriptions of images in the image set;

[0173] The initial text description is fine-grainedly rewritten using a first preset model to obtain a target text description, wherein the first preset model is obtained by training text training data.

[0174] In one embodiment, the loss function construction module 303 is further configured to use a second preset model to infer images in the image set and output an initial text description, wherein the second preset model is obtained by training image training data.

[0175] In one embodiment, the task execution module 304 is further configured to obtain a third loss function, wherein the third loss function is determined by image labels in the image set and a similarity matrix corresponding to the image set;

[0176] Based on the first loss function, the second loss function and the third loss function, a target pre-training model is determined.

[0177] In one embodiment, the task execution module 304 is further configured to obtain, for each image in the image set, an original prompt word corresponding to each image;

[0178] Calculate the similarity between each image and the original prompt word corresponding to each image to obtain the second similarity corresponding to each image;

[0179] Summarize the second similarities corresponding to each image to obtain a second similarity matrix corresponding to the image set;

[0180] A third loss function of the initial pre-trained model is constructed using the second similarity matrix and the image labels in the image collection.

[0181] In one embodiment, the task execution module 304 is further configured to perform text encoding on the original prompt word corresponding to each image to obtain a feature vector of the original prompt word;

[0182] Perform image encoding on each image to obtain the global feature vector of each image;

[0183] Based on the original prompt word feature vector and the global feature vector of each image, a second similarity corresponding to each image is calculated.

[0184] This application Figure 4 A schematic diagram of a computer device is provided. Figure 4 As shown, the computer device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned various pre-training model fine-tuning method embodiments are implemented, such as Figure 1 Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned embodiments of the pre-training model fine-tuning device are realized, for example Figure 3 Functionality of modules / units 301 to 304 is shown.

[0185] The present application also provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, it is used to implement the pre-training model fine-tuning method provided in the various embodiments described above.

[0186] Among them, the readable storage medium can be a computer storage medium or a communication medium. Communication media include any medium that facilitates the transmission of computer programs from one place to another. Computer storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0187] The present application also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the pre-trained model fine-tuning method provided in the various embodiments described above.

[0188] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in this application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0189] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A pre-training model fine-tuning method, characterized in that: include: Get the initial pre-trained model; Inputting local prompt words and target text descriptions into the input layer of the initial pre-trained model, wherein the local prompt words are used to represent prompt words for downstream tasks, and the target text descriptions are used to represent text descriptions of images in the image set; Aligning the local prompt word with an image in the image set input to the input layer to construct a first loss function of the initial pre-trained model, and regularizing the original prompt word of the input layer using the target text description to construct a second loss function of the initial pre-trained model; Based on the first loss function and the second loss function, a target pre-training model is determined to perform a multimedia resource understanding task through the target pre-training model.

2. The pre-training model fine-tuning method according to claim 1, characterized in that: The aligning the local prompt word with the image in the image set input to the input layer to construct the first loss function of the initial pre-training model includes: For each image in the image set, obtaining a local prompt word corresponding to each image; Calculating the similarity between each image and the local prompt word corresponding to each image to obtain a first similarity corresponding to each image; Summarizing the first similarities corresponding to each image to obtain a first similarity matrix corresponding to the image set; A first loss function of the initial pre-trained model is constructed using the first similarity matrix and the image labels in the image set.

3. The pre-training model fine-tuning method according to claim 2, characterized in that: The obtaining of the local prompt words corresponding to each image includes: Get the category name of each image; A prompt is added before the category name of each image, and the prompt is used as a local prompt word corresponding to each image.

4. The pre-training model fine-tuning method according to claim 2, wherein: The calculating the similarity between each image and the local prompt word corresponding to each image to obtain the first similarity corresponding to each image includes: Performing text encoding on the local prompt words corresponding to each image to obtain a local prompt word feature vector; Performing image encoding on each of the images to obtain a global feature vector of each of the images; Based on the local prompt word feature vector and the global feature vector of each image, a first similarity corresponding to each image is calculated.

5. The pre-training model fine-tuning method according to claim 4, characterized in that: The calculating the first similarity corresponding to each image based on the local prompt word feature vector and the global feature vector of each image includes: For each image block in each image, extracting a feature vector of each image block from a global feature vector of each image; Calculating similarity between the feature vector of each image block and the feature vector of the local prompt word to obtain the similarity of each image block; Summarizing the similarity of each image block to obtain the similarity of all image blocks; The similarities of all the image blocks are averaged to obtain a first similarity corresponding to each image.

6. The pre-training model fine-tuning method according to claim 1, wherein: Regularizing the original prompt words of the input layer by using the target text description to construct a second loss function of the initial pre-training model includes: Obtain target text descriptions of images in the image set; Performing text encoding on the target text description and the original prompt word to obtain a text description feature vector and an original prompt word feature vector; Calculate the similarity between the feature vector of the text description and the feature vector of the original prompt word to obtain the similarity corresponding to the original prompt word; The second loss function of the initial pre-training model is constructed using the similarity corresponding to the original prompt word.

7. The pre-training model fine-tuning method according to claim 6, characterized in that: The obtaining of target text descriptions of images in the image set includes: Obtaining initial text descriptions of images in the image collection; The initial text description is fine-grainedly rewritten using a first preset model to obtain the target text description, wherein the first preset model is obtained by training text training data.

8. The pre-training model fine-tuning method according to claim 7, wherein: The obtaining of an initial text description of an image in the image set includes: The images in the image set are inferred using a second preset model to output the initial text description, wherein the second preset model is obtained by training the image training data.

9. The pre-training model fine-tuning method according to claim 1, wherein: The determining a target pre-training model based on the first loss function and the second loss function includes: Obtaining a third loss function, wherein the third loss function is determined by image labels in the image set and a similarity matrix corresponding to the image set; Based on the first loss function, the second loss function and the third loss function, a target pre-training model is determined.

10. The pre-training model fine-tuning method according to claim 9, wherein: The obtaining of the third loss function includes: For each image in the image set, obtaining the original prompt word corresponding to each image; Calculating the similarity between each image and the original prompt word corresponding to each image to obtain a second similarity corresponding to each image; Summarizing the second similarities corresponding to each image to obtain a second similarity matrix corresponding to the image set; A third loss function of the initial pre-trained model is constructed using the second similarity matrix and the image labels in the image set.

11. The pre-training model fine-tuning method according to claim 10, wherein: The calculating the similarity between each image and the original prompt word corresponding to each image to obtain the second similarity corresponding to each image includes: Performing text encoding on the original prompt word corresponding to each image to obtain a feature vector of the original prompt word; Performing image encoding on each of the images to obtain a global feature vector of each of the images; Based on the original prompt word feature vector and the global feature vector of each image, a second similarity corresponding to each image is calculated.

12. A pre-training model fine-tuning device, characterized in that: include: Acquisition module, used to obtain the initial pre-training model; an input module, configured to input local prompt words and target text descriptions into the input layer of the initial pre-trained model, wherein the local prompt words are used to represent prompt words for downstream tasks, and the target text descriptions are used to represent text descriptions of images in the image set; a loss function construction module, configured to align the local prompt word with an image in the image set input to the input layer to construct a first loss function of the initial pre-trained model, and regularize the original prompt word of the input layer using the target text description to construct a second loss function of the initial pre-trained model; A task execution module is used to determine a target pre-training model based on the first loss function and the second loss function, so as to perform the multimedia resource understanding task through the target pre-training model.

13. A computer device, characterized in that: comprising a memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors so that the one or more processors implement the pre-training model fine-tuning method as described in claims 1 to 11.

14. A computer-readable storage medium, characterized in that It includes a program or instruction, which, when running on a computer, implements the pre-training model fine-tuning method described in claims 1 to 11.

Images