Downstream task processing method, model training method and device

Abstract

One or more embodiments of the present specification provide a downstream task processing method, a model training method and a device. The training method mainly trains the downstream task model by using prompts to fine-tune the pre-trained model. Specifically, the training method uses the training samples as the input of the pre-trained model, and uses the introduced prompt tokens to guide the pre-trained model to output shared features and proprietary features respectively. The shared features and proprietary features are trained by applying deterministic supervisory constraints respectively, so that the pre-trained model can deterministically learn shared knowledge and proprietary knowledge.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of model training technology, and in particular, to a downstream task processing method, a model training method and a device. Background Art

[0002] Multi-task learning is a machine learning method that aims to enable the learned model to handle multiple different tasks simultaneously by learning the shared knowledge of multiple related but different tasks and the proprietary knowledge of each task.

[0003] In multi-task learning, models suitable for each task can be trained from scratch. However, as the size and computational complexity of the model grow exponentially, the above-mentioned method of training models suitable for each task from scratch is gradually being abandoned due to the limitation of computing resources. Subsequently, a more efficient learning method has been proposed, which can fine-tune the pre-trained model through prompts so that the fine-tuned pre-trained model is suitable for downstream tasks.

[0004] Although the related technologies of fine-tuning pre-trained models have achieved certain results, there is still room for further optimization. Summary of the invention

[0005] In order to further improve the accuracy of the model trained by the prompt fine-tuning pre-trained model method, one or more embodiments of the present specification provide a downstream task processing method, a model training method and a device.

[0006] In a first aspect, one or more embodiments of the present specification provide a method for training a downstream task model, the method comprising: obtaining a training data set, the training data set comprising training data corresponding to at least one downstream task, the training data corresponding to the downstream task comprising multiple training samples and true labels corresponding to each of the training samples; inputting the training samples into a pre-trained model; introducing prompt words into the pre-trained model based on the training samples; guiding the pre-trained model to output shared features and proprietary features corresponding to the training samples through the prompt words; performing feature fusion processing on the shared features and the proprietary features to obtain fused features; generating predicted labels corresponding to the training samples based on the fused features; determining a first loss based on the true labels and the predicted labels; determining a second loss based on the shared features and the original features of the model; wherein the original features of the model are output features of the pre-trained model corresponding to the training samples before the pre-trained model is trained; training the pre-trained model based on the first loss and the second loss to obtain a downstream task model suitable for the at least one downstream task.

[0007] In one possible implementation, the pre-trained model includes a first encoder, the first encoder includes at least one feature learning layer, each of the feature learning layer includes a first freezing module and a first prompt word symbol generation module; the introduction of prompt words into the pre-trained model based on the training samples includes: using each of the first prompt word symbol generation modules respectively, based on the input features of the corresponding feature learning layer, to generate first prompt word symbols corresponding to each of the feature learning layers; wherein the input features of the first feature learning layer are the features generated based on the training samples, and the input features of the i+1th feature learning layer are the output features corresponding to the first freezing module in the i-th feature learning layer; wherein i is a positive integer greater than 0.

[0008] In a possible implementation, the first prompt word token generation module includes a convolution layer, a shared knowledge learning matrix, and a proprietary knowledge learning matrix corresponding to each of the downstream tasks; the method for the first prompt word token generation module in the i-th feature learning layer to generate the first prompt word token based on the input features of the corresponding feature learning layer includes: using the convolution layer to perform feature extraction processing on the input features of the i-th feature learning layer to obtain global features; determining the target task corresponding to the training sample; determining the target proprietary knowledge learning matrix corresponding to the target task; using the target proprietary knowledge learning matrix to extract features from the global features to obtain first features; using the shared knowledge learning matrix to extract features from the global features to obtain second features; generating a basic prompt word token based on the proprietary knowledge learning matrices corresponding to each of the downstream tasks and the shared knowledge learning matrix; generating the first prompt word token corresponding to the i-th feature learning layer based on the basic prompt word token, the global features, the first features, and the second features.

[0009] In one possible implementation, the target-specific knowledge learning matrix includes first network parameters, and the shared knowledge learning matrix includes second network parameters; training the pre-trained model based on the first loss and the second loss includes: optimizing the first network parameters and the second network parameters in the pre-trained model based on the first loss and the second loss.

[0010] In a possible implementation manner, the first encoder is an image encoder or a text encoder.

[0011] In a possible implementation, the training sample is an image sample, the pre-training model also includes a second encoder, the first encoder is an image encoder, the second encoder is a text encoder, the second encoder includes at least one text feature learning layer, each of the text feature learning layers includes a second freezing module and a second prompt word generation module; the introduction of prompt words in the pre-training model based on the training sample also includes: determining the target task corresponding to the image sample; determining at least one target true label included in the target task; processing each of the target true labels into a respectively corresponding description text; processing each of the description texts into a respectively corresponding text word; using each of the second prompt word generation modules, based on the input features of the corresponding text feature learning layer, to generate second prompt words corresponding to each of the text feature learning layers; wherein the input features of the first text feature learning layer are features generated based on the text words, and the input features of the i+1th text feature learning layer are output features corresponding to the second freezing module in the i-th text feature learning layer; wherein i is a positive integer greater than 0.

[0012] In a possible implementation, guiding the pre-trained model to output shared features and proprietary features corresponding to the training samples through the prompt word includes: guiding the first encoder to output a first shared feature and a first proprietary feature through the first prompt word; guiding the second encoder to output a second shared feature corresponding to each of the target real labels, and a second proprietary feature corresponding to each of the target real labels, through the second prompt word; wherein the shared features include the first shared features and the second shared features corresponding to each of the target real labels, and the proprietary features include the first proprietary features and the second proprietary features corresponding to each of the target real labels.

[0013] In a possible implementation, the shared features and the specific features of the target task are subjected to feature fusion processing to obtain fused features, including: fusing the first shared feature and the first specific feature to obtain a first fused feature; fusing each of the second shared features and the second specific features corresponding to each of the second shared features to obtain multiple second fused features; wherein the fused feature includes the first fused feature and the multiple second fused features.

[0014] In a possible implementation, generating a predicted label corresponding to the training sample based on the fusion feature includes: determining similarity values ​​corresponding to the first fusion feature and each of the second fusion features respectively; determining a maximum similarity value based on each of the similarity values; and determining the true label corresponding to the maximum similarity value as the predicted label.

[0015] In a possible implementation, determining the second loss based on the shared features and the initial shared features includes: determining the image loss based on the first shared features and the original features of the image; wherein the original features of the image are the output features of the first encoder corresponding to the training samples before training the pre-trained model; determining the text loss based on the second shared features and the original features of the text; wherein the original features of the text are the output features of the second encoder corresponding to the training samples before training the pre-trained model; determining the second loss based on the image loss and the text loss.

[0016] In a possible implementation, the downstream task includes at least one of the following: visual question answering, image description, image-text retrieval, image and text-based classification tasks, and text summarization tasks.

[0017] In a second aspect, one or more embodiments of the present specification also provide a downstream task processing method, the method comprising: obtaining task data of the downstream task to be processed; using a downstream task model obtained by any method described in the first aspect, processing the task data to obtain a processing result.

[0018] In a third aspect, one or more embodiments of the present specification further provide a training device for a downstream task model, the device comprising: a first acquisition module, for acquiring a training data set, wherein the training data set comprises training data corresponding to at least one downstream task, wherein the training data corresponding to the downstream task comprises a plurality of training samples and a true label corresponding to each of the training samples; an input module, for inputting the training samples into the pre-trained model; an introduction module, for introducing prompt tokens into the pre-trained model based on the training samples; an output module, for guiding the pre-trained model to output shared features and proprietary features corresponding to the training samples through the prompt tokens; and a fusion module, for fusion of the pre-trained model. The shared features and the proprietary features are subjected to feature fusion processing to obtain fused features; a prediction module is used to generate a predicted label corresponding to the training sample based on the fused features; a first loss determination module is used to determine a first loss based on the true label and the predicted label; a second loss determination module is used to determine a second loss based on the shared features and the original features of the model; wherein the original features of the model are the output features corresponding to the training samples of the pre-trained model before the pre-trained model is trained; a training module is used to train the pre-trained model based on the first loss and the second loss to obtain a downstream task model suitable for the at least one downstream task.

[0019] In a possible implementation, the pre-trained model includes a first encoder, the first encoder includes at least one feature learning layer, each of the feature learning layers includes a first freezing module and a first prompt word token generation module; the introduction module is used to introduce the prompt word token into the pre-trained model based on the training sample, specifically:

[0020] The introduction module is used to respectively use each of the first prompt word generation modules to generate the first prompt word corresponding to each of the feature learning layers based on the input features of the corresponding feature learning layer; wherein the input features of the first feature learning layer are the features generated based on the training samples, and the input features of the i+1th feature learning layer are the output features corresponding to the first frozen module in the i-th feature learning layer; wherein i is a positive integer greater than 0.

[0021] In a possible implementation, the first prompt word token generation module includes a convolution layer, a shared knowledge learning matrix, and a proprietary knowledge learning matrix corresponding to each of the downstream tasks; the introduction module is used for the first prompt word token generation module in the i-th feature learning layer to generate the first prompt word token based on the input features of the current feature learning layer, specifically:

[0022] The introduction module is used to use the convolution layer to perform feature extraction processing on the input features of the i-th feature learning layer to obtain global features; determine the target task corresponding to the training sample; determine the target proprietary knowledge learning matrix corresponding to the target task; use the target proprietary knowledge learning matrix to extract features from the global features to obtain a first feature; use the shared knowledge learning matrix to extract features from the global features to obtain a second feature; generate basic prompt word tokens based on the proprietary knowledge learning matrices and the shared knowledge learning matrix corresponding to each of the downstream tasks; and generate a first prompt word token corresponding to the i-th feature learning layer based on the basic prompt word token, the global feature, the first feature and the second feature.

[0023] In a possible implementation, the target-specific knowledge learning matrix includes a first network parameter, and the shared knowledge learning matrix includes a second network parameter; the training module is used to train the pre-trained model based on the first loss and the second loss, specifically:

[0024] The training module is used to optimize the first network parameter and the second network parameter in the pre-trained model based on the first loss and the second loss.

[0025] In a possible implementation, the training sample is an image sample, the pre-training model further includes a second encoder, the first encoder is an image encoder, the second encoder is a text encoder, the second encoder includes at least one text feature learning layer, each of the text feature learning layers includes a second freezing module and a second prompt token generation module; the introduction module is used to introduce the prompt token into the pre-training model based on the training sample, and is also used to:

[0026] The introduction module is used to determine the target task corresponding to the image sample; determine at least one target real label included in the target task; process each of the target real labels into a corresponding description text; process each of the description texts into a corresponding text token; respectively use each of the second prompt token generation modules to generate second prompt tokens corresponding to each of the text feature learning layers based on the input features of the corresponding text feature learning layer; wherein the input features of the first text feature learning layer are features generated based on the text tokens, and the input features of the i+1th text feature learning layer are output features corresponding to the second frozen module in the i-th text feature learning layer; wherein i is a positive integer greater than 0.

[0027] In a possible implementation, the output module is used to guide the pre-trained model to output the shared features and the specific features corresponding to the training samples through the prompt token, specifically:

[0028] The output module is configured to guide the first encoder to output a first shared feature and a first unique feature through the first prompt word; guide the second encoder to output a second shared feature corresponding to each of the target true labels, and a second unique feature corresponding to each of the target true labels, through the second prompt word;

[0029] The shared features include the first shared features and second shared features corresponding to the target real labels respectively, and the unique features include the first unique features and second unique features corresponding to the target real labels respectively.

[0030] In a possible implementation, the fusion module is used to perform feature fusion processing on the shared feature and the proprietary feature to obtain a fused feature, specifically:

[0031] The fusion module is used to fuse the first shared feature and the first specific feature to obtain a first fused feature; and to fuse each of the second shared features and the second specific features corresponding to each of the second shared features to obtain a plurality of second fused features;

[0032] The fusion feature includes the first fusion feature and the plurality of second fusion features.

[0033] In a possible implementation, the prediction module is used to generate a prediction label corresponding to the training sample based on the fusion feature, specifically:

[0034] The prediction module is used to determine the similarity values ​​respectively corresponding to the first fusion feature and each of the second fusion features; determine the maximum similarity value based on each of the similarity values; and determine the true label corresponding to the maximum similarity value as the predicted label.

[0035] In a possible implementation, the second loss determination module is used to determine the second loss based on the shared features and the original features of the model, specifically:

[0036] The second loss determination module is used to determine the image loss based on the first shared features and the original features of the image; wherein the original features of the image are the output features corresponding to the training samples of the first encoder before the pre-training model is trained; the text loss is determined based on the second shared features and the original features of the text; wherein the original features of the text are the output features corresponding to the training samples of the second encoder before the pre-training model is trained; the second loss is determined based on the image loss and the text loss.

[0037] In a possible implementation, the downstream task includes at least one of the following: visual question answering, image description, image-text retrieval, image and text-based classification tasks, and text summarization tasks.

[0038] In a fourth aspect, one or more embodiments of the present specification also provide a downstream task processing device, characterized in that the device includes: a second acquisition module, used to acquire task data of the downstream task to be processed; and a processing module, used to process the task data using a downstream task model obtained by the method described in any one of the first aspects to obtain a processing result.

[0039] In a fifth aspect, one or more embodiments of the present specification also provide an electronic device comprising a memory and a processor; the memory is used to store a computer program product; the processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the training method of the downstream task model of the first aspect or the downstream task processing method of the second aspect is implemented.

[0040] In a sixth aspect, one or more embodiments of the present specification further provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, which, when executed, implement the training method of the downstream task model of the first aspect or the downstream task processing method of the second aspect.

[0041] In summary, one or more embodiments of the present specification provide a downstream task processing method, a model training method and a device. In the training method, the output features of the pre-trained model are decomposed into shared features and proprietary features, and deterministic supervisory constraints are applied to the shared features and the proprietary features, respectively. In this way, by applying deterministic supervisory constraints to the shared features, the fine-tuned shared features can be made consistent with the output features of the original pre-trained model, which is helpful for learning shared knowledge across all tasks. By applying deterministic supervisory constraints to the proprietary features, the output of the proprietary features containing task-specific information can be deterministically guided by prompt tokens. In this way, the trained downstream task model can deterministically learn shared knowledge and proprietary knowledge.

[0042] In addition, the training method of the downstream task model provided in one or more embodiments of the present specification can assist pre-training by configuring a prompt word generation module in each feature learning layer, and generating customized prompt words for each feature learning layer based on the shared knowledge, proprietary knowledge and unique information corresponding to the input features of each feature learning layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of one or more embodiments of the present specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of one or more embodiments of the present specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A flowchart of a method for training a downstream task model provided for one or more embodiments of this specification;

[0045] Figure 2 A schematic diagram of an algorithm architecture of a training method for a downstream task model provided in one or more embodiments of this specification;

[0046] Figure 3 A schematic diagram of an algorithm architecture of a prompt word token generation method provided in one or more embodiments of this specification;

[0047] Figure 4 A flowchart of a method for generating a prompt token provided in one or more embodiments of this specification;

[0048] Figure 5 A schematic diagram of an algorithm architecture of a training method for a downstream task model based on CLIP provided in one or more embodiments of this specification;

[0049] Figure 6 A structural block diagram of a training device for a downstream task model provided in one or more embodiments of this specification;

[0050] Figure 7 A structural block diagram of a downstream task processing device provided for one or more embodiments of this specification;

[0051] Figure 8 A structural block diagram of an electronic device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0052] One or more embodiments of the present invention are further described in detail below through the accompanying drawings and examples. Through these descriptions, the features and advantages of one or more embodiments of the present invention will become clearer and more specific.

[0053] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0054] In addition, the technical features involved in different implementations of one or more embodiments of this specification described below can be combined with each other as long as they do not conflict with each other.

[0055] To facilitate understanding, the application scenarios of the technical solutions provided by one or more embodiments of this specification are first described below.

[0056] Multi-task learning is a machine learning method that aims to enable the learned model to handle multiple different tasks simultaneously by learning the shared knowledge of multiple related but different tasks and the proprietary knowledge of each task.

[0057] In multi-task learning, models suitable for each task can be trained from scratch. However, as the size and computational complexity of the model grow exponentially, the above-mentioned method of training models suitable for each task from scratch is gradually being abandoned due to the limitation of computing resources. Subsequently, a more efficient learning method has been proposed, which can fine-tune the pre-trained model through prompts so that the fine-tuned pre-trained model is suitable for downstream tasks.

[0058] Pre-trained models can be pre-trained on large-scale unlabeled corpora and can learn general language representations that can be used for other downstream tasks. For example, pre-trained visual language models such as Contrastive Language-Image Pretraining (CLIP) and Bidirectional Encoder representations from Transformers (BEIT) can understand images and text. Therefore, pre-trained visual language models can be used to process downstream tasks corresponding to natural language understanding and computer vision tasks, such as visual question answering, image description, image-text retrieval, image- and text-based classification tasks, text summarization tasks, and other downstream tasks.

[0059] In this way, by prompting fine-tuning the pre-trained model, the fine-tuned model can be applied to various specific application fields such as visual assistants, intelligent robots, online education, and online shopping.

[0060] In one possible implementation, the prompt fine-tuning pre-trained model can be implemented in the following way: first, a set of shared prompt tokens are simply trained for multiple tasks. Then, by introducing the prompt tokens into the pre-trained model, the shared knowledge and task-specific knowledge of different tasks are learned, and a feature is output, which integrates the shared knowledge and the specific knowledge. It can be seen that the above method uses the fusion feature of shared knowledge and specific knowledge as the output of the pre-trained model, which makes it difficult for the pre-trained model to deterministically learn the two types of knowledge. In addition, since there are obvious fine-grained differences between different tasks and between different samples of the same task, all tasks use the same shared prompt token, which may ignore the specific knowledge of each task, resulting in incomplete information, poor learning effect, and even negative joint learning effect.

[0061] In order to solve the technical problem that it is difficult for a pre-trained visual language model to deterministically learn two types of knowledge, one or more embodiments of this specification provide a method for training a downstream task model, which mainly trains the downstream task model by prompting fine-tuning the pre-trained model. The method decomposes the output features of the pre-trained model into shared features and proprietary features, and applies deterministic supervisory constraints to the shared features and the proprietary features, respectively. In this way, by applying deterministic supervisory constraints to the shared features, the fine-tuned shared features can be made consistent with the shared features of the original pre-trained model, which is helpful for learning shared knowledge across all tasks. By applying deterministic supervisory constraints to the proprietary features, the output of the proprietary features containing task-specific information can be deterministically guided by the prompt tokens. In this way, the trained downstream task model can deterministically learn shared knowledge and proprietary knowledge.

[0062] In order to solve the technical problem that the way of using a set of shared prompt tokens for multiple tasks will ignore the proprietary knowledge of each subtask and even lead to negative joint learning effects, one or more embodiments of this specification provide a prompt token generation module. The prompt token generation module can extract shared knowledge, proprietary knowledge and sample-specific information for different tasks, generate customized prompt tokens for each sample, and thus help the pre-trained model learn more complete knowledge.

[0063] The following describes an embodiment of a method for training a downstream task model provided by one or more embodiments of this specification.

[0064] The training method of the downstream task model provided in one or more embodiments of the present specification can be implemented by deploying a neural network model and computer program code in software form in a hardware computing environment. Available hardware computing environments include: personal computers, servers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, programmable consumer electronic devices, cloud servers, server instances, supercomputers, etc.

[0065] Figure 1 A flowchart of a method for training a downstream task model provided for one or more embodiments of this specification. Figure 2 A schematic diagram of an algorithm architecture of a training method for a downstream task model provided in one or more embodiments of this specification. Figure 1 and 2 As shown, a training method for a downstream task model provided by one or more embodiments of this specification may include the following steps:

[0066] Step S102: Obtain a training data set.

[0067] The training data set may include training data corresponding to at least one downstream task, and the training data corresponding to each downstream task may include multiple training samples and true labels corresponding to each training sample, wherein each downstream task corresponds to a different specific task and each downstream task is correlated with another.

[0068] For example, the downstream tasks in the training data set may include image classification tasks for different categories of goods in online shopping. The training data corresponding to each category of image classification task may include multiple image samples and real labels corresponding to each image sample. The real label represents the real image category corresponding to the image sample.

[0069] For example, the downstream task may be an image classification task for different categories of goods, such as food, clothing, and home appliances. Correspondingly, the training data set may include training data corresponding to different categories of goods, such as food, clothing, and home appliances. The training data for each category of goods may include sample images of the corresponding product category and image categories corresponding to each sample image. For example, the training data for clothing category goods may include 20 image categories, such as hats, gloves, clothes, and pants, as well as sample images corresponding to each image category.

[0070] The training data corresponding to each downstream task may also include a task identifier, which is used to identify the downstream task to which each training sample belongs. For example, the training data corresponding to the food category includes a task identifier task1, the training data corresponding to the clothing category includes a task identifier task2, the training data corresponding to the home appliance category includes a task identifier task3, etc., which are not listed here one by one.

[0071] Correspondingly, each training sample can carry its corresponding task identifier. For example, if training sample 1 is a training sample in the food category, then the training sample 1 can carry the task identifier task1. In this way, based on the task identifier task1 carried by training sample 1, it can be determined that the downstream task corresponding to training sample 1 is the image classification task of the food category.

[0072] It should be noted that the training samples are not limited to image samples, but may also be natural language materials, text samples, etc., and one or more embodiments of this specification do not limit this.

[0073] Step S104: input training samples into the pre-training model.

[0074] Step S106: introducing prompt tokens into the pre-training model based on the training samples.

[0075] Step S108: guiding the pre-trained model to output shared features and proprietary features corresponding to the training samples through prompt tokens.

[0076] Step S110: perform feature fusion processing on the shared features and the proprietary features to obtain fused features.

[0077] Step S112: Generate prediction labels corresponding to training samples based on fusion features .

[0078] Step S114: Based on the real label and predicted labels , determine the first loss .

[0079] Step S116: Determine the second loss based on the shared features and the original features of the model .

[0080] Step S118: Based on the first loss and the second loss , train the pre-trained model to obtain a downstream task model suitable for at least one downstream task.

[0081] The pre-trained model may be a model that has learned the shared knowledge in the relevant field. For example, the pre-trained model may be a pre-trained visual language model such as CLIP or BEIT. The pre-trained visual language model such as CLIP or BEIT can understand images and texts.

[0082] See also Figure 2 The pre-training model may include an input layer, an encoder and an output layer. The encoder may include a freezing module and a prompt token generation module HIEM.

[0083] The training samples are input to the encoder through the input layer for processing. Optionally, before the training samples are input to the encoder, the training samples can be processed into corresponding sample vectors. In this way, using the sample vectors as input features of the encoder makes it easier to integrate other modal information.

[0084] For example, if the input training sample is an image sample, the image sample can be first divided into multiple non-overlapping image patches, and the size of each image patch can be 16x16 pixels. Then, each image patch is passed through a convolutional neural network (for example, 3x3 convolution) to output a vector, which can be called the image patch token of the corresponding image patch. In this way, the corresponding sample vector can be obtained after splicing all the image patch tokens. The sample vector can represent the information of the input training sample.

[0085] After the sample vector is input into the encoder, the prompt token generation module HIEM in the encoder can generate a prompt token based on the sample vector. Then, the prompt token and the sample vector are used as inputs of the freezing module. The freezing module can learn the shared features of multiple downstream tasks based on the input sample vector, and the freezing module can learn the specific features of the downstream task based on the guidance of the prompt token.

[0086] Among them, after being processed by the above encoder, two parts of features are output, and the two parts of features are shared features and exclusive features. In other words, the output layer is used to output shared features and exclusive features respectively.

[0087] It should be noted that the encoder may include at least one feature learning layer, each feature learning layer may include a freezing module and a prompt word token generation module HIEM, wherein the prompt word token generation module HIEM is configured before the freezing module. In this way, for the first freezing module, the input of the freezing module includes the sample vector and the prompt word token generated by the first prompt word token generation module HIEM. The output features of the first freezing module include intermediate shared features and intermediate proprietary features. For the second freezing module, the input of the second freezing module includes the intermediate shared features output by the first freezing module, and the prompt word token generated by the second prompt word token generation module HIEM based on the intermediate proprietary features.

[0088] By analogy, the i+1th prompt word symbol generation module HIEM can generate prompt word symbols based on the intermediate proprietary features output by the i-th freezing module. The i+1th freezing module takes the prompt word symbols generated by the i+1th prompt word symbol generation module HIEM and the intermediate shared features output by the i-th freezing module as input, and then outputs the intermediate shared features and intermediate proprietary features corresponding to the i+1th freezing module. Until the last freezing module outputs shared features and proprietary features. Among them, i can be a positive integer greater than 0.

[0089] It should also be noted that during the training process of the downstream task model, there is no need to fine-tune the parameters of the frozen module to retain the shared knowledge previously learned by the pre-trained model. For different pre-trained models, the network structure corresponding to the frozen module can be different. For example, the frozen module can be a network structure such as a recurrent neural network (RNN), a long short-term memory network (LSTM), a convolutional sequence to sequence (Conv-Seq2Seq), a Transformer, or a convolutional neural network (CNN).

[0090] It should also be noted that the prompt tokens can include semantic information used to characterize the content of the training sample to guide the pre-trained model to generate the required output in a specific direction. For example, if the input training sample is an image including a fork, the corresponding prompt token can be a vector used to represent the fork image. In this way, the prompt tokens in vector form can interact with the features of the frozen module in the pre-trained model for learning. In other words, during the training process, these prompt tokens are learnable and gradually absorb and encode semantic features of different categories, thereby improving the performance of the model.

[0091] Since the freezing module can map the sample vector to the corresponding representation space, a shared representation vector, i.e., a shared feature, can be generated based on the sample vector. For each specific downstream task, the unique visual and semantic information required can be generated by the prompt token generation module HIEM to generate the corresponding prompt token. Then, the generated prompt token is introduced into the input of the freezing module. In this way, the corresponding proprietary features can be output by introducing the prompt token.

[0092] Furthermore, one or more embodiments of the present specification introduce two loss functions to perform deterministic supervised learning on the output shared features and proprietary features. The first loss function can be used to guide the prompt tokens to fit the downstream tasks using the true labels of the training samples. The second loss function can be used to constrain the shared features so that the learned shared features are consistent with the representation of the original features of the pre-trained model, thereby retaining the shared knowledge originally learned by the pre-trained model. Exemplarily, the first loss function can use a cross entropy loss function, and the second loss function can use a consistency constraint function. In this way, the cross entropy loss function can be used to constrain the shared features based on the true labels. and predicted labels , determine the first loss And, using the consistency constraint function, based on the shared features and the original features of the model, determine the second loss The original model features refer to the shared features learned by the pre-trained model before the pre-trained model is fine-tuned, that is, the output features corresponding to the input training samples of the pre-trained model before the pre-trained model is fine-tuned.

[0093] In this way, the pre-trained model can be trained based on the first loss and the second loss. For example, the sum of the first loss and the second loss, or the weighted sum of the first loss and the second loss can be used as the model loss. If the model loss is greater than or equal to the loss threshold, training based on the pre-trained model continues until the model loss of the task processing model obtained by training is less than the loss threshold.

[0094] Since the prompt token in one or more embodiments of the present specification is a learnable prompt token. That is to say, the prompt token includes optimizable network parameters. In this way, the above training process can be understood as a process of continuously optimizing the network parameters in the prompt token. If the model loss is greater than or equal to the loss threshold, continue to train based on the pre-trained model to continuously optimize the network parameters in the prompt token. If the model loss is less than the loss threshold, stop training the pre-trained model, obtain the downstream task model, and use the network parameters obtained from the last training as the network parameters of the prompt token.

[0095] Thus, one or more embodiments of the present specification provide a training method for a downstream task model, which divides the output features into shared features and proprietary features, and performs deterministic supervised learning on the two output features. This enables the pre-trained model to deterministically learn the shared features and proprietary features of the downstream task, which is conducive to learning more refined proprietary knowledge of the downstream task, so as to more comprehensively represent the fine-grained information unique to the downstream task.

[0096] One or more embodiments of the present specification also provide a prompt word token generation method, which can be applied to the prompt word token generation module HIEM in the above embodiments, so that the prompt word token generation module HIEM can generate customized prompt words for each feature learning layer, thereby helping the pre-trained model to learn more complete knowledge.

[0097] Figure 3 A schematic diagram of an algorithm architecture of a prompt word token generation method provided in one or more embodiments of this specification, Figure 4 A flowchart of a method for generating a prompt token provided in one or more embodiments of the present specification.

[0098] like Figure 3 As shown, the prompt token generation module HIEM may include a convolution layer, a shared knowledge learning matrix, and a proprietary knowledge learning matrix corresponding to each downstream task. For example, if the training sample set includes training data corresponding to four downstream tasks, the prompt token generation module HIEM may be configured with four proprietary knowledge learning matrices, wherein the four proprietary knowledge learning matrices correspond to the four downstream tasks one by one.

[0099] It should be noted that the model architecture corresponding to the prompt word token generation module HIEM in each feature learning layer is the same, and the difference is that the input features corresponding to each prompt word token generation module HIEM are different. For example, for the first prompt word token generation module HIEM, its input feature is a sample vector. For subsequent prompt word token generation modules HIEM, their input is the intermediate proprietary feature output by the previous frozen module.

[0100] The following is an example of a method for generating prompt tokens by the prompt token generation module HIEM in the i-th feature learning layer.

[0101] like Figure 4 As shown, a method for generating a prompt token provided by one or more embodiments of this specification may include the following steps:

[0102] Step S202: Use the convolution layer to perform feature extraction processing on the input features of the i-th feature learning layer to obtain global features.

[0103] Step S204: determine the target task corresponding to the training sample.

[0104] Step S206: Determine the target-specific knowledge learning matrix corresponding to the target task.

[0105] Step S208: extracting the global features using the target-specific knowledge learning matrix to obtain the first features.

[0106] Step S210: extracting the global features using the shared knowledge learning matrix to obtain the second features.

[0107] Step S212: Generate basic prompt tokens based on the proprietary knowledge learning matrix and the shared knowledge learning matrix corresponding to each downstream task.

[0108] Step S214: Generate a prompt word token corresponding to the i-th feature learning layer based on the basic prompt word token, the global feature, the first feature and the second feature.

[0109] Combination Figure 3 and Figure 4 In step S202, the input of the i-th prompt token generation module HIEM is ,in, After that, you can input Perform feature extraction to obtain global features ,in, , represents the number of all downstream tasks, Indicates the characteristic length.

[0110] For example, an efficient convolution layer (e.g., 1×1 convolution) can be used to Perform convolution calculation to obtain global features .

[0111] Optional, in the Before performing convolution calculation, you can first Perform average pooling. Average pooling can compress the input into a vector that can represent the global semantic features of the entire input feature.

[0112] In step S204, the target task refers to the downstream task corresponding to the input training sample.

[0113] In a possible implementation, the training data may include task identifiers corresponding to the training samples, so that the downstream task to which the input training sample belongs can be determined based on the correspondence between the training sample and the task identifier.

[0114] In a possible implementation, the training sample may carry a task identifier, so that the downstream task to which the input training sample belongs can be determined based on the task identifier carried by the input training sample.

[0115] In step S206, the proprietary knowledge learning matrix is ​​used to learn the proprietary knowledge that is most relevant to the training samples.

[0116] The proprietary knowledge learning matrix corresponding to all downstream tasks can be expressed as = ,in, .in, Indicates downstream task 1 and downstream task 2. The corresponding proprietary knowledge learning matrices respectively.

[0117] For example, if The corresponding target task is the downstream task k, then the target proprietary knowledge learning matrix corresponding to the downstream task k can be determined as ,in, .

[0118] Determine the target proprietary knowledge learning matrix as After that, the target-specific knowledge can be used to learn the matrix , learn proprietary knowledge based on global features and obtain the first feature corresponding to the downstream task k ,in, The first feature Includes proprietary knowledge features corresponding to the downstream task k.

[0119] Among them, the first feature corresponding to the downstream task k satisfies the following relationship (1).

[0120] = Relation (1)

[0121] In step S210, the same shared knowledge learning matrix is ​​used for any training sample, wherein the shared knowledge learning matrix is ​​used to learn the shared knowledge that is most relevant to each training sample.

[0122] The shared knowledge learning matrix corresponding to all downstream tasks can be expressed as = ,in, .in, Indicates downstream task 1 and downstream task 2. The corresponding shared knowledge learning matrices respectively.

[0123] For example, for the input , the shared knowledge learning matrix can be used ,study The corresponding second feature .in, , the second feature Includes shared knowledge features corresponding to each downstream task.

[0124] Among them, the second feature corresponding to the downstream task k , satisfying the following relationship (2).

[0125] Relation (2)

[0126] In step S212, the proprietary knowledge learning matrix Learning Matrix with Shared Knowledge The dimensions are the same. In this way, the proprietary knowledge learning matrix of the same dimension can be Learning Matrix with Shared Knowledge Add and process to get the basic prompt word ,in, .

[0127] In step S214, the basic prompt word is obtained , global features , the first feature And the second feature Then, we can use the relation (3) to generate Corresponding prompt word . After that, the above prompt word and Can be used as input to the i+1th freezing module.

[0128] =P + + Relation (3)

[0129] It can be seen that the above prompt word generation method can use the proprietary knowledge learning matrix Learning Matrix with Shared Knowledge , learning and input In this way, by configuring the above-mentioned prompt word generation module before each freezing module of the encoder, the corresponding proprietary knowledge features and shared knowledge features of each layer input can be learned, so that the encoder can learn more complete knowledge.

[0130] Among them, the prompt word symbols corresponding to each freezing module can refer to the description of the prompt word symbol generation method provided in the above embodiment, which will not be repeated here.

[0131] It should be noted that the proprietary knowledge learning matrix introduced in the above scheme and shared knowledge learning matrix It can be understood as two trainable matrices. Among them, the proprietary knowledge learning matrix It can include first network parameters, shared knowledge learning matrix The second network parameter may be included. The first network parameter and the second network parameter are learnable network parameters. Therefore, the process of training the pre-trained model based on the first loss and the second loss can be understood as continuously optimizing the proprietary knowledge learning matrix based on the first loss and the second loss. and shared knowledge learning matrix The process of the first network parameter and the second network parameter in.

[0132] It should also be noted that one or more embodiments of this specification do not limit the specific forms of the proprietary knowledge learning matrix and the shared knowledge learning matrix. For example, the proprietary knowledge learning matrix and the shared knowledge learning matrix can be task embedding matrices, and the task embedding matrix defines a task embedding vector for each downstream task, and all downstream task embedding vectors can be spliced ​​into a matrix. In this way, by multiplying the task embedding matrix with the input global feature, the corresponding proprietary knowledge feature or shared knowledge feature can be obtained.

[0133] It should also be noted that the pre-trained model may include one or more encoders. One or more embodiments of this specification do not limit the number of encoders, and the number of encoders may be specifically determined by the pre-trained model. For example, the pre-trained model may include an encoder, such as a first encoder, and the first encoder may be an image encoder or a text encoder. For another example, for the pre-trained model CLIP, CLIP may include a first encoder and a second encoder, wherein the first encoder may be an image encoder and the second encoder may be a text encoder.

[0134] The following takes CLIP as an example of a pre-trained model to exemplify the training method of the downstream task model provided in one or more embodiments of this specification.

[0135] Figure 5 A schematic diagram of the algorithm architecture of a CLIP-based downstream task model training method provided for one or more embodiments of this specification.

[0136] like Figure 5 As shown, the pre-trained model may include an image encoder (also referred to as a first encoder) and a text encoder (also referred to as a second encoder), and the corresponding pre-trained parameters may be expressed as ( , ).

[0137] The image encoder may include at least one image feature learning layer, and each image feature learning layer may include a first freezing module and a first prompt token generation module HIEM. For example, the image encoder may include a first freezing module LayerV 1 、The first freezing module LayerV 2 , ..., the first freezing module LayerV n Wherein, a first prompt word symbol generating module HIEM may be respectively configured before each first freezing module.

[0138] Similarly, the text encoder may include at least one text feature learning layer, and each text feature learning layer may include a second freezing module and a second prompt token generation module HIEM. 1 , the second freezing module LayerL 2 , ..., the second freezing module LayerL n Wherein, a second prompt word generating module HIEM may be respectively configured before each second freezing module.

[0139] The training sample set may include sample images corresponding to the downstream task and true labels corresponding to each sample image.

[0140] After obtaining the training sample set, the training sample set may be preprocessed to obtain inputs of the image encoder and the text encoder respectively.

[0141] For an image encoder, a sample image may be processed into image block tokens, and then the image block tokens are used as inputs of the image encoder.

[0142] For the text encoder, the target task corresponding to the image sample can be determined first. Then, all the target true labels corresponding to the target task are determined. After that, each target true label is processed into a corresponding description text, and each description text is processed into a corresponding text token. Finally, each text token is used as the input of the text encoder.

[0143] For example, the input sample image X is a sample image in the downstream task k, and the downstream task k includes 64 image categories (i.e., 64 real labels). The sample image X can be mapped to the corresponding image block token { Then, the image block token { } as the input of the image encoder. Then, the 64 real labels can be respectively passed through the text template to generate the corresponding description text. For example, the description text includes "a photo of a {class label1}", "a photo of a {class label2}", etc., where classlabel1 and class label2 are different real labels. Then, each description text is converted into the text token { }, and replace the text token { } as the input of the text encoder.

[0144] The processing flow of the image encoder is as follows: image block token { After the image encoder is input, the first prompt token generation module HIEM in the image encoder is based on the image block token { }, generate the first prompt token ={ Then, the first prompt token and the image block token are used as the input of the first freezing module. For example, the input of the first freezing module is { },in, represents a learnable category token. The first frozen module in the image encoder converts { }Mapped into a latent visual representation space , output the first shared feature and a unique feature generated based on the first prompt token ,in, , Among them, the first shared feature Responsible for preserving the shared knowledge learned by the original CLIP model pre-training, and also learning the shared knowledge of downstream tasks. Then, the first hint token introduced is fully utilized to output the specialized knowledge of each downstream task. Further, for the image sample of downstream task k, the first shared feature The first unique feature corresponding to the downstream task k Perform fusion processing to obtain the first fusion feature The first fusion feature It can contain more complete information, thereby improving the generalization of the model and the downstream migration effect.

[0145] Among them, the first fusion feature It can be obtained through the following relations (4) and (5):

[0146] = ={ } Relation (4)

[0147] Relation (5)

[0148] in, represents the unique features determined according to the downstream task k.

[0149] The processing flow of the text encoder is as follows: text token { After the text encoder is input, the second prompt token generation module HIEM in the text encoder is based on the text token { }, generate the second prompt token Then, the second prompt token and the text block token are used as the input of the second freezing module. For example, the input of the second freezing module is ,in, represents a learnable category token, and Represent the start and end tokens of the learnable words respectively. The second frozen module in the text encoder will Mapping to latent text feature space , output the second shared features corresponding to each category And unique features ,in, , , . Further, for the image samples of downstream task k, the second shared features of each category are With the second unique feature Perform fusion processing to obtain the second fusion features corresponding to each category , ,……, . Where N represents the number of image categories contained in the downstream task k, and N is a positive integer greater than 1.

[0150] Among them, the second fusion feature It can be obtained through the following relations (6) and (7):

[0151] = , ={ } Relation (6)

[0152] Relation (7)

[0153] in, represents the unique features determined according to the downstream task k.

[0154] Get the first fusion feature and the second fusion feature , ,……, After that, the similarity values ​​corresponding to the first fusion feature and each second fusion feature can be calculated. , , Then, based on the similarity values, the maximum similarity value is determined, and the true label corresponding to the maximum similarity value is determined as the predicted label .

[0155] For example, the relationship (8) can be used to determine the predicted label based on the similarity value .

[0156] Relation (8)

[0157] Among them, sim(·) represents the cosine similarity calculation formula, Represents the temperature coefficient, which is used to adjust the sensitivity of the similarity calculation.

[0158] The loss calculation method is as follows: The cross entropy loss can be determined based on the true label and the predicted label (Also called the first loss). And, based on the first shared feature and the original image feature, determine the consistency constraint loss (Also called image loss), based on the second shared feature and the original text feature, determine the consistency constraint loss (Also called text loss).

[0159] Specifically, the cross entropy loss can be calculated using equation (9): , using equation (10) to calculate the consistency constraint loss , using equation (11) to calculate the consistency constraint loss .

[0160] L(sim( , ), y) Relationship (9)

[0161] Relation (10)

[0162] Relation (11)

[0163] in, represents the original shared features of the image encoder, represents the original shared features of the text encoder, and m represents the number of training samples.

[0164] Afterwards, the cross entropy loss can be , consistency constraint loss and The sum is the total loss of the model . That is, the total loss of the model The relationship (12) can be satisfied.

[0165] = + + Relation (12)

[0166] In this way, when the total loss of the model When it is less than the loss threshold, the training of the pre-trained model can be stopped to obtain the downstream task model.

[0167] It should be noted that the processing process of the input image features by the first freezing module and the first prompt word generating module of the image encoder, and the processing process of the input text features by the second freezing module and the second prompt word generating module of the text encoder can be referred to in Figure 1 and Figure 2 The description of the corresponding embodiments will not be repeated here.

[0168] In order to prove that the training method of the downstream task model provided in one or more embodiments of this specification is effective, the training method of the downstream task model provided in one or more embodiments of this specification is verified based on two mainstream data sets, Office-Home and MiniDomainNet.

[0169] The Office-Home dataset contains images corresponding to the following four different downstream tasks: Art, Clipart, Product, and Real. The Office-Home dataset covers 65 object categories in office and home environments. There are approximately 15,500 images in total.

[0170] The MiniDomainNet dataset is a subset of the challenging multi-task learning dataset DomainNet. The MiniDomainNet dataset includes images corresponding to the following four different downstream tasks: Clipart, Painting, Sketch, and Real. The MiniDomainNet dataset contains 140,000 images distributed in 126 categories.

[0171] In the validation experiment, the training data selected 10% (6 samples per class) and 20% (12 samples per class) samples from the Office-Home dataset, and 1% (3 samples per class) and 2% (6 samples per class) samples from the MiniDomainNet dataset. The remaining samples were used to test the model effect.

[0172] Experiment 1

[0173] The purpose of Experiment 1 is to compare the effects of different training methods on the accuracy of the model.

[0174] Specifically, the training method provided by one or more embodiments of this specification, zero-shot learning (Zero-shot), the adaptation method of the visual-language pre-training model (C-Adapter), the collaborative multi-task learning method based on CLIP (CoOp-MT), the visual prompt multi-task learning method based on CLIP (VPT-MT), the masked language modeling driven multi-task learning method (MaPLe-MT), and the multimodal adaptation prompt method (MmAP) are respectively used to train based on the same training data as above. Then, based on the test data, the accuracy corresponding to each downstream task is tested using each trained model.

[0175] Among them, for the training methods corresponding to Zero-shot, C-Adapter, CoOp-MT, VPT-MT, MaPLe-MT, and MmAP, please refer to the description of the prior art and will not be repeated here.

[0176] Using the Office-Home dataset as training data, the accuracy of each training method is shown in Table 1. Using the MiniDomainNet dataset as training data, the accuracy of each training method is shown in Table 2.

[0177] Table 1

[0178]

[0179] As can be seen from Table 1 above, in the experiments using 10% and 20% of the office-Home dataset as training samples respectively, the training method provided by one or more embodiments of the present specification performs well in the accuracy of each downstream task and has significant advantages over other training methods.

[0180] Table 2

[0181]

[0182] Comparing Table 1 and Table 2, it can be seen that compared with the downstream tasks corresponding to Office-Home, the difficulty of MiniDomainNet in downstream tasks has increased significantly. In addition, the training data used in Table 2 is significantly less than the training data used in Table 1. However, the training method provided by one or more embodiments of this specification performs well in the accuracy of each downstream task, and has significant advantages over other training methods.

[0183] Experiment 2

[0184] The purpose of Experiment 2 is to compare the effects of output features and supervision methods on the accuracy of the model.

[0185] Specifically, in Experiment 2, four groups of experimental examples are used to compare the impact of output features and supervision methods on the accuracy of the model. In Experiment 1, only shared features are output, no dedicated features are output, and only As the loss function. In Experiment 2, only the dedicated features are output, no shared features are output, and only As the loss function. In Experiment 3, the dedicated features and shared features are output separately, and only As the loss function. In Experiment 4, the dedicated features and shared features are output respectively, and and as a loss function.

[0186] It should be noted that in the above Experimental Examples 1, 2, 3 and 4, except that the output features and loss functions are set differently as shown in Table 3, other training processes and parameters are the same.

[0187] In this way, the model is trained based on the settings of Experimental Example 1, Experimental Example 2, Experimental Example 3 and Experimental Example 4 in Table 3. The accuracy rates of the models trained by the four experimental examples are shown in Table 3.

[0188] Table 3

[0189]

[0190] As can be seen from Table 3, when shared or proprietary features are used as output in Experimental Examples 1 and 2, the accuracy of the model is significantly lower. The accuracy of Experimental Example 3 is lower than that of Experimental Example 4. As a loss function, the shared features do not use the consistency loss function to retain common knowledge, resulting in the performance of the model being affected. The training method corresponding to Experimental Example 4 is the training method provided by one or more embodiments of this specification. As can be seen from Table 3, one or more embodiments of this specification use shared features and proprietary features as outputs, and + As a supervision method of the loss function, it can make the performance of the trained model better.

[0191] Experiment 3

[0192] The purpose of Experiment 3 is to compare the effect of the cue token on the accuracy of the model.

[0193] Specifically, in Experiment 3, two groups of experimental examples are used to compare the effect of prompt tokens on the accuracy of the model. In Experiment 5, the prompt tokens of the pre-trained model are obtained by random initialization during training. In Experiment 6, the prompt tokens of the pre-trained model are introduced during training based on Figure 3 and Figure 4 The method shown is generated.

[0194] It should be noted that, in the above Experimental Example 5 and Experimental Example 6, except for the different settings shown in Table 4, other training methods and parameters are the same.

[0195] In this way, the model is trained based on the settings of Experimental Example 5 and Experimental Example 6 in Table 4. The accuracy of the models trained in Experimental Example 5 and Experimental Example 6 is shown in Table 4.

[0196] Table 4

[0197]

[0198] It can be seen from Table 4 above that, during training, using the prompt word token generation method provided by one or more embodiments of this specification can significantly improve the accuracy of the model.

[0199] In summary, the training method of the downstream task model provided by one or more embodiments of the present specification decomposes the output features of the pre-trained model into shared features and proprietary features, and applies deterministic supervisory constraints to the shared features and the proprietary features, respectively. In this way, by applying deterministic supervisory constraints to the shared features, the fine-tuned shared features can be made consistent with the output features corresponding to the original pre-trained model, which is helpful for learning shared knowledge across all tasks. By applying deterministic supervisory constraints to the proprietary features, the output of the proprietary features containing task-specific information can be deterministically guided by the prompt tokens. In this way, the trained downstream task model can deterministically learn shared knowledge and proprietary knowledge.

[0200] In addition, the training method of the downstream task model provided in one or more embodiments of this specification can configure a prompt word token generation module in each feature learning layer, and generate customized prompt word tokens for each feature learning layer according to the shared knowledge, proprietary knowledge and unique information corresponding to the input features of each feature learning layer, thereby helping the pre-trained model to learn more complete knowledge. Experimental data show that this prompt word token generation method can significantly improve the accuracy of the model.

[0201] One or more embodiments of the present specification also provide a downstream task processing method, which can first obtain task data of the downstream task to be processed; then, use the downstream task model trained by the above embodiment to process the task data to obtain a processing result corresponding to the task data.

[0202] It should be noted that the downstream task to be processed may be any downstream task in at least one downstream task of the training. For example, the downstream task may include at least one of the following: visual question answering, image description, image and text retrieval, image and text-based classification tasks, and text summarization tasks.

[0203] Exemplarily, the trained downstream task model can be used to process text-based sentiment classification tasks. In this way, the downstream task to be processed can be a text-based sentiment classification task, the task data can be text, and the processing result can be the sentiment type corresponding to the input text.

[0204] It should be noted that the process of processing the task data by using the downstream task model trained by the above embodiment can be referred to the description of the embodiment of the training method of the downstream task model, which will not be repeated here. The difference is that when the downstream task model trained by the above embodiment is used to process the task data, the obtained prediction label is the processing result. In other words, when the downstream task model trained by the above embodiment is used to process the task data, there is no need to perform loss calculation.

[0205] It can be understood that the above embodiments are only examples and can be modified in actual implementation. Those skilled in the art can understand that the modification methods of the above embodiments without creative labor fall within the protection scope of one or more embodiments of this specification and will not be repeated in the embodiments.

[0206] Based on the same inventive concept, one or more embodiments of the present specification also provide a training device for a downstream task model. Since the principle of solving the problem by the training device for the downstream task model is similar to that of the aforementioned training method for the downstream task model, the implementation of the training device for the downstream task model can refer to the implementation of the aforementioned training method for the downstream task model, and the repeated parts will not be repeated.

[0207] See also Figure 6 , Figure 6 A structural block diagram of a training device for a downstream task model provided in one or more embodiments of this specification. Figure 6 As shown, the training device 300 of the downstream task model may include: a first acquisition module 301, an input module 302, an introduction module 303, an output module 304, a fusion module 305, a prediction module 306, a first loss determination module 307, a second loss determination module 308, and a training module 309.

[0208] The first acquisition module 301 is used to acquire a training data set, where the training data set includes training data corresponding to at least one downstream task, where the training data corresponding to the downstream task includes multiple training samples and true labels corresponding to each of the training samples.

[0209] The input module 302 is used to input the training samples into the pre-training model.

[0210] The introduction module 303 is used to introduce prompt tokens into the pre-training model based on the training samples.

[0211] The output module 304 is used to guide the pre-trained model to output the shared features and the specific features corresponding to the training samples through the prompt token.

[0212] The fusion module 305 is used to perform feature fusion processing on the shared features and the specific features to obtain fused features.

[0213] The prediction module 306 is used to generate a prediction label corresponding to the training sample based on the fusion feature.

[0214] The first loss determination module 307 is used to determine a first loss based on the true label and the predicted label.

[0215] The second loss determination module 308 is used to determine the second loss based on the shared features and the original features of the model; wherein the original features of the model are the output features corresponding to the training samples of the pre-trained model before the pre-trained model is trained.

[0216] The training module 309 is used to train the pre-training model based on the first loss and the second loss to obtain a downstream task model suitable for the at least one downstream task.

[0217] In a possible implementation, the pre-trained model includes a first encoder, the first encoder includes at least one feature learning layer, each of the feature learning layers includes a first freezing module and a first prompt token generation module; the introduction module 303 is used to introduce the prompt token into the pre-trained model based on the training sample, specifically:

[0218] The introduction module 303 is used to respectively use each of the first prompt word generation modules to generate the first prompt word corresponding to each of the feature learning layers based on the input features of the corresponding feature learning layer; wherein the input features of the first feature learning layer are the features generated based on the training samples, and the input features of the i+1th feature learning layer are the output features corresponding to the first frozen module in the i-th feature learning layer; wherein i is a positive integer greater than 0.

[0219] In a possible implementation, the first prompt word token generation module includes a convolution layer, a shared knowledge learning matrix, and a proprietary knowledge learning matrix corresponding to each of the downstream tasks; the introduction module is used for the first prompt word token generation module in the i-th feature learning layer to generate the first prompt word token based on the input features of the current feature learning layer, specifically:

[0220] The introduction module is used to use the convolution layer to perform feature extraction processing on the input features of the i-th feature learning layer to obtain global features; determine the target task corresponding to the training sample; determine the target proprietary knowledge learning matrix corresponding to the target task; use the target proprietary knowledge learning matrix to extract features from the global features to obtain a first feature; use the shared knowledge learning matrix to extract features from the global features to obtain a second feature; generate basic prompt word tokens based on the proprietary knowledge learning matrices and the shared knowledge learning matrix corresponding to each of the downstream tasks; and generate a first prompt word token corresponding to the i-th feature learning layer based on the basic prompt word token, the global feature, the first feature and the second feature.

[0221] In a possible implementation, the target-specific knowledge learning matrix includes a first network parameter, and the shared knowledge learning matrix includes a second network parameter; the training module 309 is used to train the pre-trained model based on the first loss and the second loss, specifically:

[0222] The training module 309 is used to optimize the first network parameter and the second network parameter in the pre-trained model based on the first loss and the second loss.

[0223] In a possible implementation, the training sample is an image sample, the pre-training model further includes a second encoder, the first encoder is an image encoder, the second encoder is a text encoder, the second encoder includes at least one text feature learning layer, each of the text feature learning layers includes a second freezing module and a second prompt token generation module; the introduction module 303 is used to introduce the prompt token into the pre-training model based on the training sample, and is also used to:

[0224] The introduction module 303 is used to determine the target task corresponding to the image sample; determine at least one target real label included in the target task; process each of the target real labels into a corresponding description text; process each of the description texts into a corresponding text token; respectively use each of the second prompt token generation modules to generate second prompt tokens corresponding to each of the text feature learning layers based on the input features of the corresponding text feature learning layer; wherein the input features of the first text feature learning layer are features generated based on the text tokens, and the input features of the i+1th text feature learning layer are output features corresponding to the second frozen module in the i-th text feature learning layer; wherein i is a positive integer greater than 0.

[0225] In a possible implementation, the output module 304 is used to guide the pre-trained model to output the shared features and the specific features corresponding to the training samples through the prompt token, specifically:

[0226] The output module 304 is used to guide the first encoder to output the first shared feature and the first unique feature through the first prompt word; guide the second encoder to output the second shared feature corresponding to each of the target real labels, and the second unique feature corresponding to each of the target real labels, through the second prompt word;

[0227] The shared features include the first shared features and second shared features corresponding to the target real labels respectively, and the unique features include the first unique features and second unique features corresponding to the target real labels respectively.

[0228] In a possible implementation, the fusion module 305 is used to perform feature fusion processing on the shared feature and the proprietary feature to obtain a fused feature, specifically:

[0229] The fusion module 305 is used to fuse the first shared feature and the first specific feature to obtain a first fused feature; and to fuse each of the second shared features and the second specific features corresponding to each of the second shared features to obtain a plurality of second fused features;

[0230] The fusion feature includes the first fusion feature and the plurality of second fusion features.

[0231] In a possible implementation, the prediction module 306 is used to generate a prediction label corresponding to the training sample based on the fusion feature, specifically:

[0232] The prediction module 306 is used to determine the similarity values ​​respectively corresponding to the first fused feature and each of the second fused features; determine the maximum similarity value based on each of the similarity values; and determine the true label corresponding to the maximum similarity value as the predicted label.

[0233] In a possible implementation, the second loss determination module 308 is used to determine the second loss based on the shared features and the original features of the model, specifically:

[0234] The second loss determination module 308 is used to determine the image loss based on the first shared features and the original image features, wherein the original image features are the output features of the first encoder based on the input training samples before the pre-training model is trained. The text loss is determined based on the second shared features and the original text features, wherein the original text features are the output features of the second encoder based on the input training samples before the pre-training model is trained; the second loss is determined based on the image loss and the text loss.

[0235] In a possible implementation, the downstream task includes at least one of the following: visual question answering, image description, image-text retrieval, image and text-based classification tasks, and text summarization tasks.

[0236] See also Figure 7 , Figure 7 This is a structural block diagram of a downstream task processing device provided in one or more embodiments of this specification. Figure 7 As shown, the downstream task processing device 400 may include: a second acquisition module 401 and a processing module 402.

[0237] The second acquisition module 401 is used to acquire task data of the downstream task to be processed.

[0238] The processing module 402 is used to process the task data using the downstream task model trained by the training method of the downstream task model provided in one or more embodiments of this specification to obtain a processing result.

[0239] See also Figure 8 , Figure 8 This is a structural block diagram of an electronic device provided in one or more embodiments of this specification. Figure 8As shown, the electronic device 500 may include a processor 501 and a memory 502; the memory 502 may be coupled to the processor 501. It is worth noting that the Figure 8 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0240] In a possible implementation, the functions of the downstream task model training device 300 or the downstream task processing device 400 may be integrated into the processor 501. The processor 501 may be configured to perform the following operations:

[0241] Acquire a training data set, wherein the training data set includes training data corresponding to at least one downstream task, and the training data corresponding to the downstream task includes multiple training samples and true labels corresponding to each of the training samples;

[0242] Inputting the training sample into the pre-training model;

[0243] Introducing prompt tokens into the pre-trained model based on the training samples;

[0244] Guiding the pre-trained model to output shared features and proprietary features corresponding to the training samples through the prompt token;

[0245] Performing feature fusion processing on the shared features and the proprietary features to obtain fused features;

[0246] Based on the fusion features, generating a prediction label corresponding to the training sample;

[0247] Determine a first loss based on the true label and the predicted label;

[0248] Determine a second loss based on the shared features and the original features of the model; wherein the original features of the model are output features of the pre-trained model corresponding to the training sample before the pre-trained model is trained;

[0249] Based on the first loss and the second loss, the pre-training model is trained to obtain a downstream task model suitable for the at least one downstream task.

[0250] Alternatively, the processor 501 may be configured to perform the following operations:

[0251] Get task data of downstream tasks to be processed.

[0252] The task data is processed using a downstream task model trained using a downstream task model training method as provided in one or more embodiments of this specification to obtain a processing result.

[0253] In another possible implementation, the training device 300 of the downstream task model or the downstream task processing device 400 can be configured separately from the processor 501. For example, the training device 300 of the downstream task model or the downstream task processing device 400 can be configured as a chip connected to the processor 501, and the training of the downstream task model or the downstream task processing can be implemented through the control of the processor 501.

[0254] In addition, in some optional implementations, the electronic device 500 may also include: a communication module, an input unit, an audio processor, a display, a power supply, etc. It is worth noting that the electronic device 500 does not necessarily have to include Figure 8 In addition, the electronic device 500 may also include Figure 8 For components not shown, reference may be made to the prior art.

[0255] In some optional implementations, the processor 501 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 500.

[0256] The memory 502 may be, for example, a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The memory 502 may store the information related to the training device 300 or the downstream task processing device 400 of the downstream task model, and may also store a program for executing the related information. The processor 501 may execute the program stored in the memory 502 to implement information storage or processing.

[0257] The input unit can provide input to the processor 501. The input unit is, for example, a key or a touch input device. The power supply can be used to provide power to the electronic device 500. The display can be used to display display objects such as images and text. The display can be, for example, an LCD display, but is not limited thereto.

[0258] The memory 502 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 502 may also be some other type of device. The memory 502 includes a buffer memory (sometimes referred to as a buffer). The memory 502 may include an application / function storage unit for storing application programs and function programs or processes for executing the operation of the electronic device 500 through the processor 501.

[0259] The memory 502 may also include a data storage unit for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit of the memory 502 may include various drivers for the communication function of the computer device and / or for executing other functions of the computer device (such as a messaging application, a contact book application, etc.).

[0260] The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor 501 to provide input signals and receive output signals, which may be the same as in a conventional mobile communication terminal.

[0261] Based on different communication technologies, multiple communication modules may be provided in the same computer device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to the processor 501, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0262] One or more embodiments of the present specification also provide a computer-readable storage medium capable of implementing all steps in the training method of the downstream task model or the downstream task processing method in the above embodiments, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the downstream task model or the downstream task processing method in the above embodiments is implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0263] Acquire a training data set, wherein the training data set includes training data corresponding to at least one downstream task, and the training data corresponding to the downstream task includes multiple training samples and true labels corresponding to each of the training samples;

[0264] Inputting the training sample into the pre-training model;

[0265] Introducing prompt tokens into the pre-trained model based on the training samples;

[0266] Guiding the pre-trained model to output shared features and proprietary features corresponding to the training samples through the prompt token;

[0267] Performing feature fusion processing on the shared features and the proprietary features to obtain fused features;

[0268] Based on the fusion features, generating a prediction label corresponding to the training sample;

[0269] Determine a first loss based on the true label and the predicted label;

[0270] Determine a second loss based on the shared features and the original features of the model; wherein the original features of the model are output features of the pre-trained model corresponding to the training sample before the pre-trained model is trained;

[0271] Based on the first loss and the second loss, the pre-training model is trained to obtain a downstream task model suitable for the at least one downstream task.

[0272] Alternatively, the processor implements the following steps when executing the computer program:

[0273] Get task data of downstream tasks to be processed.

[0274] The task data is processed using a downstream task model trained using a downstream task model training method as provided in one or more embodiments of this specification to obtain a processing result.

[0275] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0276] Those skilled in the art should understand that the embodiments of this specification may be provided as methods, devices (systems) or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0277] One or more embodiments of this specification are described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to one or more embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0278] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0279] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0280] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0281] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meaning of the above terms in one or more embodiments of this specification can be understood according to the specific circumstances.

[0282] It should be noted that, in the absence of conflict, one or more embodiments of this specification and the features in the embodiments may be combined with each other. One or more embodiments of this specification are not limited to any single aspect, nor to any single embodiment, nor to any combination and / or replacement of these aspects and / or embodiments. Moreover, each aspect and / or embodiment of one or more embodiments of this specification may be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0283] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of one or more embodiments of this specification, rather than to limit them. Although one or more embodiments of this specification have been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of one or more embodiments of this specification, and they should all be included in the scope of the claims and the specification of one or more embodiments of this specification.

[0284] One or more embodiments of this specification are described above in conjunction with optional implementation methods, but these implementation methods are only exemplary and serve only as an illustration. On this basis, multiple replacements and improvements can be made to one or more embodiments of this specification, all of which fall within the scope of protection of one or more embodiments of this specification.

Claims

1. A training method for a downstream task model, characterized in that: The method comprises: Acquire a training data set, wherein the training data set includes training data corresponding to at least one downstream task, wherein the training data corresponding to the downstream task includes multiple training samples and true labels corresponding to each of the training samples; the downstream task includes at least one of the following: visual question answering, image description, image-text retrieval, image- and text-based classification tasks, and text summarization tasks; Inputting the training samples into the pre-training model; Introducing prompt tokens into the pre-trained model based on the training samples; Guiding the pre-trained model to output shared features and proprietary features corresponding to the training samples through the prompt token; Performing feature fusion processing on the shared features and the proprietary features to obtain fused features; Based on the fusion features, generating a prediction label corresponding to the training sample; Determine a first loss based on the true label and the predicted label; Determine a second loss based on the shared features and the original features of the model; wherein the original features of the model are output features of the pre-trained model corresponding to the training sample before the pre-trained model is trained; Based on the first loss and the second loss, the pre-training model is trained to obtain a downstream task model suitable for the at least one downstream task.

2. The method according to claim 1, characterized in that The pre-trained model includes a first encoder, the first encoder includes at least one feature learning layer, each of the feature learning layers includes a first freezing module and a first prompt token generation module; the introducing the prompt token into the pre-trained model based on the training sample includes: The first prompt word tokens corresponding to each of the feature learning layers are generated respectively by using the first prompt word token generation modules based on the input features of the corresponding feature learning layer; wherein the input features of the first feature learning layer are the features generated based on the training samples, and the input features of the i+1th feature learning layer are the output features corresponding to the first frozen module in the i-th feature learning layer; wherein i is a positive integer greater than 0.

3. The method according to claim 2, characterized in that The first prompt word token generation module includes a convolution layer, a shared knowledge learning matrix, and a proprietary knowledge learning matrix corresponding to each of the downstream tasks; the method for generating the first prompt word token by the first prompt word token generation module in the i-th feature learning layer based on the input features of the corresponding feature learning layer includes: Using the convolutional layer to perform feature extraction processing on the input features of the i-th feature learning layer to obtain global features; Determining a target task corresponding to the training sample; Determining a target-specific knowledge learning matrix corresponding to the target task; Using the target-specific knowledge learning matrix to perform feature extraction on the global feature to obtain a first feature; Using the shared knowledge learning matrix to extract features from the global features to obtain a second feature; Generate basic prompt tokens based on the proprietary knowledge learning matrix and the shared knowledge learning matrix corresponding to each of the downstream tasks; Based on the basic prompt word token, the global feature, the first feature and the second feature, a first prompt word token corresponding to the i-th feature learning layer is generated.

4. The method according to claim 3, characterized in that The target-specific knowledge learning matrix includes first network parameters, and the shared knowledge learning matrix includes second network parameters; and the pre-training model is trained based on the first loss and the second loss, including: Based on the first loss and the second loss, the first network parameter and the second network parameter in the pre-trained model are optimized.

5. The method according to any one of claims 2 to 4, characterized in that The first encoder is an image encoder or a text encoder.

6. The method according to claim 2, characterized in that The training sample is an image sample, the pre-training model further includes a second encoder, the first encoder is an image encoder, the second encoder is a text encoder, the second encoder includes at least one text feature learning layer, each of the text feature learning layers includes a second freezing module and a second prompt token generation module; the introducing the prompt token into the pre-training model based on the training sample also includes: Determining a target task corresponding to the image sample; Determining at least one target true label included in the target task; Processing each of the target true labels into corresponding description texts; Processing each of the description texts into corresponding text tokens; The second prompt word tokens corresponding to each of the text feature learning layers are generated by using each of the second prompt word token generation modules respectively based on the input features of the corresponding text feature learning layer; wherein the input features of the first text feature learning layer are the features generated based on the text word tokens, and the input features of the i+1th text feature learning layer are the output features corresponding to the second frozen module in the i-th text feature learning layer; wherein i is a positive integer greater than 0.

7. The method according to claim 6, characterized in that The step of guiding the pre-trained model to output shared features and specific features corresponding to the training samples through the prompt token includes: Guiding the first encoder to output a first shared feature and a first unique feature through the first prompt word; Guiding the second encoder to output second shared features corresponding to each of the target true labels and second unique features corresponding to each of the target true labels through the second prompt word; The shared features include the first shared features and second shared features corresponding to the target real labels respectively, and the unique features include the first unique features and second unique features corresponding to the target real labels respectively.

8. The method according to claim 7, characterized in that The performing feature fusion processing on the shared features and the specific features of the target task to obtain fused features includes: fusing the first shared feature and the first specific feature to obtain a first fused feature; fusing each of the second shared features and the second specific features corresponding to each of the second shared features to obtain a plurality of second fused features; The fusion feature includes the first fusion feature and the plurality of second fusion features.

9. The method according to claim 8, characterized in that The generating a prediction label corresponding to the training sample based on the fusion feature includes: Determine the similarity values ​​respectively corresponding to the first fused feature and each of the second fused features; Based on the similarity values, determining a maximum similarity value; The true label corresponding to the maximum similarity value is determined as the predicted label.

10. The method according to claim 7, characterized in that The determining of the second loss based on the shared features and the original features of the model includes: Determine the image loss based on the first shared feature and the original image feature; wherein the original image feature is the output feature of the first encoder corresponding to the training sample before training the pre-trained model; Determine the text loss based on the second shared features and the original features of the text; wherein the original features of the text are the output features of the second encoder corresponding to the training samples before training the pre-training model; The second loss is determined based on the image loss and the text loss.

11. A downstream task processing method, characterized in that: The method comprises: Get task data of downstream tasks to be processed; The task data is processed using the downstream task model obtained by the method according to any one of claims 1 to 10 to obtain a processing result.

12. A training device for a downstream task model, characterized in that: The device comprises: A first acquisition module is used to acquire a training data set, wherein the training data set includes training data corresponding to at least one downstream task, and the training data corresponding to the downstream task includes multiple training samples and true labels corresponding to each of the training samples; the downstream task includes at least one of the following: visual question answering, image description, image-text retrieval, image- and text-based classification tasks, and text summarization tasks; An input module, used for inputting the training samples into the pre-training model; An introduction module, used for introducing prompt tokens into the pre-trained model based on the training samples; An output module, used to guide the pre-trained model to output shared features and specific features corresponding to the training samples through the prompt token; A fusion module, used for performing feature fusion processing on the shared features and the proprietary features to obtain fused features; A prediction module, used to generate a prediction label corresponding to the training sample based on the fusion feature; A first loss determination module, configured to determine a first loss based on the true label and the predicted label; A second loss determination module, configured to determine a second loss based on the shared features and the original features of the model; wherein the original features of the model are output features of the pre-trained model corresponding to the training samples before the pre-trained model is trained; A training module is used to train the pre-training model based on the first loss and the second loss to obtain a downstream task model suitable for the at least one downstream task.

13. The device according to claim 12, characterized in that The pre-trained model includes a first encoder, the first encoder includes at least one feature learning layer, each of the feature learning layers includes a first freezing module and a first prompt token generation module; the introduction module is used to introduce the prompt token into the pre-trained model based on the training sample, specifically: The introduction module is used to generate first prompt word tokens corresponding to each of the feature learning layers respectively by using each of the first prompt word token generation modules based on input features of the corresponding feature learning layer; in, The input features of the first feature learning layer are features generated based on the training samples, and the input features of the i+1th feature learning layer are output features corresponding to the first frozen module in the i-th feature learning layer; wherein i is a positive integer greater than 0.

14. The device according to claim 13, characterized in that The first prompt word token generation module includes a convolution layer, a shared knowledge learning matrix, and a proprietary knowledge learning matrix corresponding to each of the downstream tasks; the introduction module is used for the first prompt word token generation module in the i-th feature learning layer to generate the first prompt word token based on the input features of the corresponding feature learning layer, specifically: The introduction module is used to use the convolution layer to perform feature extraction processing on the input features of the i-th feature learning layer to obtain global features; Determine the target task corresponding to the training sample; determine the target-specific knowledge learning matrix corresponding to the target task; extract the global feature using the target-specific knowledge learning matrix to obtain a first feature; The shared knowledge learning matrix is ​​used to extract features from the global features to obtain a second feature; basic prompt word symbols are generated based on the proprietary knowledge learning matrices and the shared knowledge learning matrix corresponding to each of the downstream tasks; and a first prompt word symbol corresponding to the i-th feature learning layer is generated based on the basic prompt word symbol, the global features, the first feature and the second feature.

15. The device according to claim 14, characterized in that The target-specific knowledge learning matrix includes first network parameters, and the shared knowledge learning matrix includes second network parameters; the training module is used to train the pre-training model based on the first loss and the second loss, specifically: The training module is used to optimize the first network parameter and the second network parameter in the pre-trained model based on the first loss and the second loss.

16. The device according to claim 13, characterized in that The training sample is an image sample, the pre-training model further includes a second encoder, the first encoder is an image encoder, the second encoder is a text encoder, the second encoder includes at least one text feature learning layer, each of the text feature learning layers includes a second freezing module and a second prompt token generation module; the introduction module is used to introduce the prompt token into the pre-training model based on the training sample, and is also used to: The introduction module is used to determine the target task corresponding to the image sample; determine at least one target real label included in the target task; process each of the target real labels into a corresponding description text; process each of the description texts into a corresponding text token; respectively use each of the second prompt token generation modules to generate second prompt tokens corresponding to each of the text feature learning layers based on the input features of the corresponding text feature learning layer; wherein the input features of the first text feature learning layer are features generated based on the text tokens, and the input features of the i+1th text feature learning layer are output features corresponding to the second frozen module in the i-th text feature learning layer; wherein i is a positive integer greater than 0.

17. The device according to claim 16, characterized in that The output module is used to guide the pre-trained model to output the shared features and the specific features corresponding to the training samples through the prompt token, specifically: The output module is used to guide the first encoder to output a first shared feature and a first specific feature through the first prompt word symbol; Guiding the second encoder to output second shared features corresponding to each of the target true labels and second unique features corresponding to each of the target true labels through the second prompt word; The shared features include the first shared features and second shared features corresponding to the target real labels respectively, and the unique features include the first unique features and second unique features corresponding to the target real labels respectively.

18. The device according to claim 17, characterized in that The fusion module is used to perform feature fusion processing on the shared feature and the proprietary feature to obtain the fusion feature, specifically: The fusion module is used to fuse the first shared feature and the first specific feature to obtain a first fused feature; and to fuse each of the second shared features and the second specific features corresponding to each of the second shared features to obtain a plurality of second fused features; The fusion feature includes the first fusion feature and the plurality of second fusion features.

19. The device according to claim 18, characterized in that The prediction module is used to generate prediction tags corresponding to the training sample based on the fusion feature, specifically: The prediction module is used to determine the similarity values ​​respectively corresponding to the first fusion feature and each of the second fusion features; determine the maximum similarity value based on each of the similarity values; and determine the true label corresponding to the maximum similarity value as the predicted label.

20. The device according to claim 17, characterized in that The second loss determination module is used to determine the second loss based on the shared feature and the original feature of the model, specifically: The second loss determination module is used to determine the image loss based on the first shared features and the original features of the image; wherein the original features of the image are the output features corresponding to the training samples of the first encoder before the pre-training model is trained; the text loss is determined based on the second shared features and the original features of the text; wherein the original features of the text are the output features corresponding to the training samples of the second encoder before the pre-training model is trained; the second loss is determined based on the image loss and the text loss.

21. A downstream task processing device, characterized in that: The device comprises: The second acquisition module is used to obtain task data of downstream tasks to be processed; The processing module is used to process the task data using the downstream task model obtained by any of claims 1 to 10 to obtain processing results.

22. An electronic device, characterized in that: The electronic device comprises: A memory for storing a computer program product; A processor is used to execute a computer program product stored in the memory, and when the computer program product is executed, it implements the method of any one of the above claims 1-10 or 11.

23. A computer-readable storage medium, characterized in that: Computer program instructions are stored on the computer readable storage medium, and when the computer program instructions are executed, the method according to any one of the above claims 1-10 or 11 is implemented.

Images