Multimodal training method, electronic device, and storage medium

Abstract

This application provides a multimodal training method, electronic device, and storage medium, relating to the field of artificial intelligence. The method includes: training a multimodal pre-trained model using real samples from a current training sample set; constructing pseudo-samples in the form of negative samples during training, and replaying the pseudo-samples to enable the multimodal pre-trained model to relearn based on the pseudo-samples. The method provided in this application helps reduce knowledge forgetting during multimodal training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a multimodal training method, electronic device, and storage medium. Background Technology

[0002] Vision-language pretraining is a multimodal training method based on images and text. Vision-language pretrained models can generalize to many different downstream tasks, such as image classification, image recall, and text recall. Therefore, vision-language pretrained models are attracting increasing attention.

[0003] In the process of training vision-language pre-trained models, the dataset is usually collected in advance before training. However, in real-world scenarios, data is often continuously collected. Therefore, existing algorithms are not good at rapidly and effectively iterating on new data, and knowledge is easily forgotten. Summary of the Invention

[0004] This application provides a multimodal training method, electronic device, and storage medium that help reduce the forgetting of knowledge during multimodal training.

[0005] In a first aspect, embodiments of this application provide a multimodal training method applied to a multimodal pre-trained model, comprising:

[0006] The multimodal pre-trained model is trained using real samples from this batch of training samples;

[0007] During training, pseudo-samples are constructed in the form of negative samples and replayed so that the multimodal pre-trained model can relearn based on the pseudo-samples.

[0008] In this embodiment, pseudo-samples are constructed in the form of negative samples and then replayed. This can help the multimodal pre-trained model to enhance the playback of historical knowledge during continuous training, thereby helping to reduce the forgetting of knowledge.

[0009] In one possible implementation, the multimodal pre-trained model is a vision-language pre-trained model, where real samples are a combination of real images and real text, and pseudo samples are a combination of pseudo text and real images, or a combination of real text and pseudo images, or a combination of pseudo text and pseudo images.

[0010] In one possible implementation, the pseudo-sample is a combination of pseudo-text and real images. The pseudo-sample is constructed in the form of negative samples, including:

[0011] Obtain the first image and the first text. The first image is any real image in the current batch of training samples, and the first text is the initialization pseudo-text.

[0012] The second text encoder extracts features from the first text to obtain the second text features. The second text encoder is the text encoder of the previous time step adjacent to the first text encoder. The first text encoder is used to extract features from the real text in the current batch of training samples to obtain the first text features.

[0013] Based on the second image features and the second text features, pseudo-text is constructed in the form of negative samples. The combination of pseudo-text and the first image is used as a pseudo-sample. The second image features include features extracted from each real image in the current batch of training samples by the second image encoder. The second image encoder is the image encoder of the previous time step adjacent to the first image encoder. The first image encoder is used to extract features from the real images in the current batch of training samples to obtain the first image features.

[0014] In this embodiment of the application, feature extraction is performed through a historical encoder, thereby constructing pseudo-samples based on historical knowledge, which is helpful for learning historical knowledge.

[0015] One possible implementation involves constructing pseudo-text based on second image features and second text features, using negative samples:

[0016] Similarity is calculated based on the second image features and the second text features;

[0017] The first loss is calculated based on the first loss function and similarity.

[0018] The first text is updated based on the first loss to construct pseudo-text in the form of negative samples.

[0019] In this embodiment of the application, pseudo-text is constructed by using a first loss function, which can generate high-quality pseudo-text and thus enable the learning of more historical knowledge.

[0020] In one possible implementation, the similarity between the pseudo-text in the pseudo-sample and the first image is higher than a preset similarity threshold.

[0021] In this embodiment of the application, by constructing negative samples with high similarity, the complexity of learning is increased, thereby enabling the learning of more historical knowledge.

[0022] In one possible implementation, after replaying the pseudo-sample, the method further includes:

[0023] Learn knowledge based on pseudo-samples;

[0024] Knowledge distillation based on pseudo-samples.

[0025] In this embodiment of the application, knowledge distillation can both educate current knowledge and preserve historical knowledge, thereby helping to avoid forgetting historical knowledge.

[0026] In one possible implementation, the pseudo-samples are a combination of pseudo-text and real images, and the knowledge learned based on the pseudo-samples includes:

[0027] The second loss is calculated based on the second loss function, where the second loss is used to characterize the learning of matching relationships based on pseudo-samples. The second loss includes the average loss of text recall and the average loss of image recall.

[0028] In this embodiment of the application, the current knowledge can be effectively learned through the second loss function.

[0029] In one possible implementation, the pseudo-sample is a combination of pseudo-text and real images, and knowledge distillation based on the pseudo-sample includes:

[0030] The third loss is calculated based on the third loss function, where the third loss is used to characterize knowledge distillation based on pseudo-samples. The third loss includes the loss between the multimodal pre-trained model at the current time step and the multimodal pre-trained model at the previous time step.

[0031] In this embodiment of the application, the third loss function can be used to effectively learn historical knowledge, thereby helping to avoid forgetting historical knowledge.

[0032] In one possible implementation, the third loss includes the loss between the current multimodal pre-trained model and the previous multimodal pre-trained model based on the first dimension, and the loss between the current multimodal pre-trained model and the previous multimodal pre-trained model based on the second dimension. The first dimension is used to represent the dimension from image to text, and the second dimension is used to represent the dimension from text to image.

[0033] In this embodiment, by considering the bidirectional loss between text-to-image and image-to-text, the above-mentioned bidirectional loss can be integrated, thereby enabling better learning of historical matching knowledge between text and images.

[0034] Secondly, embodiments of this application provide a multimodal training apparatus, including one or more functional modules, which are used to implement the multimodal training method as described in the first aspect.

[0035] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, wherein the memory is used to store a computer program; and the processor is used to run the computer program to implement the multimodal training method as described in the first aspect.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to implement the multimodal training method as described in the first aspect. Attached Figure Description

[0037] Figure 1 A flowchart illustrating an embodiment of the multimodal training method provided in this application;

[0038] Figure 2 A flowchart illustrating another embodiment of the multimodal training method provided in this application;

[0039] Figure 3 A flowchart illustrating another embodiment of the multimodal training method provided in this application;

[0040] Figure 4 A schematic diagram illustrating the framework of the multimodal training method provided in this application embodiment;

[0041] Figure 5 This is a schematic diagram of the structure of the multimodal training device provided in the embodiments of this application;

[0042] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] In this embodiment of the application, unless otherwise stated, the character " / " indicates that the preceding and following objects are in an OR relationship. For example, A / B can represent A or B. "AND / OR" describes the relationship between the associated objects, indicating that three relationships can exist. For example, A AND / OR B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0044] It should be noted that the terms "first" and "second" used in the embodiments of this application are used only for distinguishing descriptive purposes and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor should they be construed as indicating or implying order.

[0045] In the embodiments of this application, "at least one" refers to one or more items, and "more than one" refers to two or more items. Furthermore, "at least one of the following" or similar expressions refer to any combination of these items, which may include any combination of a single item or a plurality of items. For example, at least one of A, B, or C can represent: A, B, C, A and B, A and C, B and C, or A, B, and C. Each of A, B, and C can be an element itself or a set containing one or more elements.

[0046] In this application, terms such as "exemplary," "in some embodiments," and "in another embodiment" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0047] In the embodiments of this application, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction, their meanings are consistent. Similarly, in the embodiments of this application, "communication" and "transmission" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction, their meanings are consistent. For example, transmission can include sending and / or receiving, and can be a noun or a verb.

[0048] In the embodiments of this application, the term "equal to" can be used in conjunction with "greater than" to apply to technical solutions employing the condition of "greater than", and can also be used in conjunction with "less than" to apply to technical solutions employing the condition of "less than". It should be noted that when "equal to" is used with "greater than", it cannot be used with "less than"; and when "equal to" is used with "less than", it cannot be used with "greater than".

[0049] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0050] 1. Visual-Language Pre-trained Model. In this application, the visual-language pre-trained model refers to a model trained on a large-scale image-text dataset. This model can generalize to downstream visual-language tasks, such as visual-language question answering and image-text retrieval. The visual-language pre-trained model can also be called a language-visual pre-trained model.

[0051] 2. Catastrophic Forgetting. In the embodiments of this application, catastrophic forgetting refers to the catastrophic and significant forgetting of old knowledge that occurs when the model sequentially learns from the data stream.

[0052] 3. Token: In the embodiments of this application, a token refers to the most basic unit used to represent text in natural language processing.

[0053] 4. Encoder: The encoder in this embodiment is used to extract features from the input. The input format may differ for different encoders; for example, the input of an image encoder is an image, and the input of a text encoder is text.

[0054] 5. Knowledge Distillation: In the embodiments of this application, knowledge distillation refers to transferring knowledge from one model to another.

[0055] 6. Stability-plasticity trade-off: In the embodiments of this application, stability-plasticity trade-off refers to the fact that the model needs to be plastic enough to absorb new knowledge, while also needing to be stable enough to avoid forgetting previous knowledge. However, these two are in conflict with each other.

[0056] Existing visual-language pre-trained models typically assume that the training environment is static, meaning that the dataset is collected only once and the visual-language pre-trained model only needs to be trained once. Therefore, their design does not consider how to continuously update the data to cope with the constantly collected new data, which can cause the visual-language pre-trained model to forget its knowledge during continuous training.

[0057] To address the aforementioned issues, this application proposes a multimodal training method. This method constructs pseudo-samples using negative samples and replays these pseudo-samples. This helps the multimodal pre-trained model enhance its recall of historical knowledge during continuous training, thereby reducing knowledge forgetting. It is understood that multimodal training can be image-and-text based, video-and-text based, or other forms of multimodal training; this application does not impose any specific limitations on these methods. This document uses an image-and-text-based visual-language pre-trained model as an example, but this does not constitute a limitation on the embodiments of this application. In some embodiments, the multimodal training method of this application is also applicable to other forms of multimodal pre-trained models.

[0058] Now combined Figures 1-4 The multimodal training method provided in the embodiments of this application will be described.

[0059] Figure 1 This is a flowchart illustrating an embodiment of the multimodal training method provided in this application. Figure 1 The illustrated embodiment is applied to a vision-language pre-trained model and specifically includes the following steps:

[0060] Step 101: Train the visual-language pre-trained model using real samples from this batch of training samples.

[0061] Specifically, a visual-language pre-trained model can be trained using multiple batches of training sample sets. These batches are obtained by grouping the training dataset. Understandably, the current batch of training sample sets can be used for the current training, and the previous batch can be used for the previous training. After each training run using these sample sets, the model's parameters and encoder can be updated, allowing it to learn knowledge based on the data in the training sample sets.

[0062] In this context, a real sample can be a sample from the training sample set. The training sample set can be a collection containing multiple training samples. Taking a vision-language pre-trained model as an example, the training sample set can contain a collection of multiple image-text combinations. That is, an image-text combination can be considered a training sample, or a real sample. A real sample contains a real image and a real text; in other words, a real sample can be a combination of a real image and a real text.

[0063] By training the visual-language pre-trained model using the aforementioned real samples, the parameters in the visual-language pre-trained model can be trained. This allows the model to learn knowledge from real samples during the training process, such as the matching relationship between images and text, thereby optimizing the visual-language pre-trained model.

[0064] Step 102: During the training process, pseudo-samples are constructed in the form of negative samples. The pseudo-samples are then replayed so that the visual-language pre-trained model can relearn based on the pseudo-samples.

[0065] Specifically, if a vision-language pre-trained model is trained using only real samples from the training sample set, the uncertainty inherent in these pre-collected samples may lead to forgetting of past knowledge if the same information is learned over a long period. Therefore, pseudo-samples can be constructed during training. These pseudo-samples can be generated based on real samples.

[0066] In some optional embodiments, the pseudo-sample can be a combination of pseudo-text and real image, wherein the pseudo-text can be generated based on the features of the real text and the features of the real image.

[0067] Alternatively, a pseudo-sample can be a combination of real text and pseudo-images. In the combination of real text and pseudo-images, the pseudo-image can be generated based on the features of the real text and the features of the real image.

[0068] Alternatively, a pseudo-sample can be a combination of pseudo-text and pseudo-image. In the combination of pseudo-text and pseudo-image, the pseudo-image and pseudo-text can be generated based on the features of real text and real image.

[0069] Understandably, to enhance the visual-language pre-trained model's retention of historical knowledge and prevent knowledge forgetting, pseudo-samples can be generated as negative samples. Simultaneously, these pseudo-samples exhibit high similarity between the pseudo-text and the real image; for example, the similarity can exceed a preset similarity threshold. It is understandable that more knowledge is learned from pseudo-samples with high similarity than from pseudo-samples with low similarity, thereby strengthening the visual-language pre-trained model's learning of knowledge from pseudo-samples and effectively preventing knowledge forgetting.

[0070] Next, combined Figure 2 The example is illustrated using a combination of pseudo-sample text and real images. Figure 2 As shown, the method of constructing pseudo-text in the form of negative samples as described above specifically includes the following steps:

[0071] Step 201: Obtain the first image and the first text. The first image is any real image in the current batch of training samples, and the first text is the initialization pseudo-text.

[0072] Specifically, we can first select any real image from the training sample set. For ease of explanation, we will refer to the randomly selected real image as the first image. We can also initialize a text. For ease of explanation, we will refer to the initialized text as the first text. It can be understood that the first text initialized here is a newly created text.

[0073] Step 202: Extract features from the first text using the second text encoder to obtain the second text features. The second text encoder is the text encoder of the previous time step adjacent to the first text encoder. The first text encoder is used to extract features from the real text in the current batch of training samples to obtain the first text features.

[0074] Specifically, after acquiring the first image and the first text, features can be extracted from the first text using a second text encoder to obtain features for the second text. It is understandable that before feature extraction from the first text, word segmentation can be performed on the first text to obtain words, thereby enabling the second text encoder to extract features from the first text based on these words.

[0075] The second text encoder is the text encoder of the previous time step adjacent to the first text encoder. The second text encoder is used to extract features from the real text in the previous batch of training sample set so as to train based on the previous batch of training sample set; the first text encoder is used to extract features from the real text in the current batch of training sample set so as to obtain the first text features.

[0076] It is understandable that the first text encoder is trained on the current batch of training samples, or it can be considered as trained on the training samples at the current time step. The second text encoder is trained on the previous batch of training samples, or it can be considered as trained on the training samples at the previous time step. Here, the current batch of training samples and the previous batch of training samples are different training samples. That is, after training on the previous time step or the previous batch of training samples, the second text encoder can be updated, thus obtaining the first text encoder for training on the current time step or the current batch of training samples. Let g be the first text encoder for training on the current time step t or the current batch of training samples. t For example, the second image encoder used for training at the previous time step t-1 or the previous batch of training samples is g. t-1 This allows us to extract the knowledge implicit in the encoder of the previous time step into the pseudo-text, and then train it using the encoder of the current time step, thereby enhancing the learning and memorization of historical knowledge.

[0077] Step 203: Based on the second image features and the second text features, construct pseudo-text in the form of negative samples, and use the combination of pseudo-text and the first image as pseudo-samples. The second image features include features extracted from each real image in the current batch of training samples by the second image encoder. The second image encoder is the image encoder of the previous time step adjacent to the first image encoder. The first image encoder is used to extract features from the real images in the current batch of training samples to obtain the first image features.

[0078] Specifically, after obtaining the second text features, the second image features can also be obtained, and the similarity between the image and the text can be calculated based on the second image features and the second text features. For example, the specific way to calculate the similarity can be by calculating the similarity through cosine distance, but it is not limited to this. The similarity can also be calculated in other ways, such as Euclidean distance. This application embodiment does not make any special limitation on this.

[0079] Specifically, the second image features can be obtained by extracting features from each real image in the current batch of training samples using a second image encoder. In other words, the second image features include the features extracted from each real image in the current batch of training samples using the second image encoder. The second image encoder is the image encoder from the previous time step adjacent to the first image encoder. The first image encoder is used to extract features from the real images in the current batch of training samples to obtain the first image features. It can be understood that the aforementioned first image features and first text features can be used to learn the matching relationship between real images and real text.

[0080] The first image encoder is trained on the current batch of training samples, or it can be considered as an image encoder trained on the training samples at the current time step. The second image encoder is trained on the previous batch of training samples, or it can be considered as an image encoder trained on the training samples at the previous time step. Here, the current batch of training samples and the previous batch of training samples are different training samples. That is, after training based on the previous time step or the previous batch of training samples, the second image encoder can be updated, thus obtaining the first image encoder for training at the current time step or the current batch of training samples. Let f be the first image encoder for training at the current time step t or the current batch of training samples. t For example, the second image encoder used for training at the previous time step t-1 or the previous batch of training samples is f. t-1 .

[0081] Since the second image features include features extracted from each real image in the current batch of training samples by the second image encoder, the similarity between the second text features and the second image features includes the similarity between the features extracted from each real image and the second text features.

[0082] Next, the first loss can be calculated based on the first loss function and the calculated similarity. The first text can then be updated using gradient descent until the first loss converges. For example, after calculating the first loss, if the first loss is greater than the preset target loss, the first text can be updated. For instance, the content of the first text can be changed to update the first text. After feature extraction, the updated first text is then compared with the features of the second image to calculate the first loss again. This process is repeated until the first loss is less than or equal to the preset target loss, meaning the first loss has converged. At this point, the updated first text can be used as pseudo-text, and the combination of this pseudo-text and the first image can be used as a pseudo-sample.

[0083] Understandably, previously learned knowledge is stored in the model's parameters. The generation and replay of pseudo-samples can be used to extract knowledge implicit in the model (e.g., the encoder). By transforming this hidden knowledge into training samples, knowledge can resist forgetting. Therefore, to extract historical knowledge implicit in the image encoder and text encoder, the aforementioned pseudo-samples can be constructed in the form of negative samples. Taking a combination of a real image and pseudo-text as an example, the pseudo-text can be constructed as a negative sample, and the similarity between the pseudo-text and the real image is high. That is, during the update of the first text, text with high similarity to the first image can be selected. By constructing highly similar negative samples, the historical knowledge implicit in the image encoder and text encoder can be better extracted and used for training.

[0084] It should be noted that the above examples only illustrate the combination of pseudo-samples and real images, but do not constitute a limitation on the embodiments of this application. The specific implementation of obtaining pseudo-images in the combination of pseudo-samples and real images, as well as the specific implementation of obtaining pseudo-text and pseudo-images in the combination of pseudo-samples and pseudo-samples, can refer to the above-described method of obtaining pseudo-text, and will not be repeated here.

[0085] Next, the pseudo-samples can be replayed so that the visual-language pre-trained model can relearn based on the pseudo-samples.

[0086] In this embodiment, pseudo-samples are constructed in the form of negative samples and then replayed. This can help the multimodal pre-trained model to enhance the playback of historical knowledge during continuous training, thereby helping to reduce the forgetting of knowledge.

[0087] The above text passed Figure 1 and Figure 2 An illustrative example is provided of how to construct pseudo-samples for relearning. To learn knowledge from new data and simultaneously retain knowledge from both new and old data to avoid forgetting, comparative learning and knowledge distillation can also be performed. The following section will illustrate this further. Figure 3 The methods of comparative learning and knowledge distillation are illustrated by example.

[0088] Figure 3 This is a flowchart illustrating another embodiment of the multimodal training method provided in this application. Figure 3 The illustrated embodiment can be used for comparative learning and knowledge distillation. In step 102, after replaying the pseudo-sample, the following steps may also be included:

[0089] Step 301: Learn knowledge based on pseudo-samples.

[0090] Specifically, knowledge learning based on pseudo-samples can be achieved by comparing and learning the knowledge in the samples and pseudo-samples of the current training sample set. Taking a multimodal pre-trained model as a vision-language pre-trained model as an example, knowledge learning based on pseudo-samples can be achieved by comparing and learning the matching relationship between images and text in the samples of the current training sample set, and comparing and learning the matching relationship between images and text in pseudo-samples.

[0091] After replaying the pseudo-samples, a second loss function can be used to learn knowledge based on the pseudo-samples through comparison. This second loss function is used to learn the alignment between the image and text. Taking a combination of pseudo-text and real images as an example, the second loss function can be used to calculate the second loss, which represents the comparative learning of the matching relationship based on the pseudo-samples. The second loss includes the average loss for text recall and the average loss for image recall. It can be understood that the text recall loss refers to the recall loss for all texts in a given image, where all texts include both real and pseudo-texts from the training sample set. The image recall loss refers to the recall loss for all images in a given text, where the given text includes both real and pseudo-texts from the training sample set. Therefore, the average loss for text recall can be the mean of the text recall losses for all images, and the average loss for image recall can be the mean of the image recall losses for all texts.

[0092] Understandably, since the relationship between images and text is symmetrical, incorporating these two relationships (e.g., image-to-text and text-to-image) into the second loss function allows the visual-language pre-trained model to better learn the matching relationships under these two relationships. For example, image recall can determine which image best matches the text, or text recall can determine which text best matches the image. This allows for a comprehensive analysis of the losses under these two relationships, leading to more accurate learning of knowledge under these two relationships.

[0093] Step 302: Perform knowledge distillation based on pseudo-samples.

[0094] Specifically, the execution order of step 302 and step 301 is not important. For example, this step can be executed before step 301, after step 301, or simultaneously with step 301. This application embodiment does not impose any special limitations on this.

[0095] The specific implementation of the above knowledge distillation can be as follows: the third loss is calculated through the third loss function, where the third loss function is used to ensure that the visual-language pre-trained model does not forget the old image-text matching relationship while learning the new image-text matching relationship. The third loss is used to characterize knowledge distillation based on pseudo-samples. The third loss includes the loss between the visual-language pre-trained model at the current time and the visual-language pre-trained model at the previous time.

[0096] Understandably, the aforementioned third loss function reduces knowledge forgetting by requiring the current visual-language pre-trained model to maintain consistency with the previous visual-language pre-trained model's predictions of the image-text relationship. Since the image-text relationship is symmetrical, the third loss can also calculate a two-dimensional loss, where the two dimensions can include a first dimension and a second dimension. The first dimension ensures the image-to-text dimension, and the second dimension represents the text-to-image dimension. Therefore, the third loss can include the loss between the current and previous visual-language pre-trained models based on the first dimension, and the loss between the current and previous visual-language pre-trained models based on the second dimension.

[0097] In this embodiment, knowledge based on pseudo-text is learned through a second loss function and knowledge based on pseudo-text is distilled through a third loss function. This allows for the learning of new knowledge based on pseudo-text while retaining both new and old knowledge based on pseudo-text, thus achieving a better balance between stability and plasticity.

[0098] Next, taking a multimodal pre-trained model as a vision-language pre-trained model, and pseudo-samples as a combination of real images and pseudo-text as an example, and combining... Figure 4 Further illustrative examples are provided below.

[0099] First, a training dataset can be selected. For example, 2 million image-text combinations can be randomly selected from the Conceptual Caption 12M dataset as the training dataset, where each image-text combination includes one image and one text. It is understood that other datasets can also be selected as the training dataset, and this embodiment of the application does not impose any special limitations on this.

[0100] To simulate the scenario of continuous pre-training of vision-language, the 2 million image-text combinations in the above training dataset can be randomly divided into multiple parts, for example, 8 parts, to obtain a training sample set. Each part contains 250,000 image-text combinations, and each part can be considered a training sample set. That is, each training sample set includes 250,000 image-text combinations, and each image-text combination is a sample.

[0101] Assuming the visual-language pre-trained model has been trained on the previous batch of data, it needs to be trained on the current batch of data, where a batch of data refers to the data in a training sample set. After training on the previous batch of data is complete, the text encoder and image encoder in the visual-language pre-trained model are updated. The text encoder trained on the previous batch of data can be the second text encoder g. t-1 The g t-1 This can also be referred to as the text encoder trained at the previous time step, while the image encoder trained on the previous batch of data can be called the second image encoder f. t-1 The f t-1 This can also be referred to as the image encoder trained at the previous time step. The text encoder trained on the current batch of data can be called the first text encoder g. t The g t This can also be referred to as the text encoder trained at the current moment, while the image encoder trained on this batch of data can be called the first image encoder f. t The f t It can also be called an image encoder trained at the current moment.

[0102] Next, refer to Figure 4 When training on this batch of data, a first image encoder can be used to extract features from the images in this batch of data to obtain first image features u; alternatively, a first text encoder can be used to extract features from the text in this batch of data to obtain first text features v. Furthermore, pseudo-samples can be constructed during the training process of this batch of data. Taking the construction of a pseudo-sample as an example, an image (e.g., the first image) can be arbitrarily selected from this batch of data, and a pseudo-text (e.g., the first text) can be initialized. Then, a second image encoder can be used to extract features from each image in this batch of data to obtain second image features u', and a second text encoder can be used to extract features from the first text to obtain second text features v'.

[0103] After obtaining the second image feature u' and the second text feature v', the similarity s between each image and the first text can be calculated based on the second image feature u' and the second text feature v'. For example, the similarity can be calculated using cosine distance or Euclidean distance. This application embodiment does not specifically limit the specific method for calculating the similarity. Next, the first text can be updated using a first loss function and the similarity. This first loss function can also be called a text generation loss function. For example, the expression of the first loss function is as follows:

[0104]

[0105] Among them, L G The first loss is denoted by B, where B is the total number of images in the current training sample set, and s is the total number of images. min A preset first similarity threshold is used to characterize a preset minimum similarity, s max A second similarity threshold is preset to represent the maximum preset similarity.

[0106] The specific method for updating the first text can be: modifying the words in the first text to update it; this embodiment does not specifically limit this method. It is understood that, to enhance the memory of historical knowledge, the pseudo-text is a negative sample, and its similarity to the first image can be high. Therefore, the first text can be updated according to the above principles. After updating the first text, if the first loss does not meet the convergence condition, the above steps can be repeated, for example, updating the first text until the first loss converges; if the first loss meets the convergence condition after updating the first text, the updated first text can be used as pseudo-text, and the combination of pseudo-text and the first image can be used as pseudo-sample. Through the construction and replay of pseudo-samples, knowledge implicit in the model can be extracted, and the knowledge implicit in the model can be transformed into training data to achieve knowledge resistance to forgetting.

[0107] Then, after constructing the pseudo-samples, contrastive learning can be performed to learn knowledge from the new data. The new data includes both samples from the current dataset and the constructed pseudo-samples. Specifically, this learning can be achieved by calculating a second loss based on a second loss function. This second loss characterizes the learning of knowledge from the new data and includes the average loss for text recall and the average loss for image recall. The second loss function, also known as the contrastive learning loss function, is used to learn the alignment between images and text. For example, the expression for the second loss function is as follows:

[0108]

[0109] Among them, L cFor the second loss, L c I2T The average loss for text recall, L c I2T This refers to the average loss for text recall, L, for a given image. c T2I The average loss for image recall, L c T2I This refers to the average loss for image recall given text. α is a weighting coefficient used to control different directions (e.g., from text to image, and from image to text).

[0110] For example, the expression for the function used to calculate the average loss of text recall is as follows:

[0111]

[0112] Among them, P I2T Let B be the probability that each text matches a given image, and let B be the total number of images in the batch of data. i For the features of the i-th image, v j For the features of the j-th text, B T y represents the total number of text and pseudo-text data in this batch. i I2T For the text that best matches the given i-th image, τ is the temperature value in the normalization function, which can be the softmax function.

[0113] For example, the expression for the function used to calculate the average loss for image recall is as follows:

[0114]

[0115] Among them, P T2I Let y be the probability that each image matches a given text. i T2I Find the image that best matches the given i-th text.

[0116] The second loss function described above can be used to learn the alignment between images and text, including the alignment between images and text in the samples of this batch of data, as well as the alignment between images and text in pseudo-samples.

[0117] Furthermore, after constructing pseudo-samples, knowledge distillation can be performed to simultaneously retain new and old knowledge. New knowledge can be acquired from the current batch of data, while old knowledge can be acquired from the previous batch. Specifically, knowledge distillation can be performed by calculating a third loss function. This third loss function ensures that the visual-language pre-trained model learns new image-text matching relationships without forgetting old ones. The third loss includes the loss between the visual-language pre-trained model at time t and the visual-language pre-trained model at time t-1. By maintaining consistency between the image-text relationship predictions of the current and previous visual-language pre-trained models at time t-1, both new and old knowledge are retained, thus reducing the forgetting of historical knowledge. An example expression for the third loss function is as follows:

[0118]

[0119] Among them, L d I2T L represents the loss between the visual-language pre-trained model at the current time t based on the first dimension and the visual-language pre-trained model at the previous time t-1. d T2I Let η be the loss between the current time t and the previous time t-1, based on the second dimension. The first dimension represents the image-to-text dimension, and the second dimension represents the text-to-image dimension. η is a weighting coefficient used to control the loss for different dimensions. For example, the expression for the function used to calculate the loss based on the first dimension is as follows:

[0120]

[0121] Where KL is a metric function used to calculate two parameters (e.g., P). I2T and P T2I The distance between θ t Let τ be the model parameters at the current time t. t Let θ be the temperature magnitude in the normalized function at the current time t. t-1 Let τ be the model parameters at the previous time t-1. t-1 This represents the temperature magnitude in the normalized function of the previous time step t-1.

[0122] For example, the expression for the function used to calculate the loss based on the second dimension is as follows:

[0123]

[0124] By using the third loss function mentioned above, we can learn new knowledge without forgetting old knowledge.

[0125] Furthermore, the total loss can be calculated based on the second and third losses. For example, the expression for the function used to calculate the total loss is as follows:

[0126] L overall =L c +λL d .

[0127] Among them, L overall Let λ be the total loss, and λ be the weighting coefficients controlling the second and third losses.

[0128] By combining the second and third losses mentioned above, we can learn new knowledge while not forgetting old knowledge, thus achieving a better balance between stability and plasticity.

[0129] Finally, during the training of the visual-language pre-trained model, the parameters of the visual-language pre-trained model are continuously adjusted so that the total loss mentioned above converges. This completes the learning process described above, thereby strengthening the memory of knowledge learned based on pseudo-text and retaining the strengthened knowledge to avoid forgetting.

[0130] It is understood that the above description uses a visual-language pre-trained model as an example, but it does not constitute a limitation on the embodiments of this application. In some embodiments, the above multimodal training method is also applicable to other forms of multimodal pre-trained models. The specific implementation of the training method of other forms of multimodal pre-trained models can refer to the training method of the above visual-language pre-trained model, which will not be repeated here.

[0131] Figure 5 This is a schematic diagram of the structure of one embodiment of the multimodal training device of this application, as shown below. Figure 5 As shown, the multimodal training device 50 described above is applied to a multimodal pre-trained model and may include: a training module 51 and a replay module 52; wherein,

[0132] Training module 51 is used to train the multimodal pre-trained model using real samples from the current batch of training samples;

[0133] The replay module 52 is used to construct pseudo-samples in the form of negative samples during training and replay the pseudo-samples so that the multimodal pre-trained model can relearn based on the pseudo-samples.

[0134] In one possible implementation, the multimodal pre-trained model is a vision-language pre-trained model, where real samples are a combination of real images and real text, and pseudo samples are a combination of pseudo text and real images, or a combination of real text and pseudo images, or a combination of pseudo text and pseudo images.

[0135] In one possible implementation, the pseudo sample is a combination of pseudo text and real image. The replay module 52 is also used to acquire the first image and the first text. The first image is any real image in the current batch of training sample set, and the first text is the initialization pseudo text.

[0136] The second text encoder extracts features from the first text to obtain the second text features. The second text encoder is the text encoder of the previous time step adjacent to the first text encoder. The first text encoder is used to extract features from the real text in the current batch of training samples to obtain the first text features.

[0137] Based on the second image features and the second text features, pseudo-text is constructed in the form of negative samples. The combination of pseudo-text and the first image is used as a pseudo-sample. The second image features include features extracted from each real image in the current batch of training samples by the second image encoder. The second image encoder is the image encoder of the previous time step adjacent to the first image encoder. The first image encoder is used to extract features from the real images in the current batch of training samples to obtain the first image features.

[0138] In one possible implementation, the replay module 52 is further used to calculate similarity based on the second image features and the second text features;

[0139] The first loss is calculated based on the first loss function and similarity.

[0140] The first text is updated based on the first loss to construct pseudo-text in the form of negative samples.

[0141] In one possible implementation, the similarity between the pseudo-text in the pseudo-sample and the first image is higher than a preset similarity threshold.

[0142] In one possible implementation, the multimodal training device 50 further includes:

[0143] Learning module 53 is used to learn knowledge based on pseudo-samples;

[0144] The knowledge distillation module 54 is used for knowledge distillation based on pseudo-samples.

[0145] In one possible implementation, the pseudo-sample is a combination of pseudo-text and real images. The learning module 53 is specifically used to calculate the second loss based on the second loss function. The second loss is used to characterize the learning of the matching relationship based on the pseudo-sample. The second loss includes the average loss of text recall and the average loss of image recall.

[0146] In one possible implementation, the pseudo-sample is a combination of pseudo-text and real image. The knowledge distillation module 54 is specifically used to calculate the third loss based on the third loss function. The third loss is used to characterize the knowledge distillation based on the pseudo-sample. The third loss includes the loss between the multimodal pre-trained model at the current time and the multimodal pre-trained model at the previous time.

[0147] In one possible implementation, the third loss includes the loss between the current multimodal pre-trained model and the previous multimodal pre-trained model based on the first dimension, and the loss between the current multimodal pre-trained model and the previous multimodal pre-trained model based on the second dimension. The first dimension is used to represent the dimension from image to text, and the second dimension is used to represent the dimension from text to image.

[0148] Figure 6 This is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of this application. The electronic device 600 may include: at least one processor; and at least one memory communicatively connected to the processor. The electronic device 600 may be a computer or a server. The memory stores program instructions executable by the processor, which can invoke the program instructions to perform actions performed by the electronic device provided in this embodiment of the application.

[0149] like Figure 6 As shown, the electronic device 600 can be represented as a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: one or more processors 610, memory 620, communication bus 640 and communication interface 630 connecting different system components (including memory 620 and processor 610).

[0150] Communication bus 640 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0151] Electronic device 600 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 600, including volatile and non-volatile media, removable and non-removable media.

[0152] Memory 620 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 600 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Although Figure 6 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the communication bus 640 via one or more data media interfaces. The memory 620 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0153] A program / utility having a set (at least one) of program modules can be stored in memory 620. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0154] Electronic device 600 can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through communication interface 630. Furthermore, electronic device 600 can also communicate through a network adapter (… Figure 6 (Not shown) communicates with one or more networks (e.g., Local Area Network (LAN), Wide Area Network (WAN), and / or public networks, such as the Internet). The aforementioned network adapter can communicate with other modules of the electronic device via the communication bus 640. It should be understood that, although... Figure 6 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Drives (RAID) systems, tape drives, and data backup storage systems.

[0155] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0156] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal training method, characterized in that, The method, applied to multimodal pre-trained models, includes: The multimodal pre-trained model is trained using real samples from this batch of training samples; During training, pseudo-samples are constructed in the form of negative samples, and the pseudo-samples are replayed so that the multimodal pre-trained model can relearn based on the pseudo-samples. The multimodal pre-trained model is a vision-language pre-trained model, the real sample is a combination of real images and real text, the pseudo sample is a combination of pseudo text and real images, or the pseudo sample is a combination of real text and pseudo images, or the pseudo sample is a combination of pseudo text and pseudo images. The construction of pseudo-samples in the form of negative samples includes: Obtain a first image and a first text, wherein the first image is any real image in the current batch of training samples, and the first text is initialization pseudo-text; The first text is processed by a second text encoder to extract features, which are used to obtain second text features. The second text encoder is the text encoder of the previous time step that is adjacent to the first text encoder. The first text encoder is used to extract features from the real text in the current batch of training samples to obtain first text features. Based on the second image features and the second text features, pseudo-text is constructed in the form of negative samples. The combination of the pseudo-text and the first image is used as a pseudo-sample. The second image features include features extracted by the second image encoder from each real image in the current batch of training samples. The second image encoder is the image encoder of the previous time step adjacent to the first image encoder. The first image encoder is used to extract features from the real images in the current batch of training samples to obtain the first image features.

2. The method according to claim 1, characterized in that, The step of constructing pseudo-text based on the second image features and the second text features, and in the form of negative samples, includes: Similarity is calculated based on the second image features and the second text features; The first loss is calculated based on the first loss function and the similarity. The first text is updated based on the first loss to construct pseudo-text in the form of negative samples.

3. The method according to claim 1 or 2, characterized in that, The similarity between the pseudo-text in the pseudo-sample and the first image is higher than a preset similarity threshold.

4. The method according to claim 1, characterized in that, After replaying the pseudo-sample, the method further includes: Learn knowledge based on the pseudo-samples; Knowledge distillation is performed based on the pseudo-samples.

5. The method according to claim 4, characterized in that, The pseudo-samples are a combination of pseudo-text and real images, and the knowledge learned based on the pseudo-samples includes: A second loss is calculated based on a second loss function, wherein the second loss is used to characterize the learning of the matching relationship based on the pseudo-samples, and the second loss includes the average loss of text recall and the average loss of image recall.

6. The method according to claim 4, characterized in that, The pseudo-sample is a combination of pseudo-text and real images, and the knowledge distillation based on the pseudo-sample includes: The third loss is calculated based on the third loss function, wherein the third loss is used to characterize the knowledge distillation based on the pseudo-sample, and the third loss includes the loss between the multimodal pre-trained model at the current time and the multimodal pre-trained model at the previous time.

7. The method according to claim 6, characterized in that, The third loss includes the loss between the current multimodal pre-trained model and the previous multimodal pre-trained model based on the first dimension, and the loss between the current multimodal pre-trained model and the previous multimodal pre-trained model based on the second dimension. The first dimension is used to represent the dimension from image to text, and the second dimension is used to represent the dimension from text to image.

8. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program; the processor being used to run the computer program to implement the multimodal training method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, implements the multimodal training method as described in any one of claims 1-7.

Images