Post-pretraining method for image-text model to video-text model

Abstract

The present invention relates to multi-modal learning, and relates to a post-pretraining method for image-text model to video-text model, which is used for solving the problem that most existing video-text models require a large amount of computing resources and data, and full utilization cannot be achieved due to distribution shift when a pre-trained image-text model is used as a video-text model. According to the technical solution of the present invention, a video-text model is obtained from an image-text model from the three aspects of training, tasks, and data: first, a trained image-text model is used for post-pretraining, solving the problem of distribution shift by reducing computing resources and data as much as possible; second, in tasks, the robustness and sustainability of a model are improved by using text masks; and finally, by using a random video block loss technique, redundant features of video data are matched, so that the requirement on computing resources is further reduced.

Description

A post-pretraining method from image-text model to video-text model Technical Field

[0001] The present invention relates to multimodal learning, and in particular to a post-pretraining method from an image-text model to a video-text model. Background Art

[0002] In the field of video-language understanding, self-supervised pre-training on large-scale data, followed by fine-tuning on downstream supervised tasks, has become the mainstream paradigm. Early methods directly used pre-extracted video and text features, while more recent methods have achieved end-to-end pre-training. Popular methods often use two or three pre-training tasks, such as text mask modeling, frame ordering modeling, video-text matching, video-text contrastive learning, and video-text mask learning. Pre-training video-text multimodal models often requires millions of video-text pairs, requiring extensive computational resources and data for effective training. Among them, the work of Wang et al. (Wang, Alex et al. "All in One: Exploring Unified Video-Language Pre-training." ArXiv abs / 2203.07303(2022): n.pag.) achieved large-scale pre-training of multiple tasks, but failed to utilize the pre-trained image-text model; and the work of Xue et al. (Xue, Hongwei et al. "CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-LanguageRepresentation Alignment." ArXivabs / 2209.06430(2022): n.pag.) further pre-trained on the pre-trained image-text model, but still required billions of additional data for training.

[0003] Summary of the Invention

[0004] Most existing video-text multimodal models are trained from scratch, requiring extensive computational resources and data, unlike conventional supervised video model training, which leverages existing image-text models. While a small number of studies have utilized pre-trained image-text models, these models are underutilized due to distribution shift.

[0005] In order to solve the above-mentioned problems existing in the prior art, the purpose of the present invention is to propose a post-pretraining method, device and system, which fully utilizes the pre-trained image-text model and uses video-text data for efficient post-pretraining to obtain an effective video-text model.

[0006] In order to achieve the above objectives, the technical solutions of the present invention are as follows.

[0007] In a first aspect, the present invention provides a post-pretraining method for obtaining a video-text model from an image-text model, the method comprising the following steps:

[0008] Use the pre-trained image-text model parameters to randomly initialize the video-text model;

[0009] The initialized video-text model is pre-trained using video-text data. During the training process, the model weights are integrated in stages to obtain a trained video-text model.

[0010] In the above technical solution, the distribution shift problem is overcome by weight integration, but the distribution shift from image-text to video-text is much larger than the text shift between images in different domains. In particular, the post-pre-training requires a long training cycle, which makes the shift more serious. By gradually integrating the model weights instead of compensating after the training process, the above problem can be overcome. By guiding the model weight integration in stages

[0011] In the above technical solution, the model weight integration is guided in stages, and the steps include:

[0012] Set the number of training stages and the training period in each stage;

[0013] After each stage of training, according to the set of parameters to be integrated;

[0014] For each parameter in the parameter set, obtain its weight value saved in the previous stage, integrate them with the weight value after the end of training in the current stage, and use them as the weight value to be saved in the current stage.

[0015] In the above technical solution, the loss of the video-text model during training consists of contrastive learning loss and mask prediction loss; where:

[0016] The contrastive learning loss is the contrastive loss of video-to-text similarity and text-to-video similarity;

[0017] The mask prediction loss is the loss generated by predicting the mask part of the text;

[0018] Text masking is to perform a masking operation on the text data in the video-text pair data used for training, randomly replacing text tokens with mask tokens.

[0019] In the above technical solution, during the training of the video-text model, the video data in the video-text pair data used are randomly discarded for each video data.

[0020] In a second aspect, the present invention proposes a computer-readable storage device. The computer-readable storage device includes a readable storage medium storing a computer program that can be loaded by a processor and execute any one of the methods proposed in the present invention.

[0021] In the third aspect, according to the method proposed in the present invention, a post-pretraining system for obtaining a video-text model from an image-text model is implemented. The system uses the pre-trained image-text model parameters to randomly initialize the video-text model, and uses the video-text data to post-pretrain the initialized video-text model. During the training process, the model weight integration is guided in stages to obtain a trained video-text model.

[0022] Compared with the prior art, the present invention has the following technical effects:

[0023] (1) Adopting post-pre-training integration method: By integrating during the training process, the distribution shift is effectively alleviated without the need for large-scale training resources and data, overcoming the previous video-text pre-training or post-pre-training requirements for training resources and data.

[0024] (2) Text masking: By introducing a text masking module to perform random masking on text, the capabilities of the pre-trained image-text model are expanded, achieving the purpose of modal fusion, so that post-pre-training can enable the model to complete tasks such as video question answering.

[0025] (3) Random video block discarding technology: By discarding random blocks of the input video to match the redundant characteristics of the video, the requirements for computing resources are greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic diagram of a post-pretraining method on weight integration in a specific embodiment;

[0027] Figure 2 is a schematic diagram of the post-pre-training method in processing tasks and using data in a specific implementation. DETAILED DESCRIPTION

[0028] In supervised learning, using pre-trained image models when training video models is now a mainstream approach. Existing video-text multimodal models are mostly trained from scratch, requiring extensive computing resources and data, unlike conventional supervised video model training, which leverages existing image-text models. While a small number of studies have used pre-trained image-text models, these models are not fully utilized due to distribution shift.

[0029] To address these issues, we explored how to use pre-trained multimodal image-text pre-training models for further self-supervised pre-training on videos, and proposed a post-pretraining method. This process of further self-supervised pre-training on videos using pre-trained multimodal image-text pre-training models is referred to as post-pretraining. The goal of post-pretraining is to fully utilize existing pre-trained models, resulting in an effective, robust, and efficient model. This post-pretraining method not only allows us to leverage existing image-text pre-training models, but also eliminates the distribution shift from image-text models to video-text models. Furthermore, it eliminates the need for extensive computational resources and data, enabling large-scale video-text pre-training models that are user-friendly to small research groups and avoid energy waste.

[0030] The following explains the key implementation points of the technical solution from three aspects: training, tasks and data.

[0031] (1) Training

[0032] The weight integration method proposed by Wortsman et al. consists of two steps: standard fine-tuning training and weight space integration. This method involves fine-tuning image-text pre-trained models in the image domain. Although simple and effective, it is not as effective in the video-text domain. The weight integration method is intended to address the distribution shift problem, but the distribution shift from image-text to video-text is much larger than the text shift between images in different domains. Long training cycles can make the shift even more severe.

[0033] Weight integration can solve the distribution shift problem, but when training on different modalities, the structure of the desired model may differ from the original model, so directly weight integration of the model may not achieve the desired effect. In addition, the distribution shift from image-text to video-text due to the long training cycle may be much larger than the text shift between images in different domains. Therefore, in terms of training, the present invention adopts a guided weight integration method during the training process. Specifically, it is a staged guided model weight integration to obtain a trained video-text model, rather than compensating for it after the training process.

[0034] The steps in one embodiment of the staged guided model weight integration include:

[0035] Set the number of training stages and the training period in each stage;

[0036] After each stage of training, according to the set of parameters to be integrated;

[0037] For each parameter in the parameter set, obtain its weight value saved in the previous stage, integrate them with the weight value after the end of training in the current stage, and use them as the weight value to be saved in the current stage.

[0038] As shown in Figure 1, the model is initialized. First, its parameters are randomly initialized. Then, the parameters of the pretrained image-text model are randomly initialized. The order of these two random initialization steps can be changed. Three stages are set, with 1000 training cycles in each stage. The initialized video-text model is post-trained using video-text data. After the first stage, weights are calculated based on the model output and saved. Parameters are selected for weight integration and the resulting weight values ​​are saved, resulting in Model 1. In the second stage, after training Model 1, the weights for the current stage are calculated based on the model output. Combined with the weights saved from the first stage, parameters are selected for weight integration and the resulting weight values ​​are saved, resulting in Model 2. In the third stage, after training Model 2, the weights for the current stage are calculated based on the model output. Combined with the weights saved from the first and second stages, parameters are selected for weight integration and the resulting weight values ​​are saved, resulting in Model 3. Model 3, shown on the far right, is the result of training Model 3 and then integrating it with Models 1 and 2. This result is also referred to as Model 3.

[0039] It should be noted that the parameters of the video-text model and the image-text model can be the same or different. When they are different, the initial pre-trained model is ignored for the additional parameters. During the post-pre-training process, the parameters in the original weight integration method can be removed to make the method more concise; additional parameters can also be integrated. Therefore, the post-pre-training weight integration method can prevent the model from overfitting during the post-pre-training process. Since the number of stages and the number of training cycles in each stage are set, the model integration effect can be guided by adjusting the integrated training cycle and the number of integrated weights, so that even on a relatively small data set, the final model obtained will not produce serious distribution shifts.

[0040] Although the above example uses the post-pretraining method of obtaining a video-text model from an image-text model, the post-pretraining method can be applied to other post-pretraining methods that utilize cross-models.

[0041] (2) Task

[0042] Image-text pre-training models based on contrastive learning often only model a single modality and only perform comparisons in the feature space at the end, lacking the ability to fuse models. This design limits its applicability to downstream tasks.

[0043] Therefore, in terms of the task, in one implementation, a text masking module is used to mask the text data in the video-text pair data used for training, and the text tokens are randomly replaced with mask tokens. When restoring the text mask, the features of the corresponding video are used to perform modal fusion and amplify the capabilities of the original pre-trained image-text model.

[0044] Further improvements have been made to the task. For each video data point in the video-text pair data, random video blocks are discarded to match the redundancy of the video data. This significantly reduces the computing resource requirements and makes post-pre-training more efficient. Unlike mainstream mask learning, the random video block discarding technique in this invention does not require remodeling of the discarded blocks, thus significantly saving computing resources. When the masking rate reaches 90%, the required graphics card memory is only one-eighth of the original amount.

[0045] The post-trained model for the video-to-text model implemented using the aforementioned task is shown in Figure 2. It takes text-video pairs as input. A text masking operation is performed on the input text data, and then text features are extracted using a model built solely for text. After randomly discarding video blocks, the video features are extracted using a model built solely for video. Using both video and text features, the text is decoded to predict the replaced text in the text mask. Text decoding can utilize a cross-modal decoder, which acts as a cross-modal fusion. By introducing cross-modal fusion, the post-trained video-to-text model can be better applied to multimodal tasks requiring modal fusion, such as video question answering, without the need for large-scale retraining. The cross-modal decoder, by using text masking during training, makes the model robust and sustainable.

[0046] The pre-training loss for the video-to-text model implemented using the above task consists of a contrastive learning loss and a mask prediction loss. The mask prediction loss is the loss incurred when predicting the masked portion of the text, while the contrastive learning loss is the contrastive loss between video-to-text similarity and text-to-video similarity. The contrastive learning loss is calculated using the InfoNCE Loss function.

[0047] The video-to-text similarity is recorded as NCE(z v , z t ), the text-to-video similarity is recorded as NCE(z t , z v ), z v is the video segment feature, z t is the text segment feature, then the contrastive learning loss L is:

[0048] Where: B is the video-text pair data set, v is the video segment data, and t is the text segment data.

[0049] Since text-to-video similarity is symmetrical to video-to-text similarity, the following example provides a formula for calculating video-to-text similarity:

[0050] Where: T is the temperature hyperparameter. is covered in the video segment feature z v The positive embedding text segment features in It is the negative embedded text segment feature implied by its text segment feature, and z is the set of positive embedded text segment features and negative embedded text segment features.

[0051] According to the above method, the method can be implemented as a corresponding system. In one specific embodiment, the system uses pre-trained image-text model parameters to randomly initialize the video-text model, and then uses video-text data to post-pretrain the initialized video-text model. During the training process, the model weights are integrated in stages to obtain a trained video-text model.

[0052] Through the description of the above implementation methods, those skilled in the art can clearly understand that the methods and systems disclosed herein can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by means of dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present disclosure, in most cases, software program implementation is a better implementation method. In summary, the present invention has the following technical effects:

[0053] (1) Adopting post-pre-training integration method: By integrating during the training process, the distribution shift is effectively alleviated without the need for large-scale training resources and data, overcoming the previous video-text pre-training or post-pre-training requirements for training resources and data.

[0054] (2) Text masking: By introducing a text masking module to perform random masking on text, the capabilities of the pre-trained image-text model are expanded, achieving the purpose of modal fusion, so that post-pre-training can enable the model to complete tasks such as video question answering.

[0055] (3) Random video block discarding technology: By discarding random blocks of the input video to match the redundant characteristics of the video, the requirements for computing resources are greatly reduced.

[0056] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. A post-pretraining method for obtaining a video-text model from an image-text model, characterized in that: The method comprises the following steps: Use the pre-trained image-text model parameters to randomly initialize the video-text model; The initialized video-text model is post-pretrained using video-text data. During the training process, the model weights are guided to be integrated in stages to obtain a trained video-text model.

2. The method according to claim 1, characterized in that The model weight integration is guided in stages, the steps include: Set the number of training stages and set the training cycle in each stage; After each stage of training, according to the set of parameters to be integrated; For each parameter in the parameter set, obtain its weight value saved in the previous stage, integrate them with the weight value after the training of the current stage, and use them as the weight value to be saved in the current stage.

3. The method according to claim 1, characterized in that The loss of the video-text model during training consists of contrastive learning loss and mask prediction loss; where: The contrastive learning loss is the contrastive loss of video-to-text similarity and text-to-video similarity; The mask prediction loss is the loss caused by predicting the mask part of the text; Text masking is to perform a masking operation on the text data in the video-text pair data used for training, and randomly replace the text tokens with mask tokens.

4. The method according to claim 1, characterized in that: During the training of the video-text model, random video blocks are discarded for each video data in the video-text pair data used.

5. A computer-readable storage device, characterized in that: The readable storage device includes a readable storage medium storing a computer program that can be loaded by a processor and executes the method according to any one of claims 1 to 4.

6. A post-pretraining system for obtaining a video-text model from an image-text model, characterized in that: The system uses pre-trained image-text model parameters to randomly initialize the video-text model, uses video-text data to post-pretrain the initialized video-text model, and during the training process, guides the model weight integration in stages to obtain a trained video-text model.

7. The system according to claim 1, characterized in that The model weight integration is guided in stages, the steps include: Set the number of training stages and set the training cycle in each stage; After each stage of training, according to the set of parameters to be integrated; For each parameter in the parameter set, obtain its weight value saved in the previous stage, integrate them with the weight value after the training of the current stage, and use them as the weight value to be saved in the current stage.

8. The system according to claim 1, characterized in that The loss of the video-text model during training consists of contrastive learning loss and mask prediction loss; where: The contrastive learning loss is the contrastive loss of video-to-text similarity and text-to-video similarity; The mask prediction loss is the loss caused by predicting the mask part of the text; Text masking is to perform a masking operation on the text data in the video-text pair data used for training, and randomly replace the text tokens with mask tokens.

9. The system according to claim 1, characterized in that During the training of the video-text model, random video blocks are discarded for each video data in the video-text pair data used.

Images