A video representation method and device based on an unsupervised pre-training model

Abstract

The application provides a video representation method based on an unsupervised pre-training model, comprising the following steps: obtaining a video sample set; taking video frame embedding of the video sample set and text labels of video titles as inputs, pre-training a video representation model by using a mask framework modeling and a mask language modeling respectively, and obtaining a pre-training model; reconstructing the text labels of the video titles in the video sample set by using a contrastive learning method, and obtaining a positive sample set, a medium sample set and a negative sample set; performing contrastive training on the pre-training model by using the positive sample set, the medium sample set and the negative sample set by using a dynamic queue training method, and obtaining a completed video representation model; and obtaining a video to be labeled, and performing content extraction by using the completed video representation model.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of big data artificial intelligence deep learning. BACKGROUND

[0002] With the rise of short videos, a large number of short videos are produced and uploaded to major video platforms every day. In the face of a large number of short videos, how to better understand the video content and recommend it to the appropriate user is an important issue for all short video platforms. Short video information is rich, generally composed of short video text title, cover picture, video content and other elements. In order to more accurately extract content tags, it is necessary to comprehensively utilize these multi-modal information.

[0003] Of course, using artificially annotated data can achieve better training results, but the number of short videos is so large that relying solely on artificially annotated data is unrealistic. On short video platforms, millions of videos are uploaded every day, and artificial annotation consumes a lot of time and money. In most cases, a small amount of manually annotated data is used for fine-tuning. Therefore, we propose a self-supervised learning pre-training method to solve this problem. This pre-training method can perform self-supervised learning without using any artificially annotated data and achieve good results.

[0004] Pre-training tasks are divided into task-specific pre-training and task-agnostic pre-training. Task-specific pre-training is usually based on a specific scenario, and its effect is often better than that of task-agnostic pre-training in a specific scenario. However, some task-agnostic pre-training, such as Mask Language model, can help enhance the understanding ability of the model. Therefore, our pre-training method is divided into two stages. In the first stage, the model is trained using Mask Language Modeling (MLM) and Mask Frame Modeling (MFM). In the second stage, the model is trained using contrastive learning. Our pre-training method achieves good performance on the video similarity task. Referring to the success of contrastive learning in learning embeddings, we enhance the data and construct three kinds of positive samples, medium samples and negative samples. The positive pair adopts the same sentence, and their embeddings only differ in the dropout mask. We use word deletion to construct medium samples. In each training session, 30% of the text in the title is randomly deleted.

[0005] Our pre-training faces a challenge: training efficiency. For large-scale pre-training, training efficiency can be important. In contrastive learning, more negative instances often lead to better performance.

[0006] However, the traditional batch negative sampling is limited by the limited GPU memory. In order to have more negative samples during training, we design a dynamic queue so that the negative samples of the last batch can be used for the next batch training. When we calculate the gradient, we only calculate the gradient of one of the two channels. Therefore, the GPU memory will only increase a small amount during training.

[0007] The present application proposes a self-supervised pre-training method to solve the above problems, which achieves remarkable results without using any manually annotated data; and proposes a method for constructing positive samples and medium samples; and proposes a loss function to achieve competitive performance. SUMMARY

[0008] The present application aims to at least partially solve one of the technical problems in the related art.

[0009] To this end, the present application aims to propose a video representation method based on an unsupervised pre-training model for more accurately extracting content labels of short videos.

[0010] To achieve the above-mentioned purposes, the first aspect of the present application proposes a video representation method based on an unsupervised pre-training model, comprising:

[0011] Obtaining a video sample set, embedding video frames of the video sample set and text labels of video titles as inputs, pre-training a video representation model using mask framework modeling and mask language modeling respectively, to obtain a pre-training model;

[0012] Reconstructing the text labels of the video titles in the video sample set by a contrastive learning method to obtain a positive sample set, a medium sample set and a negative sample set;

[0013] Using the positive sample set, the medium sample set and the negative sample set to conduct contrastive training on the pre-training model by a dynamic queue training method to obtain a completed video representation model;

[0014] Obtaining a video to be labeled, and completing content extraction by the completed video representation model.

[0015] In addition, the video representation method based on the unsupervised pre-training model according to the above-mentioned embodiments of the present application can also have the following additional technical features:

[0016] Further, in an embodiment of the present application, the embedding of the video frames of the video sample set and the text labels of the video titles as inputs, pre-training the video representation model using mask framework modeling and mask language modeling respectively, comprises:

[0017] embedding the video frame through a fully connected layer to make the video frame embedding size consistent with the size of the text label;

[0018] connecting the special label, the video frame embedding, the special label and the text label;

[0019] inputting the special label, the video frame embedding, the special label and the text label into a video representation model to learn a multi-modal representation;

[0020] using a masked framework model for the video frame embedding and a masked language model for the text label as a baseline to pre-train the video representation model.

[0021] Further, in an embodiment of the present application, the text label of the video title in the video sample set is reconstructed by a contrastive learning method to obtain a positive sample set, a medium sample set and a negative sample set, comprising:

[0022] passing the same text label to the encoder of the pre-training model twice, obtaining two different embeddings as a positive sample pair by applying standard dropout twice, and constructing the positive sample set;

[0023] On the basis of the positive sample set, a predetermined number of text labels are randomly deleted to construct the medium sample set.

[0024] Further, in an embodiment of the present application, the pre-training model is trained by a dynamic queue training method using the positive sample set, the medium sample set and the negative sample set, comprising:

[0025] constructing a queue with a maximum capacity M, when a positive sample pair is trained, the final embedding obtained is added as a negative example to the queue to construct the negative sample set;

[0026] wherein, when the capacity of the queue reaches the maximum value, the first added negative example is deleted.

[0027] Further, in an embodiment of the present application, the pre-training model is trained by a dynamic queue training method using the positive sample set, the medium sample set and the negative sample set, further comprising:

[0028] optimizing the video representation model by introducing a ranking loss function; wherein, define x i and is a positive sample pair, x i and is a medium sample pair, is a negative sample, h i and represents xi and denotes denotes The loss function is defined as:

[0029]

[0030] where λ1, λ2, λ3 are temperature hyperparameters, is the cosine similarity

[0031] To achieve the above purpose, an embodiment of the second aspect of the present application provides a video representation device based on an unsupervised pre-training model, comprising the following modules:

[0032] A pre-training module is configured to obtain a video sample set, embed video frames of the video sample set and text labels of video titles as inputs, and pre-train a video representation model using a mask framework modeling and a mask language modeling respectively to obtain a pre-trained model.

[0033] A construction module is configured to re-construct the text labels of the video titles in the video sample set by a contrastive learning method to obtain a positive sample set, a medium sample set and a negative sample set.

[0034] A training module is configured to use the positive sample set, the medium sample set and the negative sample set to conduct contrastive training on the pre-trained model by a dynamic queue training method to obtain a completed video representation model.

[0035] An extraction module is configured to obtain a video to be labeled and complete content extraction by using the completed video representation model.

[0036] Further, in an embodiment of the present application, the pre-training module is further configured to:

[0037] Embed the video frame through a full connection layer to make the size of the video frame embedding consistent with the size of the text label.

[0038] Connect the special label, the video frame embedding, the special label and the text label.

[0039] Input the special label, the video frame embedding, the special label and the text label into a video representation model to learn a multi-modal representation.

[0040] Pre-train the video representation model using a mask framework model for the video frame embedding and a mask language model for the text label as a baseline.

[0041] Further, in an embodiment of the present application, the configuration module is further configured to:

[0042] The same text mark is transmitted to the encoder of the pre-trained model twice, two different embeddings are obtained as a positive sample pair by applying standard dropout twice, and the positive sample set is constructed;

[0043] On the basis of the positive sample set, a preset number of text marks are randomly deleted, and the medium sample set is constructed.

[0044] Further, in an embodiment of the present application, the training module is further configured to:

[0045] A queue with a maximum capacity M is constructed, when a positive sample pair training is completed, the final embedding obtained is added to the queue as a negative example, and the negative sample set is constructed;

[0046] When the capacity of the queue reaches the maximum value, the first added negative example is deleted.

[0047] To achieve the above purpose, the third aspect of the embodiment of the present application provides a computer device, characterized by comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to realize the video representation method based on the unsupervised pre-training model as described above.

[0048] To achieve the above purpose, the fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, characterized in that the computer program is executed by a processor to realize the video representation method based on the unsupervised pre-training model as described above.

[0049] The video representation method based on unsupervised contrast learning provided by the embodiment of the present application solves the problem of difficult labeling in the video representation task through unsupervised contrast learning. Compared with the prior art, the present application has the advantages of: effectively utilizing unlabeled data, improving the accuracy of video representation, saving the time of manual labeling, and improving the efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and more readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0051] Figure 1 The flowchart of the video representation method based on the unsupervised pre-training model provided by the embodiment of the present application.

[0052] Figure 2This is a schematic diagram of the structure of a video representation device based on an unsupervised pre-trained model provided in an embodiment of the present invention. Detailed Implementation

[0053] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0054] The following describes a video representation method based on an unsupervised pre-trained model according to an embodiment of the present invention, with reference to the accompanying drawings.

[0055] Figure 1 This is a flowchart illustrating a video representation method based on an unsupervised pre-trained model provided in an embodiment of the present invention.

[0056] like Figure 1 As shown, this video representation method based on an unsupervised pre-trained model includes the following steps:

[0057] S101: Obtain a video sample set, take the video frame embeddings and video title text tags of the video sample set as input, and pre-train the video representation model using mask frame modeling and mask language modeling respectively to obtain the pre-trained model;

[0058] S102: The text tags of video titles in the video sample set are reconstructed using a contrastive learning method to obtain a positive sample set, a medium sample set, and a negative sample set;

[0059] S103: The pre-trained model is trained by comparing positive, medium and negative sample sets using a dynamic queue training method to obtain a complete video representation model.

[0060] S104: Obtain the video to be labeled and extract the content using the completed video representation model.

[0061] Furthermore, in one embodiment of the present invention, the video frame embeddings and video title text tags of the video sample set are used as input, and the video representation model is pre-trained using masked frame modeling and masked language modeling, respectively, including:

[0062] Video frames are embedded through a fully connected layer, making the embedded video frame size consistent with the size of the text markup;

[0063] Connect special tags, video frame embeddings, special tags, and text tags;

[0064] Special tags, video frame embeddings, special tags, and text tags are input into the video representation model to learn multimodal representations;

[0065] The mask framework model is adopted for video frame embedding, and the mask language model is adopted for text token embedding as a baseline for pre-training of the video representation model.

[0066] Specifically, given a video sample, the model takes video frame embedding and text token embedding of the video title as input. First, the video frame embedding is passed through a fully connected layer to make its size consistent with that of the text. Then, the special token [CLS], the video frame embedding, the special token [SEP] and the text token embedding are connected, and then they are input into the model to learn the multi-modal representation. The input embedding is the sum of the token embedding and the position embedding.

[0067] The mask framework model (MFM) is adopted for video, and the mask language model (MLM) is adopted for text as a baseline. Unlike the language model with a fixed vocabulary usually with special [MASK] tokens, the video tokens are infinite in continuous space, and the video tokens are masked by setting them to all zeros and requiring the encoder to recover the video tokens.

[0068] Further, in an embodiment of the present application, the text tokens of the video title in the video sample set are reconstructed by a contrastive learning method to obtain a positive sample set, a medium sample set and a negative sample set, comprising:

[0069] The same text tokens are passed to the encoder of the pre-trained model twice, and two different embeddings are obtained as a positive sample pair by applying standard dropout twice, so as to construct the positive sample set.

[0070] On the basis of the positive sample set, a predetermined number of text tokens are randomly deleted, so as to construct the medium sample set.

[0071] For the construction of the positive sample, there are two methods, one is to use word repetition to construct the positive sample. The word repetition mechanism randomly duplicates some words in the sentence. We repeat 15% of the words in the sentence. The other is to pass the same sample to the pre-trained encoder twice, and two different embeddings can be obtained as a positive pair by applying standard dropout twice. It is found that the effect of using the second method is better than that of the first method.

[0072] DaPI was originally proposed in SimCSE as a simple strategy to perturb the intermediate representation, and thus can be used as data augmentation. The present invention applies a dropout rate of 0.1 to the fully connected layers and attention probabilities in the Transformer encoder, just like in BERT. The same input is fed into the encoder twice to obtain two representations, one of which is used as a positive instance for the other. The dropout strategy outperforms all commonly used discrete perturbation techniques, including clipping, word deletion, mask language modeling, and synonym replacement. Unlike SimCSE, the present invention only computes the gradient for one of the two channels.

[0073] The present invention uses word deletion to construct medium samples. On the basis of positive samples, 30% of the text is randomly deleted. This is not static deletion; the example will not delete the same text every time. Then randomly sample other samples in the same mini-batch as the negative.

[0074] At the beginning of training, the present invention establishes a queue with a maximum capacity of M. When a batch is trained, its final embedding is added to the queue as a negative example for the next round of loss calculation. When its capacity reaches the maximum, the first added negative example will be deleted.

[0075] Further, in an embodiment of the present invention, the pre-trained model is contrastively trained using the positive sample set, the medium sample set and the negative sample set by the dynamic queue training method, comprising:

[0076] A queue with a maximum capacity of M is constructed, and when a positive sample pair is trained, the final embedding obtained is added to the queue as a negative example, thereby constructing the negative sample set;

[0077] Wherein, when the capacity of the queue reaches the maximum, the first added negative example is deleted.

[0078] Unlike other loss functions, such as cross-entropy loss or mean squared error loss, whose goal is to learn to directly predict the label, value, or set of values for a given input, the ranking loss in the present invention is to predict the relative distance between inputs.

[0079] Further, in an embodiment of the present invention, the pre-trained model is contrastively trained using the positive sample set, the medium sample set and the negative sample set by the dynamic queue training method, further comprising:

[0080] Optimizing the video representation model by introducing a ranking loss function; wherein, define x i and is a positive sample pair, x i and is a medium sample pair, is a negative sample, h i and represents x i and represents represents The loss function is defined as:

[0081]

[0082] where λ1, λ2, λ3 are temperature hyperparameters, is the cosine similarity

[0083] The value of λ depends on the distance between samples. Based on the idea of ranking, only the order of each sample needs to be determined, but the distance between their orders may be very different. For example, student A scores 90, student B scores 50, and student C scores 40. Although the order between them is a before B, and B before C, they are not suitable for processing with the same weight. When the distance between the positive sample and the medium sample is not large, λ1 should be set to be small. If the distance between the medium sample and the negative sample is not large, λ2 must be set to be small. If there is no medium sample, only positive samples and negative samples, then λ1 and λ2 can be set to 0.

[0084] Since the negative samples are constructed by randomly batching other samples, it is difficult to guarantee that all negative samples are truly negative samples. Therefore, if the label of the negative sample is set to 0, the usual consequence is over-learning, thereby losing the generalization ability, or the optimization is too difficult, resulting in no learning at all. The loss function of the present application only tells the model the order between samples, and does not depend on the specific value, which greatly reduces this problem.

[0085] The video representation method based on unsupervised contrastive learning provided by the embodiment of the present application solves the problem of difficult labeling in the video representation task through unsupervised contrastive learning. Compared with the prior art, the present application has the advantages of: effectively utilizing unlabeled data, improving the accuracy of video representation, saving the time of manual labeling, and improving the efficiency.

[0086] In order to realize the above-mentioned embodiment, the present application further provides a video representation device based on an unsupervised pre-training model.

[0087] Figure 2 A structural schematic diagram of a video representation device based on an unsupervised pre-training model provided by the embodiment of the present application.

[0088] As Figure 2 shown, the video representation device based on the unsupervised pre-training model comprises a pre-training module 100, a construction module 200, a training module 300, and an extraction module 400.

[0089] a pre-training module, configured to obtain a video sample set, embed video frames of the video sample set and text labels of video titles as inputs, pre-train a video representation model using a mask framework modeling and a mask language modeling respectively, and obtain a pre-trained model;

[0090] a construction module, configured to re-construct the text labels of the video titles in the video sample set through a contrastive learning method, and obtain a positive sample set, a medium sample set and a negative sample set;

[0091] a training module, configured to train the pre-trained model using the positive sample set, the medium sample set and the negative sample set through a dynamic queue training method, and obtain a completed video representation model;

[0092] an extraction module, configured to obtain a video to be labeled, and complete content extraction through the completed video representation model.

[0093] Further, in an embodiment of the present application, the pre-training module is further configured to:

[0094] pass the video frame embedding through a full connection layer to make the size of the video frame embedding consistent with the size of the text label;

[0095] connect the special label, the video frame embedding, the special label and the text label;

[0096] input the special label, the video frame embedding, the special label and the text label into the video representation model to learn a multi-modal representation;

[0097] pre-train the video representation model using a mask framework model for the video frame embedding and a mask language model for the text label as a baseline.

[0098] Further, in an embodiment of the present application, the construction module is further configured to:

[0099] pass the same text label to an encoder of the pre-trained model twice, obtain two different embeddings as a positive sample pair through two applications of standard dropout, and thereby construct the positive sample set;

[0100] on the basis of the positive sample set, randomly delete a preset number of text labels, and thereby construct the medium sample set.

[0101] Further, in an embodiment of the present application, the training module is further configured to:

[0102] construct a queue with a maximum capacity of M, when a positive sample pair is trained, add the obtained final embedding as a negative example to the queue, and thereby construct the negative sample set;

[0103] wherein, when the capacity of the queue reaches the maximum value, the first added negative example is deleted.

[0104] To achieve the above object, the third aspect of the present application provides a computer device, characterized in comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video representation method based on unsupervised pre-training model as described above.

[0105] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the present application and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0106] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0107] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A video representation method based on an unsupervised pre-trained model, characterized in that, Includes the following steps: A video sample set is obtained. The video frame embeddings and text tags of the video titles from the video sample set are used as input. A pre-trained video representation model is then performed using masked frame modeling and masked language modeling, respectively, to obtain a pre-trained model. This process includes: passing the video frame embeddings through a fully connected layer to make the size of the video frame embeddings consistent with the size of the text tags; connecting the special tags, the video frame embeddings, and the text tags; inputting the special tags, the video frame embeddings, and the text tags into the video representation model to learn multimodal representations; and using a masked frame model for the video frame embeddings and a masked language model for the text tags as a baseline for pre-training the video representation model. The text tags of video titles in the video sample set are reconstructed using a contrastive learning method to obtain a positive sample set, a medium sample set, and a negative sample set. This includes: passing the same text tags to the encoder of the pre-trained model twice, applying standard dropout twice to obtain two different embeddings as positive sample pairs, thereby constructing the positive sample set; and randomly deleting a preset number of text tags from the positive sample set to construct the medium sample set. The pre-trained model is trained by comparing the positive sample set, the medium sample set, and the negative sample set using the dynamic queue training method to obtain the complete video representation model. The video to be labeled is obtained, and the content is extracted using the completed video representation model.

2. The method according to claim 1, characterized in that, The step of using the dynamic queue training method to comparatively train the pre-trained model using the positive sample set, the medium sample set, and the negative sample set includes: Construct a queue with a maximum capacity of M. When a positive sample pair is trained, add the final embedding as a negative example to the queue to construct the negative sample set. Specifically, when the capacity of the queue reaches its maximum value, the first added negative instance is deleted.

3. The method according to claim 1, characterized in that, The step of using the dynamic queue training method to comparatively train the pre-trained model using the positive sample set, the medium sample set, and the negative sample set also includes: The video representation model is optimized by introducing a ranking loss function; where, defined and These are positive sample pairs. and It is a medium-sized sample pair. , … It is a negative sample. and express and , express , , … express , … The loss function is then defined as: ， in , , It is a temperature hyperparameter, sim( , () is cosine similarity .

4. A video representation device based on an unsupervised pre-trained model, characterized in that, Includes the following modules: The pre-training module is used to acquire a video sample set, taking the video frame embeddings and video title text tags of the video sample set as input, and pre-training the video representation model using masked frame modeling and masked language modeling respectively to obtain a pre-trained model. It is also used to pass the video frame embeddings through a fully connected layer to make the size of the video frame embeddings consistent with the size of the text tags; connect the special tags, the video frame embeddings, and the text tags; input the special tags, the video frame embeddings, and the text tags into the video representation model to learn multimodal representations; and use a masked frame model for the video frame embeddings and a masked language model for the text tags as a baseline to pre-train the video representation model. The construction module is used to reconstruct the text tags of video titles in the video sample set through a contrastive learning method to obtain a positive sample set, a medium sample set, and a negative sample set. It is also used to pass the same text tags to the encoder of the pre-trained model twice, and obtain two different embeddings as positive sample pairs by applying standard dropout twice, thereby constructing the positive sample set. Based on the positive sample set, a preset number of text tags are randomly deleted to construct a medium sample set; The training module is used to perform comparative training on the pre-trained model using the positive sample set, medium sample set, and negative sample set through a dynamic queue training method to obtain a complete video representation model. The extraction module is used to acquire the video to be labeled and to extract content using the completed video representation model.

5. The apparatus according to claim 4, characterized in that, The training module is also used for: Construct a queue with a maximum capacity of M. When a positive sample pair is trained, add the final embedding as a negative example to the queue to construct the negative sample set. Specifically, when the capacity of the queue reaches its maximum value, the first added negative instance is deleted.

6. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the video representation method based on an unsupervised pre-trained model as described in any one of claims 1-3.

Images