Fine tuning training method and device for video language model and electronic equipment

Abstract

The embodiment of the invention discloses a fine tuning training method for a video language model, and the method comprises the steps: adding a corresponding text adaptation network and a video adaptation network into a text encoder and a video encoder of the video language model; and the text adaptation network and the video adaptation network comprise shared mapping layers with the same structure and model parameters, so that the relation between videos and texts can be kept during fine tuning training, and the situation that a text encoder and a video encoder are separately trained is avoided. Moreover, during fine tuning training, only the model parameters of the text adaptation network and the video adaptation network are adjusted, and the model parameters in the original video language model are not adjusted, so that the original generalization ability of the video language model can be maintained, and therefore, for any fine-grained classification task, the classification efficiency is improved. The text adaptive network and the video adaptive network which are adaptive to the task only need to be added and trained on the basis of the original video language model, and a large amount of training cost does not need to be consumed.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, device, and electronic device for fine-tuning a video language model. Background Art

[0002] Video language models can be used to understand and generate textual information related to the input video content. They have been widely used in various fields, including video understanding, subtitle generation, video search, and video classification. Similar to most machine learning models, video language models are categorized into generative and discriminative models. Discriminative video language models are primarily used in video classification, for example, to determine the riskiness of an input video.

[0003] A discriminative video language model is generally composed of a text encoder and a video encoder. The text encoder is used to encode the input text to obtain text features, and the video encoder is used to encode the input video to obtain video features. Finally, based on the video features and text features, it is determined whether the input video and text match, so as to classify the video into a certain text category.

[0004] Pre-trained video language models usually have high zero-sample reasoning capabilities and strong generalization capabilities, and can be used as a base for many downstream fine-grained classification tasks.

[0005] Even so, when applying pre-trained video language models to downstream, more fine-grained classification tasks, they still need to be fine-tuned to improve the accuracy of their output results. Existing techniques for fine-tuning video language models often employ a full fine-tuning approach, adjusting all model parameters within the video language model. This not only incurs significant training costs and reduces fine-tuning efficiency, but more importantly, it severely impacts the video language model's inherent zero-shot reasoning capabilities, significantly reducing its generalization capabilities.

[0006] Therefore, how to fine-tune the video language model so that it can maintain its original generalization ability and quickly adapt to more fine-grained classification tasks has become an urgent problem to be solved. Summary of the Invention

[0007] The embodiments of this specification provide a method, device, storage medium, and electronic device for fine-tuning a video language model to partially solve the problems existing in the above-mentioned prior art.

[0008] The embodiments of this specification adopt the following technical solutions:

[0009] This specification provides a fine-tuning training method for a video language model, wherein the video language model includes a text encoder and a video encoder, wherein the text encoder includes a text encoding network and a text adaptation network, and wherein the video encoder includes a video encoding network and a video adaptation network, wherein the text adaptation network and the video adaptation network include the same shared mapping layer; the method includes:

[0010] Obtaining a sample video and a plurality of sample texts, and determining an annotated text in each sample text that matches the sample video;

[0011] Inputting the sample video into the video encoder, obtaining common video features of the sample video through the video encoding network, and obtaining aggregated video features of the sample video through the shared mapping layer in the video adaptation network; and inputting each sample text into the text encoder, obtaining common text features of each sample text through the text encoding network, and obtaining aggregated text features of each sample text through the shared mapping layer in the text adaptation network;

[0012] According to the common video features and aggregated video features of the sample video, and the common text features and aggregated text features of each sample text, determining, in each sample text, a text that matches the sample video as an estimated text;

[0013] Fine-tuning the text adaptation network and the video adaptation network is performed based on the estimated text and the annotated text.

[0014] This specification provides a fine-tuning training device for a video language model. The video language model includes a text encoder and a video encoder. The text encoder includes a text encoding network and a text adaptation network. The video encoder includes a video encoding network and a video adaptation network. The text adaptation network and the video adaptation network include the same shared mapping layer. The device includes:

[0015] A sample acquisition module, configured to acquire a sample video and a plurality of sample texts, and determine an annotated text in each sample text that matches the sample video;

[0016] a video encoding module, configured to input the sample video into the video encoder, obtain common video features of the sample video through the video encoding network, and obtain aggregated video features of the sample video through a shared mapping layer in the video adaptation network;

[0017] A text encoding module, configured to input each sample text into the text encoder, obtain the common text features of each sample text through the text encoding network, and obtain the aggregated text features of each sample text through the shared mapping layer in the text adaptation network;

[0018] a matching module for determining, from each sample text, a text that matches the sample video as an estimated text based on the common video features and aggregated video features of the sample video and the common text features and aggregated text features of each sample text;

[0019] A fine-tuning training module is used to fine-tune the text adaptation network and the video adaptation network based on the estimated text and the annotated text.

[0020] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the fine-tuning training method of the video language model.

[0021] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the fine-tuning training method for the video language model described above is implemented.

[0022] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0023] The embodiments of this specification disclose a fine-tuning training method for a video language model. The method adds corresponding text adaptation networks and video adaptation networks to the text encoder and video encoder of the video language model, and the text adaptation network and the video adaptation network contain a shared mapping layer with the same structure and model parameters, which can maintain the connection between video and text during fine-tuning training, avoiding the need to train the text and video encoders separately. Moreover, during fine-tuning training, only the model parameters of the text adaptation network and the video adaptation network are adjusted, without adjusting the model parameters in the original video language model, so that the video language model can maintain its original generalization ability. In this way, for any fine-grained classification task, it is only necessary to add and train a text adaptation network and a video adaptation network adapted to the task on the basis of the original video language model, without consuming a large amount of training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0025] Figure 1 A flow chart of the fine-tuning training method for the video language model provided in the embodiments of this specification;

[0026] Figure 2 A schematic diagram of the structure of the video language model provided in the embodiments of this specification;

[0027] Figure 3 A schematic diagram of the structure of a coding layer pair and its corresponding adaptation network provided in an embodiment of this specification;

[0028] Figure 4 A schematic diagram of a fine-tuning training device for a video language model provided in an embodiment of this specification;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0030] When fine-tuning the video language model, we need to face three technical challenges simultaneously:

[0031] The first is how to ensure that the fine-tuned video language model maintains its original generalization ability;

[0032] The second is how to improve training efficiency so that the video language model can quickly adapt to fine-grained classification tasks;

[0033] The third is how to model the connection between the video and text encoders in the video language model. This is because the role of the video language model (discriminant) is to classify the input video into the category corresponding to a certain text among several input texts. It neither encodes the video nor the text separately, but deeply explores the connection between the video and the text. Therefore, once the modeling of the connection between the video and the text is lost, the video language model after fine-tuning training will inevitably be split into a separate video encoder and a separate text encoder, which will seriously affect the accuracy of the video language model.

[0034] Based on this, the embodiment of this specification adds a text adaptation network to the text encoder and a video adaptation network to the video encoder on the basis of the original video language model. The text adaptation network and the video adaptation network contain a shared mapping layer whose structure and model parameters remain consistent during the fine-tuning training process, so as to maintain the connection between the video and the text, thereby solving the third problem mentioned above. During fine-tuning training, only the model parameters of the text adaptation network and the video adaptation network are adjusted, and the model parameters of the original video language model are not adjusted, so that it retains its original generalization ability, thereby solving the first problem mentioned above. Moreover, for any classification task, it is only necessary to add and train the corresponding text adaptation network and video adaptation network to make the video language model adapt to the classification task, without the need for full fine-tuning, thereby solving the second problem mentioned above.

[0035] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0036] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0037] Figure 1 The flow chart of the fine-tuning training method of the video language model provided in the embodiment of this specification includes the following steps:

[0038] S100: Obtain a sample video and several sample texts, and determine the annotated text in each sample text that matches the sample video.

[0039] In the embodiment of this specification, the Figure 1 The fine-tuning training method illustrated can be performed by a server or a server cluster consisting of multiple servers. Those skilled in the art will appreciate that any electronic device capable of fine-tuning the video language model can be used, and this specification does not impose any restrictions. The following description uses a server as an example.

[0040] The server can first obtain a pre-trained video language model, which also includes two paths: a text encoder and a video encoder. Then, a text adaptation network is added to the text encoder, and a video adaptation network is added to the video encoder. In this specification, the original network in the text encoder is referred to as the text encoding network, and the original network in the video encoder is referred to as the video encoding network. In this way, after adding the corresponding adaptation networks to the two encoders, the text encoder of the video language model includes a text encoding network and a text adaptation network, and the video encoder includes a video encoding network and a video adaptation network.

[0041] It should be noted that the text adaptation network added to the text encoder and the video adaptation network added to the video encoder in the embodiments of this specification contain the same shared mapping layer. The structure and model parameters of this shared mapping layer remain consistent throughout the fine-tuning training process.

[0042] After obtaining the video language model with the adaptation network added, the server can obtain sample videos and several sample texts for supervised training, and identify the sample texts that match the sample video among these sample texts as the annotated texts. The sample texts can be annotated manually or automatically by injecting other machine learning models, etc., which is not limited in this embodiment.

[0043] For example, if the sample video obtained is a video clip of a slam dunk goal in a basketball game, the sample texts may be: 3-point shot, pass, slam dunk, layup, free throw, etc. Obviously, the labeled text is "slam dunk".

[0044] After obtaining the sample video and each sample text and determining the annotated text in the sample text, the server can synchronously use the video encoder (the video encoders described below are all video encoders with a video adaptation network added) and the text encoder (the text encoders described below are all text encoders with a text adaptation network added) to encode the sample video and each sample text respectively, that is, the execution order of the subsequent steps S102 and S104 is not in particular order and can be executed synchronously.

[0045] S102: Input the sample video into the video encoder, obtain common video features of the sample video through the video encoding network, and obtain aggregated video features of the sample video through the shared mapping layer in the video adaptation network.

[0046] For the video encoder, the video encoder can input the sample video into the original video encoding network to obtain the common video features of the sample video, and on the other hand obtain the aggregated video features of the sample video through the newly added video adaptation network.

[0047] Since the shared mapping layer in the video adaptation network always maintains the same structure and model parameters as the shared mapping layer in the text adaptation network, the video features obtained after processing by the shared mapping layer in the video adaptation network include the connection and interaction between the sample video and the sample text, and are video features that can reflect the text features of the sample text to a certain extent. It is equivalent to a video feature that integrates at least part of the text features. Therefore, this manual refers to the video features obtained by the video adaptation network as aggregated video features.

[0048] S104: Input each sample text into the text encoder, obtain the common text features of each sample text through the text encoding network, and obtain the aggregated text features of each sample text through the shared mapping layer in the text adaptation network.

[0049] Similarly, for the text encoder, the text encoder can also input the sample text into the original text encoding network to obtain the ordinary text features of the sample text, and obtain the aggregated text features of the sample text through the newly added text adaptation network.

[0050] Since the shared mapping layer in the text adaptation network always maintains the same structure and model parameters as the shared mapping layer in the video adaptation network, the text features obtained after processing by the shared mapping layer in the text adaptation network include the connection and interaction between the sample text and the sample video. They are text features that can reflect the video features of the sample video to a certain extent, which is equivalent to text features that integrate at least part of the video features. Therefore, this manual refers to the text features obtained by the text adaptation network as aggregated text features.

[0051] S106: Based on the common video features and aggregated video features of the sample video, and the common text features and aggregated text features of each sample text, determine, in each sample text, a text that matches the sample video as an estimated text.

[0052] In the embodiment of the present specification, after obtaining the common video features and the aggregated video features of the sample video in step S102, the common video features and the aggregated video features may be fused to obtain the fused video features of the sample video.

[0053] Similarly, for each sample text, after obtaining the common text features and aggregated text features of the sample text in step S104, the common text features and aggregated text features may be fused to obtain the fused text features of the sample text.

[0054] Among them, for the fusion method of fusing ordinary video features and aggregated video features, and the fusion method of fusing ordinary text features and aggregated text features, the two fusion methods need to be consistent, and the specific fusion method may include but is not limited to splicing, pooling or matrix addition methods, which are not limited in the embodiments of this specification.

[0055] After obtaining the fused video features of the sample video and the fused text features of each sample text, the text that matches the sample video can be determined in each sample text as the estimated text. Specifically, the similarity between the fused video features of the sample video and the fused text features of each sample text can be determined separately, and the sample text corresponding to the fused text feature with the greatest similarity to the fused video features of the sample video can be determined as the estimated text. Among them, the method for determining the similarity between the fused video features and the fused text features includes but is not limited to determining the similarity between the fused video features and the fused text features based on the Euclidean distance or cosine similarity between the fused video features and the fused text features, and the embodiments of this specification do not limit this.

[0056] S108: Fine-tune the text adaptation network and the video adaptation network according to the estimated text and the annotated text.

[0057] In an embodiment of the present specification, the text adaptation network and the video adaptation network can be fine-tuned and trained by a supervised training method. That is, the estimated text and the annotated text are input into a preset first loss function to obtain a first loss value output by the first loss function, wherein the smaller the difference between the estimated text and the annotated text, the smaller the first loss value, and conversely, the greater the difference between the estimated text and the annotated text, the greater the first loss value; with the goal of at least reducing the first loss value, the model parameters of the text adaptation network and the video adaptation network are adjusted. In other words, with the training goal of minimizing the difference between the estimated text and the annotated text output by the video language model, the model parameters of the text adaptation network and the video adaptation network are adjusted, while the model parameters of the original text encoding network and the video encoding network are fixed unchanged until the entire video language model converges. Wherein, when adjusting the model parameters of the text adaptation network and the video adaptation network, the model parameters of the shared mapping layer in the text adaptation network and the shared mapping layer in the video adaptation network need to always remain the same.

[0058] Through the above method, corresponding text adaptation networks and video adaptation networks are added to the text encoder and video encoder of the video language model. These networks contain shared mapping layers with identical structures and model parameters. This maintains the connection between video and text during fine-tuning training, avoiding the need to train the text and video encoders separately. Furthermore, during fine-tuning training, only the model parameters of the text adaptation network and video adaptation network are adjusted, without adjusting the model parameters of the original video language model. This allows the video language model to maintain its original generalization capabilities. This allows any fine-grained classification task to be performed by simply adding and training a text adaptation network and video adaptation network adapted to the task on the basis of the original video language model, without incurring significant training costs.

[0059] Figure 2 This is a schematic diagram of the structure of the video language model provided in the embodiment of this specification. Figure 2 In the video language model shown, the text encoding network in the text encoder is composed of N text encoding layers cascaded in sequence, where N is a positive integer. Similarly, the video encoding network in the video encoder is also composed of N video encoding layers cascaded in sequence. It can be seen that the number of text encoding layers and video encoding layers is the same, both N.

[0060] In the embodiments of this specification, the (i-j)-th text encoding layer of the text encoding network is used as the specified text encoding layer, and each specified text encoding layer has a corresponding text adaptation network. Correspondingly, the (i-j)-th video encoding layer of the video encoding network is used as the specified video encoding layer, and each specified video encoding layer has a corresponding video adaptation network. Here, 1 < i < j < N, and both i and j are positive integers.

[0061] That is to say, Figure 2 From the text encoding network and the video encoding network shown, the text encoding layers in the text encoding network and the video encoding layers in the video encoding network are paired in twos. In the embodiments of this specification, the concept of an encoding layer pair is introduced, that is, in the order from front to back (front and back refer to the time sequence of processing the data of the input video language model), the text encoding layer and the video encoding layer with the same sorting in the text encoding network and the video encoding network form an encoding layer pair. Or rather, the i-th text encoding layer in the text encoding network and the i-th video encoding layer in the video encoding network form an encoding layer pair.

[0062] In the (i-j)-th encoding layer pair in the embodiments of this specification, there are corresponding text adaptation network and video adaptation network, while other encoding layer pairs do not have corresponding adaptation networks. Specifically, the encoding layer pairs with corresponding adaptation networks can be set to be relatively later. For example, when i > N / 2, that is, select the encoding layer pair as far back as possible to set the corresponding adaptation networks.

[0063] Furthermore, in the embodiments of this specification, the text adaptation network corresponding to the text encoding layer in an encoding layer pair and the video adaptation network corresponding to the video encoding layer in this encoding layer pair include the same shared mapping layer. That is, the text adaptation network and the video adaptation network corresponding to the same encoding layer pair include shared mapping layers whose structures and model parameters always remain consistent, and there are no restrictions on the shared mapping layers included in the text adaptation networks and video adaptation networks corresponding to different encoding layer pairs. Or rather, the text adaptation network corresponding to the n-th text encoding layer of the text encoding network and the video adaptation network corresponding to the n-th video encoding layer of the video encoding network include shared mapping layers whose structures and model parameters always remain consistent, where i ≤ n ≤ j and n is a positive integer.

[0064] For example Figure 2 The text adaptation network and the video adaptation network within the dashed box in are the text adaptation network and the video adaptation network corresponding to the same encoding layer pair, and the structures and model parameters of the shared mapping layers they respectively include always remain consistent.

[0065] Furthermore, Figure 3 is a schematic structural diagram of an encoding layer pair and its corresponding adaptation network provided by the embodiments of this specification. As Figure 3In the embodiment of this specification, for a specified text encoding layer, the input end of the corresponding text adaptation network is the same as the input end of the specified text encoding layer, and the output end is connected to the output end of the specified text encoding layer, so that the output result of the text adaptation network is integrated with the output result of the specified text encoding layer. Figure 1 In the step S106 shown, for each sample text, when the ordinary text features and the aggregated text features of the sample text obtained in step S104 are fused, for each specified text coding layer, the output features of the previous text coding layer of the specified text coding layer can be input into the specified text coding layer and the text adaptation network corresponding to the specified text coding layer respectively to obtain the ordinary text features output by the specified text coding layer and the aggregated text features output by the text adaptation network corresponding to the specified text coding layer, and then the ordinary text features output by the specified text coding layer and the aggregated text features output by the text adaptation network corresponding to the specified text coding layer are fused to obtain the output features of the specified text coding layer, until the output features of the last text coding layer of the entire text coding network are obtained, and the output features of the last text coding layer are used as the fused text features.

[0066] Similarly, for a specified video coding layer, the input end of the corresponding video adaptation network is the same as the input end of the specified video coding layer, and the output end is connected to the output end of the specified video coding layer, so that the output result of the video adaptation network is integrated with the output result of the specified video coding layer. Figure 1 In step S106 shown, when the ordinary video features and the aggregated video features of the sample video obtained in step S102 are fused, for each specified video coding layer, the output features of the previous video coding layer of the specified video coding layer can be input into the specified video coding layer and the video adaptation network corresponding to the specified video coding layer respectively to obtain the ordinary video features output by the specified video coding layer and the aggregated video features output by the video adaptation network corresponding to the specified video coding layer, and then the ordinary video features output by the specified video coding layer and the aggregated video features output by the video adaptation network corresponding to the specified video coding layer are fused to obtain the output features of the specified video coding layer, until the output features of the last video coding layer of the entire video coding network are obtained, and the output features of the last video coding layer are used as the fused video features.

[0067] Furthermore, although the structure and model parameters of the shared mapping layer included in the corresponding text adaptation network and video adaptation network of one coding layer in the embodiment of this specification are always consistent ( Figure 3The two shared mapping layers in the dotted box indicate that the structures and model parameters of the two shared mapping layers are always consistent), but the encoding layer for the corresponding text adaptation network and video adaptation network not only contains the same shared mapping layer, but also includes other network layers. Therefore, the encoding layer for the corresponding text adaptation network and video adaptation network are not the same as a whole, such as Figure 3 shown.

[0068] exist Figure 3 In the example, the text adaptation network corresponding to the specified text encoding layer includes not only the shared mapping layer, but also the dimensionality reduction processing layer and the dimensionality increase processing layer. The video adaptation network corresponding to the specified video encoding layer includes not only the shared mapping layer, but also the dimensionality reduction processing layer, the time dimensionality increase processing layer and the space dimensionality increase processing layer. Figure 3 The shared mapping layer in the corresponding text adaptation network and video adaptation network is the same for one encoding layer pair, but the other network layers are different.

[0069] exist Figure 3 In the text adaptation network shown, the text adaptation network is composed of a cascade of dimensionality reduction processing layers, shared mapping layers, and dimensionality increase processing layers. For a sample text, the process of outputting the aggregated text features of the sample text through the text adaptation network corresponding to the specified text encoding layer is as follows: inputting the output features of the text encoding layer preceding the specified text encoding layer into the dimensionality reduction processing layer to obtain the reduced output features; inputting the reduced output features into the shared mapping layer of the text adaptation network corresponding to the specified text encoding layer to obtain the aggregated text features to be increased in dimension; and inputting the aggregated text features to be increased in dimension into the dimensionality increase processing layer to obtain the aggregated text features output by the text adaptation network corresponding to the specified text encoding layer.

[0070] Unlike the text adaptation network, Figure 3The video adaptation network shown in the figure consists of a dimensionality reduction layer, a shared mapping layer, a temporal dimensionality increase layer, and a spatial dimensionality increase layer. The dimensionality reduction layer and the shared mapping layer are cascaded sequentially, with the output of the shared mapping layer connected to the inputs of both the temporal and spatial dimensionality increase layers. The output of the temporal and spatial dimensionality increase layers is fused to form the output of the video adaptation network. The process of outputting the aggregated video features of the sample video through the video adaptation network corresponding to the specified video coding layer is as follows: inputting the output features of the previous video coding layer into the dimensionality reduction processing layer to obtain the output features after dimensionality reduction, inputting the output features after dimensionality reduction into the shared mapping layer in the video adaptation network corresponding to the specified video coding layer to obtain the aggregated video features to be upgraded, inputting the aggregated video features to be upgraded into the time dimensionality upgrading processing layer and the space dimensionality upgrading processing layer respectively to obtain the time dimensionality upgrading aggregated video features output by the time dimensionality upgrading processing layer and the space dimensionality upgrading aggregated video features output by the space dimensionality upgrading processing layer, fusing the time dimensionality upgrading aggregated video features and the space dimensionality upgrading aggregated video features to obtain the aggregated video features output by the video adaptation network corresponding to the specified video coding layer.

[0071] The temporal dimensionality upscaling layer is used to extract and upscale the temporal features of the sample video from the aggregated video features to be upscaled. This is because the sample video is essentially composed of frames of images. For a continuous action, such as a slam dunk, the video is also represented by a "decomposed action" represented by a series of "freeze-frame images." Only by continuously playing these frozen decomposed actions can the user recognize the action as a slam dunk. Therefore, the video adaptation network needs to extract the temporal features of the sample video through the temporal dimensionality upscaling layer.

[0072] Specifically, for a pixel region in a frame of a sample video, the temporal dimensionality increase processing layer can obtain temporal dimensionality increase aggregate features based on the pixel regions in other adjacent frames of the sample video that have the same position as the pixel region. For example, a 3D convolution kernel is used to perform a 3D convolution on the sample video to obtain the temporal dimensionality increase aggregate features.

[0073] The spatial dimensionality increase processing layer is used to extract and increase the spatial features of the sample video from the aggregated video features to be increased in dimension. A spatial feature represents the visual effect exhibited by a single frame, independent of other frames. Therefore, for a pixel region in a frame of the sample video, the spatial dimensionality increase processing layer can obtain an aggregated spatial dimensionality increase feature based on adjacent pixel regions in the same frame. For example, a two-dimensional convolution kernel is used to perform a two-dimensional convolution on each frame of the sample video to obtain an aggregated spatial dimensionality increase feature.

[0074] It should be noted that the dimensions of the temporal up-scaled aggregated video features obtained by the temporal up-scaled processing layer and the spatial up-scaled aggregated video features obtained by the spatial up-scaled processing layer are consistent. Thus, after obtaining the temporal up-scaled aggregated video features and the spatial up-scaled aggregated video features, they can be fused to obtain the aggregated video features. Specific fusion methods can also include, but are not limited to, splicing, pooling, or matrix addition, and are not limited in this embodiment.

[0075] In the embodiments of this specification, Figure 2 and Figure 3 The model structure shown includes several text adaptation networks and video adaptation networks. Figure 1 When fine-tuning the video language model in step S108, the estimated text and the annotated text can be input into a preset first loss function to obtain a first loss value output by the first loss function. The smaller the difference between the estimated text and the annotated text, the smaller the first loss value, and the greater the difference between the estimated text and the annotated text, the larger the first loss value. The model parameters of each text adaptation network and each video adaptation network in the video language model are adjusted with the goal of at least reducing the first loss value. When adjusting the model parameters of the text adaptation network and the video adaptation network, the model parameters of the shared mapping layer in the corresponding text adaptation network and video adaptation network of the same coding layer must always remain the same.

[0076] In order to further improve the model accuracy of the video language model in the embodiments of this specification, in addition to using the difference between the estimated text and the annotated text as a basis during fine-tuning training, a comparative learning method can also be used to further constrain the video language model that has been added with the text adaptation network and the video adaptation network. Specifically, for each sample text input into the video language model, the description information of each sample text can be obtained, and the description information of each sample text can be input into the comparative encoder to obtain the comparative text features of each sample text output by the comparative encoder. Among them, the comparative encoder described in this specification only includes a text encoding network, which is the original text encoding network of the text encoder of the above-mentioned video language model, that is, a text encoding network without adding any text adaptation network.

[0077] After obtaining the above-mentioned contrasting text features, the contrasting text features of each sample text and the fused text features of each sample text can be input into the second loss function to obtain a second loss value output by the second loss function, wherein the smaller the difference between the contrasting text features and the fused text features of the same sample text, the smaller the second loss value, and the greater the difference between the contrasting text features and the fused text features of the same sample text, the greater the second loss value; the final loss value is determined based on the first loss value and the second loss value, and the model parameters of each text adaptation network and each video adaptation network are adjusted with the goal of reducing the final loss value. In determining the final loss, the first loss value and the second loss value can be weighted according to the preset weight to obtain the final loss value.

[0078] Therefore, the video language model fine-tuned and trained using the above method has two losses to constrain: one is to minimize the difference between the estimated text output by the video language model and the predetermined annotated text; the other is to minimize the difference between the fused text features obtained based on the sample text and through the text encoder with the text adaptation network added, and the contrast text features obtained based on the description information of the sample text and through the original text encoder.

[0079] The descriptive information of the sample text can be obtained using a large language model. Specifically, each sample text is input into a pre-trained large language model, and the large language model outputs the descriptive information of each sample text. Alternatively, the sample text and prompt information can be input into the large language model, causing the large language model to output the descriptive information of the sample text based on the prompt information. The prompt information can be used to prompt the large language model to output a desired type of descriptive information.

[0080] For example, if the sample text is "slam dunk", then "slam dunk" can be input into the large language model, and "Please use 3 sentences to describe the steps of slam dunk in time" can be used as a prompt information to make the large language model output descriptive information with temporal characteristics, or "Please use 3 sentences to describe the visual representation of slam dunk" can be used as a prompt information to make the large language model output descriptive information with spatial characteristics.

[0081] Since the description information of the sample text contains more detailed descriptions than the sample text itself, inputting this description information into the contrastive encoder to obtain contrastive text features can effectively improve the accuracy of the text encoder by constraining the fused text features obtained by encoding the sample text itself. Furthermore, since the video encoder and text encoder have already been associated and modeled using the same shared mapping layer in the video adaptation network and text adaptation network, contrastive learning can indirectly improve the accuracy of the video encoder, further improving the accuracy of the entire video language model.

[0082] It should be noted that the contrastive encoder described above is only used to fine-tune the video language model. After fine-tuning the video language model using the above method, the fine-tuned video language model (excluding the contrastive encoder) can be directly applied to the corresponding fine-grained classification task. The following example uses the fine-tuned video language model to identify the risk type of a video.

[0083] When the fine-tuned video language model is applied to identify the risk type of a video, the video to be identified and the risk type texts used to describe each risk type can be obtained first, the video to be identified is input into the video encoder, the ordinary video features of the video to be identified are obtained through the video encoding network, and the aggregated video features of the video to be identified are obtained through the shared mapping layer in the video adaptation network; and each risk type text is input into the text encoder, the ordinary text features of each risk type text are obtained through the text encoding network, and the aggregated text features of each risk type text are obtained through the shared mapping layer in the text adaptation network; based on the ordinary video features and aggregated video features of the video to be identified, as well as the ordinary text features and aggregated text features of each risk type text, the risk type text that matches the video to be identified is determined in each risk type text, and it is determined that the risk type of the video to be identified is the risk type corresponding to the matched risk type text.

[0084] It can be seen that the method of applying the fine-tuned video language model to the classification task is similar to Figure 1 The steps S100 to S106 shown are identical and will not be described in detail here.

[0085] The above is a fine-tuning training of a video language model provided in an embodiment of this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0086] Figure 4 A schematic diagram of a fine-tuning training device for a video language model provided in an embodiment of this specification, wherein the video language model includes a text encoder and a video encoder, wherein the text encoder includes a text encoding network and a text adaptation network, and wherein the video encoder includes a video encoding network and a video adaptation network, wherein the text adaptation network and the video adaptation network include the same shared mapping layer; the device includes:

[0087] The sample acquisition module 401 is used to acquire a sample video and a plurality of sample texts, and determine the annotated text in each sample text that matches the sample video;

[0088] A video encoding module 402 is configured to input the sample video into the video encoder, obtain common video features of the sample video through the video encoding network, and obtain aggregated video features of the sample video through a shared mapping layer in the video adaptation network;

[0089] A text encoding module 403 is configured to input each sample text into the text encoder, obtain the common text features of each sample text through the text encoding network, and obtain the aggregated text features of each sample text through the shared mapping layer in the text adaptation network;

[0090] A matching module 404 is configured to determine, from each sample text, a text that matches the sample video based on the common video features and aggregated video features of the sample video and the common text features and aggregated text features of each sample text, as an estimated text;

[0091] The fine-tuning training module 405 is used to fine-tune the text adaptation network and the video adaptation network according to the estimated text and the annotated text.

[0092] Optionally, the matching module 404 is specifically used to fuse the common video features and aggregated video features of the sample video to obtain fused video features; and, for each sample text, fuse the common text features and aggregated text features of the sample text to obtain fused text features of the sample text; and determine the text that matches the sample video based on the fused video features of the sample video and the fused text features of each sample text.

[0093] Optionally, the text encoding network comprises N text encoding layers cascaded in sequence, and the video encoding network comprises N video encoding layers cascaded in sequence; the number of the text encoding layers is the same as the number of the video encoding layers;

[0094] The ijth text coding layer of the text coding network is used as the designated text coding layer, and each designated text coding layer corresponds to a text adaptation network; the ijth video coding layer of the video coding network is used as the designated video coding layer, and each designated video coding layer corresponds to a video adaptation network; i and j are both greater than 1 and less than N, and i is less than j;

[0095] The text adaptation network corresponding to the nth text coding layer of the text coding network and the video adaptation network corresponding to the nth video coding layer of the video coding network include the same shared mapping layer, where n is greater than or equal to i and less than or equal to j.

[0096] Optionally, the matching module 404 is specifically used to, for each specified video coding layer, input the output features of the previous video coding layer of the specified video coding layer into the specified video coding layer and the video adaptation network corresponding to the specified video coding layer, respectively, to obtain the ordinary video features output by the specified video coding layer and the aggregated video features output by the video adaptation network corresponding to the specified video coding layer; fuse the ordinary video features output by the specified video coding layer and the aggregated video features output by the video adaptation network corresponding to the specified video coding layer to obtain the output features of the specified video coding layer, until the output features of the last video coding layer of the video coding network are obtained, and the output features of the last video coding layer are used as the fused video features.

[0097] Optionally, the matching module 404 is specifically used to, for each specified text coding layer, input the output features of the previous text coding layer of the specified text coding layer into the specified text coding layer and the text adaptation network corresponding to the specified text coding layer, respectively, to obtain the ordinary text features output by the specified text coding layer and the aggregated text features output by the text adaptation network corresponding to the specified text coding layer; fuse the ordinary text features output by the specified text coding layer and the aggregated text features output by the text adaptation network corresponding to the specified text coding layer to obtain the output features of the specified text coding layer, until the output features of the last text coding layer of the text coding network are obtained, and the output features of the last text coding layer are used as the fused text features.

[0098] Optionally, the video adaptation network corresponding to each designated video coding layer further includes a dimensionality reduction processing layer, a temporal dimensionality increase processing layer, and a spatial dimensionality increase processing layer;

[0099] The matching module 404 is specifically used to input the output features of the previous video coding layer into the dimensionality reduction processing layer to obtain the output features after dimensionality reduction; input the output features after dimensionality reduction into the shared mapping layer in the video adaptation network corresponding to the specified video coding layer to obtain the aggregated video features to be upgraded; input the aggregated video features to be upgraded into the time dimensionality upgrading processing layer and the space dimensionality upgrading processing layer respectively to obtain the time dimensionality upgrading aggregated video features output by the time dimensionality upgrading processing layer and the space dimensionality upgrading aggregated video features output by the space dimensionality upgrading processing layer; fuse the time dimensionality upgrading aggregated video features and the space dimensionality upgrading aggregated video features to obtain the aggregated video features output by the video adaptation network corresponding to the specified video coding layer.

[0100] Optionally, the text adaptation network corresponding to each specified text encoding layer further includes a dimensionality reduction processing layer and a dimensionality increase processing layer;

[0101] The matching module 404 is specifically used to input the output features of the previous text encoding layer into the dimensionality reduction processing layer to obtain the output features after dimensionality reduction; input the output features after dimensionality reduction into the shared mapping layer in the text adaptation network corresponding to the specified text encoding layer to obtain the aggregated text features to be upgraded; input the aggregated text features to be upgraded into the dimensionality increasing processing layer to obtain the aggregated text features output by the text adaptation network corresponding to the specified text encoding layer.

[0102] Optionally, the fine-tuning training module 405 is specifically used to input the estimated text and the annotated text into a preset first loss function to obtain a first loss value output by the first loss function, wherein the smaller the difference between the estimated text and the annotated text, the smaller the first loss value; at least with the goal of reducing the first loss value, the model parameters of each text adaptation network and each video adaptation network are adjusted.

[0103] Optionally, the fine-tuning training module 405 is also used to obtain descriptive information of each sample text; input the descriptive information of each sample text into a comparison encoder to obtain the comparison text features of each sample text output by the comparison encoder; the comparison encoder only includes the text encoding network; input the comparison text features of each sample text and the fusion text features of each sample text into a second loss function to obtain a second loss value output by the second loss function, wherein the smaller the difference between the comparison text features and the fusion text features of the same sample text, the smaller the second loss value; determine the final loss value based on the first loss value and the second loss value; and adjust the model parameters of each text adaptation network and each video adaptation network with the goal of reducing the final loss value.

[0104] Optionally, the fine-tuning training module 405 is specifically configured to input each sample text into a pre-trained large language model to obtain description information of each sample text output by the large language model.

[0105] This specification also provides a computer-readable storage medium, wherein the storage medium stores a computer program, which can be used to perform the above-mentioned Figure 1 Provides a fine-tuning training method for the video language model.

[0106] based on Figure 1 The fine-tuning training method of the video language model shown in the embodiment of this specification also provides Figure 5 The structural diagram of the electronic device shown in FIG. Figure 5 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The fine-tuning training method of the video language model.

[0107] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A fine-tuning training method for a video language model, wherein the video language model comprises a text encoder and a video encoder, wherein the text encoder comprises a text encoding network and a text adaptation network, and wherein the video encoder comprises a video encoding network and a video adaptation network, wherein the text adaptation network and the video adaptation network comprise the same shared mapping layer; the method comprising: Obtaining a sample video and a plurality of sample texts, and determining an annotated text in each sample text that matches the sample video; Inputting the sample video into the video encoder, obtaining common video features of the sample video through the video encoding network, and obtaining aggregated video features of the sample video through the shared mapping layer in the video adaptation network; Furthermore, each sample text is input into the text encoder, the common text features of each sample text are obtained through the text encoding network, and the aggregated text features of each sample text are obtained through the shared mapping layer in the text adaptation network; According to the common video features and aggregated video features of the sample video, and the common text features and aggregated text features of each sample text, determining, in each sample text, a text that matches the sample video as an estimated text; Fine-tuning the text adaptation network and the video adaptation network is performed based on the estimated text and the annotated text.

2. The method according to claim 1, further comprising determining, in each sample text, a text matching the sample video based on the common video features and aggregated video features of the sample video and the common text features and aggregated text features of each sample text, the text matching the sample video, comprising: Fusing the common video features and the aggregated video features of the sample video to obtain fused video features; And, for each sample text, the ordinary text features and the aggregated text features of the sample text are fused to obtain the fused text features of the sample text; The text matching the sample video is determined based on the fused video features of the sample video and the fused text features of each sample text.

3. The method of claim 2, wherein the text encoding network comprises N text encoding layers cascaded in sequence, and the video encoding network comprises N video encoding layers cascaded in sequence; the number of the text encoding layers is the same as the number of the video encoding layers; The ijth text coding layer of the text coding network is used as the designated text coding layer, and each designated text coding layer corresponds to a text adaptation network; the ijth video coding layer of the video coding network is used as the designated video coding layer, and each designated video coding layer corresponds to a video adaptation network; i and j are both greater than 1 and less than N, and i is less than j; The text adaptation network corresponding to the nth text coding layer of the text coding network and the video adaptation network corresponding to the nth video coding layer of the video coding network include the same shared mapping layer, where n is greater than or equal to i and less than or equal to j.

4. The method according to claim 3, wherein the common video features and the aggregated video features of the sample video are fused to obtain fused video features, specifically comprising: For each designated video coding layer, the output features of the previous video coding layer of the designated video coding layer are input into the designated video coding layer and the video adaptation network corresponding to the designated video coding layer, respectively, to obtain the common video features output by the designated video coding layer and the aggregated video features output by the video adaptation network corresponding to the designated video coding layer; The ordinary video features output by the specified video coding layer and the aggregated video features output by the video adaptation network corresponding to the specified video coding layer are fused to obtain the output features of the specified video coding layer, until the output features of the last video coding layer of the video coding network are obtained, and the output features of the last video coding layer are used as the fused video features.

5. The method according to claim 3, wherein the ordinary text features and the aggregated text features of the sample text are fused to obtain the fused text features of the sample text, specifically comprising: For each designated text coding layer, the output features of the previous text coding layer of the designated text coding layer are respectively input into the designated text coding layer and the text adaptation network corresponding to the designated text coding layer to obtain the ordinary text features output by the designated text coding layer and the aggregated text features output by the text adaptation network corresponding to the designated text coding layer; The ordinary text features output by the specified text encoding layer and the aggregated text features output by the text adaptation network corresponding to the specified text encoding layer are fused to obtain the output features of the specified text encoding layer, until the output features of the last text encoding layer of the text encoding network are obtained, and the output features of the last text encoding layer are used as the fused text features.

6. The method according to claim 4, wherein the video adaptation network corresponding to each designated video coding layer further comprises a dimensionality reduction processing layer, a temporal dimensionality increase processing layer, and a spatial dimensionality increase processing layer; Obtain the aggregated video features output by the video adaptation network corresponding to the specified video coding layer, specifically including: Inputting the output features of the previous video encoding layer into the dimensionality reduction processing layer to obtain output features after dimensionality reduction; Inputting the output features after dimensionality reduction into the shared mapping layer in the video adaptation network corresponding to the specified video coding layer to obtain the aggregated video features to be upgraded; Inputting the to-be-upscaled aggregated video features into the temporal dimensionality-upscaled processing layer and the spatial dimensionality-upscaled processing layer respectively, obtaining the temporal dimensionality-upscaled aggregated video features output by the temporal dimensionality-upscaled processing layer and the spatial dimensionality-upscaled aggregated video features output by the spatial dimensionality-upscaled processing layer; The temporal dimension-raising aggregated video features and the spatial dimension-raising aggregated video features are fused to obtain the aggregated video features output by the video adaptation network corresponding to the specified video coding layer.

7. The method according to claim 5, wherein the text adaptation network corresponding to each designated text encoding layer further comprises a dimensionality reduction processing layer and a dimensionality increase processing layer; Obtain the aggregated text features output by the text adaptation network corresponding to the specified text encoding layer, specifically including: Inputting the output features of the previous text encoding layer into the dimensionality reduction processing layer to obtain output features after dimensionality reduction; Inputting the output features after dimensionality reduction into the shared mapping layer in the text adaptation network corresponding to the specified text encoding layer to obtain the aggregated text features to be upgraded; The aggregated text features to be upgraded are input into the dimensionality upgrading processing layer to obtain the aggregated text features output by the text adaptation network corresponding to the specified text encoding layer.

8. The method according to claim 3, further comprising fine-tuning the text adaptation network and the video adaptation network based on the estimated text and the annotated text, comprising: Inputting the estimated text and the annotated text into a preset first loss function to obtain a first loss value output by the first loss function, wherein the smaller the difference between the estimated text and the annotated text, the smaller the first loss value; At least with the goal of reducing the first loss value, the model parameters of each text adaptation network and each video adaptation network are adjusted.

9. The method of claim 8, further comprising: Get the description information of each sample text; Inputting the description information of each sample text into a comparison encoder to obtain a comparison text feature of each sample text output by the comparison encoder; The contrastive encoder only includes the text encoding network; At least with the goal of reducing the loss value, the model parameters of each text adaptation network and each video adaptation network are adjusted, specifically including: Inputting the contrasting text features of each sample text and the fused text features of each sample text into a second loss function to obtain a second loss value output by the second loss function, wherein the smaller the difference between the contrasting text features and the fused text features of the same sample text, the smaller the second loss value; determining a final loss value according to the first loss value and the second loss value; With the goal of reducing the final loss value, the model parameters of each text adaptation network and each video adaptation network are adjusted.

10. The method according to claim 9, wherein obtaining description information of each sample text comprises: Each sample text is input into a pre-trained large language model to obtain description information of each sample text output by the large language model.

11. A fine-tuning training device for a video language model, the video language model comprising a text encoder and a video encoder, the text encoder comprising a text encoding network and a text adaptation network, the video encoder comprising a video encoding network and a video adaptation network, the text adaptation network and the video adaptation network comprising the same shared mapping layer; the device comprising: A sample acquisition module, configured to acquire a sample video and a plurality of sample texts, and determine an annotated text in each sample text that matches the sample video; a video encoding module, configured to input the sample video into the video encoder, obtain common video features of the sample video through the video encoding network, and obtain aggregated video features of the sample video through a shared mapping layer in the video adaptation network; A text encoding module, configured to input each sample text into the text encoder, obtain the common text features of each sample text through the text encoding network, and obtain the aggregated text features of each sample text through the shared mapping layer in the text adaptation network; a matching module for determining, from each sample text, a text that matches the sample video as an estimated text based on the common video features and aggregated video features of the sample video and the common text features and aggregated text features of each sample text; A fine-tuning training module is used to fine-tune the text adaptation network and the video adaptation network based on the estimated text and the annotated text.

12. A computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the program.

Images