Video and text based model training method and apparatus

Abstract

The application discloses a model training method and device based on video and text, and the method comprises the following steps: determining a training video and corresponding description text for training a model; performing frame extraction on the training video to obtain a plurality of training video frames corresponding to the training video; inputting the plurality of training video frames and the description text into a video reconstruction prediction model based on a Transformer network structure to perform training, calculating a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training, optimizing model parameters of the video reconstruction prediction model according to the loss function value until convergence is achieved, and obtaining the trained video reconstruction prediction model. It can be seen that the application can utilize the algorithm advantages of the Transformer network structure, so that the model obtained by training can realize the effect of reconstructing a video according to text.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of algorithm model training, and in particular to a model training method and device based on video and text. BACKGROUND

[0002] With the development of algorithm technology, more and more enterprises begin to use algorithm models to perform video-related data prediction tasks, such as video reconstruction. Such tasks require algorithm models to fully extract the features of videos and learn. However, the existing technology does not consider the relationship between the description text and the video, nor does it consider the algorithm advantages of the Transformer network structure for video-related task training. Therefore, the model trained by the existing technology cannot achieve good prediction results. It can be seen that the existing technology has defects and needs to be solved. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a model training method and device based on video and text, which can utilize the algorithm advantages of the Transformer network structure, so that the model trained can achieve the effect of reconstructing a video according to a text.

[0004] To solve the above technical problems, the first aspect of the present application discloses a model training method based on video and text, which comprises:

[0005] determining a training video and a corresponding description text for training a model;

[0006] frame extraction is performed on the training video to obtain a plurality of training video frames corresponding to the training video;

[0007] The plurality of training video frames and the description text are input into a video reconstruction prediction model based on a Transformer network structure for training. In the training, a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input is calculated. The model parameters of the video reconstruction prediction model are optimized according to the loss function value until convergence, and the trained video reconstruction prediction model is obtained.

[0008] As an optional implementation, in the first aspect of the present application, the frame extraction is performed on the training video to obtain a plurality of training video frames corresponding to the training video, comprising:

[0009] determining a first frame extraction interval corresponding to the training video according to a video parameter of the training video;

[0010] frame the training video according to the first frame interval to obtain a plurality of candidate video frames corresponding to the training video;

[0011] As an optional implementation, in the first aspect of the present application, the frame operation on the training video according to the first frame interval to obtain a plurality of candidate video frames corresponding to the training video comprises:

[0012] frame the training video according to the first frame interval to obtain a plurality of candidate video frames corresponding to the training video;

[0013] For any two adjacent candidate video frames, calculate the picture similarity between the two candidate video frames;

[0014] determine whether the picture similarity meets a preset similarity threshold condition;

[0015] If the determination result is yes, determine the two candidate video frames as key video frames;

[0016] determine a plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames.

[0017] As an optional implementation, in the first aspect of the present application, the determination of a plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames comprises:

[0018] frame the other candidate video frames in the plurality of candidate video frames except the key video frames according to a second frame interval to obtain a plurality of extracted video frames; the second frame interval is greater than the first frame interval;

[0019] determine all the key video frames and the extracted video frames as the plurality of training video frames corresponding to the training video.

[0020] As an optional implementation, in the first aspect of the present application, after the frame operation on the training video to obtain a plurality of training video frames corresponding to the training video, the method further comprises:

[0021] determine whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0022] If yes, divide the plurality of training video frames into at least two divided training video frame groups with a video frame number less than or equal to the first frame number threshold; each divided training video frame group is used as single-time input training data when training a video reconstruction prediction model;

[0023] and / or,

[0024] determining whether the number of the plurality of training video frames is less than a preset second frame number threshold value;

[0025] If yes, extracting video frames from the training video to fill into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold value.

[0026] As an optional implementation form, in the first aspect of the present application, before the plurality of training video frames and the description text are input into the video reconstruction prediction model based on the Transformer network structure for training, the method further comprises:

[0027] disordering the arrangement order of the plurality of training video frames arranged in the original time sequence to obtain disordered training video frames; the disordered training video frames are used for input into an encoder of the video reconstruction prediction model; the encoding output data of the disordered training video frames are used for input into a decoder of the video reconstruction prediction model after being restored to the original time sequence;

[0028] and / or,

[0029] determining the mask video frames in the plurality of training video frames;

[0030] deleting the mask video frames, determining the remaining training video frames in the plurality of training video frames as input video frames; the input video frames are used for input into an encoder of the video reconstruction prediction model; the mask video frames are used for input into a decoder of the video reconstruction prediction model together with the encoding output data of the input video frames; the ratio of the number of the mask video frames to the total number of the plurality of training video frames is greater than 50%.

[0031] As an optional implementation form, in the first aspect of the present application, the video reconstruction prediction model comprises an encoder; the encoder comprises a video embedding layer, a text embedding layer, a feature fusion layer and a first Transformer layer; the video embedding layer is used for receiving the training video frames and processing to obtain video features; the text embedding layer is used for receiving the description text and processing to obtain text features; the feature fusion layer is used for fusing the video features and the text features to obtain training features, and inputting the training features into the first Transformer layer.

[0032] As an optional implementation form, in the first aspect of the present application, the loss function value between the plurality of predicted video frames and the plurality of input training video frames is calculated as follows:

[0033] For any of the predicted video frames, a frame loss function value between the predicted video frame and the corresponding training video frame is calculated;

[0034] An average of the frame loss function values of all the predicted video frames is calculated to obtain a loss function value between the predicted video frames and the input training video frames.

[0035] The second aspect of the present application discloses a video and text-based model training device, which comprises:

[0036] A data determination module is configured to determine training videos and corresponding description texts for training a model;

[0037] A frame extraction operation module is configured to perform frame extraction operation on the training videos to obtain a plurality of training video frames corresponding to the training videos;

[0038] A model training module is configured to input the plurality of training video frames and the description texts into a video reconstruction prediction model based on a Transformer network structure for training, calculate a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the input training video frames during the training, optimize model parameters of the video reconstruction prediction model according to the loss function value until convergence, and obtain a trained video reconstruction prediction model.

[0039] As an optional implementation, in the second aspect of the present application, the frame extraction operation module comprises:

[0040] An interval determination unit is configured to determine a first frame extraction interval corresponding to the training videos according to video parameters of the training videos;

[0041] A frame extraction operation unit is configured to perform frame extraction operation on the training videos according to the first frame extraction interval to obtain a plurality of training video frames corresponding to the training videos.

[0042] As an optional implementation, in the second aspect of the present application, the specific manner in which the frame extraction operation unit performs frame extraction operation on the training videos according to the first frame extraction interval to obtain a plurality of training video frames corresponding to the training videos comprises:

[0043] The frame extraction operation unit performs frame extraction operation on the training videos according to the first frame extraction interval to obtain a plurality of candidate video frames corresponding to the training videos;

[0044] For any two adjacent candidate video frames, a picture similarity between the two candidate video frames is calculated;

[0045] It is judged whether the picture similarity meets a preset similarity threshold condition;

[0046] If the determination result is yes, the two candidate video frames are determined as key video frames.

[0047] According to all the key video frames in the plurality of candidate video frames, a plurality of training video frames corresponding to the training video are determined.

[0048] As an optional implementation, in the second aspect of the present application, the specific manner in which the frame extraction operation unit determines the plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames comprises:

[0049] The candidate video frames other than the key video frames are subjected to frame extraction operation according to a second frame extraction interval to obtain a plurality of extracted video frames; the second frame extraction interval is greater than the first frame extraction interval.

[0050] All the key video frames and the extracted video frames are determined as the plurality of training video frames corresponding to the training video.

[0051] As an optional implementation, in the second aspect of the present application, the device further comprises a frame number correction module, which is configured to perform the following steps after the frame extraction operation module performs frame extraction operation on the training video to obtain the plurality of training video frames corresponding to the training video:

[0052] Determine whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0053] If yes, the plurality of training video frames are divided into at least two divided training video frame groups each having a number of video frames less than or equal to the first frame number threshold; each of the divided training video frame groups is used as single-time input training data when training the video reconstruction prediction model;

[0054] and / or,

[0055] Determine whether the number of the plurality of training video frames is less than a preset second frame number threshold;

[0056] If yes, video frames are extracted from the training video and filled into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

[0057] As an optional implementation, in the second aspect of the present application, the device further comprises a preprocessing module, which is configured to perform the following steps before the model training module trains the video reconstruction prediction model based on the plurality of training video frames and the description text:

[0058] Shuffling the arrangement order of the plurality of training video frames arranged in the original time sequence to obtain shuffled training video frames; the shuffled training video frames are used as input to an encoder of the video reconstruction prediction model; the encoder outputs data according to the encoding of the shuffled training video frames, which are used as input to a decoder of the video reconstruction prediction model after being restored to the original time sequence;

[0059] and / or,

[0060] Determining a mask video frame in the plurality of training video frames;

[0061] Deleting the mask video frame, and determining the remaining training video frames of the plurality of training video frames as input video frames; the input video frames are used as input to an encoder of the video reconstruction prediction model; the mask video frame is used as input to a decoder of the video reconstruction prediction model together with the encoding output data of the input video frames according to the encoder; the ratio of the number of the mask video frame to the total number of the plurality of training video frames is greater than 50%.

[0062] As an optional implementation, in the second aspect of the present application, the video reconstruction prediction model comprises an encoder; the encoder comprises a video embedding layer, a text embedding layer, a feature fusion layer and a first Transformer layer; the video embedding layer is used to receive the training video frames and process to obtain video features; the text embedding layer is used to receive the description text and process to obtain text features; the feature fusion layer is used to fuse the video features and the text features to obtain training features, and input the training features to the first Transformer layer.

[0063] As an optional implementation, in the second aspect of the present application, the loss function value between the plurality of predicted video frames and the input plurality of training video frames is calculated as follows:

[0064] For any of the predicted video frames, the frame loss function value between the predicted video frame and the corresponding training video frame is calculated;

[0065] The average value of the frame loss function values of all the predicted video frames is calculated to obtain the loss function value between the plurality of predicted video frames and the input plurality of training video frames.

[0066] The third aspect of the present application discloses another video and text-based model training device, which comprises:

[0067] A memory storing executable program codes;

[0068] A processor coupled with the memory;

[0069] The processor invokes the executable program code stored in the memory to perform part or all of the steps in the video and text-based model training method disclosed in the first aspect of the embodiment of the present application.

[0070] The fourth aspect of the present application discloses a computer storage medium, which stores computer instructions, when invoked, for performing part or all of the steps in the video and text-based model training method disclosed in the first aspect of the embodiment of the present application.

[0071] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0072] In the embodiment of the present application, a training video and corresponding description text used for training a model are determined; frame extraction is performed on the training video to obtain a plurality of training video frames corresponding to the training video; the plurality of training video frames and the description text are input into a video reconstruction prediction model based on a Transformer network structure for training, a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training is calculated, and model parameters of the video reconstruction prediction model are optimized according to the loss function value until convergence, thereby obtaining the trained video reconstruction prediction model. As can be seen, the video and text corresponding to the video obtained by frame extraction using the training video can be used to train an algorithm model based on a Transformer network structure, so that the advantages of the algorithm based on the Transformer network structure are utilized, and the model obtained by training can realize the effect of reconstructing a video according to text. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0074] Figure 1 is a flowchart of a video and text-based model training method disclosed by the embodiment of the present application;

[0075] Figure 2 is a flowchart of another video and text-based model training method disclosed by the embodiment of the present application;

[0076] Figure 3 is a structural diagram of a video and text-based model training device disclosed by the embodiment of the present application;

[0077] Figure 4is a structural schematic view of another video and text-based model training device disclosed by the embodiment of the present application.

[0078] Figure 5 is a structural schematic view of still another video and text-based model training device disclosed by the embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor fall within the scope of protection of the present application.

[0080] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or end including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product, or end.

[0081] In this document, the reference to "embodiment" means that the particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with other embodiments.

[0082] The present application discloses a video and text-based model training method and device, which can train an algorithm model based on a Transformer network structure by using video frames obtained by frame extraction from a training video and description text corresponding to the video, so as to utilize the algorithm advantages of the Transformer network structure, and enable the trained model to realize the effect of reconstructing a video according to text. The following will be described in detail respectively.

[0083] Embodiment one

[0084] Please refer to Figure 1 , Figure 1 is a flowchart of a video and text-based model training method disclosed by the embodiment of the present application. In the method, Figure 1The described method is applied to a video data processing device, which can be a corresponding processing terminal, processing equipment or processing server, and the server can be a local server or a cloud server, and the embodiments of the present application are not limited. As shown in the figure, the video and text-based model training method can include the following operations: Figure 1 The video and text-based model training method can include the following operations:

[0085] 101. Determine the training video and the corresponding description text for training the model.

[0086] Optionally, the description text is related to the content of the training video, which can be written by an operator according to the content of the training video, or automatically predicted and generated by other video content algorithm, and the present application is not limited.

[0087] Optionally, the video content of the training video can be determined according to the target processing video content of the model to be trained, and it can also include various types of video content to improve the adaptive characteristics of the trained model. Optionally, the video content of the training video can be a continuous action or a continuous scene. Since the scheme of the present application adds the auxiliary of the description text, the picture lens or scene in the video content of the training video can exist switching.

[0088] 102. Frame extraction operation is performed on the training video to obtain a plurality of training video frames corresponding to the training video.

[0089] 103. Input the plurality of training video frames and the description text into the video reconstruction prediction model based on the Transformer network structure for training. In the training, the loss function value between the plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input is calculated, and the model parameters of the video reconstruction prediction model are optimized according to the loss function value until convergence, thereby obtaining the trained video reconstruction prediction model.

[0090] Optionally, the loss function value can be an L1 loss function value or an L2 loss function value, or other loss function values suitable for calculating the similarity between images, and the present application is not limited.

[0091] As can be seen, the method described in the embodiments of the present application can utilize the video frames obtained by frame extraction of the training video and the description text corresponding to the video to train the algorithm model based on the Transformer network structure, thereby utilizing the algorithm advantages of the Transformer network structure, so that the trained model can realize the effect of reconstructing the video according to the text.

[0092] As an optional implementation, the video reconstruction prediction model comprises an encoder and a decoder, wherein the encoder is configured to extract features, and the decoder is configured to restore the features. Optionally, the encoder comprises a video embedding layer, a text embedding layer, a feature fusion layer, and a first Transformer layer. The video embedding layer is configured to receive a training video frame and process the training video frame to obtain video features, the text embedding layer is configured to receive a description text and process the description text to obtain text features, the feature fusion layer is configured to fuse the video features and the text features to obtain training features, and the training features are input into the first Transformer layer.

[0093] The video embedding layer comprises a two-dimensional convolution layer and / or a three-dimensional convolution layer.

[0094] Optionally, a two-dimensional convolutional patch embedding layer can be constructed as the video embedding layer, which is configured to replace high-dimensional original image features with a lower-dimensional vector. Optionally, the patch embedding layer is composed of one convolutional layer, the size of the convolution kernel is equal to the size of the step, and the number of output channels is 768 (or other numbers).

[0095] Optionally, a three-dimensional convolutional video patch embedding layer, i.e., a 3D patch embedding layer, can also be constructed as the video embedding layer. The three-dimensional convolutional layer is configured to perform convolution in the spatial and temporal dimensions, thereby extracting additional correlation features between the previous and subsequent frames. Specifically, in order to ensure that each frame has features related to the previous and subsequent frames, the size of the convolution step is not necessarily equal to the size of the convolution depth, i.e., the step size can be equal to 1. For example, the convolution depth is equal to 2, and the step size is equal to 1. In this case, the first frame and the second frame are convolved to obtain a feature vector, and then the second frame and the third frame are also convolved to obtain a feature vector. If the convolution depth is equal to 2 and the step size is also equal to 2, the first frame and the second frame are convolved to obtain a feature vector, and then the convolution feature between the third frame and the fourth frame is calculated without calculating the feature between the second frame and the third frame.

[0096] Specifically, the video embedding layer converts the input video frames into corresponding feature vectors. Taking a specific implementation of the two-dimensional convolutional patch embedding layer as an example, for example, 30 video frames, each frame has an image size of 224*224, and the dimension is (30, 3, 224, 224), 3 represents RGB three channels, the convolution kernel size is (16, 16), and the step size is 16. After conversion, it becomes (30, 768, 14, 14), and then it is transformed into a feature vector (30, 768, 196). Subsequently, the order of the dimensions is adjusted, and a class label is added at the initial position of the 196 dimension to process downstream tasks such as classification. Finally, the dimension of the obtained feature is (30, 197, 768).

[0097] Optionally, the text embedding layer can include a Tokenizer module and an Embedding module, wherein the Tokenizer module is configured to convert the text characters of the input description text into token indexes, and then the Embedding module encodes the token indexes into feature vectors.

[0098] Optionally, the feature fusion layer can include a first fully connected layer, a second fully connected layer, a feature fusion module, and a GELU activation layer. The input feature dimensions of the first fully connected layer and the second fully connected layer need to be equal in order to fuse the features of the two. The first fully connected layer is configured to receive the output features of the video embedding layer and convert the feature space, and the second fully connected layer is configured to receive the output features of the text embedding layer and convert the feature space. The feature fusion module can use feature addition mean fusion, splicing fusion, or other fusion methods to fuse the features of the first fully connected layer and the second fully connected layer, and then pass through the GELU activation layer for transformation to obtain the final fused features.

[0099] In a specific scheme, in order to better fuse the video frame features and the text features together, two fully connected layers are respectively constructed for feature space conversion. The feature vector dimension of the video frame is (30, 197, 768), and the text feature does not add a position mark. When the maximum text length is 196, the corresponding feature vector dimension is (1, 196, 768). The above two features are spliced in the first dimension (starting from 0), and then the dimension order of the spliced features is transformed, i.e., (30, 197+196, 768) -> (30, 768, 197+196). Subsequently, the features are input into a fully connected layer, and the output dimension is (30, 768, 197). Finally, a GELU activation layer is used for nonlinear transformation to obtain the fused features, and the dimension is transformed to (30, 197, 768) again.

[0100] Optionally, the first Transformer layer includes a plurality of stacked Transformer modules, and the structure thereof can refer to the encoder structure of the VIT (Vision Transformer) network.

[0101] As can be seen, by implementing the optional embodiment, an encoder structure can be constructed to sufficiently extract and fuse the features of the video frame and the description text, so that the prediction effect of the model is better.

[0102] As an optional embodiment, the decoder includes a first fully connected layer, a second Transformer layer, and a second fully connected layer.

[0103] Specifically, the decoder first adopts a fully connected layer to fuse and convert the features of the encoder. Then, the decoder is also composed of multiple Transformer modules, and finally, a fully connected layer is adopted to generate the pixel-level video frame.

[0104] It can be seen that by implementing the optional embodiment, a decoder structure can be constructed which can fully reconstruct the features of the video frame, so that the prediction effect of the model is better.

[0105] As an optional embodiment, before the step of inputting the multiple training video frames and the description text into the video reconstruction prediction model based on the Transformer network structure for training, the method further comprises:

[0106] The multiple training video frames are normalized, so that the pixel values of the pixel points in each training video frame are in a preset pixel value interval.

[0107] Preferably, the original input video frame is normalized before being processed by the encoder to obtain the features, so that the pixel value range of the image is transformed to [0, 1]. Therefore, the pixel value range of the video frame output by the decoder is also [0, 1], which can effectively improve the calculation efficiency and training efficiency.

[0108] As an optional embodiment, before the step of training the video reconstruction prediction model based on the Transformer network structure according to the multiple training video frames, the method further comprises:

[0109] The arrangement order of the multiple training video frames arranged in the original time sequence is disturbed to obtain disordered training video frames.

[0110] The disordered training video frames are used for input to the encoder of the video reconstruction prediction model. Specifically, the encoding output data of the encoder according to the disordered training video frames is used for input to the decoder of the video reconstruction prediction model after being restored to the original time sequence.

[0111] Through the above setting, the video frames can be disturbed to train the model, which can effectively improve the prediction effect of the trained model.

[0112] As an optional embodiment, before the step of inputting the multiple training video frames and the description text into the video reconstruction prediction model based on the Transformer network structure for training, the method further comprises:

[0113] The mask video frame in the multiple training video frames is determined;

[0114] The mask video frame is deleted, and the remaining training video frames of the plurality of training video frames are determined as input video frames.

[0115] The input video frame is used for input to an encoder of a video reconstruction prediction model, and specifically, the mask video frame is used for input to a decoder of the video reconstruction prediction model together with encoded output data of the input video frame.

[0116] Specifically, the ratio of the number of mask video frames to the total number of the plurality of training video frames is greater than 50%. The purpose of this setting is that, due to the constraint of the text description information, the Mask proportion of the video frame can be increased, the larger Mask proportion reduces the size of the input feature vector, thereby greatly reducing the parameter quantity and the operation quantity, and reducing the performance requirement of the related hardware device. At the same time, the larger Mask proportion will not cause training abnormalities, but will make the model reconstruct more rich video effects based on the existing video frames according to the text description.

[0117] For example, taking the above specific implementation as an example, there are a total of 30 video frames, corresponding to a feature vector of (30, 197, 768). After 50% Mask operation, the video frame order is randomly disturbed, and a feature vector with a dimension of (15, 197, 768) is obtained. The text description corresponds to a feature vector of (1, 196, 768). Then, the feature fusion is performed according to the fusion method proposed above.

[0118] As an optional implementation, the loss function value between the plurality of predicted video frames and the input plurality of training video frames is calculated as follows:

[0119] For any predicted video frame, the frame loss function value between the predicted video frame and the corresponding training video frame is calculated.

[0120] The average value of the frame loss function values of all predicted video frames is calculated to obtain the loss function value between the plurality of predicted video frames and the input plurality of training video frames.

[0121] Specifically, the SmoothL1 loss between the original video frame received by the encoder and the corresponding output video frame of the decoder can be calculated, the loss between each frame is calculated, and then the average value is calculated as the loss of video frame reconstruction. Since the above method has integrated the text features and the video frame features, the loss part does not need to use an additional video text semantic loss, but only needs to calculate the loss based on the original video frame and the corresponding decoded video frame.

[0122] Preferably, the model trained through the above steps can generate more creative video effects in the existing few video frames according to the text description. Moreover, the performance requirement of the hardware device in the model training process can be reduced due to the use of a larger Mask ratio.

[0123] Embodiment Two

[0124] Referring to Figure 2 , Figure 2 is another flowchart of a video and text based model training method according to an embodiment of the present application. In the flowchart, Figure 2 The method described in the flowchart can be applied to a video data processing device, which can be a corresponding processing terminal, processing device or processing server, and the server can be a local server or a cloud server, and the present application does not limit the same. As shown in Figure 2 The video and text based model training method can include the following operations:

[0125] 201. Determine a training video and corresponding description text for training a model.

[0126] 202. Determine a first frame extraction interval corresponding to the training video according to a video parameter of the training video.

[0127] Optionally, the video parameter can include a picture change parameter and / or a video scene parameter, which can be used to represent the complexity of the video content. When the complexity is high, the first frame extraction interval should be shortened to obtain more video frames, so that the video frames can fully reflect the video content. When the complexity is low, the first frame extraction interval can be appropriately increased to obtain fewer video frames.

[0128] 203. Perform a frame extraction operation on the training video according to the first frame extraction interval to obtain a plurality of training video frames corresponding to the training video.

[0129] Optionally, the frame extraction interval described in the present application can be a time interval or a frame number interval.

[0130] Optionally, a plurality of frame extraction time points can be determined according to a time sequence or a frame number sequence with the first frame extraction interval as the interval, and then the video frames corresponding to the frame extraction time points in the training video are obtained to obtain the plurality of training video frames corresponding to the training video. It should be noted that the time interval or frame number interval between adjacent video frames in the plurality of training video frames is not necessarily the first frame extraction interval described above, because when the frame extraction is performed according to the time sequence or the frame number interval, the last frame can be directly determined as a training video frame when the interval at the end is insufficient.

[0131] Optionally, after obtaining multiple training video frames corresponding to the training video, the multiple training video frames can be bound to and saved with the identifier of the training video (such as storage path, video ID, etc.) for subsequent training.

[0132] 204. Input multiple training video frames and descriptive text into the video reconstruction prediction model based on the Transformer network structure for training. During training, calculate the loss function values ​​between the multiple predicted video frames output by the video reconstruction prediction model and the multiple training video frames input. Optimize the model parameters of the video reconstruction prediction model based on the loss function values ​​until convergence, and obtain the trained video reconstruction prediction model.

[0133] For the specific technical details and explanations of technical terms in steps 201 and 204 above, please refer to the descriptions of steps 101 and 103 in Implementation 1, which will not be repeated here.

[0134] As can be seen, implementing the method described in the embodiments of the present invention can determine a reasonable frame extraction interval, thereby extracting appropriate video frames that can represent the video content, resulting in better prediction performance of the trained model.

[0135] As an optional implementation, the first frame extraction interval includes multiple different first frame extraction intervals. Correspondingly, the step described above, performing frame extraction on the training video according to the first frame extraction interval to obtain multiple training video frames corresponding to the training video, may include:

[0136] The training video is subjected to frame extraction operations at multiple different frame extraction intervals to obtain multiple training video frame groups corresponding to the training video.

[0137] Optionally, each group of training video frames is used as training data as a single input when training the video reconstruction prediction model.

[0138] Optionally, each group of training video frames may include multiple training video frames.

[0139] Specifically, to augment the dataset, the same training video may be sampled at various intervals, such as 5 frames per second and 2 frames per second, resulting in a difference in the speed of the video frame sequence. Optionally, frame sampling of the training video can be performed before the start of the entire training process. Early frame sampling can significantly reduce training time. If dynamic frame sampling intervals are used to sample short videos during the training phase, although this can greatly enrich the dataset and achieve data augmentation, the frame sampling speed is often slow, requiring re-sampling for each training iteration, which undoubtedly severely hinders the entire training process.

[0140] As an optional implementation, in the step of frame extracting the training video according to the first frame extraction interval to obtain a plurality of candidate video frames corresponding to the training video, the step includes:

[0141] frame extracting the training video according to the first frame extraction interval to obtain a plurality of candidate video frames corresponding to the training video;

[0142] calculating a picture similarity between any two adjacent candidate video frames;

[0143] judging whether the picture similarity meets a preset similarity threshold condition;

[0144] if the judging result is yes, determining the two candidate video frames as key video frames;

[0145] determining a plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames.

[0146] Optionally, the picture similarity can be a structural similarity parameter or a cosine similarity parameter. Optionally, the similarity threshold condition can be that the picture similarity is greater than a certain similarity threshold, at this time, the two candidate video frames are not similar, and it can be considered that a picture switching occurs between the two frames, and the two frames are retained as critical key frames.

[0147] As an optional implementation, in the step of determining a plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames, the step includes:

[0148] frame extracting other candidate video frames in the plurality of candidate video frames except the key video frames according to a second frame extraction interval to obtain a plurality of extracted video frames; wherein the second frame extraction interval is greater than the first frame extraction interval;

[0149] determining all the key video frames and the extracted video frames as the plurality of training video frames corresponding to the training video.

[0150] Optionally, the determination of the second frame extraction interval can be:

[0151] determining a total frame number of all the other candidate video frames;

[0152] determining the second frame extraction interval according to the total frame number and a preset frame number-interval corresponding relationship.

[0153] wherein the second frame extraction interval is proportional to the total frame number, that is, the more the total frame number, that is, the more the remaining video content, and therefore the larger the second frame extraction interval, and the fewer the frame extraction quantity, because at this time, for the video with more content but less picture transformation, too many frames do not need to be extracted, and vice versa.

[0154] Optionally, the plurality of training video frames corresponding to the training video are determined according to all the key video frames in the plurality of candidate video frames, and the plurality of training video frames can also include:

[0155] For the candidate video frames other than the key video frames in the plurality of candidate video frames, the picture similarity between at least two other candidate video frames is calculated;

[0156] And the candidate video frames that satisfy the similarity threshold condition between each other are retained to determine the plurality of training video frames corresponding to the training video.

[0157] Specifically, the video can be first framed with a smaller first frame extraction interval, for example, if the frame rate of the video is 30, a frame extraction interval of 10 frames per second is selected, and 5 seconds of video can obtain 50 corresponding frames, then the picture similarity between each two consecutive frames in the 50 frames is calculated, when the similarity is greater than a set maximum threshold m, it is considered that a picture switch occurs between the two frames, and the two frames are retained as critical key frames. After all the key frames in the video that occur picture switching are retained by the above method, the continuous frames other than the key frames are screened and framed at equal intervals, at this time the second frame extraction interval can be determined according to the final sequence length, or it can be adaptively selected by calculating the picture similarity between the continuous frames, for example, if the similarity between the first frame and the second frame is less than a set minimum threshold n, it is considered that the difference between the two pictures is small and the picture redundancy is high, then only the first frame is retained and the second frame is discarded; the third frame is compared with the first frame, as long as it is less than the set threshold n, the latter frame is discarded, until the latter frame is greater than the minimum threshold n or the critical key frame, and then it is retained.

[0158] The key frame selection method for picture switching proposed above is to enable the model to learn the feature change process of the picture switching, so that the finally reconstructed video can also exhibit a certain picture switching special effect or picture continuity.

[0159] As an optional implementation, in the above step, the first frame extraction interval corresponding to the training video is determined according to the video parameters of the training video, which includes:

[0160] The picture change parameter of the training video is determined;

[0161] The first frame extraction interval corresponding to the training video is determined according to the picture change parameter and the preset parameter threshold condition.

[0162] Optionally, the picture change parameter can be the optical flow value parameter between different frames of the training video, for example, the average value or the maximum value or the weighted average value of the optical flow motion amount between all adjacent frames. Correspondingly, the parameter threshold condition can be the optical flow value threshold condition.

[0163] Preferably, the optical flow value between each frame can be calculated, and the first frame skipping interval is determined by limiting the optical flow movement amount between each frame, such as setting a threshold value of the optical flow change amount, and after statistics, it is found that the optical flow change amount of every K frames just exceeds the threshold value, and then the first frame skipping interval can be set as K frames. If the optical flow movement amount between each frame is larger, a smaller first frame skipping interval can be selected.

[0164] It can be seen that by implementing the optional embodiment, a more reasonable frame skipping interval can be determined according to the picture change parameter, so that the video content of the video frame can be reasonably and efficiently determined, thereby improving the training efficiency of the model.

[0165] As an optional embodiment, in the above step, the first frame skipping interval corresponding to the training video is determined according to the video parameter of the training video, comprising:

[0166] determining the video scene parameter of the training video;

[0167] determining the first frame skipping interval corresponding to the training video according to the video scene parameter and the preset scene frame skipping corresponding relationship.

[0168] Optionally, the video scene parameter can be the scene type of the training video. Optionally, the scene frame skipping corresponding relationship is used to indicate the first frame skipping interval corresponding to different types of scenes. For example, a short video mainly shows some changes of human body, such as a person's slam dunk action, the whole action speed is fast, and the change between each frame is relatively obvious, so a smaller frame skipping interval can be selected. If the content change speed of the short video is slow or has certain rules, such as a car slowly driving on a mountain path, a dashcam records the road conditions in front and the trees on both sides move regularly towards the lens; such regular or slow change can select a larger frame skipping interval. Specifically, for different task scenes, the frame skipping interval is different.

[0169] Optionally, the number of different types of scenes appearing in the training video can be used to indicate the degree of scene change or the complexity of video content. Optionally, the scene frame skipping corresponding relationship is used to indicate the frame skipping interval corresponding to the number of different types of scenes. Generally speaking, the frame skipping interval is inversely proportional to the number of scenes, that is, the more the number of scenes, the more the scene changes, and the more complex the content, at this time, the frame skipping interval is smaller, and the number of video frames obtained is more, and vice versa.

[0170] It can be seen that by implementing the optional embodiment, a more reasonable frame skipping interval can be determined according to the video scene parameter, so that the video content of the video frame can be reasonably and efficiently determined, thereby improving the training efficiency of the model.

[0171] As an optional implementation, after the step of performing frame extraction on the training video according to the first frame extraction interval to obtain a plurality of training video frames corresponding to the training video, the method further comprises:

[0172] determining whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0173] if yes, dividing the plurality of training video frames into at least two divided training video frame groups each having a number of video frames less than or equal to the first frame number threshold.

[0174] Each divided training video frame group is used as single-time input training data when training the video reconstruction prediction model.

[0175] Specifically, after frame extraction is performed on the video, the final number of frames is counted. In order to avoid excessive memory occupation during training, a first frame number threshold, such as 30 frames, is set. When the result of frame extraction of a certain video is 54, the first 30 frames are taken as a sequence, and the last 24 frames are taken as a second sequence to divide the single-time input training data.

[0176] It can be seen that, by implementing the optional implementation, the plurality of training video frames can be divided into at least two divided training video frame groups each having a number of video frames less than or equal to the first frame number threshold, so that the single-time input training data can be reasonably and efficiently determined to reduce the training cost and improve the training efficiency of the model.

[0177] As an optional implementation, after the step of performing frame extraction on the training video according to the first frame extraction interval to obtain a plurality of training video frames corresponding to the training video, the method further comprises:

[0178] determining whether the number of the plurality of training video frames is less than a preset second frame number threshold;

[0179] if yes, extracting video frames from the training video and filling into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

[0180] Specifically, after frame extraction is performed on the video, the final number of frames is counted. In order to avoid that the number of frames of some videos is too small to achieve the expected training effect, a second frame number threshold, such as 30 frames, is set. If the total number of frames of a video is less than 30, such as 24 frames in total, 6 frames are randomly selected and copied and inserted in the original time order to complete 30 frames.

[0181] It can be seen that by implementing the optional embodiment, video frames can be extracted from the training video to fill in the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold, so that the amount of training data input at a time can be reasonably and efficiently completed, and the training efficiency and effect of the model can be improved.

[0182] Embodiment three

[0183] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of a model training device based on video and text disclosed by the embodiment of the application. Wherein, Figure 3 The described device can be applied to a corresponding video data processing device, which can be a corresponding processing terminal, processing equipment or processing server, and the server can be a local server or a cloud server, and the embodiment of the application does not limit it. As Figure 3 indicated, the device can include:

[0184] The data determination module 301 is configured to determine a training video and corresponding description text for training a model.

[0185] The frame extraction operation module 302 is configured to perform frame extraction operation on the training video to obtain a plurality of training video frames corresponding to the training video.

[0186] The model training module 303 is configured to input the plurality of training video frames and the description text into a video reconstruction prediction model based on a Transformer network structure for training, calculate a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training, optimize model parameters of the video reconstruction prediction model according to the loss function value until convergence, and obtain a trained video reconstruction prediction model.

[0187] As an optional embodiment, as Figure 4 indicated, the frame extraction operation module 302 includes:

[0188] The interval determination unit 3021 is configured to determine a first frame extraction interval corresponding to the training video according to a video parameter of the training video.

[0189] The frame extraction operation unit 3022 is configured to perform frame extraction operation on the training video according to the first frame extraction interval to obtain a plurality of training video frames corresponding to the training video.

[0190] As an optional embodiment, the specific manner in which the frame extraction operation unit 3022 performs frame extraction operation on the training video according to the first frame extraction interval to obtain the plurality of training video frames corresponding to the training video includes:

[0191] frame the training video according to a first frame interval to obtain a plurality of candidate video frames corresponding to the training video;

[0192] For any two adjacent candidate video frames, calculate the picture similarity between the two candidate video frames;

[0193] Determine whether the picture similarity meets a preset similarity threshold condition;

[0194] If the determination result is yes, the two candidate video frames are determined as key video frames;

[0195] According to all the key video frames in the plurality of candidate video frames, determine a plurality of training video frames corresponding to the training video.

[0196] As an optional implementation, the frame extraction operation unit 3022 determines the specific manner of the plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames, including:

[0197] According to a second frame interval, frame the other candidate video frames in the plurality of candidate video frames except the key video frames to obtain a plurality of extracted video frames; the second frame interval is greater than the first frame interval;

[0198] Determine all the key video frames and the extracted video frames as the plurality of training video frames corresponding to the training video.

[0199] As an optional implementation, as shown in Figure 4 The device further includes a frame number correction module 304, which is configured to perform the following steps after the frame extraction operation module 302 performs the frame extraction operation on the training video to obtain the plurality of training video frames corresponding to the training video:

[0200] Determine whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0201] If yes, divide the plurality of training video frames into at least two divided training video frame groups, each of which has a number of video frames less than or equal to the first frame number threshold; each divided training video frame group is used as single-time input training data when training the video reconstruction prediction model;

[0202] and / or,

[0203] Determine whether the number of the plurality of training video frames is less than a preset second frame number threshold;

[0204] If yes, extract video frames from the training video and fill them into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

[0205] As an optional implementation, as shown in Figure 4As shown, the device further comprises a preprocessing module 305, configured to perform the following steps before the model training module 303 trains the plurality of training video frames and the description text into the video reconstruction prediction model based on the Transformer network structure:

[0206] Shuffling the arrangement order of the plurality of training video frames arranged in the original time sequence to obtain shuffled training video frames; the shuffled training video frames are used for input into an encoder of the video reconstruction prediction model; the encoding output data of the shuffled training video frames are used for input into a decoder of the video reconstruction prediction model after being restored to the original time sequence;

[0207] and / or,

[0208] Determining the mask video frames in the plurality of training video frames;

[0209] Deleting the mask video frames, and determining the remaining training video frames in the plurality of training video frames as input video frames; the input video frames are used for input into the encoder of the video reconstruction prediction model; the mask video frames are used for input into the decoder of the video reconstruction prediction model together with the encoding output data of the input video frames; the ratio of the number of the mask video frames to the total number of the plurality of training video frames is greater than 50%.

[0210] As an optional implementation, the video reconstruction prediction model comprises an encoder; the encoder comprises a video embedding layer, a text embedding layer, a feature fusion layer and a first Transformer layer; the video embedding layer is configured to receive the training video frames and process to obtain video features; the text embedding layer is configured to receive the description text and process to obtain text features; the feature fusion layer is configured to fuse the video features and the text features to obtain training features, and input the training features into the first Transformer layer.

[0211] As an optional implementation, the loss function value between the plurality of predicted video frames and the input plurality of training video frames is calculated as follows:

[0212] For any predicted video frame, the frame loss function value between the predicted video frame and the corresponding training video frame is calculated;

[0213] The average value of the frame loss function values of all predicted video frames is calculated to obtain the loss function value between the plurality of predicted video frames and the input plurality of training video frames.

[0214] Embodiment Four

[0215] Please refer to Figure 5 , Figure 5 is another structure schematic diagram of a video and text based model training device disclosed by the embodiments of the present application. As shown in Figure 5As shown, the apparatus can include:

[0216] a memory 401 storing executable program code;

[0217] a processor 402 coupled with the memory 401;

[0218] The processor 402 invokes the executable program code stored in the memory 401 to perform part or all of the steps of the video and text based model training method disclosed in Embodiment One or Embodiment Two.

[0219] Embodiment Five

[0220] The computer storage medium provided by the embodiment of the present application stores computer instructions, which, when invoked, are used to perform part or all of the steps of the video and text based model training method disclosed in Embodiment One or Embodiment Two.

[0221] The above describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which they are recited, and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily have to be performed in the specific order shown and / or sequentially, as some of the processes can be performed in parallel or concurrently. In some embodiments, multitasking and parallel processing can be advantageous.

[0222] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the apparatus, device, and non-volatile computer readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0223] The apparatus, device, non-volatile computer readable storage medium, and method provided by the embodiments of the present specification are corresponding, and therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be described here.

[0224] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by ordering a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit that implements the logical method flow can be easily obtained.

[0225] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0226] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0227] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in implementing the present specification.

[0228] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the embodiments of the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0229] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flow or multiple flows and / or blocks Figure 1 one or more flow or multiple flows and / or blocks

[0230] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flow or multiple flows and / or blocks Figure 1 one or more flow or multiple flows and / or blocks

[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flow or multiple flows and / or blocks Figure 1 one or more flow or multiple flows and / or blocks

[0232] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0233] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer-readable media.

[0234] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0235] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0236] The specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The specification can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0237] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0238] It should be noted that the model training method and device based on video and text disclosed in the embodiments of the present application are only the preferred embodiments of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones. The modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a model based on video and text, characterized in that, The method comprises: determining a training video and corresponding description text for training a model; determining a first frame extraction interval corresponding to the training video according to video parameters of the training video, wherein the video parameters comprise video scene parameters; performing frame extraction on the training video according to the first frame extraction interval to obtain a plurality of candidate video frames corresponding to the training video; calculating a picture similarity between any two adjacent candidate video frames; judging whether the picture similarity meets a preset similarity threshold condition, wherein the preset similarity threshold condition is used to indicate that picture switching occurs between the two adjacent candidate video frames; if the judgment result is yes, determining the two candidate video frames as key video frames; determining a plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames; inputting the plurality of training video frames and the description text into a video reconstruction prediction model based on a Transformer network structure for training, calculating a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training, and optimizing model parameters of the video reconstruction prediction model according to the loss function value until convergence to obtain a trained video reconstruction prediction model. 2.The video and text based model training method of claim 1, wherein, The determination of the plurality of training video frames corresponding to the training video according to all the key video frames in the plurality of candidate video frames comprises: performing frame extraction on other candidate video frames except the key video frames in the plurality of candidate video frames according to a second frame extraction interval to obtain a plurality of extracted video frames; the second frame extraction interval is greater than the first frame extraction interval; determining all the key video frames and the extracted video frames as the plurality of training video frames corresponding to the training video. 3.The video and text based model training method of claim 1, wherein, After the frame extraction on the training video to obtain the plurality of training video frames corresponding to the training video, the method further comprises: judging whether the number of the plurality of training video frames is greater than a preset first frame number threshold; if yes, dividing the plurality of training video frames into at least two divided training video frame groups with a number of video frames less than or equal to the first frame number threshold; each divided training video frame group is used as single-time input training data when training a video reconstruction prediction model; and / or, judging whether the number of the plurality of training video frames is less than a preset second frame number threshold; if yes, extracting video frames from the training video to fill into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold. 4.The video and text based model training method of claim 1, wherein, Before the inputting of the plurality of training video frames and the description text into the video reconstruction prediction model based on the Transformer network structure for training, the method further comprises: Shuffling the arrangement order of the plurality of training video frames arranged in the original time sequence to obtain out-of-order training video frames; the out-of-order training video frames are used as input to an encoder of the video reconstruction prediction model; the encoder outputs data according to the encoding of the out-of-order training video frames, which are used as input to a decoder of the video reconstruction prediction model after being restored to the original time sequence; and / or, Determining a mask video frame from the plurality of training video frames; Deleting the mask video frame, and determining the remaining training video frames from the plurality of training video frames as input video frames; the input video frames are used as input to an encoder of the video reconstruction prediction model; the mask video frame is used as input to a decoder of the video reconstruction prediction model together with the encoding output data of the input video frames according to the encoder; the ratio of the number of mask video frames to the total number of the plurality of training video frames is greater than 50%. 5.The video and text based model training method of claim 1, wherein, The video reconstruction prediction model includes an encoder; the encoder includes a video embedding layer, a text embedding layer, a feature fusion layer, and a first Transformer layer; the video embedding layer is used to receive the training video frames and process to obtain video features; the text embedding layer is used to receive the description text and process to obtain text features; the feature fusion layer is used to fuse the video features and the text features to obtain training features, and input the training features to the first Transformer layer. 6.The video and text based model training method of claim 1, wherein, The loss function value between the plurality of predicted video frames and the input plurality of training video frames is calculated as follows: For any predicted video frame, calculate the frame loss function value between the predicted video frame and the corresponding training video frame; Calculate the average of the frame loss function values of all predicted video frames to obtain the loss function value between the plurality of predicted video frames and the input plurality of training video frames.

7. A video and text based model training apparatus, characterized by, The device is used to perform the video and text-based model training method according to any one of claims 1-6, and the device includes: A data determination module is configured to determine training videos and corresponding description texts for training a model; A frame extraction operation module is configured to perform frame extraction operation on the training videos to obtain a plurality of training video frames corresponding to the training videos; A model training module is configured to input the plurality of training video frames and the description texts into a video reconstruction prediction model based on a Transformer network structure for training, calculate a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the input plurality of training video frames during the training, optimize model parameters of the video reconstruction prediction model according to the loss function value until convergence, and obtain a trained video reconstruction prediction model.

8. A video and text based model training apparatus, characterized by, The device includes: A memory storing executable program codes; A processor coupled to the memory; The processor invokes the executable program codes stored in the memory to perform the video and text-based model training method according to any one of claims 1-6.

Images