Video Frame-Based Model Training Method and Apparatus

Abstract

This invention discloses a model training method and apparatus based on video frames. The method includes: determining multiple training videos for training the model; determining the frame extraction interval corresponding to the training videos based on the video parameters of the training videos; performing frame extraction on the training videos according to the frame extraction interval to obtain multiple training video frames corresponding to the training videos; training a video reconstruction prediction model based on a Transformer network structure using the multiple training video frames; calculating the loss function value between the multiple predicted video frames output by the video reconstruction prediction model and the multiple training video frames input during training; optimizing the model parameters of the video reconstruction prediction model based on the loss function value until convergence, thereby obtaining the trained video reconstruction prediction model. It is evident that this invention can improve training efficiency and utilizes the algorithmic advantages of the Transformer network structure to achieve better prediction results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of algorithm model training, in particular to a model training method and device based on video frames. BACKGROUND

[0002] With the development of algorithm technology, more and more enterprises begin to use algorithm models to perform video-related data prediction tasks, such as video reconstruction, which requires algorithm models to fully extract the features of the video and learn. However, the existing technology does not consider the frame interval when extracting frames from the video, nor does it consider the advantages of using the Transformer network structure to train video-related tasks, so it cannot reasonably extract frames to obtain reasonable video frames for training, and the training effect is poor. Therefore, the existing technology has defects and needs to be improved. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a model training method and device based on video frames, which can improve the training efficiency and make the prediction effect better by using the algorithm advantages of the Transformer network structure.

[0004] To solve the above technical problems, the first aspect of the present application discloses a model training method based on video frames, which comprises:

[0005] determining a plurality of training videos for training a model;

[0006] determining the frame interval corresponding to the training video according to the video parameters of the training video;

[0007] performing a frame extraction operation on the training video according to the frame interval to obtain a plurality of training video frames corresponding to the training video;

[0008] training a video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, calculating the loss function value between the plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training, optimizing the model parameters of the video reconstruction prediction model according to the loss function value until convergence, and obtaining the trained video reconstruction prediction model.

[0009] As an optional implementation, in the first aspect of the present application, the determination of the frame interval corresponding to the training video according to the video parameters of the training video comprises:

[0010] determining the picture change parameter of the training video;

[0011] determining the frame extraction interval of the training video according to the picture change parameter and a preset parameter threshold condition;

[0012] and / or,

[0013] determining the video scene parameter of the training video;

[0014] determining the frame extraction interval of the training video according to the video scene parameter and a preset scene-frame extraction corresponding relationship.

[0015] As an optional implementation, in the first aspect of the present application, the frame extraction interval includes a plurality of different frame extraction intervals; and the frame extraction operation on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video includes:

[0016] respectively performing frame extraction operation on the training video according to the plurality of different frame extraction intervals to respectively obtain a plurality of first training video frame groups corresponding to the training video; each of the first training video frame groups is used as single-time input training data when training the video reconstruction prediction model.

[0017] As an optional implementation, in the first aspect of the present application, after the frame extraction operation on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video, the method further includes:

[0018] determining whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0019] if yes, dividing the plurality of training video frames into at least two second training video frame groups with a video frame number less than or equal to the first frame number threshold; each of the second training video frame groups is used as single-time input training data when training the video reconstruction prediction model.

[0020] and / or,

[0021] determining whether the number of the plurality of training video frames is less than a preset second frame number threshold;

[0022] if yes, extracting video frames from the training video to fill into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

[0023] As an optional implementation, in the first aspect of the present application, before the training of the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, the method further includes:

[0024] normalize the plurality of training video frames, so that pixel values of pixel points in each of the plurality of training video frames are in a preset pixel value interval;

[0025] and / or,

[0026] shuffle an arrangement order of the plurality of training video frames arranged in an original time sequence to obtain shuffled training video frames; the shuffled training video frames are used for input to an encoder of the video reconstruction prediction model; encoded output data of the shuffled training video frames are used for input to a decoder of the video reconstruction prediction model after being recovered to the original time sequence;

[0027] and / or,

[0028] determine a mask video frame in the plurality of training video frames;

[0029] delete the mask video frame, and determine remaining training video frames of the plurality of training video frames as input video frames; the input video frames are used for input to an encoder of the video reconstruction prediction model; the mask video frame is used for input to a decoder of the video reconstruction prediction model together with encoded output data of the input video frames.

[0030] As an optional implementation, in the first aspect of the present application, the loss function value between the plurality of prediction video frames and the plurality of input training video frames is calculated in the following manner:

[0031] for any one of the prediction video frames, calculate a frame loss function value between the prediction video frame and the corresponding training video frame;

[0032] calculate an average value of the frame loss function values of all the prediction video frames to obtain the loss function value between the plurality of prediction video frames and the plurality of input training video frames.

[0033] As an optional implementation, in the first aspect of the present application, the video reconstruction prediction model comprises an encoder; the encoder comprises an embedding layer, a position encoding layer and a first Transformer layer; the embedding layer comprises a two-dimensional convolution layer and / or a three-dimensional convolution layer.

[0034] As an optional implementation, in the first aspect of the present application, the video reconstruction prediction model comprises a decoder; the decoder comprises a first fully connected layer, a second Transformer layer and a second fully connected layer.

[0035] The second aspect of the present application discloses a model training device based on video frames, and the device comprises:

[0036] a video determination module configured to determine a plurality of training videos for training a model;

[0037] an interval determination module configured to determine, according to a video parameter of the training video, a frame extraction interval corresponding to the training video;

[0038] a frame extraction operation module configured to perform frame extraction operation on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video;

[0039] a model training module configured to train a video reconstruction prediction model based on a Transformer network structure according to the plurality of training video frames, calculate, in the training, a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input, optimize model parameters of the video reconstruction prediction model according to the loss function value until convergence, and obtain the trained video reconstruction prediction model.

[0040] As an optional implementation, in the second aspect of the present application, the specific manner in which the interval determination module determines the frame extraction interval corresponding to the training video according to the video parameter of the training video includes:

[0041] determining a picture change parameter of the training video;

[0042] determining the frame extraction interval corresponding to the training video according to the picture change parameter and a preset parameter threshold condition;

[0043] and / or,

[0044] determining a video scene parameter of the training video;

[0045] determining the frame extraction interval corresponding to the training video according to the video scene parameter and a preset scene-frame extraction corresponding relationship.

[0046] As an optional implementation, in the second aspect of the present application, the frame extraction interval includes a plurality of different frame extraction intervals; and the specific manner in which the frame extraction operation module performs frame extraction operation on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video includes:

[0047] performing frame extraction operation on the training video according to the plurality of different frame extraction intervals to obtain a plurality of first training video frame groups corresponding to the training video respectively; each of the first training video frame groups is used as single-time input training data when training the video reconstruction prediction model.

[0048] As an optional implementation, in the second aspect of the present application, the device further comprises a frame number adjustment module, which is configured to perform the following steps after the frame extraction operation module extracts the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video:

[0049] determine whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0050] if yes, divide the plurality of training video frames into at least two second training video frame groups with a number of video frames less than or equal to the first frame number threshold; each of the second training video frame groups is used as single-time input training data when training the video reconstruction prediction model;

[0051] and / or,

[0052] determine whether the number of the plurality of training video frames is less than a preset second frame number threshold;

[0053] if yes, extract video frames from the training video to fill into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

[0054] As an optional implementation, in the second aspect of the present application, the device further comprises a preprocessing module, which is configured to perform the following steps before the model training module trains the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames:

[0055] perform normalization operation on the plurality of training video frames so that the pixel value of each pixel point in each of the training video frames is in a preset pixel value interval;

[0056] and / or,

[0057] shuffle the arrangement order of the plurality of training video frames arranged in the original time sequence to obtain disordered training video frames; the disordered training video frames are used as input to an encoder of the video reconstruction prediction model; the encoding output data of the disordered training video frames are used as input to a decoder of the video reconstruction prediction model after being restored to the original time sequence;

[0058] and / or,

[0059] determine the mask video frame in the plurality of training video frames;

[0060] The mask video frame is deleted, and remaining training video frames of the plurality of training video frames are determined as input video frames; the input video frames are used for input to an encoder of the video reconstruction prediction model; and the mask video frame is used for input to a decoder of the video reconstruction prediction model together with encoded output data of the input video frames according to the encoder.

[0061] As an optional implementation, in the second aspect of the present application, the loss function value between the plurality of predicted video frames and the plurality of input training video frames is calculated in the following manner:

[0062] For any predicted video frame, a frame loss function value between the predicted video frame and the corresponding training video frame is calculated.

[0063] An average value of the frame loss function values of all the predicted video frames is calculated to obtain the loss function value between the plurality of predicted video frames and the plurality of input training video frames.

[0064] As an optional implementation, in the second aspect of the present application, the video reconstruction prediction model comprises an encoder; the encoder comprises an embedding layer, a position encoding layer and a first Transformer layer; and the embedding layer comprises a two-dimensional convolution layer and / or a three-dimensional convolution layer.

[0065] As an optional implementation, in the second aspect of the present application, the video reconstruction prediction model comprises a decoder; the decoder comprises a first fully connected layer, a second Transformer layer and a second fully connected layer.

[0066] The third aspect of the present application discloses another video frame-based model training device, which comprises:

[0067] A memory storing executable program codes;

[0068] A processor coupled with the memory;

[0069] The processor invokes the executable program codes stored in the memory to execute part or all steps of the video frame-based model training method disclosed in the first aspect of the present application.

[0070] The fourth aspect of the present application discloses a computer storage medium storing computer instructions, which are invoked to execute part or all steps of the video frame-based model training method disclosed in the first aspect of the present application.

[0071] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0072] In the embodiment of the present application, a plurality of training videos for training a model are determined; according to the video parameters of the training videos, the frame extraction intervals corresponding to the training videos are determined; the frame extraction operation is performed on the training videos according to the frame extraction intervals, and a plurality of training video frames corresponding to the training videos are obtained; the video reconstruction prediction model based on the Transformer network structure is trained according to the plurality of training video frames, the loss function value between the plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training is calculated in the training, the model parameters of the video reconstruction prediction model are optimized according to the loss function value until convergence, and the trained video reconstruction prediction model is obtained. It can be seen that the video frames obtained by frame extraction of the training videos based on a specific interval can be used to train the algorithm model based on the Transformer network structure, so that on the one hand, the video content of the video frames can be reasonably and efficiently determined to improve the training efficiency of the model, and on the other hand, the algorithm advantage of the Transformer network structure is utilized, so that the prediction effect of the model trained is better. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0074] Figure 1 is a flow diagram of a model training method based on video frames disclosed by the embodiment of the present application;

[0075] Figure 2 is a flow diagram of another model training method based on video frames disclosed by the embodiment of the present application;

[0076] Figure 3 is a structural diagram of a model training device based on video frames disclosed by the embodiment of the present application;

[0077] Figure 4 is a structural diagram of another model training device based on video frames disclosed by the embodiment of the present application;

[0078] Figure 5 is a structural diagram of another model training device based on video frames disclosed by the embodiment of the present application. DETAILED DESCRIPTION

[0079] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of, rather than all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of the present application.

[0080] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, rather than to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or end including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product, or end.

[0081] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor does it necessarily refer to a separate or alternative embodiment in isolation from other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0082] The present application discloses a model training method and device based on video frames, which can train an algorithm model based on a Transformer network structure by using video frames obtained by frame extraction of a training video based on a specific interval, thereby on the one hand reasonably and efficiently determining the video content of the video frames to improve the training efficiency of the model, and on the other hand utilizing the algorithm advantages of the Transformer network structure to make the prediction effect of the trained model better. The following will be described in detail respectively.

[0083] Embodiment one

[0084] Please refer to Figure 1 , Figure 1 is a flowchart of a model training method based on video frames disclosed by an embodiment of the present application. Among them, Figure 1 The described method is applied to a video data processing device, which can be a corresponding processing terminal, processing equipment or processing server, and the server can be a local server or a cloud server, and the embodiments of the present application are not limited. As Figure 1 The model training method based on video frames can include the following operations:

[0085] 101. Determine a plurality of training videos for training the model.

[0086] Optionally, the video content of the training video can be determined according to the target processing video content of the model to be trained, and can also include various types of video content to improve the adaptive characteristics of the trained model. Optionally, the video content of the training video can be a continuous action or a continuous scene, and the picture lens can move continuously and slowly, but the number of lens switching in the video content should be as small as possible, which can be set to be less than a preset number threshold, which can be 0.

[0087] 102. Determine the frame extraction interval corresponding to the training video according to the video parameters of the training video.

[0088] Optionally, the video parameters can include picture change parameters and / or video scene parameters, which can be used to represent the complexity of the video content. When the complexity is high, the frame extraction interval should be shortened to obtain more video frames, so that the video frames can fully reflect the video content. When the complexity is low, the frame extraction interval can be appropriately increased to obtain fewer video frames.

[0089] 103. Frame extraction is performed on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video.

[0090] Optionally, the frame extraction interval can be a time interval or a frame number interval.

[0091] Optionally, a plurality of frame extraction time points can be determined according to the time sequence or the frame number sequence with the frame extraction interval as the interval, and then the video frames corresponding to the frame extraction time points in the training video are obtained to obtain a plurality of training video frames corresponding to the training video. It should be noted that the time interval or the frame number interval between adjacent video frames in the plurality of training video frames is not necessarily the above-mentioned frame extraction interval, because when frame extraction is performed according to the time sequence or the frame number interval, the last frame can be directly determined as a training video frame when the interval at the end is insufficient.

[0092] Optionally, after obtaining the plurality of training video frames corresponding to the training video, the plurality of training video frames can be bound and saved with the identifier (such as the storage path, the video ID, etc.) of the training video for subsequent training.

[0093] 104. Train the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames. In the training, the loss function value between the plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input is calculated, and the model parameters of the video reconstruction prediction model are optimized according to the loss function value until convergence, to obtain the trained video reconstruction prediction model.

[0094] Optionally, the loss function value can be an L1 loss function value or an L2 loss function value, or other loss function values suitable for calculating the similarity between images, which are not limited by the present application.

[0095] It can be seen that the method described in the embodiments of the present application can use the video frames obtained by frame extraction of the training video based on a specific interval to train the algorithm model based on the Transformer network structure, thereby on the one hand, the video content of the video frames can be reasonably and efficiently determined to improve the training efficiency of the model, and on the other hand, the algorithm advantage of the Transformer network structure is utilized, so that the prediction effect of the trained model is better.

[0096] As an optional implementation, the video reconstruction prediction model includes an encoder and a decoder, wherein the encoder is used to extract features, and the decoder is used to restore the features. Optionally, the encoder includes an embedding layer, a position encoding layer and a first Transformer layer, wherein the embedding layer includes a two-dimensional convolution layer and / or a three-dimensional convolution layer.

[0097] Optionally, a two-dimensional convolutional image block embedding layer (Patch Embedding) can be constructed, which is used to replace the high-dimensional original image features with a lower-dimensional vector. Optionally, the image block embedding layer is composed of a convolution layer, the size of the convolution kernel and the size of the step are equal, and the output channel number is selected as 768 (other values are also available).

[0098] Optionally, a three-dimensional convolutional video block embedding layer, i.e., a 3D Patch Embedding layer, can also be constructed, which performs convolution in the spatial and temporal dimensions through a three-dimensional convolution layer, thereby additionally extracting the correlation features between the previous and subsequent frames. Specifically, in order to ensure that each frame has the correlation features between the previous and subsequent frames, the size of the convolution step is not necessarily equal to the size of the convolution depth, i.e., the step can be equal to 1, for example, the convolution depth is equal to 2 and the step is equal to 1, at this time, the first frame and the second frame will obtain a feature vector through three-dimensional convolution, and then the second frame and the third frame will also obtain a feature vector. If the convolution depth is equal to 2 and the step is also equal to 2, at this time, the first frame and the second frame will obtain a feature vector after convolution, and then the convolution features between the third frame and the fourth frame will be calculated directly without calculating the features between the second and third frames.

[0099] Specifically, the input video frames are converted into corresponding feature vectors by the embedding layer. Taking a specific embodiment of a two-dimensional convolutional image block embedding layer as an example, for example, 30 video frames, each with an image size of 224*224, the dimension is (30, 3, 224, 224), 3 represents RGB three channels, the convolution kernel size is (16, 16), and the step is 16. After conversion, it becomes (30, 768, 14, 14), and then it is converted into a feature vector (30, 768, 196). Subsequently, the order of the dimensions is exchanged and a class label is added at the initial position of the 196 dimension to process downstream tasks such as classification. Finally, the dimension of the feature obtained is (30, 197, 768).

[0100] Optionally, the position encoding layer adds a position encoding vector initialized by cosine to the feature to represent the position information of each image block in the image.

[0101] Optionally, the first Transformer layer includes a plurality of stacked Transformer modules, and the structure can refer to the encoder structure of the VIT (Vision Transformer) network.

[0102] As can be seen, by implementing the optional embodiment, an encoder structure that can fully extract the features of the video frames can be constructed, so that the prediction effect of the model is better.

[0103] As an optional embodiment, the decoder includes a first fully connected layer, a second Transformer layer, and a second fully connected layer.

[0104] Specifically, the decoder first uses a fully connected layer to fuse and convert the features of the encoder. Subsequently, it is also composed of a plurality of Transformer modules, and finally, it will also use a fully connected layer to generate pixel-level video frames.

[0105] As can be seen, by implementing the optional embodiment, a decoder structure that can fully reconstruct the features of the video frames can be constructed, so that the prediction effect of the model is better.

[0106] As an optional embodiment, in the above step, the frame extraction interval corresponding to the training video is determined according to the video parameters of the training video, including:

[0107] Determine the picture change parameter of the training video.

[0108] According to the picture change parameter and the preset parameter threshold condition, the frame extraction interval corresponding to the training video is determined.

[0109] Optionally, the picture change parameter can be an optical flow value parameter between different frames of the training video, for example, an average value, a maximum value or a weighted average value of the optical flow motion amount between all adjacent frames. Correspondingly, the parameter threshold condition can be an optical flow value threshold condition.

[0110] Preferably, the optical flow value between each frame can be calculated, and the frame skipping interval can be determined by limiting the optical flow motion amount between each frame, for example, by setting an optical flow change threshold value. After statistics, it is found that the optical flow change amount of every K frames just exceeds the threshold value, and then the frame skipping interval can be set as K frames. If the optical flow motion amount between each frame is larger, a smaller frame skipping interval can be selected.

[0111] It can be seen that by implementing the optional embodiment, a more reasonable frame skipping interval can be determined according to the picture change parameter, so that the video content of the video frame can be reasonably and efficiently determined, thereby improving the training efficiency of the model.

[0112] As an optional embodiment, in the above step, the frame skipping interval corresponding to the training video is determined according to the video parameter of the training video, including:

[0113] determining a video scene parameter of the training video;

[0114] determining the frame skipping interval corresponding to the training video according to the video scene parameter and a preset scene frame skipping corresponding relationship.

[0115] Optionally, the video scene parameter can be a scene type of the training video. Optionally, the scene frame skipping corresponding relationship is used to indicate the frame skipping interval corresponding to different types of scenes. For example, a short video mainly shows some changes of human body, such as a person's slam dunk action. The whole action is fast, and the change between each frame is relatively obvious. Therefore, a smaller frame skipping interval can be selected. If the content change speed of the short video is slow or has a certain rule, for example, a car slowly drives on a mountain path, and a dashcam records the road conditions in front and the trees on both sides regularly move towards the lens. For such changes with rules or slow speed, a larger frame skipping interval can be selected. Specifically, for different task scenes, the frame skipping interval is different.

[0116] Optionally, the number of different types of scenes appearing in the training video can be used to indicate the degree of scene change or the complexity of video content. Optionally, the scene frame skipping corresponding relationship is used to indicate the frame skipping interval corresponding to the number of different types of scenes. Generally, the frame skipping interval is inversely proportional to the number of scenes, that is, the more the number of scenes, the more the scene changes, and the more complex the content. At this time, the frame skipping interval is smaller, and the number of video frames obtained is more, and vice versa.

[0117] It can be seen that, by implementing the optional embodiment, a more reasonable frame extraction interval can be determined according to the video scene parameters, so that the video content of the video frame can be reasonably and efficiently determined, thereby improving the training efficiency of the model.

[0118] As an optional embodiment, after the step of performing frame extraction on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video, the method further comprises:

[0119] determining whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0120] if yes, dividing the plurality of training video frames into at least two second training video frame groups each having a number of video frames less than or equal to the first frame number threshold.

[0121] Each second training video frame group is used as single-input training data when training the video reconstruction prediction model.

[0122] Specifically, after frame extraction is performed on the video, the final frame number is counted. In order to avoid excessive memory occupation during training, a first frame number threshold, such as 30 frames, is set. When the result of frame extraction of a certain video is 54, the first 30 frames are taken as a sequence, and the last 24 frames are taken as a second sequence for segmentation of single-input training data.

[0123] It can be seen that, by implementing the optional embodiment, the plurality of training video frames can be divided into at least two second training video frame groups each having a number of video frames less than or equal to the first frame number threshold, so that single-input training data can be reasonably and efficiently determined, thereby reducing the training cost and improving the training efficiency of the model.

[0124] As an optional embodiment, after the step of performing frame extraction on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video, the method further comprises:

[0125] determining whether the number of the plurality of training video frames is less than a preset second frame number threshold;

[0126] if yes, extracting video frames from the training video and filling into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

[0127] Specifically, after frame extraction is performed on the video, the final frame number is counted. In order to avoid that the frame number of some videos is too small to achieve the expected training effect, a second frame number threshold, such as 30 frames, is set. If the total frame number of a video is less than 30, such as 24 frames in total, 6 frames are randomly selected and copied and inserted in the original time order, respectively, to complete 30 frames.

[0128] It can be seen that by implementing the optional embodiment, video frames can be extracted from the training video to fill the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold, so that the amount of training data input at a time can be reasonably and efficiently completed, and the training efficiency and effect of the model can be improved.

[0129] As an optional embodiment, before the step of training the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, the method further comprises:

[0130] The plurality of training video frames are subjected to a normalization operation, so that the pixel values of the pixel points in each training video frame are in a preset pixel value interval.

[0131] Preferably, the original input video frame is subjected to a normalization operation before being processed by the encoder to obtain the features, so that the pixel value range of the image is transformed to [0, 1]. Therefore, the pixel value range of the video frame output by the decoder is also [0, 1], which can effectively improve the calculation efficiency and training efficiency.

[0132] As an optional embodiment, before the step of training the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, the method further comprises:

[0133] The arrangement order of the plurality of training video frames arranged in the original time sequence is disturbed to obtain disordered training video frames.

[0134] The disordered training video frames are used for input to the encoder of the video reconstruction prediction model, and specifically, the encoding output data of the encoder according to the disordered training video frames is used for input to the decoder of the video reconstruction prediction model after being restored to the original time sequence.

[0135] Through the above setting, the video frames can be disturbed to train the model, which can effectively improve the prediction effect of the model obtained by training.

[0136] As an optional embodiment, before the step of training the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, the method further comprises:

[0137] The mask video frames in the plurality of training video frames are determined;

[0138] The mask video frames are deleted, and the remaining training video frames of the plurality of training video frames are determined as input video frames.

[0139] The input video frame is used for input to an encoder of a video reconstruction prediction model. Specifically, the mask video frame is used for input to a decoder of the video reconstruction prediction model together with encoded output data of the input video frame.

[0140] Specifically, taking the above specific implementation as an example, there are a total of 30 video frames, corresponding to a feature vector of (30, 197, 768). However, during the training process, a part of the video frames will be randomly Masked, that is, a part of the 30 dimensions will be randomly removed. For example, the Masking ratio is 33%, and finally a feature vector of (20, 197, 768) will be used for training. Specifically, the Masking operation adopts a random Masking method, that is, a part of the 30 frames is randomly Masked.

[0141] Then, the order of the video frames after being randomly Masked is shuffled, and then input into the encoder and decoder, so that the model learns the video features. The encoder is responsible for learning the consistency features and variability features of the 20 visible video frames. Since the encoder is composed of multiple Transformer modules, the output dimension is consistent with the input dimension. After passing through the encoder, the output dimension is still (20, 197, 768).

[0142] Then, the decoder not only receives the output of the above-mentioned encoder, but also receives the part of the video frames that have been Masked, that is, all the video frames, and all the input video frames are arranged in the original time sequence. In the decoder, the input feature vector of (30, 197, 768) is first converted in the feature space through a fully connected layer, and then input into the Transformer of the decoder. Finally, the same (30, 197, 768) feature vector is output. At this time, the feature vector corresponding to the class label is removed, that is, (30, 196, 768), and the dimension of 196 is transformed into 14x14. The final output dimension is (30, 14, 14, 768), where 30 represents 30 frames, 14x14 represents 196 image blocks, each image block contains 768 pixels (3x256, 3 represents the number of image channels RGB, and 256 represents 16x16 pixels), and the transformed shape is (30, 3, 224, 224). As can be seen, after the Masking operation of a part of the frame information of the input 30 video frames, the feature learning is performed through the encoder, and then the 30 frames are reconstructed through the decoder.

[0143] As an optional implementation, the calculation method of the loss function value between the plurality of predicted video frames and the plurality of input training video frames is as follows:

[0144] For any predicted video frame, the frame loss function value between the predicted video frame and the corresponding training video frame is calculated.

[0145] The average value of the frame loss function values of all predicted video frames is calculated to obtain the loss function value between the plurality of predicted video frames and the plurality of training video frames input.

[0146] Specifically, the SmoothL1 loss between the original video frame received by the encoder and the corresponding output video frame of the decoder is calculated, the loss between each frame is calculated, and then the average value is calculated as the loss of video frame reconstruction. Since the loss calculation is only based on the original video frame and the corresponding decoded video frame, no additional label information is required, and an unsupervised learning method can be implemented.

[0147] Preferably, after the pre-training of the model is completed according to the above embodiment, if the decoder is used to reconstruct the video, the output result of the decoder can be de-normalized to obtain the reconstructed RGB video frame, and then the continuous video frames are converted into a video. If it is used for other downstream tasks such as classification tasks, only a small amount of supervised data set needs to be prepared, and then only the encoder is used to obtain the output feature vector, and the 0th dimension of the 197 dimensions is taken, that is, the classification label, which is input into a new fully connected layer for classification. That is, the trained model can be used for a variety of other downstream tasks.

[0148] Embodiment two

[0149] Please refer to Figure 2 , Figure 2 is another flowchart of a model training method based on video frames disclosed in the embodiments of the present application. Among them, Figure 2 The method described is applied to a video data processing device, which can be a corresponding processing terminal, processing equipment or processing server, and the server can be a local server or a cloud server, and the embodiments of the present application are not limited. As Figure 2 The model training method based on video frames can include the following operations:

[0150] 201, determine a plurality of training videos for training the model.

[0151] 202, determine the frame extraction interval corresponding to the training video according to the video parameter of the training video.

[0152] Specifically, the frame extraction interval includes a plurality of different frame extraction intervals.

[0153] 203, respectively according to a plurality of different frame extraction intervals, frame extraction operation is performed on the training video to obtain a plurality of first training video frame groups corresponding to the training video.

[0154] Optionally, each first training video frame group is used as single-time input training data when training the video reconstruction prediction model.

[0155] Optionally, each first training video frame group includes a plurality of training video frames.

[0156] Specifically, in order to expand the data set, the same training video is extracted at different frame intervals, for example, 5 frames per second and 2 frames per second, and the corresponding video frame sequence shows a slow and fast difference. Optionally, the frame extraction of the training video can be performed before the entire training is started. Early frame extraction can greatly reduce the training time. If dynamic frame extraction interval is selected for short video extraction during the training stage, although the data set can be greatly enriched and data augmentation can be achieved, the extraction speed is often slow, and each training iteration needs to be re-extracted, which undoubtedly seriously hinders the entire training process.

[0157] 204. Training the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, calculating the loss function value between the plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input during training, and optimizing the model parameters of the video reconstruction prediction model according to the loss function value until convergence to obtain the trained video reconstruction prediction model.

[0158] The specific technical details and technical terms of the above steps 201-202 and 204 can be referred to the description of steps 101-102 and 104 in the first embodiment, which will not be repeated here.

[0159] It can be seen that the method described in the embodiment of the application can effectively perform data augmentation using multiple different intervals, thereby increasing the amount of training data at low cost, and the prediction effect of the model obtained by training is better.

[0160] Embodiment three

[0161] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of a model training device based on video frames disclosed by the embodiment of the application. Wherein, Figure 3 The device described can be applied to a corresponding video data processing device, which can be a corresponding processing terminal, processing equipment or processing server, and the server can be a local server or a cloud server, and the embodiment of the application does not limit it. As Figure 3 As shown in the figure, the device can include:

[0162] The video determination module 301 is configured to determine a plurality of training videos for training the model.

[0163] The interval determination module 302 is configured to determine the frame extraction interval corresponding to the training video according to the video parameter of the training video.

[0164] The frame extraction operation module 303 is configured to perform frame extraction operation on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video.

[0165] The model training module 304 is configured to train the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, calculate the loss function value between the plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training, optimize the model parameters of the video reconstruction prediction model according to the loss function value until convergence, and obtain the trained video reconstruction prediction model.

[0166] As an optional implementation, the interval determination module 302 determines the specific manner of the frame extraction interval corresponding to the training video according to the video parameter of the training video, which includes:

[0167] determining the picture change parameter of the training video;

[0168] determining the frame extraction interval corresponding to the training video according to the picture change parameter and the preset parameter threshold condition;

[0169] and / or,

[0170] determining the video scene parameter of the training video;

[0171] determining the frame extraction interval corresponding to the training video according to the video scene parameter and the preset scene-frame extraction corresponding relationship.

[0172] As an optional implementation, the frame extraction interval includes a plurality of different frame extraction intervals; the frame extraction operation module 303 performs frame extraction operation on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video in a specific manner, which includes:

[0173] performing frame extraction operation on the training video according to the plurality of different frame extraction intervals to obtain a plurality of first training video frame groups corresponding to the training video respectively; each first training video frame group is used as single-time input training data when training the video reconstruction prediction model.

[0174] As an optional implementation, as shown in Figure 4 The device further includes a frame number adjustment module 305 configured to perform the following steps after the frame extraction operation module 303 performs frame extraction operation on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video:

[0175] determine whether the number of the plurality of training video frames is greater than a preset first frame number threshold;

[0176] If yes, the plurality of training video frames are divided into at least two second training video frame groups each having a number of video frames less than or equal to the first frame number threshold; each of the second training video frame groups is used as single-time input training data when training the video reconstruction prediction model;

[0177] and / or,

[0178] determining whether the number of the plurality of training video frames is less than a preset second frame number threshold;

[0179] If yes, video frames are extracted from the training video and filled into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

[0180] As an optional implementation, as shown in Figure 4 The apparatus further includes a preprocessing module 306 configured to perform the following steps before the model training module 304 trains the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames:

[0181] normalizing the plurality of training video frames so that pixel values of pixel points in each training video frame are in a preset pixel value interval;

[0182] and / or,

[0183] shuffling the arrangement order of the plurality of training video frames arranged in the original time sequence to obtain shuffled training video frames; the shuffled training video frames are used as input to an encoder of the video reconstruction prediction model; the encoding output data of the encoder according to the shuffled training video frames are used as input to a decoder of the video reconstruction prediction model after being restored to the original time sequence;

[0184] and / or,

[0185] determining mask video frames from the plurality of training video frames;

[0186] deleting the mask video frames and determining remaining training video frames from the plurality of training video frames as input video frames; the input video frames are used as input to an encoder of the video reconstruction prediction model; the mask video frames are used as input to a decoder of the video reconstruction prediction model together with the encoding output data of the encoder according to the input video frames.

[0187] As an optional implementation, the loss function value between the plurality of prediction video frames and the plurality of input training video frames is calculated as follows:

[0188] for any prediction video frame, calculating a frame loss function value between the prediction video frame and a corresponding training video frame;

[0189] An average of frame loss function values of all predicted video frames is calculated to obtain loss function values between the plurality of predicted video frames and the plurality of training video frames inputted.

[0190] As an optional implementation, the video reconstruction prediction model comprises an encoder; the encoder comprises an embedding layer, a position encoding layer and a first Transformer layer; the embedding layer comprises a two-dimensional convolution layer and / or a three-dimensional convolution layer.

[0191] As an optional implementation, the video reconstruction prediction model comprises a decoder; the decoder comprises a first fully connected layer, a second Transformer layer and a second fully connected layer.

[0192] Embodiment four

[0193] Please refer to Figure 5 , Figure 5 is another structure diagram of the video frame-based model training device disclosed by the embodiments of the present application. As shown in the figure, the device can include: Figure 5

[0194] a memory 401 storing executable program codes;

[0195] a processor 402 coupled with the memory 401;

[0196] The processor 402 invokes the executable program codes stored in the memory 401 to execute part or all of the steps of the video frame-based model training method disclosed by the embodiments one or two of the present application.

[0197] Embodiment five

[0198] The embodiments of the present application disclose a computer storage medium storing computer instructions, which when invoked, are used to execute part or all of the steps of the video frame-based model training method disclosed by the embodiments one or two of the present application.

[0199] The above describes certain embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order and still accomplish desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0200] ​The various embodiments described in the specification are progressive, and the same or similar parts between the various embodiments can be mutually referred to. Each of the embodiments focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0201] The device, equipment, non-volatile computer readable storage medium, and method provided by the embodiments of the specification are corresponding, and therefore, the device, equipment, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding device, equipment, and non-volatile computer storage medium will not be described here.

[0202] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by ordering a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit that implements the logical method flow can be easily obtained.

[0203] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0204] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0205] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in one or more software and / or hardware in the implementation of the present specification.

[0206] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the embodiments of the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0207] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flow or multiple flows and / or blocks Figure 1 one or more flow or multiple flows and / or blocks

[0208] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flow or multiple flows and / or blocks Figure 1 one or more flow or multiple flows and / or blocks

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flow or multiple flows and / or blocks Figure 1 one or more flow or multiple flows and / or blocks

[0210] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0211] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0212] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0213] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to encompass non-exclusive inclusion, such that processes, methods, articles or devices that comprise a list of elements not only include those elements, but also include other elements not expressly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0214] The specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0215] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0216] It should be noted that the model training method and device based on video frames disclosed in the embodiments of the present application are only the preferred embodiments of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones. The modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for model training based on video frames, characterized in that, The method comprises: determining a plurality of training videos for training a model; determining a frame extraction interval corresponding to the training video according to a video parameter of the training video; performing frame extraction on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video; training a video reconstruction prediction model based on a Transformer network structure according to the plurality of training video frames, calculating a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training, and optimizing model parameters of the video reconstruction prediction model according to the loss function value until convergence to obtain a trained video reconstruction prediction model; wherein the determining of the frame extraction interval corresponding to the training video according to the video parameter of the training video comprises: determining a picture change parameter of the training video; the picture change parameter is an optical flow value parameter between different frames of the training video; determining the frame extraction interval corresponding to the training video according to the picture change parameter and a preset parameter threshold condition; the parameter threshold condition is an optical flow value threshold condition; and / or, determining a video scene parameter of the training video; the video scene parameter is a scene type of the training video; determining the frame extraction interval corresponding to the training video according to the video scene parameter and a preset scene-frame extraction corresponding relationship; the scene-frame extraction corresponding relationship is used to indicate that the number of different types of scenes corresponds to the frame extraction interval, and the frame extraction interval is inversely proportional to the number of scenes.

2. The method of claim 1, wherein, The frame extraction interval comprises a plurality of different frame extraction intervals. The performing of the frame extraction on the training video according to the frame extraction interval to obtain the plurality of training video frames corresponding to the training video comprises: performing frame extraction on the training video according to the plurality of different frame extraction intervals to respectively obtain a plurality of first training video frame groups corresponding to the training video; each first training video frame group is used as single-time input training data when training the video reconstruction prediction model. 3.The method of claim 1, wherein, After the performing of the frame extraction on the training video according to the frame extraction interval to obtain the plurality of training video frames corresponding to the training video, the method further comprises: judging whether the number of the plurality of training video frames is greater than a preset first frame number threshold; if yes, dividing the plurality of training video frames into at least two second training video frame groups with a number of video frames less than or equal to the first frame number threshold; each second training video frame group is used as single-time input training data when training the video reconstruction prediction model; and / or, judging whether the number of the plurality of training video frames is less than a preset second frame number threshold; if yes, extracting video frames from the training video to fill into the plurality of training video frames until the number of the plurality of training video frames is equal to the second frame number threshold.

4. The method of claim 1, wherein, Before the training of the video reconstruction prediction model based on the Transformer network structure according to the plurality of training video frames, the method further comprises: normalize each of the training video frames so that pixel values of pixels in each of the training video frames are in a preset pixel value interval; and / or, shuffle an arrangement order of the plurality of training video frames arranged in an original time sequence to obtain disordered training video frames; the disordered training video frames are used as input to an encoder of the video reconstruction prediction model; encoded output data of the encoder according to the disordered training video frames are used as input to a decoder of the video reconstruction prediction model after being recovered to the original time sequence; and / or, determine a mask video frame from the plurality of training video frames; delete the mask video frame, and determine remaining training video frames from the plurality of training video frames as input video frames; the input video frames are used as input to the encoder of the video reconstruction prediction model; the mask video frame is used as input to the decoder of the video reconstruction prediction model together with the encoded output data of the input video frames.

5. The method of claim 1, wherein, The loss function value between the plurality of predicted video frames and the plurality of input training video frames is calculated in the following manner: For any one of the predicted video frames, calculate a frame loss function value between the predicted video frame and the corresponding training video frame; Calculate an average value of the frame loss function values of all the predicted video frames to obtain the loss function value between the plurality of predicted video frames and the plurality of input training video frames.

6. The method of claim 1, wherein, The video reconstruction prediction model comprises an encoder; the encoder comprises an embedding layer, a position encoding layer, and a first Transformer layer; the embedding layer comprises a two-dimensional convolution layer and / or a three-dimensional convolution layer.

7. The method of claim 1, wherein, The video reconstruction prediction model comprises a decoder; the decoder comprises a first fully connected layer, a second Transformer layer, and a second fully connected layer.

8. A model training apparatus based on video frames, characterized by, The device is used to perform the model training method based on video frames according to any one of claims 1-7, and the device comprises: a video determination module configured to determine a plurality of training videos for training a model; an interval determination module configured to determine a frame extraction interval corresponding to the training videos according to video parameters of the training videos; an extraction operation module configured to perform frame extraction on the training videos according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training videos; a model training module configured to train a video reconstruction prediction model based on a Transformer network structure according to the plurality of training video frames, calculate a loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of input training video frames during the training, optimize model parameters of the video reconstruction prediction model according to the loss function value until convergence, and obtain the trained video reconstruction prediction model.

9. A model training apparatus based on video frames, characterized by, The device comprises: a memory storing executable program codes; a processor coupled to the memory; the processor invokes the executable program codes stored in the memory to perform the model training method based on video frames according to any one of claims 1-7.

Images