Model training, synthetic frame generation, video recognition methods and devices, and media

Abstract

This disclosure provides a model training, synthetic frame generation, video recognition method and apparatus, as well as a storage medium. The training method includes: compressing a video frame training sequence using a synthetic frame generation model to obtain a two-dimensional synthetic frame; processing the two-dimensional synthetic frame using a joint training model and constructing a first loss function based on the processing result; constructing a second loss function based on the color information of the video frame training sequence and the two-dimensional synthetic frame; and jointly training the synthetic frame generation model and the joint training model based on the first and second loss functions. The method, apparatus, and storage medium of this disclosure generate two-dimensional synthetic frames that have the ability to restore video visual details, video categories, and video action information, and are close to normal images in terms of visual details and color space, thus retaining more video information in the synthetic frame; it can reduce the computational load of video recognition and the workload of model training.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for model training, synthetic frame generation, and video recognition, as well as a storage medium. Background Technology

[0002] The goal of video recognition is to identify the main content in a video, including objects, actions, and events. Existing video recognition technologies are mainly divided into two categories: handcrafted feature-based video recognition and deep learning-based video recognition. Handcrafted feature-based video recognition technology has low recognition performance. Deep learning-based video recognition technology typically applies two-dimensional convolutional neural networks used in image recognition to video recognition, and improves the performance of the neural network by using recurrent neural networks or three-dimensional convolutions. However, deep learning-based video recognition technology is usually computationally intensive and time-consuming, making it difficult to apply to time-sensitive scenarios. Summary of the Invention

[0003] In view of this, one technical problem to be solved by the present invention is to provide a method and apparatus for model training, synthetic frame generation, video recognition, and storage medium.

[0004] According to a first aspect of this disclosure, a model training method is provided, comprising: compressing a video frame training sequence using a synthetic frame generation model to obtain a two-dimensional synthetic frame; processing the two-dimensional synthetic frame using a joint training model and constructing a first loss function based on the processing result; constructing a second loss function based on the color information of the video frame training sequence and the two-dimensional synthetic frame; and jointly training the synthetic frame generation model and the joint training model based on the first loss function and the second loss function.

[0005] Optionally, the step of compressing the video frame training sequence using the synthetic frame generation model to obtain a two-dimensional synthetic frame includes: determining keyframes in the video frame training sequence; determining the motion vectors and residuals of other frames in the video frame training sequence corresponding to the keyframes; generating a video compression sequence based on the keyframes, the motion vectors, and the residuals; and generating the two-dimensional synthetic frame using the synthetic frame generation model and based on the video compression sequence.

[0006] Optionally, the joint training model includes a visual information reconstruction model; the first loss function includes a visual information reconstruction loss function; the step of using the joint training model to process the two-dimensional synthetic frame and constructing the first loss function based on the processing result includes: using the visual information reconstruction model to generate a restored keyframe corresponding to the two-dimensional synthetic frame; and constructing the visual information reconstruction loss function based on the difference information between the restored keyframe and the keyframe.

[0007] Optionally, the joint training model includes a video classification model; the first loss function includes a video classification loss function; the step of using the joint training model to process the two-dimensional synthetic frame and constructing the first loss function based on the processing result includes: using the video classification model to generate a category prediction probability corresponding to the two-dimensional synthetic frame; and constructing the video classification loss function based on the category prediction probability and the category labeling information of the synthetic frame.

[0008] Optionally, the joint training model includes a dynamic information prediction model; the first loss function includes a dynamic information prediction loss function; the step of using the joint training model to process the two-dimensional synthetic frame and constructing the first loss function based on the processing result includes: using the dynamic information prediction model and based on the two-dimensional synthetic frame, generating a predicted motion vector and a predicted residual corresponding to the key frame; determining a first difference between the predicted motion vector and the motion vector corresponding to the key frame; determining a second difference between the predicted residual and the residual corresponding to the key frame; and constructing the dynamic information prediction loss function based on the first difference and the second difference.

[0009] Optionally, the joint training model includes a discriminator model; the first loss function includes a synthetic frame generation loss function; the step of using the joint training model to process the two-dimensional synthetic frame and constructing the first loss function based on the processing result includes: using the discriminator model to make a decision on the two-dimensional synthetic frame generated by the synthetic frame generation model and the key frame; and generating the synthetic frame generation loss function based on the discrimination result.

[0010] Optionally, the second loss function includes a color consistency loss function; the step of constructing the second loss function based on the color information of the video frame training sequence and the two-dimensional synthesized frame includes: determining the average value of the first color channel of each video frame in the video frame training sequence; and constructing the color consistency loss function based on the difference between the average value of the second color channel and the average value of the first color channel of the two-dimensional synthesized frame.

[0011] Optionally, the synthetic frame generation model and the joint training model include a neural network model.

[0012] According to a second aspect of this disclosure, a method for generating composite frames is provided, comprising: compressing a video frame sequence using a composite frame generation model to obtain a two-dimensional composite frame; wherein the composite frame generation model is trained using the training method described above.

[0013] According to a third aspect of this disclosure, a video recognition method is provided, comprising: compressing a video frame sequence using a synthetic frame generation model to obtain a two-dimensional synthetic frame; wherein the synthetic frame generation model is trained by the training method described above; and using the trained video recognition model and based on the two-dimensional synthetic frame to obtain a content recognition result corresponding to the video frame sequence.

[0014] Optionally, the video recognition model includes: a two-dimensional convolutional neural network and a three-dimensional convolutional neural network.

[0015] According to a fourth aspect of this disclosure, a model training apparatus is provided, comprising: a first synthetic frame generation module, configured to compress a video frame training sequence using a synthetic frame generation model to obtain a two-dimensional synthetic frame; a first loss function construction module, configured to process the two-dimensional synthetic frame using a joint training model and construct a first loss function based on the processing result; a second loss function construction module, configured to construct a second loss function based on the color information of the video frame training sequence and the two-dimensional synthetic frame; and a model joint training model, configured to jointly train the synthetic frame generation model and the joint training model based on the first loss function and the second loss function.

[0016] Optionally, the first synthetic frame generation module is configured to: determine keyframes in the video frame training sequence; determine motion vectors and residuals corresponding to the keyframes in other frames of the video frame training sequence; generate a video compression sequence based on the keyframes, the motion vectors, and the residuals; and generate the two-dimensional synthetic frame using the synthetic frame generation model and based on the video compression sequence.

[0017] Optionally, the joint training model includes a visual information reconstruction model; the first loss function includes a visual information reconstruction loss function; the first loss function construction module is further configured to use the visual information reconstruction model to generate a restored keyframe corresponding to the two-dimensional synthesized frame; and to construct the visual information reconstruction loss function based on the difference information between the restored keyframe and the keyframe.

[0018] Optionally, the joint training model includes a video classification model; the first loss function includes a video classification loss function; the first loss function construction module is further configured to use the video classification model to generate a category prediction probability corresponding to the two-dimensional synthetic frame; and to construct the video classification loss function based on the category prediction probability and the category labeling information of the synthetic frame.

[0019] Optionally, the joint training model includes a dynamic information prediction model; the first loss function includes a dynamic information prediction loss function; the first loss function construction module is further configured to use the dynamic information prediction model and, based on the two-dimensional synthesized frame, generate a predicted motion vector and a predicted residual corresponding to the key frame; determine a first difference between the predicted motion vector and the motion vector corresponding to the key frame; determine a second difference between the predicted residual and the residual corresponding to the key frame; and construct the dynamic information prediction loss function based on the first difference and the second difference.

[0020] Optionally, the joint training model includes: a discriminator model; the first loss function includes: a synthetic frame generation loss function; the first loss function construction module is further configured to use the discriminator model to perform decision processing on the two-dimensional synthetic frame generated by the synthetic frame generation model and the key frame; and generate the synthetic frame generation loss function based on the discrimination result.

[0021] Optionally, the second loss function includes: a color consistency loss function; a second loss function construction module, specifically used to determine the average value of the first color channel of each video frame in the video frame training sequence; and to construct the color consistency loss function based on the difference between the average value of the second color channel and the average value of the first color channel of the two-dimensional synthesized frame.

[0022] Optionally, the synthetic frame generation model and the joint training model include a neural network model.

[0023] According to a fifth aspect of this disclosure, a composite frame generation apparatus is provided, comprising: a second composite frame generation module, configured to compress a video frame sequence using a composite frame generation model to obtain a two-dimensional composite frame; wherein the composite frame generation model is trained using the training method described above.

[0024] According to a sixth aspect of this disclosure, a video recognition apparatus is provided, comprising: a third composite frame generation module, configured to compress a video frame sequence using a composite frame generation model to obtain a two-dimensional composite frame; wherein the composite frame generation model is trained using the training method described above; and a recognition processing module, configured to use the trained video recognition model and, based on the two-dimensional composite frame, obtain a content recognition result corresponding to the video frame sequence.

[0025] Optionally, the video recognition model includes: a two-dimensional convolutional neural network and a three-dimensional convolutional neural network.

[0026] According to a seventh aspect of this disclosure, a model training apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method described above based on instructions stored in the memory.

[0027] According to an eighth aspect of this disclosure, a composite frame generation apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method described above based on instructions stored in the memory.

[0028] According to a ninth aspect of this disclosure, a video recognition apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method described above based on instructions stored in the memory.

[0029] According to a tenth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions which are executed by a processor using the method described above.

[0030] The model training, synthetic frame generation, video recognition method and apparatus disclosed herein, as well as the storage medium, compress the video frame training sequence to obtain a two-dimensional synthetic frame, and construct a loss function to jointly train the synthetic frame generation model and the joint training model. The generated two-dimensional synthetic frame has the ability to restore video visual details, video category, and video action information, and is close to normal images in terms of visual details and color space, and can retain more video information in the synthetic frame. It can reduce the computational workload of video recognition and model training, improve the efficiency and accuracy of model training and video recognition, has good applicability and robustness, and improves the user experience. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating an embodiment of the model training method according to the present disclosure;

[0033] Figure 2This is a schematic diagram of a training framework according to an embodiment of the model training method of this disclosure;

[0034] Figure 3 This is a flowchart illustrating an embodiment of the video recognition method according to the present disclosure;

[0035] Figure 4A and 4B This is a schematic diagram illustrating video recognition using synthesized frames according to an embodiment of the video recognition method of this disclosure;

[0036] Figure 5 This is a schematic diagram of a module of an embodiment of a model training apparatus according to the present disclosure;

[0037] Figure 6 This is a schematic diagram of a module of an embodiment of a composite frame generation apparatus according to the present disclosure;

[0038] Figure 7 This is a schematic diagram of a module of a video recognition device according to an embodiment of the present disclosure;

[0039] Figure 8 This is a schematic diagram of a module according to another embodiment of the model training apparatus according to the present disclosure;

[0040] Figure 9 This is a schematic diagram of a module according to another embodiment of the composite frame generation apparatus of the present disclosure;

[0041] Figure 10 This is a schematic diagram of a module of another embodiment of a video recognition device according to the present disclosure. Detailed Implementation

[0042] The present disclosure will now be described more fully with reference to the accompanying drawings, which illustrate exemplary embodiments of the present disclosure. The technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative effort are within the scope of protection of the present disclosure.

[0043] The terms "first" and "second" used in the following text are only used to describe the differences between them and have no other special meaning.

[0044] Existing video recognition technologies based on handcrafted features typically first detect spatiotemporal points of interest in the video, then describe these points of interest using local features, and finally perform content recognition. The local features used for description include grayscale histograms, grayscale gradient histograms, optical flow histograms, and dense trajectory features. These features are all handcrafted and not optimized for video recognition tasks, resulting in low performance in many applications.

[0045] In deep learning-based video recognition technology, two-dimensional convolutional neural networks (CNNs) used in image recognition are typically applied to video recognition. Recurrent neural networks (RNNs) or three-dimensional convolutions are then used to improve the performance of the neural network in video recognition. Generally, higher-performance neural network structures require greater computational resources.

[0046] For example, the computational cost of a video recognition system based on a two-dimensional convolutional neural network is directly linearly related to the frame sampling rate. When sampling a large number of frames, it requires significantly more computation than image recognition. Similarly, video recognition systems based on three-dimensional convolutional neural networks have much larger computational costs and model parameter sizes than two-dimensional convolutional neural networks, which greatly limits the application of video recognition in real-world scenarios.

[0047] Figure 1 This is a flowchart illustrating an embodiment of the model training method according to the present disclosure, as shown below. Figure 1 As shown:

[0048] Step 101: Compress the video frame training sequence using the synthetic frame generation model to obtain a two-dimensional synthetic frame.

[0049] In one embodiment, the synthetic frame generation model can be various models, such as a neural network model. The video frame training sequence can be multiple video frame sequences selected by the user. The synthetic frame generation model compresses the information in the video frame sequence into a two-dimensional synthetic frame. The two-dimensional synthetic frame is a two-dimensional image that contains both dynamic information and visual detail information from the video. Through the design of the loss function during training, the generated synthetic frame contains the original information such as dynamic information and visual detail information from the video. When the synthetic frame is used as input to a neural network, the neural network can predict the visual structure and dynamic information in the original video.

[0050] Step 102: Use the joint training model to process the two-dimensional synthetic frame accordingly, and construct the first loss function based on the processing results.

[0051] In one embodiment, the number of jointly trained models can be one or more, used to process the two-dimensional synthetic frames accordingly. The jointly trained models can be various models, such as neural network models.

[0052] Step 103: Construct a second loss function based on the video frame training sequence and the color information of the two-dimensional synthesized frame.

[0053] Step 104: Based on the first loss function and the second loss function, jointly train the synthetic frame generation model and the joint training model. The parameters of the synthetic frame generation model and the joint training model can be adjusted based on the first and second loss functions to achieve the training objective.

[0054] After the synthetic frame generation model and the joint training model are trained, a two-dimensional synthetic frame is generated using the trained synthetic frame generation model. The two-dimensional synthetic frame contains both dynamic information and visual details from the video. When using the two-dimensional synthetic frame for recognition processing, both spatial visual information and temporal dynamic changes of the video can be obtained from the two-dimensional synthetic frame.

[0055] In one embodiment, obtaining a two-dimensional composite frame can be achieved using various methods. For example, keyframes can be identified in a video frame training sequence, and the video frame training sequence can be determined accordingly.

[0056] The motion vectors and residuals corresponding to the keyframes in the video are used to generate other frames. Based on the keyframes, motion vectors, and residuals, a compressed video sequence is generated. A composite frame generation model is then used to generate two-dimensional composite frames based on the compressed video sequence.

[0057] like Figure 2 As shown, the synthetic frame generation model can be a convolutional neural network, such as a convolutional encoder-decoder network. The convolutional encoder-decoder network transforms the video frame training sequence into a single (two-dimensional) synthetic frame. For a video sequence (video frame training sequence) containing T frames, X = {x...} t |t=1,...,T}, where each frame is an image with a resolution of H×W and containing C channels.

[0058] Information compression of frame sequences is achieved by using a convolutional coding-decoding neural network F (a synthetic frame generation model). in These are the composite frames output by the system. Generally speaking, adjacent frames in a video frame sequence are very similar, and a frame in a video can be represented as the difference between a single frame and a nearby keyframe.

[0059] For example, the first frame x1 of the input video frame sequence is used as the keyframe, and the other frames x1 in the video frame sequence are used as keyframes. t Then x1 can be used through the motion vector m t and residual r t The result of the conversion is:

[0060] x t =x1*mt +r t (1-1);

[0061] Here, * indicates pixel-by-pixel displacement via the value of the motion vector. In the field of video compression, x1 is usually referred to as an I-frame (intra-coded frame), while {m t r t} represents the difference between a P-frame (predicted frame) and an I-frame. A complete video sequence X can be expressed as the reconstructed video sequence {x1, m2, r2, ..., m}. T r T This type of video compression sequence can be directly extracted from videos encoded in video formats such as MPEG-4, H.264, and HEVC.

[0062] In one embodiment, the jointly trained model includes a visual information reconstruction model, and the first loss function includes a visual information reconstruction loss function. The visual information reconstruction model is used to generate restored keyframes corresponding to the two-dimensional synthesized frames, and the visual information reconstruction loss function is constructed based on the difference information between the restored keyframes and the keyframes.

[0063] like Figure 2 As shown, compared with the input video data, the synthesized frames Since it contains only 1 / T of data, the best way to preserve valuable information in the synthesized frame is to make the frame compression process F as reversible as possible, meaning that the input original video X can be compressed from the synthesized frame. The visual information in a compressed domain format video sequence is mainly stored in keyframe x1, {m t r t The file only stores the construction of x. t The required motion information was thus used to construct another convolutional encoder-decoder neural network for the visual information reconstruction task. (Visual information reconstruction model) from synthesized frames Restore keyframe x1.

[0064] The loss function for appearance reconstruction represents the mean squared error between the input keyframe x1 and the reconstructed keyframe:

[0065]

[0066] in, This represents the mean squared error of the entire image. The visual information reconstruction loss function enables the synthesized frame to retain as much spatial detail as possible from the input video.

[0067] In one embodiment, the jointly trained model includes a video classification model, and the first loss function includes a video classification loss function. The video classification model is used to generate class prediction probabilities corresponding to the two-dimensional synthesized frames. Based on the class prediction probabilities and the class annotation information of the synthesized frames, a video classification loss function is constructed.

[0068] like Figure 2 As shown, video categorization is a task that classifies the spatiotemporal content of a video. Video categorization requires category labeling. A convolutional neural network C (the video classification model) is used to determine the category of the input video from synthesized frames. Given a set of video sequences and their corresponding category labels, the cross-entropy loss function is used to measure the difference between the predicted probability and the labeled category.

[0069] L cat (F, C)=CrossEntropy{y, C(F(X))} (1-3);

[0070] Where X is the input video sequence, y is the video category label, C is the video classification model, F is the synthetic frame generation model, and CrossEntroPY is the cross-entropy loss function. C(F(X)) is the probability of the convolutional neural network predicting each category, including people, actions, etc., and these probabilities are normalized to (0, 1) through a softmax operation.

[0071] In one embodiment, the jointly trained model includes a dynamic information prediction model, and the first loss function includes a dynamic information prediction loss function. Using the dynamic information prediction model and based on a two-dimensional synthesized frame, predicted motion vectors and prediction residuals corresponding to keyframes are generated. A first difference between the predicted motion vectors and the motion vectors corresponding to the keyframes is determined. A second difference between the prediction residuals and the residuals corresponding to the keyframes is determined. Based on the first and second differences, the dynamic information prediction loss function is constructed.

[0072] like Figure 2 As shown, optical flow is typically used to measure pixel-level displacement between two consecutive frames. The entire optical flow graph can be represented as a transformation function that transforms one frame into another. The motion estimation task uses a similar approach to optical flow graphs, allowing information in the synthesized frame to transform the input keyframe x1 into the subsequent t-th frame x. t Since the video is represented as a sequence of frames in the compressed domain, this transformation capability is equivalent to reconstructing the motion vector m. t and residual r t The motion vectors and residuals only contain x. t Differences between keyframe x1 and keyframe x1.

[0073] Construct a convolutional encoder-decoder neural network (Dynamic Information Prediction Model) predicts the motion vectors and residuals of all P-frames through a single forward computation. Therefore, the dynamic information prediction loss function for the dynamic information prediction task can be expressed as the mean square error between the motion vectors / residuals of the input video and the predicted motion vectors / residuals:

[0074]

[0075] Where X is the input video frame training sequence, and F is the synthetic frame generation model. For the reconstructed motion vector With residual sequence.

[0076]

[0077] m t r t The inputs are the motion vector and the residual sequence, and T is the length of the video frame training sequence.

[0078] As shown above, a total of T-1 motion vector maps and T-1 residual maps need to be restored. This amount of data is much greater than the amount of data contained in the synthesized frame. This information "bottleneck" structure can summarize valuable dynamic information in the synthesized frame as much as possible and remove redundant information between frames.

[0079] In one embodiment, the jointly trained model includes a discriminator model, and the first loss function includes a synthetic frame generation loss function. The discriminator model is used to make a decision on the two-dimensional synthetic frames and keyframes generated by the synthetic frame generation model, and the synthetic frame generation loss function is generated based on the decision result.

[0080] like Figure 2 As shown, the goal of adversarial learning regularization is to make the generated synthetic frames as visually similar to real frames as possible. A convolutional neural network D (discriminator model) is constructed as a discriminator to distinguish between real frames and synthetic frames, while a frame compression network F (synthetic frame generation model) generates high-quality synthetic frames to "deceive" the discriminator as much as possible.

[0081] By constructing a pair of generative and discriminative models and having them compete against each other, this regularization of the visual quality of the synthesized frames enables the synthesized frames to resemble real images in terms of texture, pattern, and structure. For the generated synthesized frame F(X) and the input keyframe x1, the loss function of adversarial learning (the synthesized frame generation loss function) is defined as:

[0082]

[0083] Here, x1 is a keyframe.

[0084]

[0085] Where D,(·) represents the discriminator's rating of whether an image is real. Similar to adversarial generative networks, the training process of adversarial learning is an adversarial process between model F and discriminative model D, eventually reaching a good equilibrium state, allowing F to generate synthetic frames that are visually close to normal images.

[0086] In one embodiment, the second loss function includes a color consistency loss function. The average value of the first color channel of each video frame in the training sequence is determined, and the color consistency loss function is constructed based on the difference between the average value of the second color channel and the average value of the first color channel of the two-dimensional synthesized frame.

[0087] like Figure 2 As shown, color consistency regularization is used to standardize the physical meaning of different channels in the synthesized frame. The video frames in the training sequence contain three color channels: red, green, and blue. The synthesized frame also has three color channels. If the three color channels of the synthesized frame are not constrained during training, their physical meaning will be random and unpredictable, affecting the stability of the synthesized frame and resulting in the loss of specific color information of objects.

[0088] The physical meaning of each color channel is aligned by constraining the average value of each color channel in the video frames of the training sequence to synthesize the video frames. This regularized loss function (color consistency loss function) minimizes the mean square error of the average value of each channel between the synthesized frame and the input frame.

[0089]

[0090] Where Ave(·)∈R C This represents the average value of each color channel across the entire image. X represents the video frames in the training sequence, and T is the length of the training sequence. t Let t be the t-th video frame in the training sequence of video frames.

[0091] In one embodiment, the first and second loss functions are used as the objective function. Joint training using the objective function and regularization ensures the performance of the synthesized frames in subsequent applications. Joint end-to-end training is performed using the first and second loss functions:

[0092] L = L app +L cat +L mot +R adv +Rcolor (1-9);

[0093] Among them, L app For the visual information reconstruction loss function, L cat For video classification loss function, L mot For dynamic information prediction loss function, R adv To counteract the learning loss function, R color The color consistency loss function is used. All five loss functions can use the same weights. All five loss functions apply simultaneously to the entire system (the system includes, for example, color consistency loss functions). Figure 2 The five neural network models in the model are jointly trained. Existing methods can be used to adjust the parameters of each neural network model. The first four loss functions are assisted by four neural network models in calculating the corresponding loss functions, but their final gradients all work together on the neural network model that generates the synthetic frame, thus improving the quality of the synthetic frame.

[0094] In one embodiment, a composite frame generation model is used to compress a video frame sequence to obtain a two-dimensional composite frame. The composite frame generation model is trained using the model training method described in any of the above embodiments.

[0095] Figure 3 This is a flowchart illustrating an embodiment of the video recognition method according to the present disclosure, as follows: Figure 3 As shown:

[0096] Step 301: Compress the video frame sequence using the composite frame generation model to obtain a two-dimensional composite frame. The composite frame generation model is trained using the training method described in any of the above embodiments.

[0097] Step 302: Using the trained video recognition model and based on two-dimensional synthesized frames, obtain the content recognition results corresponding to the video frame sequence.

[0098] Video recognition models include 2D convolutional neural networks and 3D convolutional neural networks. These models can be trained using various existing training methods, and the content recognition results identify objects, actions, events, etc., within the video.

[0099] In one embodiment, video recognition can be performed using both two-dimensional convolutional neural networks and three-dimensional convolutional networks. For example... Figure 4A As shown, a video frame sequence is converted into a single image (a two-dimensional composite frame), and a two-dimensional convolutional neural network is used to identify the content of the composite frame. Alternatively, 12 frames can be combined into a single two-dimensional image, and a two-dimensional residual network can be used as the classifier.

[0100] like Figure 4BAs shown, longer video frame sequences are converted into shorter synthetic frame sequences, and a 3D convolutional neural network is used to identify the content of the synthetic frame sequences. For example, a 96-frame long video sequence is converted into an 8-frame synthetic frame sequence using a sliding window method, and then a pseudo-3D residual network is used for content recognition.

[0101] In one embodiment, such as Figure 5 As shown, this disclosure provides a model training device 50, including a first synthetic frame generation module 51, a first loss function construction module 52, a second loss function construction module 53, and a joint training model 54. The first synthetic frame generation module 51 uses the synthetic frame generation model to compress the video frame training sequence to obtain a two-dimensional synthetic frame.

[0102] The first loss function construction module 52 processes the two-dimensional synthesized frame using the joint training model and constructs a first loss function based on the processing result. The second loss function construction module 53 constructs a second loss function based on the video frame training sequence and the color information of the two-dimensional synthesized frame. The joint training model 54 jointly trains the synthesized frame generation model and the joint training model based on the first and second loss functions.

[0103] In one embodiment, the first composite frame generation module 51 determines keyframes in the video frame training sequence and determines the motion vectors and residuals corresponding to the keyframes in other frames of the video frame training sequence. Based on the keyframes, motion vectors, and residuals, the first composite frame generation module 51 generates a video compression sequence, uses a composite frame generation model, and generates a two-dimensional composite frame based on the video compression sequence.

[0104] The joint training model includes a visual information reconstruction model, and the first loss function includes a visual information reconstruction loss function. The first loss function construction module 52 uses the visual information reconstruction model to generate restored keyframes corresponding to the two-dimensional synthesized frames. Based on the difference information between the restored keyframes and the keyframes, the first loss function construction module 52 constructs the visual information reconstruction loss function.

[0105] The joint training model includes a video classification model, and the first loss function includes a video classification loss function. The first loss function construction module 52 uses the video classification model to generate class prediction probabilities corresponding to the two-dimensional synthetic frames, and constructs the video classification loss function based on the class prediction probabilities and the class labeling information of the synthetic frames.

[0106] The joint training model includes a dynamic information prediction model, and the first loss function includes a dynamic information prediction loss function. The first loss function construction module 52 uses the dynamic information prediction model and, based on a two-dimensional synthesized frame, generates predicted motion vectors and prediction residuals corresponding to keyframes. The first loss function construction module 52 determines a first difference between the predicted motion vector and the motion vector corresponding to the keyframe, and determines a second difference between the prediction residual and the residual corresponding to the keyframe. Based on the first and second differences, the first loss function construction module 52 constructs the dynamic information prediction loss function.

[0107] The joint training model includes a discriminator model, and the first loss function includes a synthetic frame generation loss function. The first loss function construction module 52 uses the discriminator model to perform decision processing on the two-dimensional synthetic frames and keyframes generated by the synthetic frame generation model, and generates the synthetic frame generation loss function based on the decision results.

[0108] The second loss function includes a color consistency loss function. The second loss function construction module 53 determines the average value of the first color channel for each video frame in the video frame training sequence. The second loss function construction module 53 constructs the color consistency loss function based on the difference between the average value of the second color channel and the average value of the first color channel of the two-dimensional synthesized frame.

[0109] In one embodiment, such as Figure 6 As shown, this disclosure provides a composite frame generation apparatus 60, including a second composite frame generation module 61. The second composite frame generation module 61 uses a composite frame generation model to compress a video frame sequence to obtain a two-dimensional composite frame.

[0110] In one embodiment, such as Figure 7 As shown, this disclosure provides a video recognition device 70, including a third composite frame generation module 71 and a recognition processing module 72. The third composite frame generation module 71 uses a composite frame generation model to compress a video frame sequence to obtain a two-dimensional composite frame. The recognition processing module 72 uses a trained video recognition model and, based on the two-dimensional composite frame, obtains a content recognition result corresponding to the video frame sequence.

[0111] Figure 8 This is a schematic diagram of a module according to another embodiment of the model training apparatus of this disclosure. Figure 8 As shown, the device may include a memory 81, a processor 82, a communication interface 83, and a bus 84. The memory 81 is used to store instructions, and the processor 82 is coupled to the memory 81. The processor 82 is configured to execute the above-described model training method based on the instructions stored in the memory 81.

[0112] The memory 81 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 81 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 82 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the model training method of this disclosure.

[0113] Figure 9 This is a schematic diagram of a module according to another embodiment of the composite frame generation apparatus of the present disclosure. Figure 9 As shown, the device may include a memory 91, a processor 92, a communication interface 93, and a bus 94. The memory 91 is used to store instructions, and the processor 92 is coupled to the memory 91. The processor 92 is configured to execute the above-described composite frame generation method based on the instructions stored in the memory 91.

[0114] The memory 91 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 91 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 92 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the composite frame generation method of this disclosure.

[0115] Figure 10 This is a schematic diagram of a module according to another embodiment of the video recognition device based on the present disclosure. Figure 10 As shown, the device may include a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001 is used to store instructions, and the processor 1002 is coupled to the memory 1001. The processor 1002 is configured to execute the video recognition method described above based on the instructions stored in the memory 1001.

[0116] The memory 1001 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 1001 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 1002 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the video recognition method of this disclosure.

[0117] In one embodiment, this disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the model training method, synthetic frame generation method, or video recognition method as described in any of the preceding embodiments.

[0118] The model training, synthetic frame generation, video recognition method and device, and storage medium in the above embodiments compress the video frame training sequence to obtain a two-dimensional synthetic frame, and construct a loss function to jointly train the synthetic frame generation model and the joint training model. The generated two-dimensional synthetic frame has the ability to restore video visual details, video category, and video action information, and is close to normal images in terms of visual details and color space, and can retain more video information in the synthetic frame. It can reduce the computational workload of video recognition and model training, improve the efficiency and accuracy of model training and video recognition, has good applicability and robustness, and improves the user experience.

[0119] The methods and systems of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0120] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A model training method, comprising: The video frame training sequence is compressed using a synthetic frame generation model to obtain a two-dimensional synthetic frame. The two-dimensional synthesized frame is processed using a joint training model, and a first loss function is constructed based on the processing results. The joint training model includes a dynamic information prediction model; the first loss function includes a dynamic information prediction loss function; the step of using the joint training model to process the two-dimensional synthesized frame and constructing the first loss function based on the processing result includes: Using the dynamic information prediction model and based on the two-dimensional synthesized frame, a predicted motion vector and a prediction residual corresponding to the key frame are generated; a first difference between the predicted motion vector and the motion vector corresponding to the key frame is determined; a second difference between the prediction residual and the residual corresponding to the key frame is determined; and the dynamic information prediction loss function is constructed based on the first difference and the second difference. A second loss function is constructed based on the video frame training sequence and the color information of the two-dimensional synthesized frame; Based on the first loss function and the second loss function, the synthetic frame generation model and the joint training model are jointly trained to adjust the parameters of the synthetic frame generation model and the joint training model.

2. The method as described in claim 1, wherein, The step of compressing the video frame training sequence using a synthetic frame generation model to obtain a two-dimensional synthetic frame includes: Keyframes are determined in the video frame training sequence; Determine the motion vectors and residuals of other frames in the video frame training sequence corresponding to the key frame; Based on the keyframes, the motion vectors, and the residuals, a video compression sequence is generated, and the two-dimensional composite frame is generated using the composite frame generation model and based on the video compression sequence.

3. The method as described in claim 1, wherein, The joint training model includes: a visual information reconstruction model; the first loss function includes: a visual information reconstruction loss function; the step of using the joint training model to process the two-dimensional synthesized frame accordingly, and constructing the first loss function based on the processing result includes: The visual information reconstruction model is used to generate restored keyframes corresponding to the two-dimensional synthetic frames; Based on the difference information between the restored keyframes, the visual information reconstruction loss function is constructed.

4. The method of claim 1, wherein the joint training model comprises: Video classification model; The first loss function includes: a video classification loss function; the step of using a jointly trained model to process the two-dimensional synthesized frame and constructing the first loss function based on the processing result includes: The video classification model is used to generate the category prediction probability corresponding to the two-dimensional synthetic frame; The video classification loss function is constructed based on the predicted category probability and the category labeling information of the synthesized frame.

5. The method of claim 1, wherein the joint training model comprises: Discriminator model; The first loss function includes: a synthetic frame generation loss function; the step of processing the two-dimensional synthetic frame using a jointly trained model and constructing the first loss function based on the processing result includes: The discriminator model is used to make a decision on the two-dimensional composite frame generated by the composite frame generation model and the key frame; The synthetic frame generation loss function is generated based on the judgment result.

6. The method of claim 1, wherein the second loss function comprises: Color consistency loss function; The step of constructing the second loss function based on the video frame training sequence and the color information of the two-dimensional synthesized frame includes: Determine the average value of the first color channel of each video frame in the video frame training sequence; The color consistency loss function is constructed based on the difference between the average value of the second color channel and the average value of the first color channel of the two-dimensional composite frame.

7. The method of claim 1, wherein, The synthetic frame generation model and the joint training model include a neural network model.

8. A method for generating composite frames, comprising: The video frame sequence is compressed using a composite frame generation model to obtain a two-dimensional composite frame. The synthetic frame generation model is trained using the training method described in any one of claims 1 to 7.

9. A video recognition method, comprising: The video frame sequence is compressed using a composite frame generation model to obtain a two-dimensional composite frame; wherein the composite frame generation model is trained by the training method described in any one of claims 1 to 7. Using the trained video recognition model and based on the two-dimensional synthesized frame, the content recognition result corresponding to the video frame sequence is obtained.

10. The method of claim 9, wherein, The video recognition model includes: two-dimensional convolutional neural network and three-dimensional convolutional neural network.

11. A model training device, comprising: The first synthetic frame generation module is used to compress the video frame training sequence using the synthetic frame generation model to obtain a two-dimensional synthetic frame. The first loss function construction module is used to process the two-dimensional synthetic frame using the joint training model and construct the first loss function based on the processing result. The joint training model includes a dynamic information prediction model; the first loss function includes a dynamic information prediction loss function; the first loss function construction module is used to generate a predicted motion vector and a prediction residual corresponding to a key frame using the dynamic information prediction model and based on the two-dimensional synthesized frame; determine a first difference between the predicted motion vector and the motion vector corresponding to the key frame; determine a second difference between the prediction residual and the residual corresponding to the key frame; and construct the dynamic information prediction loss function based on the first difference and the second difference. The second loss function construction module is used to construct a second loss function based on the video frame training sequence and the color information of the two-dimensional synthesized frame; The model joint training model is used to jointly train the synthetic frame generation model and the joint training model based on the first loss function and the second loss function, so as to adjust the parameters of the synthetic frame generation model and the joint training model.

12. A composite frame generation apparatus, comprising: The second composite frame generation module is used to compress the video frame sequence using a composite frame generation model to obtain a two-dimensional composite frame; wherein the composite frame generation model is trained by the training method of any one of claims 1 to 7.

13. A video recognition device, comprising: The third composite frame generation module is used to compress the video frame sequence using a composite frame generation model to obtain a two-dimensional composite frame; wherein, the composite frame generation model is trained by the training method of any one of claims 1 to 7. The recognition processing module is used to obtain content recognition results corresponding to the video frame sequence using a trained video recognition model and based on the two-dimensional synthesized frame.

14. A model training device, comprising: Memory; And a processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 7 based on instructions stored in the memory.

15. A composite frame generation apparatus, comprising: Memory; and a processor coupled to the memory, the processor being configured to perform the method of claim 8 based on instructions stored in the memory.

16. A video recognition device, comprising: Memory; and a processor coupled to the memory, the processor being configured to perform the method as described in claim 9 or 10 based on instructions stored in the memory.

17. A computer-readable storage medium storing computer instructions that are executed by a processor according to any one of claims 1 to 10.

Images