Video generation method, device, equipment and computer readable storage medium

Abstract

The present application provides a video generation method, device, equipment and computer-readable storage medium; it relates to artificial intelligence technology; the method comprises: obtaining a historical video frame sequence corresponding to the current time step; feature encoding the last video frame in the historical video frame sequence to obtain the graphic features corresponding to the last video frame; predicting the position features corresponding to the current time step based on the historical video frame sequence; feature alignment processing of the position features and the graphic features to obtain the features to be decoded; wherein the features to be decoded describe the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame; feature decoding of the features to be decoded to obtain the predicted video frame of the current time step. Through the present application, the quality of the generated video can be improved.

Description

Technical Field

[0001] The present application relates to computer vision technology in the field of artificial intelligence, and in particular to a video generation method, apparatus, device and computer-readable storage medium. Background Art

[0002] Video generation is an important direction of artificial intelligence. Video generation can be divided into two categories: deterministic video generation and random video generation. Random video generation can generate multiple video sequences with different future states and reasonable motion. It introduces hidden variables to explain the complex random phenomena in video sequences.

[0003] However, in the related art, random video generation may result in feature misalignment, which causes the appearance of moving objects in the video to be lost, so that the appearance features constructed at future time steps will be blurred, ultimately resulting in lower quality of the generated video. Summary of the invention

[0004] The embodiments of the present application provide a video generation method, apparatus, device and computer-readable storage medium, which can improve the quality of generated videos.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a video generation method, including:

[0007] Get the historical video frame sequence corresponding to the current time step;

[0008] Performing feature encoding on the last video frame in the historical video frame sequence to obtain graphic features corresponding to the last video frame;

[0009] Predicting the position features corresponding to the current time step based on the historical video frame sequence;

[0010] Performing feature alignment processing on the position feature and the graphic feature to obtain a feature to be decoded; wherein the feature to be decoded describes the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame;

[0011] Feature decoding is performed on the feature to be decoded to obtain a predicted video frame of the current time step.

[0012] The present application provides a video generation method, including:

[0013] Acquire a training observation frame sequence, a training true value frame sequence, an initial prior distribution inference model, an initial feature prediction model, a posterior distribution inference model, and training graphic features corresponding to the training preset frames in the training observation frame sequence;

[0014] Using the posterior distribution inference model, inferring the posterior distribution of the training true value features obtained after feature encoding of the training true value frame sequence, to obtain an approximate posterior distribution;

[0015] Performing feature encoding on the training observation frame sequence to obtain training observation features corresponding to the training observation frame sequence;

[0016] Based on the approximate posterior distribution and the training observation features, the initial prior distribution inference model is trained until a training stop condition is reached to obtain a preset prior distribution inference model; the preset prior distribution inference model is used to learn the timing information of the historical video frame sequence;

[0017] The initial feature prediction model is trained according to the approximate posterior distribution, the training observation features and the training graphic features until a training stop condition is reached to obtain a preset feature prediction model; the preset feature prediction model is used to predict a model of unknown features.

[0018] The embodiment of the present application provides a video generation device, including:

[0019] An information acquisition module is used to obtain the historical video frame sequence corresponding to the current time step;

[0020] A feature encoding module, used for performing feature encoding on the last video frame in the historical video frame sequence to obtain a graphic feature corresponding to the last video frame;

[0021] A feature prediction module, used to predict the position feature corresponding to the current time step based on the historical video frame sequence;

[0022] A feature alignment module, used for performing feature alignment processing on the position feature and the graphic feature to obtain a feature to be decoded; wherein the feature to be decoded describes the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame;

[0023] The feature decoding module is used to perform feature decoding on the feature to be decoded to obtain the predicted video frame of the current time step.

[0024] In some embodiments of the present application, the feature alignment module is also used to construct an attention map based on the position feature and the graphic feature; determine the alignment feature used to describe the context information of the area where the mobile object is located based on the attention map and the position feature; and use the alignment feature, the position feature and the graphic feature to fuse them into the feature to be decoded.

[0025] In some embodiments of the present application, the feature alignment module is also used to map the position feature to a first preset feature space to obtain a first mapping feature map, and to map the graphic feature to the first preset feature space to obtain a second mapping feature map; for each spatial position in the first mapping feature map, extract a first spatial vector from the first mapping feature map; extract spatial vectors for multiple target positions corresponding to each spatial position in the second mapping feature map to obtain multiple second spatial vectors; and construct the attention map based on the degree of similarity between the first spatial vector and the multiple second spatial vectors.

[0026] In some embodiments of the present application, the feature alignment module is also used to calculate the similarity between the first space vector and the multiple second space vectors respectively to obtain multiple similarity scores corresponding to the multiple second space vectors; normalize the multiple similarity scores respectively to obtain multiple normalized scores corresponding to the multiple second space vectors; use the multiple normalized scores to form the attention vector corresponding to each spatial position; use the attention vector corresponding to each spatial position to form the attention map.

[0027] In some embodiments of the present application, the feature alignment module is also used to map the position feature to a second preset feature space to obtain a third mapping feature map; for each pixel position in the third mapping feature map, extract a plurality of third space vectors from the third mapping feature map; use the plurality of third space vectors to form a feature set corresponding to each pixel position; perform matrix operations on the feature set and the attention map to obtain the alignment feature used to describe the context information of the area where the moving object is located.

[0028] In some embodiments of the present application, the feature alignment module is also used to screen the positions in the third mapping feature map that are in the same row and column as each pixel position to obtain a plurality of positions to be extracted; and extract spatial vectors for the plurality of positions to be extracted to obtain the plurality of third spatial vectors corresponding to each pixel position.

[0029] In some embodiments of the present application, the feature alignment module is further used to use the convolution kernel corresponding to the second preset feature space to perform a convolution operation on the position feature to achieve mapping, so as to obtain the third mapping feature map.

[0030] In some embodiments of the present application, the feature prediction module is also used to perform feature encoding on the historical video frame sequence to obtain historical video features; use a preset prior distribution inference model to learn the prior distribution corresponding to the current time step from the historical video features; the preset prior distribution inference model is a trained model for learning the prior distribution of the temporal information of the historical video frame sequence; sample the predicted latent variables corresponding to the current time step from the prior distribution; use a preset feature prediction model to perform feature prediction on the predicted latent variables and the historical video features to obtain the position features corresponding to the current time step; the preset feature prediction model is a trained model for predicting unknown features.

[0031] An embodiment of the present application provides a video generating device, including:

[0032] A data acquisition module, used to acquire a training observation frame sequence, a training true value frame sequence, an initial prior distribution inference model, an initial feature prediction model, a posterior distribution inference model, and training graphic features corresponding to the training preset frames in the training observation frame sequence;

[0033] A distribution inference module, used to use the posterior distribution inference model to infer the posterior distribution of the training true value features obtained after feature encoding of the training true value frame sequence, so as to obtain an approximate posterior distribution;

[0034] A training encoding module, used to perform feature encoding on the training observation frame sequence to obtain training observation features corresponding to the training observation frame sequence;

[0035] A model training module is used to train the initial prior distribution inference model based on the approximate posterior distribution and the training observation features until the training stop condition is reached, thereby obtaining a preset prior distribution inference model; the preset prior distribution inference model is used to learn the timing information of the historical video frame sequence; the initial feature prediction model is trained based on the approximate posterior distribution, the training observation features and the training graphic features until the training stop condition is reached, thereby obtaining a preset feature prediction model; the preset feature prediction model is used to predict a model of unknown features.

[0036] In some embodiments of the present application, the model training module is also used to use the initial prior distribution inference model to infer the prior distribution of the training observation features, so as to obtain the training prior distribution corresponding to the training observation frame sequence; and to use the prior loss between the approximate posterior distribution and the training prior distribution to adjust the parameters of the initial prior distribution inference model until the training stop condition is met, thereby obtaining the preset prior distribution inference model.

[0037] In some embodiments of the present application, the model training module is also used to sample training latent variables from the approximate posterior distribution, and perform feature prediction on the training latent variables and the training observation features through the initial feature prediction model to obtain training prediction features; perform feature alignment on the training prediction features and the training graphic features to obtain training features to be decoded; perform feature decoding on the training features to be decoded to obtain training prediction frames; and use the reconstruction loss between the training prediction frames and the training true value frame sequence to adjust the parameters of the initial feature prediction model until the training stop condition is reached to obtain the preset feature prediction model.

[0038] The present application provides a video generation device, including:

[0039] A first memory, used to store executable video generation instructions;

[0040] The first processor is used to implement the video generation method provided in the embodiment of the present application when executing the executable video generation instruction stored in the first memory.

[0041] The present application provides a video generation device, including:

[0042] A second memory, used for storing executable video generation instructions;

[0043] The second processor is used to implement the video generation method provided in the embodiment of the present application when executing the executable video generation instruction stored in the second memory.

[0044] An embodiment of the present application provides a computer-readable storage medium storing executable video generation instructions, which are used to cause a first processor to execute and implement the video generation method provided by the embodiment of the present application; or to cause a second processor to execute and implement the video generation method provided by the embodiment of the present application.

[0045] The embodiments of the present application have the following beneficial effects: the video generation device performs feature encoding on the last video frame in the historical video frame sequence to obtain image features, and uses the historical video frame sequence to predict the position features corresponding to the current time step, and then, through alignment of the position features and the graphic features, can obtain the area that simultaneously describes the appearance area of ​​the moving object at the current time step and the area of ​​the appearance of the moving object that actually exists in the last video frame, that is, the area where the appearance of the moving object can be truly captured from the last video frame is clarified, so that when feature decoding is performed on the features to be decoded, the complete appearance of the moving object can be extracted based on the area, so that the predicted video frame obtained by feature decoding is clearer, thereby improving the quality of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the framework of the random video generation model during training;

[0047] Figure 2 This is a schematic diagram of the framework of the random video generation model during prediction;

[0048] Figure 3 It is a schematic diagram of the dislocation of appearance features;

[0049] Figure 4 is an optional architecture diagram of a video generation system 100 provided in an embodiment of the present application;

[0050] Figure 5A The embodiment of this application provides Figure 4 A structural diagram of a server in;

[0051] Figure 5B The embodiment of this application provides Figure 4 Another structural diagram of the server in FIG.

[0052] Figure 6 This is an optional process diagram of the video generation method provided in the embodiment of the present application. Figure 1 ;

[0053] Figure 7 This is an optional process diagram of the video generation method provided in the embodiment of the present application. Figure 2 ;

[0054] Figure 8 is a schematic diagram of extracting a first space vector provided in an embodiment of the present application;

[0055] Fig. 9 This is a schematic diagram of the target location range provided in the embodiment of the present application. Figure 1 ;

[0056] Fig.10 This is a schematic diagram of the target location range provided in the embodiment of the present application. Figure 2 ;

[0057] Fig.11 is a schematic diagram of a process for generating location features provided in an embodiment of the present application;

[0058] Fig. 12A This is an optional process diagram 3 of the video generation method provided in the embodiment of the present application;

[0059] Fig. 12B It is a schematic diagram of the process of model training provided in an embodiment of the present application;

[0060] Fig.13 It is a schematic diagram of a probability graph model for random video generation provided in an embodiment of the present application;

[0061] Fig.14 is a schematic diagram of a feature alignment process provided by an embodiment of the present application;

[0062] Fig.15 It is a schematic diagram comparing the effects of the video generation method provided in this application and the video generation method in the related art on a pedestrian dataset. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0064] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0065] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0068] 1) Cloud Technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or a local area network to achieve data computing, storage, processing and sharing.

[0069] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, and intelligence is realized through cloud computing.

[0070] 2) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0071] Artificial intelligence technology is a comprehensive study that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer practical technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] 3) Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, detect and measure targets, and further perform image processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0073] 4) Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0074] 5) Video Generation, also called video prediction, is a method of predicting future sequences based on observed video sequences.

[0075] 6) Deterministic Video Generation: It is believed that there is only one possibility for the future of the observed video sequence, that is, the future of the observed video sequence is determined.

[0076] 7) Stochastic Video Generation: It is believed that the future of the observed video sequence is uncertain. For example, the speed and direction of movement are likely to change. Therefore, in the stochastic video generation method, different samples can be collected from the distribution of the latent variables at each time step to generate reasonable future video sequences with multiple motion states.

[0077] 8) Time step refers to the time unit for predicting video frames.

[0078] 9) Feature Alignment: It can give the area where the appearance of the moving object in the last video frame is actually located, so as to facilitate the complete extraction of the appearance of the moving object from the area.

[0079] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0080] Video generation is an important direction of artificial intelligence. Video generation can be divided into two categories: deterministic video generation and random video generation. The motion of objects in the video is uncertain, and deterministic video generation assumes that the future state of the observed video frame sequence is certain, and it tends to learn the average value of the future state based on the observed video frames. It can be seen that deterministic video generation is difficult to deal with future uncertainty, and always leads to the averaging of future states, which makes the predicted video sequence blurred.

[0081] Random video generation can generate multiple video sequences with different future states and reasonable motion. It introduces hidden variables to explain the complex random phenomena in video sequences. In the process of random video generation, we first need to learn the prior distribution corresponding to the features at the current time step based on the observed video frames, then sample the hidden variables in the prior distribution, decode the hidden variables into the pixel space, and then get the predicted frame sequence corresponding to the current time step. Since the hidden variables can sample different values ​​at each time step, it is possible to generate frame sequences with different future states.

[0082] For example, let x 1:T represents a sequence of T consecutive video frames, then random video generation means observing the previous k frames, learning the prior distribution corresponding to the current time step t, and sampling the hidden variable z of the current time step from the prior distribution t , and decode it into pixel space through the decoder, so as to predict a video frame of length Tk, that is, to obtain a predicted frame sequence.

[0083] In the related art, most of them use the Long Short Term Memory (LSTM) network to build a stochastic video generation (SVG) model to generate a predicted frame sequence based on the observed video frames. The SVG model uses the approximate posterior distribution q φ (z t |x 1:t ) to estimate z t The true posterior distribution of (because the true posterior distribution is difficult to obtain), the approximate posterior distribution q φ (z t |x 1:t ) is composed of a conditional Gaussian distribution Ν(μ φ (x 1:t-1 ),σ φ (x 1:t-1 )) and make the approximate posterior distribution q through the prior loss φ (z t |x 1:t ) approximates the prior distribution p at the current time step t ψ (zt |x 1:t-1 ). That is, in the SVG model, the prior distribution p ψ (z t |x 1:t-1 ) is learned based on past states.

[0084] For example, see Figure 1 , Figure 1 This is a schematic diagram of the framework of the random video generation model during training. During the training process, the SVG model first uses the encoder to generate the video frame sequence 1-a (i.e. x) at the current time step t. t is the true value) and encodes 1-1 to get x t The characteristics of x t The features are input into the approximate posterior distribution inference model 1-2, namely LSTM φ Inference model, to produce x t The approximate posterior distribution of 1-3, that is, N(μ φ(t) , σ φ(t) ), by minimizing the prior loss 1-4, we use the approximate posterior distribution to approximate the observed video frame sequence 1-b according to the historical time step (1~t-1), that is, for x t-1 The features obtained by encoding 1-5 are input into the prior distribution inference model 1-6, i.e. LSTM ψ Inference model, the learned prior distribution 1-7, that is, N(μ ψ(t) ,σ ψ(t) ). At the same time, during the training process, the SVG model also analyzes the observed video frame sequence x from the historical time step 1 to t-1. t-1 Encode 1-8 and get x t-1 The features of the latent variables 1-9 (z t ), and replace hidden variables 1-9 and x t-1 The features are input to the feature inference model 1-10, namely LSTM θ In the inference model, the predicted features are obtained, and then the predicted video frame sequence 1-c obtained by decoding the predicted features 1-11 into the pixel space through the decoder is calculated, that is, By predicting the video frame sequence The reconstruction loss 1-12 directly with the true value is constrained to be as close to x as possible t .

[0085] Continue to see Figure 2 , Figure 2It is a schematic diagram of the framework of the random video generation model during prediction. During the prediction process, the SVG model directly uses the observed video frame sequence 2-1 of the historical time step (1~t-1) for encoding 2-2, obtains the features of these observed video frames, and then inputs the features into the prior distribution inference model 2-3, namely LSTM ψ In the inference model, we get the prior distribution 2-4, that is, N(μ ψ(t) ,σ ψ(t) ), then sample from the prior distribution 2-4 to obtain the hidden variable 2-5 (i.e. z t ), the latent variable 2-5 and the features obtained by encoding the observed video frame sequence 2-1 by the encoder 2-6 are input into the feature inference model 2-7, i.e., LSTM θ In the inference model, LSTM θ The predicted features of the current time step are decoded 2-8 to obtain the predicted frame sequence 2-9 of the current time step (i.e. ).

[0086] In the related art, in addition to training and predicting in the above manner, the SVG model also uses skip connections to connect the last observed frame x in the observed video frame sequence. k The encoder of is directly connected to the decoder of the current time step t, so that the decoder of the current time step t can directly obtain the last observed frame x k The appearance features are copied to the encoder of , so as to directly use the copied appearance features to construct the appearance features of the current time step t.

[0087] However, the appearance features at different time steps are different. For example, see Figure 3 , Figure 3It is a schematic diagram of appearance feature dislocation. In the last observation frame 3-1 of a certain observation video sequence, the moving object is at the position of the dotted frame 3-11. As the time step increases, when the video frame 3-2 of the current time step is reached, the moving object should move to the position of the annotation frame 3-21. In the related art, the SVG model extracts the appearance features in the annotation frame 3-12 corresponding to the annotation frame 3-21 in the last observation frame 3-1. But obviously, there is no annotated object in the annotation frame 3-12 in the last observation frame 3-1. Directly extracting the appearance features of the moving object from the last observation frame 3-1 will lead to feature dislocation. Similarly, in the last observation frame 3-3 of another observation video sequence, the annotation frame 3-31 is the receptive field required to construct the moving object (i.e., the number 8). This receptive field corresponds to the position of the annotation frame 3-41 where the moving object is located in the video frame 3-4 of the current time step, but the moving object is not in the annotation frame 3-31. If the appearance features of the moving object are directly extracted from the last observation frame 3-3, it will lead to feature dislocation.

[0088] It can be seen that in the related art, when generating random videos, directly copying the appearance features from the encoder of the last observed frame will cause feature dislocation. As a result, as the time step increases, the appearance features of the moving object will be partially or even completely missed in the decoder. In this way, the appearance features of the future time step constructed using the last observed frame will be blurred, ultimately resulting in lower quality of the generated video.

[0089] The embodiments of the present application provide a video generation method, apparatus, device and computer-readable storage medium, which can improve the quality of the generated video. The exemplary application of the video generation device provided by the embodiments of the present application is described below. The device provided by the embodiments of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), and can also be implemented as a server, and can also be a device cluster composed of a server and a terminal in real time. Below, an exemplary application when the device is implemented as a server will be described.

[0090] See also Figure 4 , Figure 4 It is an optional architectural diagram of the video generation system 100 provided in an embodiment of the present application. To support a video generation application, the terminal (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0091] The server 200 obtains the historical video frame sequence corresponding to the current time step, and encodes the last video frame in the historical video frame to obtain the graphic features corresponding to the last video frame. Next, the server 200 predicts the position features corresponding to the current time step based on the historical video frame sequence, and then performs feature alignment processing on the position features and the graphic features to obtain the features to be decoded, wherein the features to be decoded describe the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame. The server 200 decodes the features to be decoded to obtain the predicted video frame of the current time step to achieve video generation.

[0092] Next, the server 200 may send the predicted video frame to the terminal 400 - 1 and the terminal 400 - 2 so that the predicted video frame can be displayed in the graphical interface 400 - 11 of the terminal 400 - 1 and in the graphical interface 400 - 21 of the terminal 400 - 2.

[0093] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0094] It can be understood that the video generation device provided in the embodiment of the present application can be used to generate a predicted video frame for the current time step based on the historical video frame sequence of the current time step, and can also be used to train a preset prior distribution inference model for learning the timing information of the historical video frame sequence and a preset feature prediction model for predicting position features based on the training observation frame sequence, the training true value frame sequence, the initial prior distribution model, the initial feature prediction model, the posterior distribution inference model and the training graphic features, so as to facilitate the use of the preset prior distribution inference model and the preset feature prediction model to learn the historical video frame sequence.

[0095] In some embodiments, the video generation device is used to generate a predicted video frame for the current time step. In this case, see Figure 5A , Figure 5A The embodiment of this application provides Figure 4 A structural diagram of a server in Figure 5AThe server 200 shown includes: at least one first processor 210, a first memory 250, at least one first network interface 220 and a first user interface 230. The various components in the server 200 are coupled together via a first bus system 240. It is understood that the first bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the first bus system 240 is not used in the example of FIG. Figure 5A In the figure, various buses are labeled as a first bus system 240 .

[0096] The first processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0097] The first user interface 230 includes one or more first output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The first user interface 230 also includes one or more first input devices 232, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0098] The first memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The first memory 250 may optionally include one or more storage devices physically located away from the first processor 210.

[0099] The first memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The first memory 250 described in the embodiment of the present application is intended to include any suitable type of memory.

[0100] In some embodiments, the first memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0101] The first operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0102] A first network communication module 252, used to reach other computing devices via one or more (wired or wireless) first network interfaces 220, exemplary first network interfaces 220 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), etc.;

[0103] A first presentation module 253 for enabling presentation of information (e.g., a user interface for operating peripherals and displaying content and information) via one or more first output devices 231 (e.g., a display screen, a speaker, etc.) associated with the first user interface 230;

[0104] The first input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more first input devices 232 and to translate the detected inputs or interactions.

[0105] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 5A A video generating device 255 stored in the first memory 250 is shown, which may be software in the form of a program and a plug-in, etc., including the following software modules: an information acquisition module 2551, a feature encoding module 2552, a feature prediction module 2553, a feature alignment module 2554 and a feature decoding module 2555. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0106] The functions of each module will be described below.

[0107] In other embodiments, the video generating device 255 provided in the embodiment of the present application can be implemented in hardware. As an example, the video generating device 255 provided in the embodiment of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video generating method provided in the embodiment of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0108] Exemplarily, an embodiment of the present application provides a video generating device, including:

[0109] A first memory, used to store executable video generation instructions;

[0110] The first processor is used to implement the video generation method provided in the embodiment of the present application when executing the executable video generation instruction stored in the first memory.

[0111] In other embodiments, the video generation device is used to train a preset prior distribution inference model and a preset feature prediction model used in the video generation process. In this case, see Figure 5B , Figure 5B The embodiment of this application provides Figure 4 Another structural diagram of the server in Figure 5B The server 500 shown includes: at least one second processor 510, a second memory 550, at least one second network interface 520 and a second user interface 530. The various components in the server 500 are coupled together via a second bus system 540. It is understood that the second bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the second bus system 540 is not used in the example of FIG. Figure 5B In the figure, various buses are labeled as a second bus system 540.

[0112] The second processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0113] The second user interface 530 includes one or more second output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The second user interface 530 also includes one or more second input devices 532, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0114] The second memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The second memory 550 may optionally include one or more storage devices physically located away from the second processor 510.

[0115] The second memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The second memory 550 described in the embodiment of the present application is intended to include any suitable type of memory.

[0116] In some embodiments, the second memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0117] The second operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0118] A second network communication module 552, for reaching other computing devices via one or more (wired or wireless) second network interfaces 520, exemplary second network interfaces 520 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), etc.;

[0119] a second presentation module 553 for enabling presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more second output devices 531 (e.g., a display screen, a speaker, etc.) associated with the second user interface 530;

[0120] The second input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more second input devices 532 and translate the detected inputs or interactions.

[0121] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 5B A video generating device 555 stored in a second memory 550 is shown, which may be software in the form of a program or plug-in, including the following software modules: a data acquisition module 5551, a distribution inference module 5552, a training encoding module 5553, and a model training module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0122] The functions of each module will be described below.

[0123] In other embodiments, the video generating device 555 provided in the embodiment of the present application can be implemented in hardware. As an example, the video generating device 555 provided in the embodiment of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video generating method provided in the embodiment of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0124] Exemplarily, an embodiment of the present application provides a video generating device, including:

[0125] A second memory, used for storing executable video generation instructions;

[0126] The second processor is used to implement the video generation method provided in the embodiment of the present application when executing the executable video generation instruction stored in the second memory.

[0127] Below, the video generation method provided by the embodiment of the present application will be described in combination with the exemplary application and implementation of the video generation device provided by the embodiment of the present application. It should be noted that the video generation device can be implemented as a server or a terminal, so that the video generation method provided by the embodiment of the present application can be implemented on the server or the terminal; the video generation device can also be implemented as a device cluster composed of a server and a terminal. In this case, the video generation method provided by the embodiment of the present application can be implemented by the server and the terminal with the help of cloud technology.

[0128] See also Figure 6 , Figure 6 This is an optional process diagram of the video generation method provided in the embodiment of the present application. Figure 1 , will combine Figure 6 The steps shown are explained.

[0129] S101. Obtain a historical video frame sequence corresponding to a current time step.

[0130] The embodiment of the present application is implemented in a scenario where unknown video frames are generated based on an existing video frame sequence. For example, a new video frame is generated based on a road video frame sequence recorded by a trip recorder, so as to predict the road conditions in advance during automatic driving, or a new video frame is generated based on a video frame sequence of surrounding scenes recorded by a robot's camera, so as to predict the next scene of the surrounding scene so that the robot can react quickly, etc. When the video generation device starts video generation, it needs to first obtain the historical video frame sequence of the time step before the current time step. At the same time, the video generation device also obtains the graphic features of the last video frame in the historical video frame sequence to facilitate subsequent video generation.

[0131] It can be understood that the current time step is the time step corresponding to the current moment, and the time step is the time unit for performing various operations on the video frame sequence, for example, the time unit for observing the existing video frame sequence and predicting the unknown video frame, etc. The duration of the time step can be set according to the actual situation. For example, the duration of the time step can be set to 10s, so that the video generation device can observe the existing video frame once every 10s; the duration of the time step can be set to 30s, and the video generation device will generate a new video frame every 30s, etc.

[0132] The historical video frame sequence is a sequence of video frames that existed before the current time step, arranged in chronological order. The historical video frame sequence may include real captured video frames, and may also include video frames generated before the current time step (e.g., the previous time step), which is not limited in this application.

[0133] S102: Perform feature encoding on the last video frame in the historical video frame sequence to obtain graphic features corresponding to the last video frame.

[0134] After the video generation device obtains the historical video frame sequence, it will locate the last video frame from the historical video frame sequence, and then encode the last video frame through a feature encoder to obtain the graphic features corresponding to the last video frame.

[0135] It should be noted that, in some embodiments, in addition to encoding the features of the last video frame, the video generation device may also encode video frames in a preset order other than the last one in the historical video frame sequence, so as to generate a predicted video frame using the video frames in the preset order. The video frame in the preset order may be the second to last video frame in the historical video frame sequence, or the second positive video frame in the historical video frame sequence, or other video frames in the historical video frame sequence, etc., which is not limited in this application.

[0136] It is understandable that, since there are generally one or more moving objects in a video, video generation is to predict the future state of these moving objects so that the generated predicted video frames can reasonably describe the future motion state of these moving objects. There will also be moving objects in the last video frame, so the graphic features extracted from the last video frame contain the appearance of the moving objects, so that the video generation device can directly use the appearance of the moving objects in the graphic features to generate the video.

[0137] In some embodiments, the feature encoder may be a variational auto-encoder (VAE), and in other embodiments, the feature encoder may also be a convolutional neural network (CNN). Of course, the feature encoder may also be other encoders capable of converting video frames into feature space, which is not limited in this application.

[0138] S103: predicting the position features corresponding to the current time step based on the historical video frame sequence.

[0139] The video generation device converts the historical video frame sequence into a feature space, obtains the historical video features corresponding to the historical video frame sequence, and then determines the predicted latent variables of the current time step based on the historical features, thereby combining the predicted latent variables and the historical video features to infer the video features corresponding to the current time step, and then decodes the video features of the current time step through a feature decoder to obtain position features for subsequent feature alignment. Alternatively, a moving trajectory is determined using the position of each video frame of the moving object in the historical video frame sequence, and the position features of the current time step are determined based on the moving trajectory.

[0140] It should be noted that the position feature is obtained by learning the temporal information of the moving objects in the video. Therefore, the position feature can indicate the motion state of the moving object in the current time step, which also clarifies in which area of ​​the predicted video frame the moving object will appear at the current time step.

[0141] At the same time, the predicted latent variable refers to the latent variable sampled from the learned prior distribution, which can be understood as predicted based on the historical video frame sequence. The predicted latent variable is used to explain the random phenomena in the video sequence. The predicted latent variables at different time steps are different. Therefore, the position features predicted based on the predicted latent variable contain motion states that are different from other time steps.

[0142] It should be noted that simply obtaining the motion state of the current time step is not sufficient to generate a video. The video generation device also needs to obtain the appearance of the moving object in order to generate a predicted video frame that can clearly describe the motion of the moving object.

[0143] S104: Perform feature alignment processing on the position feature and the graphic feature to obtain a feature to be decoded.

[0144] Graphic features include not only the appearance of the moving object, but also the background or the appearance of other objects. If the video generation device directly decodes the position features and image features to determine the area where the moving object appears at the current time step, then based on this area, the appearance of the moving object is directly copied from the last video frame (for example Figure 3 ), so the real appearance of the moving object cannot be guaranteed, and the quality of the final video frame sequence obtained will not be high. Therefore, in the embodiment of the present application, before generating the predicted video frame, the video generation device will first perform feature alignment on the position feature and the graphic feature, so that the video generation device can clearly identify the information of the area where the appearance of the moving object can be truly extracted in the final video frame, and then combine the information of the area with the position feature and the graphic feature to obtain the feature to be decoded.

[0145] That is to say, the feature to be decoded describes the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame, so that the video generation device will not miss or omit the appearance of the moving object in the predicted video frame generated based on the feature to be decoded, that is, the appearance of the moving object is complete. This area where the real moving object exists can be inferred based on the motion state of the moving object in the position feature, for example, it can be inferred from the moving trajectory of the moving object corresponding to the position feature, or it can be obtained by analyzing the position feature and the graphic feature through the attention mechanism, which is not limited in this application.

[0146] S105 , performing feature decoding on the features to be decoded to obtain a predicted video frame at the current time step.

[0147] The video generating device decodes the features to be decoded so as to find the area from the last video frame from which the appearance of the moving object can be truly extracted, and then extracts the features corresponding to the area in the graphic features, thereby completing the appearance of the moving object and obtaining a clearer appearance of the moving object. The extracted features and position features are then mapped together to the pixel space to obtain a clearer video frame.

[0148] It should be noted that the predicted video frame in the embodiment of the present application may refer to a single video frame or a video frame sequence, and the present application does not limit this.

[0149] In an embodiment of the present application, the video generation device performs feature encoding on the last video frame in the historical video frame sequence to obtain image features, and uses the historical video frame sequence to predict the position features corresponding to the current time step. After aligning the position features and the graphic features, it is possible to obtain the area that simultaneously describes the appearance area of ​​the moving object at the current time step and the area of ​​the appearance of the moving object that actually exists in the last video frame, that is, the area where the appearance of the moving object can be truly captured from the last video frame is clarified. In this way, when feature decoding is performed on the features to be decoded, the complete appearance of the moving object can be extracted based on the area, so that the predicted video frame obtained by feature decoding is clearer, thereby improving the quality of the generated video.

[0150] See also Figure 7 , Figure 7 This is an optional process diagram of the video generation method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, the feature alignment process of the position feature and the graphic feature to obtain the feature to be decoded, that is, the implementation process of S104, may include: S1041-S1043, as follows:

[0151] S1041. Construct an attention map based on position features and graphic features.

[0152] When the video generation device performs feature alignment processing on the position feature and the graphic feature, it will first map the position feature and the graphic feature into the same feature space to obtain a feature map corresponding to the position feature and a feature map corresponding to the graphic feature. Then, based on the feature map corresponding to the graphic feature, the corresponding feature area is found for each feature position in the feature map corresponding to the position feature, and an attention map is constructed based on the degree of correlation between each feature position and its corresponding feature area.

[0153] Feature mapping the position features and graphic features to obtain their respective feature maps only changes the feature representation method for the position features and graphic features. In essence, they still describe the timing information and appearance features. Therefore, the attention map constructed based on the correlation between each feature position in the position feature and the feature area corresponding to each feature position in the feature map corresponding to the graphic feature essentially still describes the correlation between the position feature and the graphic feature.

[0154] S1042: Determine alignment features for describing context information of the area where the moving object is located based on the attention map and the position features.

[0155] The video generation device maps the position feature to convert it into another feature map, and then collects multiple feature vectors from the other feature map, performs matrix operations on the feature set consisting of the collected multiple feature vectors and the attention map to obtain a new feature, which is the alignment feature. The alignment feature indicates the changes in the area where the moving object is located, that is, the alignment feature describes the context information of the area where the moving object is located.

[0156] It is understandable that matrix operation may refer to matrix multiplication operation or matrix addition operation, and this application does not limit this.

[0157] S1043. Use the alignment features, position features and graphic features to fuse them into features to be decoded.

[0158] After determining the alignment feature, the video generation device will fuse the rotation feature, position feature and graphic feature into one feature, and the fused feature is the feature to be decoded. Since the alignment feature provides the context information of the area where the moving object is located, it is possible to clarify the area in the last video frame where the appearance of the moving object actually exists. By fusing this feature with the position feature that can describe the area in the predicted video frame where the moving object will appear at the current time step, and the graphic feature corresponding to the last video frame, the feature to be decoded can be obtained. Therefore, when the feature to be decoded is subsequently decoded, it is possible to clarify the position where the moving object should appear in the predicted video frame, the graphic features of the last video frame, and the area in the last video frame from which the moving object can be cut out.

[0159] It is understandable that the video generation device can fuse the alignment feature, the position feature and the graphic feature by feature splicing, for example, splicing the alignment feature, the position feature and the graphic feature into a feature matrix, and this feature matrix is ​​the feature to be decoded. The video generation device can also fuse the alignment feature, the position feature and the graphic feature by weighting, which is not limited in this application.

[0160] For example, in the position feature Represented by Indicates that the alignment feature is When expressed (where C is the number of channels, N is the size of the feature map, N = H × W, Represents the domain), then the feature to be decoded is expressed as

[0161] In an embodiment of the present application, the video generation device first constructs an attention map, and then determines the contextual information that can describe the area where the moving object is located through the attention map and the position features, that is, describes the position change of the area where the moving object is located, and fuses the alignment features, position features and graphic features to obtain the features to be decoded, thereby making the area where the moving object is located more accurate, so as to facilitate the subsequent capture of the complete appearance features of the moving object from the last video frame.

[0162] In some embodiments of the present application, based on the position feature and the graphic feature, an attention map is constructed, that is, a specific implementation process of S1041, which may include: S1041a-S1041d, as follows:

[0163] S1041a, mapping the position feature to a first preset feature space to obtain a first mapping feature map, and mapping the graphic feature to the first preset feature space to obtain a second mapping feature map.

[0164] The video generation device maps the position feature and the graphic feature to the same feature space by mapping the position feature and the graphic feature to the first preset feature space. It can be understood that the first preset feature space refers to a feature map size obtained by mapping and the number of channels of the feature map obtained by mapping, so that the first mapping feature map and the second mapping feature map have the same size and the same number of channels.

[0165] Furthermore, the first mapping feature map and the second mapping feature map both refer to a set of sub-feature maps corresponding to each channel. That is, the first mapping feature map contains the same number of sub-feature maps as the number of channels, and similarly, the second mapping feature map contains the same number of sub-feature maps as the number of channels.

[0166] S1041b. For each spatial position in the first mapping feature map, extract a first spatial vector from the first mapping feature map.

[0167] The spatial position refers to each pixel position on the two-dimensional plane composed of width and height. The spatial position is independent of the channel, that is, for the sub-feature maps of different channels, their spatial positions are the same pixel position. Since there are multiple channels, for each spatial position, a feature value can be extracted from the sub-feature map corresponding to each channel in the first mapping feature map. For a certain spatial position, the video generation device extracts the feature value corresponding to the spatial position from the sub-feature maps of each channel contained in the first mapping feature map, and then uses the feature values ​​corresponding to the sub-feature map of each channel to form a vector, and the first spatial vector corresponding to the spatial position is obtained.

[0168] For example, Figure 8 Schematic diagram of extracting the first space vector provided by the embodiment of the present application. Figure 8 As shown, the first mapping feature map contains sub-feature maps of two channels, namely, sub-feature Figure 8-1 and sub-features Figure 8-2 , and the spatial position is the upper left corner, the video generation device targets the sub-feature Figure 8-1 and sub-features Figure 8-2 , and get the eigenvalue of the upper left corner (i.e., sub-feature Figure 8-1 0.5 in, sub-feature Figure 8-2 0.86 in ), thereby obtaining two eigenvalues. The video generating device combines these two eigenvalues ​​into a vector to obtain the first space vector, namely (0.5, 0.86)8-3.

[0169] S1041c. Extract spatial vectors of multiple target positions corresponding to each spatial position in the second mapping feature map to obtain multiple second spatial vectors.

[0170] The video generation device determines multiple target positions from the first preset feature space for each spatial position, and then extracts the feature values ​​in each sub-feature map of the second mapping feature map for each target position to obtain the number of channel feature values, and then the number of channel feature values ​​of each target position are combined to form a second spatial vector corresponding to each target position. When the video generation device completes the extraction of spatial vectors for multiple target positions, multiple second spatial vectors can be obtained.

[0171] In some embodiments, multiple target positions refer to positions around each spatial position, or may refer to positions in the same row and column as each spatial position, or may refer to all positions including each spatial position in the second mapping feature map, which is not limited in this application.

[0172] For example, Fig. 9 This is a schematic diagram of the target location range provided in the embodiment of the present application. Figure 1 ,like Fig. 9 As shown, for the first mapping feature Figure 9-1 At a certain spatial position 9-11, the second mapping feature Figure 9-2 Each position in is taken as the target position, and the spatial vectors of these target positions are obtained 9-3, so that the first mapping feature Figure 9-1 For each spatial position in , a number of second spatial vectors corresponding to the size of the second mapping feature map can be obtained. For example, when the size of the second mapping feature map is 5×5, 25 second spatial feature vectors are obtained for spatial positions 9-11.

[0173] S1041d. Construct an attention map according to the similarity between the first space vector and the plurality of second space vectors.

[0174] After obtaining the first spatial vector and multiple second spatial vectors, the video generation device calculates the similarity between the first spatial vector and each second spatial vector, and then constructs an attention vector corresponding to each spatial position based on the similarity, thereby forming an attention map based on these attention vectors. In this way, the video generation device can obtain an attention map.

[0175] In an embodiment of the present application, the video generating device will first map the position features and the graphic features to a first preset feature space respectively to obtain a first mapping feature map and a second mapping feature map, and then extract the first space vector corresponding to each spatial position from the first mapping feature map, and extract multiple second space vectors from the second mapping feature map, and finally construct an attention map based on the similarity between the first space vector and the multiple second space vectors, so as to facilitate subsequent feature alignment based on the attention map.

[0176] In some embodiments of the present application, the attention map is constructed according to the similarity between the first space vector and the plurality of second space vectors, that is, the specific implementation process of S1041d may include: S201-S204, as follows:

[0177] S201 . Calculate similarity between a first space vector and a plurality of second space vectors respectively to obtain a plurality of similarity scores corresponding to the plurality of second space vectors.

[0178] The video generating device calculates the similarity between each second space vector in the plurality of second space vectors and the first space vector one by one, thereby obtaining a similarity score between each second space vector and the first space vector, that is, obtaining a plurality of similarity scores.

[0179] It should be noted that, in some embodiments, the similarity calculation can be implemented by inner product, that is, the video generation device calculates the inner product of each second space vector with the first space vector, and the inner product obtained is the similarity score corresponding to each second space vector. In other embodiments, the video generation device can calculate the cosine similarity between each second space vector and the first space vector, so as to obtain the similarity score corresponding to each second space vector.

[0180] Illustratively, the embodiment of the present application provides a formula for calculating the similarity score corresponding to each second space vector, see formula (1):

[0181]

[0182] Among them, ε represents each spatial position in the first mapping feature map, Q ε is the first spatial vector corresponding to each spatial position (the value range of ε is from 1 to the number of channels of the first mapping feature map), Ω ε is a set of multiple second space vectors, Ω i,ε For each second space vector (i ranges from 1 to n, n is the total number of second space vectors, determined by the method of obtaining the target position), d i,ε is the similarity score corresponding to each second space vector.

[0183] S202 . Normalize the multiple similarity scores respectively to obtain multiple normalized scores corresponding to the multiple second space vectors.

[0184] After obtaining the similarity score corresponding to each second space vector, the video generation device can first use the similarity score corresponding to each second space vector as an exponent and the natural number e as a base to perform an exponential operation, thereby obtaining the exponential operation result corresponding to each second space vector, and then use the exponential operation result corresponding to each second space vector as a numerator and the sum of the exponential operation results corresponding to all second space vectors as a denominator to perform a ratio operation. The obtained ratio is the normalized score corresponding to each second space vector. After normalization of multiple similarity scores, multiple normalized scores can be obtained. The video generation device can also directly use the similarity score corresponding to each second space vector as a numerator and the sum of the similarity scores corresponding to all second space vectors as a denominator to perform a ratio operation, thereby obtaining the normalized score corresponding to each second space vector, and then obtaining multiple normalized scores.

[0185] Exemplarily, the present application embodiment provides a formula for normalizing the similarity score, as shown in formula (2):

[0186]

[0187] Among them, d i,ε represents the similarity score corresponding to each second space vector, exp(d i,ε ) represents the exponential operation result of the similarity score corresponding to each second space vector, l represents the index of the second space vector, n represents the total number of the second space vectors, A i,ε Represents the normalized score corresponding to each second space vector.

[0188] S203. Use multiple normalized scores to form an attention vector corresponding to each spatial position.

[0189] After obtaining the normalized scores corresponding to each second spatial vector, the video generation device will use these normalized scores to form a vector, which is the attention vector corresponding to each spatial position. It is understandable that the video generation device can arrange multiple normalized scores in the order of multiple target positions to obtain the attention vector, or arrange multiple normalized scores in a random order to obtain the attention vector.

[0190] S204. Use the attention vector corresponding to each spatial position to form an attention map.

[0191] The video generation device can arrange the attention vectors corresponding to each spatial position in rows according to the order of each spatial position in the first mapping feature map, and the resulting matrix is ​​the attention map. The video generation device can also arrange the attention vectors corresponding to each spatial position in rows according to a random order to obtain an attention map.

[0192] In the embodiment of the present application, the video generation device first calculates the similarity scores of the first space vector and each second space vector respectively, then normalizes the multiple similarity scores, and then uses the multiple normalized scores to form an attention vector corresponding to each spatial position, and then uses all the attention vectors to form an attention map. In this way, the video generation device can obtain an attention map to facilitate subsequent feature alignment.

[0193] In some embodiments of the present application, the plurality of target positions include: positions in the same row as each spatial position, and positions in the same column as each spatial position.

[0194] It can be understood that in the above content, the value range of i in formula (1) and formula (2) is determined by the method of obtaining the target position, so in the embodiment of the present application, the value range of i is from 1 to H+W-1, so formula (1) will be correspondingly changed to:

[0195]

[0196] Formula (2) becomes accordingly:

[0197]

[0198] The meaning of each parameter in formula (3) is the same as that in formula (1), and the meaning of each parameter in formula (4) is the same as that in formula (2). It can be understood that in the embodiment of the present application, the time and space complexity of constructing the attention map is O((H×W)×(H+W-1)).

[0199] For example, Fig.10 This is a schematic diagram of the target location range provided in the embodiment of the present application. Figure 2 The video generating device maps the first feature Figure 10-1 At a certain spatial position 10-11, the second mapping feature Figure 10-2 In the example, each position in the same row and column as the spatial position 10-11 is used as the target position, and the spatial vectors of these target positions are obtained, so that the first mapping feature Figure 10-1 For each spatial position in , H+W-1 second spatial vectors can be obtained. For example, when the size of the second mapping feature map is 5×5, 9 second spatial feature vectors are obtained for spatial positions 10-11, thereby greatly reducing the time and space complexity of constructing the attention map.

[0200] In an embodiment of the present application, the multiple target positions are all pixel positions in the cross area corresponding to each spatial position, so that the video generation device can control the complexity of time and space when constructing the attention map, thereby ensuring the efficiency of video generation.

[0201] In some embodiments of the present application, based on the attention map and the position feature, determining the alignment feature for describing the context information of the area where the mobile object is located, that is, the specific implementation process of S1042, may include: S1042a-S1042d, as follows:

[0202] S1042a. Map the position feature to a second preset feature space to obtain a third mapping feature map.

[0203] It should be noted that the number of channels of the second preset feature space and the first preset feature space is different, but the size of the feature map is the same, that is, the number of sub-feature maps included in the third mapping feature map and the number of sub-feature maps included in the first mapping feature map are different only in quantity.

[0204] Exemplarily, when the first mapping feature map When the third mapping feature map

[0205] S1042b. For each pixel position in the third mapping feature map, extract a plurality of third space vectors from the third mapping feature map.

[0206] The video generation device determines multiple candidate positions in the third mapping feature map for each pixel position, and then uses the feature values ​​of each candidate position in the sub-feature map of different channels to form a third space vector corresponding to each candidate position. When the spatial vectors are extracted for multiple candidate positions, multiple third space vectors are obtained.

[0207] S1042c. Use multiple third space vectors to form a feature set corresponding to each pixel position.

[0208] S1042d. Perform matrix operations on the feature set and the attention map to obtain alignment features for describing context information of the area where the moving object is located.

[0209] The video generation device arranges multiple third space vectors in rows according to a random order or the order of multiple candidate positions in the third mapping feature map, and the resulting arrangement is the feature set corresponding to each pixel position. Then, the video generation device performs a matrix multiplication operation or a matrix addition operation on the feature set and the attention map, and uses the result of the matrix operation as the alignment feature.

[0210] Exemplarily, when each pixel position in the third mapping feature map is represented by ε, then, for ε, a feature set can be obtained from the third mapping feature map: Among them, C is the number of channels of the third mapping feature map, is the dimension of each third space vector, H is the height, W is the width, and H+W-1 is the number of third space vectors. On this basis, if the attention map is used When , then the formula for calculating the alignment feature is shown in formula (5):

[0211]

[0212] After obtaining the specific values ​​of the above parameters, the video generation device will substitute the specific values ​​of the above parameters into formula (5) to obtain the alignment features.

[0213] In an embodiment of the present application, the video generating device extracts multiple third space vectors for each pixel position of the third mapping feature map obtained by mapping the position feature to the second preset feature space, and then generates a feature set for each pixel position based on the multiple third space vectors, thereby obtaining an alignment feature from the feature set and the attention map through matrix calculation, so as to facilitate the subsequent generation of features to be decoded based on the alignment feature.

[0214] In some embodiments of the present application, for each pixel position in the third mapping feature map, extracting multiple third space vectors from the third mapping feature map, that is, the implementation process of S1042b, may include: S301-S302, as follows:

[0215] S301, screening the positions in the same row and column as each pixel position in the third mapping feature map to obtain a plurality of positions to be extracted.

[0216] The cross area corresponding to each pixel is actually the position in the same row and column as each pixel. The video generating device regards these positions as multiple positions to be extracted corresponding to each pixel position.

[0217] S302: Extract spatial vectors from a plurality of positions to be extracted to obtain a plurality of third spatial vectors corresponding to each pixel position.

[0218] The video generating device extracts feature values ​​from the sub-feature maps of each channel included in the third mapping feature map for each position to be extracted, and then uses the feature values ​​of each channel to form a vector, and the obtained vector is the third space vector corresponding to each position to be extracted. When the extraction of space vectors is completed for multiple positions to be extracted, the video generating device obtains multiple third space vectors.

[0219] In an embodiment of the present application, the video generating device will first take the positions in the same row and column as each pixel position as multiple positions to be extracted, and then extract the spatial vector for each position to be extracted. In this way, the video generating device can determine multiple third spatial vectors for each pixel position, so as to facilitate the subsequent use of these third spatial vectors to form a feature set corresponding to each pixel position.

[0220] In some embodiments of the present application, the specific implementation process of mapping the position feature to the second preset feature space to obtain the third mapping feature map, that is, S1042a, may include: S401, as follows:

[0221] S401. Using the convolution kernel corresponding to the second preset feature space, perform a convolution operation on the position feature to achieve mapping, and obtain a third mapping feature map.

[0222] The video generation device obtains the convolution kernel corresponding to the second preset feature space, uses the convolution kernel to convolve the position feature, and uses the obtained convolution feature map as the third mapping feature map. It can be understood that there is a dependency relationship between the convolution kernel corresponding to the second preset feature space and the number of channels and size of the third mapping feature map, that is, when the size, moving step and number of channels of the convolution kernel of the second preset feature space are determined, the second preset feature space is also determined.

[0223] In some embodiments of the present application, the size of the convolution kernel corresponding to the second preset feature space can be 1×1, and the number of channels can be set to 16. In other embodiments, the size of the convolution kernel can be 2×, and the number of channels can be set to 32. This application does not limit this.

[0224] Exemplarily, when the convolution kernel corresponding to the second preset feature space is W v (The number of channels of the convolution kernel is C), and the position feature is When , the third mapping feature map can be expressed as (N is calculated by the width, height, convolution kernel size and moving step of the position feature map).

[0225] In some embodiments of the present application, the process of mapping the position feature to the first preset feature space and the process of mapping the graphic feature to the first preset feature space can be implemented by the convolution kernel corresponding to the first preset feature space. For example, when the convolution kernel corresponding to the second preset feature space is W q and W k (The number of channels of these two convolutions is C′, and the size is 1×1, that is, the spatial scale is the same, but the internal parameters may be different), the position feature is When the graphic feature is K, the first mapping feature map can be expressed as The second mapping feature map can be expressed as

[0226] In some embodiments of the present application, the position feature corresponding to the current time step is predicted based on the historical video frame sequence, that is, the specific implementation process of S103 may include: S1031-S1035, as follows:

[0227] S1031. Perform feature encoding on the historical video frame sequence to obtain historical video features.

[0228] The video generation device inputs the historical video frame sequence into the feature encoder, and then uses the output of the feature encoder as the historical video feature.

[0229] Exemplarily, the generation process of historical video features can be expressed by formula (6):

[0230] h t-1 =Enc(x t-1 ) (6)

[0231] Among them, x t-1 is the historical video frame sequence, Enc is the feature encoder, h t-1 For historical video features.

[0232] S1032. Using a preset prior distribution inference model, learn the prior distribution corresponding to the current time step from historical video features.

[0233] The video generation device first obtains a preset prior distribution inference model, and then inputs the historical video features into the preset prior distribution inference model, so as to use the preset prior distribution inference model to learn the prior distribution corresponding to the current time step. The preset prior distribution inference model is a trained model for learning the prior distribution of the temporal information of the historical video frame sequence.

[0234] It is understandable that the preset prior distribution inference model can be a long short-term memory network (Long Sho rt-Term Memory, LSTM), a recurrent neural network (Recurrent Neural Network, RNN), or other networks that can learn temporal information, and this application does not make specific limitations here.

[0235] Exemplarily, the present application embodiment provides a learning process of a priori distribution, as shown in formula (7):

[0236] [μ ψ (t),σ ψ (t)]=LSTM ψ (h t-1 ) (7)

[0237] Among them, ht-1 is the historical video feature, LSTM ψ is the preset prior distribution inference model, [μ ψ (t),σ ψ (t)] is the prior distribution of the current time step. Furthermore, the preset prior distribution model LSTM ψ This can be implemented with a single-layer LSTM with 256 units.

[0238] S1033. Sample the predicted latent variable corresponding to the current time step from the prior distribution.

[0239] The video generation device samples the learned current prior distribution, and the predicted latent variable obtained after sampling is the predicted latent variable corresponding to the current time step. In other words, the predicted latent variable of the current time step obeys the prior distribution of the current time step.

[0240] For example, when the learned prior distribution is N(μ ψ(t) ,σ ψ(t) ), the predicted latent variable at the current time step is z t When t ~N(μ ψ(t) ,σ ψ(t) ).

[0241] S1034: Using a preset feature prediction model, perform feature prediction on the prediction latent variables and historical video features to obtain position features corresponding to the current time step.

[0242] The video generation device obtains a preset feature inference model, and then inputs both the predicted latent variables and the historical video features into the preset feature inference model, and uses the output of the preset feature prediction model as the position feature of the current time step. Subsequently, the video generation device will input the position feature into the feature decoder so that the video generation device can obtain the position feature from the feature decoder during subsequent feature alignment. The preset feature prediction model is a trained model for predicting unknown features.

[0243] Exemplarily, the process of generating position features can be shown as formula (8):

[0244]

[0245] Among them, h t-1 is the historical video feature, z t is the predicted latent variable of the current time step, LSTM θ To preset feature inference model, is the position feature. Furthermore, the preset feature prediction model LSTM θ It can be implemented by two layers of LSTM.

[0246] For example, Fig.11 Schematic diagram of the process of generating location features provided by the embodiment of the present application. Fig.11 , the historical video frame sequence 11-1 is first input into the feature encoder 11-2 to obtain the historical video feature 11-3, and then the historical video feature 11-3 is input into the preset prior distribution inference model 11-4, and the prior distribution 11-5 corresponding to the current time step can be obtained. Next, the video generation device samples the prior distribution 11-5 to obtain the predicted latent variable 11-6, and inputs the predicted latent variable 11-6 and the historical video feature 11-3 into the preset feature prediction model 11-7. The output of the preset feature prediction model 11-7 is connected to the feature decoder 11-8, so that the video generation device can obtain the position feature from the feature decoder 11-8. Next, continue to refer to Fig.11 The video generation device will input the last video frame 11-9 into the feature encoder 11-10 to obtain the graphic features, and then input the graphic features and position features into the feature aligner 11-11 for feature alignment, and then input the output of the feature aligner 11-11, that is, the features to be decoded, back into the feature decoder 11-8 for feature decoding, so as to obtain the predicted video frame 11-12.

[0247] In the embodiment of the present application, the video generation device first performs feature encoding on the historical video frame sequence to obtain the historical video features, performs prior distribution learning, then samples the predicted latent variables of the current time step from the learned prior distribution, and then uses the predicted latent variables and the historical video features to jointly predict the position features. In this way, the video generation device can obtain the position features to facilitate subsequent feature alignment.

[0248] The embodiment of the present application also provides another video generation method, which can train the prediction prior distribution inference model and the preset feature prediction model used in the video generation process. Fig. 12A , Fig. 12A This is an optional process diagram 3 of the video generation method provided in the embodiment of the present application. The steps of the video generation method are as follows:

[0249] S501, obtaining a training observation frame sequence, a training true value frame sequence, an initial prior distribution inference model, an initial feature prediction model, a posterior distribution inference model, and training graphic features corresponding to training preset frames in the training observation frame sequence.

[0250] It should be noted that the training true value frame sequence is the real video after the training observation frame sequence; the initial prior distribution inference model and the initial feature prediction model are models that have not been trained; the training preset frame can be the last frame of the training observation frame sequence, or the second to last frame of the training observation frame sequence, etc.

[0251] S502 , using a posterior distribution inference model, inferring a posterior distribution of the training true value features obtained after feature encoding of the training true value frame sequence, and obtaining an approximate posterior distribution.

[0252] Since the true posterior distribution of the training truth frame sequence is difficult to calculate, in the embodiment of the present application, the video generation device uses the approximate posterior distribution to train the initial prior distribution inference model. In this case, the video generation device first performs feature encoding on the training truth frame sequence to obtain the training truth feature, and then inputs the training truth feature into the posterior distribution inference model to infer the approximate posterior distribution of the training truth frame sequence.

[0253] Exemplarily, the process of obtaining the approximate posterior distribution by the video generation device can be described by equations (9) and (10):

[0254] h t =Enc(x t ) (9)

[0255] [μ φ (t),σ φ (t)]=LSTM φ (h t ) (10)

[0256] Among them, x t is the training true value frame sequence, Enc is the feature encoder, h t is the training true value feature, LSTM φ is the posterior distribution inference model, [μ φ (t),σ φ (t)] is the inferred approximate posterior distribution. Further, the posterior distribution inference model LSTM φ It can be implemented by a single-layer LSTM with 256 units.

[0257] S503: Perform feature encoding on the training observation frame sequence to obtain training observation features corresponding to the training observation frame sequence.

[0258] The video generating device inputs the training observation frame sequence into the feature encoder to obtain the training observation features using the feature encoder.

[0259] It should be noted that the formula for encoding the training observation features is the same as formula (7).

[0260] S504: Based on the approximate posterior distribution and the training observation features, the initial prior distribution inference model is trained until the training stop condition is reached, thereby obtaining a preset prior distribution inference model.

[0261] The video generation device first uses the training observation features to infer the training prior distribution, and then uses the loss between the approximate posterior distribution and the training prior distribution to adjust the parameters in the initial prior distribution inference model until the training stop condition is reached to obtain the preset prior distribution inference model. The preset prior distribution inference model is used to learn the temporal information of the historical video frame sequence.

[0262] S505. Train the initial feature prediction model according to the approximate posterior distribution, the training observation features, and the training image features until a training stop condition is reached to obtain a preset feature prediction model.

[0263] The video generation device samples the training latent variables based on the approximate posterior distribution, and then uses the initial feature prediction model to predict the training observation features and the training latent variables to generate the features to be aligned, aligns the features to be aligned with the training image features to obtain the training alignment features, and finally generates the training prediction frames based on the training alignment features, and adjusts the parameters of the initial feature prediction model based on the difference between the training prediction frames and the training true value frame sequence until the training stop condition is reached, and the preset feature prediction model is obtained. Among them, the preset feature prediction model is used to predict the model of unknown features.

[0264] For example, Fig. 12B Schematic diagram of the model training process provided in the embodiment of the present application. Fig. 12B As shown, the training observation frame sequence 12-1 is first input into the feature encoder 12-2 to obtain the training observation feature 12-3, and then the training observation feature 12-3 is input into the initial prior distribution inference model 12-4 to obtain the training prior distribution 12-5 corresponding to the training. At the same time, the video generation device will input the training truth value frame sequence 12-6 into the feature encoder 12-7 to obtain the training truth value feature 12-8, and then use the posterior distribution inference model 12-9 to predict the approximate posterior distribution 12-10 of the training truth value feature 12-8, and according to the loss 12-11 between the approximate posterior distribution 12-10 and the training prior distribution 12-5, the initial prior distribution inference model is trained.

[0265] Continue to see Fig. 12BThe video generation device samples the training latent variable 12-12 from the approximate posterior distribution 12-10, and inputs the training latent variable 12-12 and the training observation feature 12-3 into the initial feature prediction model 12-13 to obtain the training prediction feature. The video generation device extracts the training prediction feature from the feature decoder 12-14, and performs feature alignment 12-17 on the training image feature obtained after feature encoding of the training preset frame 12-15 by the feature encoder 12-16, to obtain the training alignment feature, and then obtain the training feature to be decoded, and then uses the feature decoder 12-13 to feature decode the training feature to be decoded, so as to obtain the training prediction frame 12-18, thereby adjusting the parameters in the initial feature prediction model based on the loss 12-19 between the training prediction frame 12-18 and the training true value frame sequence 12-6, and obtaining the preset feature prediction model.

[0266] In some embodiments of the present application, based on the approximate posterior distribution and the training observation video sequence, the initial prior distribution inference model is trained to obtain a preset prior distribution inference model, that is, the specific implementation process of S504 may include S5041-S5042, as follows:

[0267] S5041. Utilize the initial prior distribution inference model to infer the prior distribution of the training observation features, and obtain the training prior distribution corresponding to the training observation frame sequence.

[0268] S5042. Utilize the prior loss between the approximate posterior distribution and the training prior distribution to adjust the parameters of the initial prior distribution inference model until the training stop condition is reached to obtain a preset prior distribution inference model.

[0269] The video generation device can use the divergence loss between the approximate posterior distribution and the training prior distribution as the prior loss, and then iteratively adjust the parameters in the initial prior distribution inference model so that the prior loss is small enough, thereby obtaining a preset prior distribution inference model.

[0270] Exemplarily, the present application embodiment provides a calculation formula for a priori loss, as shown in formula (11):

[0271] L KL =D KL (q φ (z t |x 1:t )||p ψ (z t |x 1:t-1 )] (11)

[0272] Among them, x 1:t represents the training true value frame sequence, x 1:t-1 represents the training observation frame sequence, qφ (z t |x 1:t ) is the approximate posterior distribution, p ψ (z t |x 1:t-1 ) represents the training prior distribution, D KL represents the calculation of divergence loss, L KL is the prior loss.

[0273] It is understandable that the training stop condition may be that the prior loss is less than the loss threshold, or the number of iterations reaches a preset number, which is not limited in this application.

[0274] In the embodiment of the present application, the initial feature prediction model is trained according to the approximate posterior distribution, the training observation features, and the training graphic features until the training stop condition is reached, and a preset feature prediction model is obtained, that is, the specific implementation process of S505 may include: S5051-S5054, as follows:

[0275] S5051. Sample training latent variables from the approximate posterior distribution, and perform feature prediction on the training latent variables and training observation features through the initial feature prediction model to obtain training prediction features.

[0276] The video generation device samples a latent variable from the approximate posterior distribution as a training latent variable, and then inputs the training latent variable and the training observation features into the initial feature prediction model, and uses the features predicted by the initial feature prediction model as the training prediction features.

[0277] S5052: Perform feature alignment on the training prediction features and the training graphic features to obtain the training features to be decoded.

[0278] It should be noted that the process of aligning the training prediction features with the training graph is similar to the process in S104 and will not be described in detail here.

[0279] S5053. Perform feature decoding on the training features to be decoded to obtain a training prediction frame.

[0280] The video generating device inputs the training features to be decoded into the feature decoder to output the training prediction frames through the feature decoder.

[0281] Exemplarily, the feature decoding process for training features to be decoded can be expressed as formula (12):

[0282] μ θ (t) = Dec(g t ) (12)

[0283] Among them, g tis the training band decoding feature, Dec is the feature decoder, μ θ (t) is the training prediction frame.

[0284] S5054. Utilize the reconstruction loss between the training prediction frame and the training true value frame sequence to adjust the parameters of the initial feature prediction model until the training stop condition is reached, thereby obtaining a preset feature prediction model.

[0285] The video generation device calculates the loss between the training prediction frame and the training true value frame sequence, uses the loss as the reconstruction loss, and iteratively adjusts the parameters in the initial feature prediction model to make the reconstruction loss small enough, thereby obtaining a preset feature prediction model.

[0286] In some embodiments, the video generation device may use the loss between the value in the frame predictor and the true value as the reconstruction loss.

[0287] Exemplarily, the embodiment of the present application provides a calculation formula for reconstruction loss, as shown in formula (13):

[0288]

[0289] Among them, p θ (x t |z t ,x 1:t-1 ) is the value of the generator during prediction, is true value.

[0290] In an embodiment of the present application, the video generation device will first use the training observation frame sequence, the training true value frame sequence and the training graphic features to train the initial feature prediction model and the initial prior distribution prediction model to obtain the preset prior distribution prediction model and the preset feature prediction model, so that the preset prior distribution model and the preset feature prediction model can be used to learn and predict the historical video frame sequence, thereby generating a predicted video frame.

[0291] In some embodiments of the present application, the video generation device can also connect the reconstruction loss and the prior loss through a hyperparameter to obtain a new loss function, and then optimize the loss function to achieve the optimization of the prior loss after the reconstruction loss.

[0292] Exemplarily, the present application embodiment provides a loss function, as shown in formula (14):

[0293]

[0294] Among them, β is a hyperparameter.

[0295] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0296] The embodiment of the present application is implemented in a scenario where a future unknown video frame sequence (predicted video frame) is generated based on an existing observed video frame sequence (historical video frame sequence). First, a probability graph model in the random video generation process is introduced to clarify the relationship between various variables.

[0297] Fig.13 Schematic diagram of the probability graph model for random video generation provided by the embodiment of the present application. Fig.13 In the inference model 13-1, the frame x to be predicted is mainly t Past frame x 1:t-1 (i.e. x1 to x t-1 ) as input to calculate a posterior distribution q φ (z t |x 1:t ) area approximates the true posterior distribution, which is the approximate posterior distribution. Then, during training, the latent variable z is sampled from the approximate posterior distribution. t It should be noted that when calculating the approximate posterior distribution of time step t, all video frames before and at time step t are input into the model. For example, when calculating the approximate posterior distribution of time step T, x1 to x T-1 are input into the model. In the generative model 13-2, the video frame x in the past time step t is 1:t-1 As input, calculate the prior distribution of time step t, that is, get p ψ (z t |x 1:t-1 ), generator p θ (x t |z t ,x 1:t-1 ) and then based on the ψ (z t |x 1:t-1 ) obtained from the latent variable z t , and all past video frames x 1:t-1 , using the predicted frame at time t

[0298] The framework of random model generation is as follows Fig.11 and Fig. 12B As shown. During the training process, the inference model LSTM φ The approximate posterior distribution N(μ φ(t) ,σ φ(t) ), using the prior loss to approximate the learned prior distribution Ν(μ ψ(t) ,σ ψ(t) ), and at the same time, the predicted frame is made To get close to the frame x to be predictedt In the prediction process, by learning the prior distribution Ν(μ ψ(t) ,σ ψ(t) ) in the sample z t To predict the frame. The encoder uses the feature h of the previous time step t-1 (historical video features) and the latent variable z sampled at the current time step t , to input into the decoder, and then from the decoder and the last observed frame x k The encoder that encodes the last video frame in the historical video frame sequence extracts the intermediate features of the decoder (position features) and the intermediate features of the encoder, and then aligns the two intermediate features to help the decoder obtain the last observed frame x. k The aligned context information at time k is obtained (essentially the context information of the area where the moving object is located), and then the predicted video frame sequence (predicted video frame) is decoded based on the result after feature alignment.

[0299] Fig.14 It is a schematic diagram of the feature alignment process provided in an embodiment of the present application. is the intermediate feature 14-1 (position feature) of the decoder at time step t, Represents the intermediate feature 14-2 (graphic feature) of the encoder at time step k, where C represents the number of channels of the feature and N=H×W represents the size of the feature map.

[0300] The random video generation workstation (video generation device) first and Mapped to the same feature space, that is, using W q right Perform convolution to obtain feature map Q14-3. (First mapping feature map), using W k right Perform convolution to obtain feature map K14-4, (Second mapping feature map). At each spatial position ε of Q, we can get a vector (first space vector), at the same time, features in the same row and column as ε can be extracted from the embedding K, and these features form a set (It contains multiple second space vectors). Represents Ω ε In the case of the i-th feature, it can be obtained by Q ε With Ω i,ε Then, all similarity scores are calculated through formula (4) to calculate the attention graph A14-5. Each element in (normalized score).

[0301] At the same time, the random video generation workstation will also use W v right Perform convolution to obtain feature map V14-6, (Third mapping feature map). Then, use Ω extracted from the feature map K ε In the same way, a vector is extracted from feature map V14-6 and a set of features in the same row and column as ε (Multiple third space vectors form a feature set corresponding to each pixel position.) Then, through matrix operations get (alignment feature), then and Fusion 14-7 finally yields feature 14-8. (Features to be decoded).

[0302] Furthermore, the encoder can adopt the VGG16 structure or the DCGAN discriminator structure, and the decoder can be the mirror structure of VGG16 or the generator structure of DCGAN. The mode field of the encoder to extract features is 128, and the dimension of each inference model composed of LSTM is also 128. The size of all frames in the training process and the prediction process is 64*64.

[0303] The following describes a comparison of the effects of the video generation method of the embodiment of the present application and the video method in the related art. Fig.15 Schematic diagram of the comparison of the video generation method provided by this application and the video generation method in the related art on the pedestrian dataset. Fig.15 In the KTH dataset, the video frame sequence generated by the video generation method of the related art is 15-1, the video frame sequence generated by the video generation method of the embodiment of the present application is 15-2, and the true value is 15-3. Fig.15 It can be seen that the video frame sequence generated by the video generation method using the related art has obvious artifacts (the part marked with a solid line frame), while the video frame sequence generated by the video generation method using the embodiment of the present application does not have artifacts and is closer to the true value, thereby obtaining a clearer and higher-quality video.

[0304] In the above manner, the similarity score between the last observed frame and the current observed frame can be calculated by performing point-by-point inner product of the intermediate features of the encoder and the intermediate features of the encoder, and the attention weight (attention vector) is obtained by normalizing the similarity score to generate an attention map. Finally, the features are calculated by weighted sum to obtain the context information of the last observed frame aligned to the current time step, so that the complete appearance features of the moving object can be obtained from the last observed frame, thereby improving the quality of the generated video.

[0305] The following is a description of an exemplary structure of the video generation device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 5A As shown, the software modules stored in the video generating device 255 of the first memory 250 may include:

[0306] Information acquisition module 2551, used to obtain the historical video frame sequence corresponding to the current time step;

[0307] A feature encoding module 2552 is used to perform feature encoding on the last video frame in the historical video frame sequence to obtain a graphic feature corresponding to the last video frame;

[0308] A feature prediction module 2553 is used to predict the position feature corresponding to the current time step based on the historical video frame sequence;

[0309] The feature alignment module 2554 is used to perform feature alignment processing on the position feature and the graphic feature to obtain a feature to be decoded; wherein the feature to be decoded describes the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame;

[0310] The feature decoding module 2555 is used to perform feature decoding on the feature to be decoded to obtain the predicted video frame of the current time step.

[0311] In some embodiments of the present application, the feature alignment module 2554 is also used to construct an attention map based on the position features and the graphic features; determine the alignment features used to describe the context information of the area where the mobile object is located based on the attention map and the position features; and use the alignment features, the position features and the graphic features to fuse them into the features to be decoded.

[0312] In some embodiments of the present application, the feature alignment module 2554 is also used to map the position feature to a first preset feature space to obtain a first mapping feature map, and to map the graphic feature to the first preset feature space to obtain a second mapping feature map; for each spatial position in the first mapping feature map, extract a first spatial vector from the first mapping feature map; extract spatial vectors for multiple target positions corresponding to each spatial position in the second mapping feature map to obtain multiple second spatial vectors; and construct the attention map based on the degree of similarity between the first spatial vector and the multiple second spatial vectors.

[0313] In some embodiments of the present application, the feature alignment module 2554 is also used to calculate the similarity between the first space vector and the multiple second space vectors respectively to obtain multiple similarity scores corresponding to the multiple second space vectors; normalize the multiple similarity scores respectively to obtain multiple normalized scores corresponding to the multiple second space vectors; use the multiple normalized scores to form the attention vector corresponding to each spatial position; and use the attention vector corresponding to each spatial position to form the attention map.

[0314] In some embodiments of the present application, the feature alignment module 2554 is also used to map the position feature to a second preset feature space to obtain a third mapping feature map; for each pixel position in the third mapping feature map, extract a plurality of third space vectors from the third mapping feature map; use the plurality of third space vectors to form a feature set corresponding to each pixel position; perform matrix operations on the feature set and the attention map to obtain the alignment feature used to describe the context information of the area where the moving object is located.

[0315] In some embodiments of the present application, the feature alignment module 2554 is also used to screen the positions in the third mapping feature map that are in the same row and column as each pixel position to obtain a plurality of positions to be extracted; extract spatial vectors for the plurality of positions to be extracted to obtain the plurality of third spatial vectors corresponding to each pixel position.

[0316] In some embodiments of the present application, the feature alignment module 2554 is further used to use the convolution kernel corresponding to the second preset feature space to perform a convolution operation on the position feature to achieve mapping, so as to obtain the third mapping feature map.

[0317] In some embodiments of the present application, the feature prediction module 2553 is also used to perform feature encoding on the historical video frame sequence to obtain historical video features; use a preset prior distribution inference model to learn the prior distribution corresponding to the current time step from the historical video features; the preset prior distribution inference model is a trained model for learning the prior distribution of the temporal information of the historical video frame sequence; sample the predicted latent variables corresponding to the current time step from the prior distribution; use a preset feature prediction model to perform feature prediction on the predicted latent variables and the historical video features to obtain the position features corresponding to the current time step; the preset feature prediction model is a trained model for predicting unknown features.

[0318] The following is a description of an exemplary structure of the video generating device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 5B As shown, the software modules stored in the video generating device 555 of the second memory 550 may include:

[0319] The data acquisition module 5551 is used to acquire the training observation frame sequence, the training true value frame sequence, the initial prior distribution inference model, the initial feature prediction model, the posterior distribution inference model, and the training graphic features corresponding to the training preset frames in the training observation frame sequence;

[0320] The distribution inference module 5552 is used to use the posterior distribution inference model to infer the posterior distribution of the training true value features obtained after feature encoding of the training true value frame sequence, so as to obtain an approximate posterior distribution;

[0321] A training encoding module 5553 is used to perform feature encoding on the training observation frame sequence to obtain training observation features corresponding to the training observation frame sequence;

[0322] Model training module 5554 is used to train the initial prior distribution inference model based on the approximate posterior distribution and the training observation features until the training stop condition is reached, thereby obtaining a preset prior distribution inference model; the preset prior distribution inference model is used to learn the timing information of the historical video frame sequence; the initial feature prediction model is trained based on the approximate posterior distribution, the training observation features and the training graphic features until the training stop condition is reached, thereby obtaining a preset feature prediction model; the preset feature prediction model is used to predict a model of unknown features.

[0323] In some embodiments of the present application, the model training module 5556 is also used to use the initial prior distribution inference model to infer the prior distribution of the training observation features, and obtain the training prior distribution corresponding to the training observation frame sequence; and use the prior loss between the approximate posterior distribution and the training prior distribution to adjust the parameters of the initial prior distribution inference model until the training stop condition is met, thereby obtaining the preset prior distribution inference model.

[0324] In some embodiments of the present application, the model training module 5556 is also used to sample training latent variables from the approximate posterior distribution, and perform feature prediction on the training latent variables and the training observation features through the initial feature prediction model to obtain training prediction features; perform feature alignment on the training prediction features and the training graphic features to obtain training features to be decoded; perform feature decoding on the training features to be decoded to obtain training prediction frames; and use the reconstruction loss between the training prediction frames and the training true value frame sequence to adjust the parameters of the initial feature prediction model until the training stop condition is reached to obtain the preset feature prediction model.

[0325] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video generation method described in the embodiment of the present application.

[0326] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable video generation instructions are stored. When the executable video generation instructions are executed by a first processor, the first processor will execute the video generation method provided by the embodiment of the present application, for example, Figure 6 The method shown in the figure, or when executed by the second processor, will cause the second processor to execute the video generation method provided in the embodiment of the present application, for example, Fig. 12A The method shown.

[0327] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0328] In some embodiments, the executable video generation instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0329] As an example, executable video generation instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0330] As an example, the executable video generation instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed at multiple sites and interconnected by a communication network.

[0331] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: include: Get the historical video frame sequence corresponding to the current time step; Performing feature encoding on the last video frame in the historical video frame sequence to obtain graphic features corresponding to the last video frame; Predicting the position features corresponding to the current time step based on the historical video frame sequence; Based on the position features and the graphic features, construct an attention map; Determining, based on the attention map and the position feature, an alignment feature for describing context information of an area where the moving object is located; The alignment feature, the position feature and the graphic feature are used to fuse into a feature to be decoded; wherein the feature to be decoded describes the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame; Feature decoding is performed on the feature to be decoded to obtain a predicted video frame of the current time step.

2. The method according to claim 1, characterized in that The step of constructing an attention map based on the position feature and the graphic feature includes: Mapping the position feature to a first preset feature space to obtain a first mapping feature map, and mapping the graphic feature to the first preset feature space to obtain a second mapping feature map; For each spatial position in the first mapping feature map, extracting a first spatial vector from the first mapping feature map; Extracting spatial vectors of a plurality of target positions corresponding to each of the spatial positions in the second mapping feature map to obtain a plurality of second spatial vectors; The attention map is constructed according to the similarity between the first space vector and the multiple second space vectors.

3. The method according to claim 2, characterized in that The constructing the attention map according to the similarity between the first space vector and the plurality of second space vectors respectively includes: Calculate similarity between the first space vector and the plurality of second space vectors respectively to obtain a plurality of similarity scores corresponding to the plurality of second space vectors; Normalizing the multiple similarity scores respectively to obtain multiple normalized scores corresponding to the multiple second space vectors; Using the multiple normalized scores, forming an attention vector corresponding to each spatial position; The attention map is formed using the attention vector corresponding to each spatial position.

4. The method according to claim 2 or 3, characterized in that: The plurality of target positions include positions in the same row as each of the spatial positions and positions in the same column as each of the spatial positions.

5. The method according to any one of claims 1 to 3, characterized in that: The step of determining, based on the attention map and the position feature, an alignment feature for describing context information of the area where the mobile object is located, comprises: Mapping the position feature to a second preset feature space to obtain a third mapping feature map; For each pixel position in the third mapping feature map, extracting a plurality of third space vectors from the third mapping feature map; Using the plurality of third space vectors, a feature set corresponding to each pixel position is formed; A matrix operation is performed on the feature set and the attention map to obtain the alignment features for describing the context information of the area where the moving object is located.

6. The method according to claim 5, characterized in that The extracting a plurality of third space vectors from the third mapping feature map for each pixel position in the third mapping feature map comprises: Screening the positions in the same row and column as each pixel position in the third mapping feature map to obtain a plurality of positions to be extracted; Extract spatial vectors from the multiple positions to be extracted to obtain the multiple third spatial vectors corresponding to each pixel position.

7. The method according to claim 5, characterized in that Mapping the position feature to a second preset feature space to obtain a third mapping feature map includes: Using the convolution kernel corresponding to the second preset feature space, a convolution operation is performed on the position feature to achieve mapping, so as to obtain the third mapping feature map.

8. The method according to any one of claims 1 to 3, characterized in that: The predicting of the position feature corresponding to the current time step based on the historical video frame sequence includes: Performing feature encoding on the historical video frame sequence to obtain historical video features; Using a preset prior distribution inference model, the prior distribution corresponding to the current time step is learned from the historical video features; the preset prior distribution inference model is a trained model for learning the prior distribution of the temporal information of the historical video frame sequence; Sampling the predicted latent variable corresponding to the current time step from the prior distribution; Using a preset feature prediction model, feature prediction is performed on the predicted latent variables and the historical video features to obtain the position features corresponding to the current time step; the preset feature prediction model is a trained model for predicting unknown features.

9. A video generation method, characterized in that: include: Acquire a training observation frame sequence, a training true value frame sequence, an initial prior distribution inference model, an initial feature prediction model, a posterior distribution inference model, and training graphic features corresponding to the training preset frames in the training observation frame sequence; Using the posterior distribution inference model, inferring the posterior distribution of the training true value features obtained after feature encoding of the training true value frame sequence, to obtain an approximate posterior distribution; Performing feature encoding on the training observation frame sequence to obtain training observation features corresponding to the training observation frame sequence; Based on the approximate posterior distribution and the training observation features, the initial prior distribution inference model is trained until a training stop condition is reached to obtain a preset prior distribution inference model; the preset prior distribution inference model is used to learn the timing information of the historical video frame sequence; The initial feature prediction model is trained according to the approximate posterior distribution, the training observation features and the training graphic features until a training stop condition is reached to obtain a preset feature prediction model; the preset feature prediction model is used to predict a model of unknown features.

10. The method according to claim 9, characterized in that The initial prior distribution inference model is trained based on the approximate posterior distribution and the training observation features until a training stop condition is reached to obtain a preset prior distribution inference model, including: Using the initial prior distribution inference model, infer the prior distribution of the training observation features to obtain the training prior distribution corresponding to the training observation frame sequence; The parameters of the initial prior distribution inference model are adjusted by utilizing the prior loss between the approximate posterior distribution and the training prior distribution until the training stop condition is reached, thereby obtaining the preset prior distribution inference model.

11. The method according to claim 9, characterized in that The initial feature prediction model is trained according to the approximate posterior distribution, the training observation features and the training graphic features until a training stop condition is reached to obtain a preset feature prediction model, including: Sampling training latent variables from the approximate posterior distribution, and performing feature prediction on the training latent variables and the training observation features through the initial feature prediction model to obtain training prediction features; Performing feature alignment on the training prediction feature and the training graphic feature to obtain a training feature to be decoded; Performing feature decoding on the training features to be decoded to obtain a training prediction frame; The parameters of the initial feature prediction model are adjusted by utilizing the reconstruction loss between the training prediction frame and the training true value frame sequence until the training stop condition is reached, thereby obtaining the preset feature prediction model.

12. A video generating device, characterized in that: include: An information acquisition module is used to obtain the historical video frame sequence corresponding to the current time step; A feature encoding module, used for performing feature encoding on the last video frame in the historical video frame sequence to obtain a graphic feature corresponding to the last video frame; A feature prediction module, used to predict the position feature corresponding to the current time step based on the historical video frame sequence; A feature alignment module is used to construct an attention map based on the position feature and the graphic feature; determine an alignment feature for describing the context information of the area where the moving object is located according to the attention map and the position feature; and fuse the alignment feature, the position feature and the graphic feature into a feature to be decoded; wherein the feature to be decoded describes the appearance area of ​​the moving object at the current time step, and the appearance area of ​​the moving object in the last video frame; The feature decoding module is used to perform feature decoding on the feature to be decoded to obtain the predicted video frame of the current time step.

13. A video generating device, characterized in that: include: A first memory, used to store executable video generation instructions; The first processor is configured to implement the method according to any one of claims 1 to 8 when executing the executable video generation instruction stored in the first memory.

14. A computer-readable storage medium, characterized in that: An executable video generation instruction is stored, which is used to implement the method described in any one of claims 1 to 8 when executed by a first processor; or is used to implement the method described in any one of claims 9 to 11 when executed by a second processor.

15. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented, or the method described in any one of claims 9 to 11 is implemented.

Images