Video generation method, device, apparatus, and computer-readable storage medium

Abstract

The present invention discloses a video generation method, apparatus, device, and computer-readable storage medium. The video generation method includes: obtaining a current frame control signal and a historical frame control signal; injecting parameters into all network layers of a trained target video generation unit network based on the current frame control signal and the historical frame control signal; and determining a preliminary generated image based on the parameter injection results; determining a historical image corresponding to the historical frame control signal and determining an optical flow and a mask based on the historical image; determining a current frame image based on the preliminary generated image, the optical flow, and the mask; and constructing a video based on the current frame image. The present invention avoids the overly complex process of video generation that relies on CG technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video generation method, apparatus, device and computer-readable storage medium. Background Art

[0002] CG, or computer graphics, is the process of converting three-dimensional objects into images or videos using computers and mathematical algorithms. With the advancement of computing power, CG technology has been widely used in film, games, advertising, and interactive systems. However, the process of generating images using CG technology is complex, requiring a series of operations such as modeling, material design, lighting layout, and rendering. Character CG is even more complex, requiring extremely detailed modeling, delicate materials, complex hair systems, and powerful rendering engines to achieve realistic character effects. Summary of the Invention

[0003] The main purpose of the present invention is to provide a video generation method, device, equipment and computer-readable storage medium, aiming to solve the technical problem of how to avoid the phenomenon of overly complex video generation process.

[0004] To achieve the above object, the present invention provides a video generation method, comprising the following steps:

[0005] Obtaining a current frame control signal and a historical frame control signal, injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, and determining a preliminary generated image based on the parameter injection results;

[0006] Determining a historical image corresponding to the historical frame control signal, and determining an optical flow and a mask based on the historical image;

[0007] A current frame image is determined according to the preliminary generated image, the optical flow, and the mask, and a video is constructed according to the current frame image.

[0008] Optionally, the step of determining the current frame image according to the preliminary generated image, the optical flow, and the mask includes:

[0009] Calculating a difference between a preset value and the mask, and performing bitwise multiplication on the preliminary generated image and the difference to obtain a first bitwise multiplication result;

[0010] determining a deformation result according to the historical image and the optical flow, and performing bitwise multiplication on the deformation result and the mask to obtain a second bitwise multiplication result;

[0011] A current frame image is determined based on the first bitwise multiplication result and the second bitwise multiplication result.

[0012] Optionally, the step of injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal includes:

[0013] Taking the current frame control signal and the historical frame control signal as control parameters, projecting the control parameters into an implicit space according to a preset multi-layer perceptron;

[0014] The control parameters in the implicit space are subjected to dimensionality increase processing to obtain control variables, and parameters are injected into all network layers in the trained target video generation unit network according to the control variables.

[0015] Optionally, the step of determining the preliminary generation of an image based on the parameter injection result includes:

[0016] An input constant tensor is obtained, and model training is performed on the constant tensor according to the parameter injection result and the upsampling layer in the target video generation unit network to obtain a preliminary generated image.

[0017] Optionally, the step of determining a historical image corresponding to the historical frame control signal, and determining an optical flow and a mask based on the historical image includes:

[0018] If the historical images corresponding to the historical frame control signal are a previous image and two previous images, determining a pixel offset between the two previous images and the previous image, and determining an optical flow based on the pixel offset;

[0019] An occlusion region between the first two frames of image and the first frame of image is determined, and a mask is determined according to the occlusion region.

[0020] Optionally, before the step of injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, the method includes:

[0021] Acquire a control signal to be trained, and determine a video sequence corresponding to the control signal to be trained;

[0022] Matching data is determined according to the control signal to be trained and the video sequence, and a preset video generation unit network is optimized according to the matching data and a preset loss function to obtain a trained target video generation unit network.

[0023] Optionally, the loss function includes at least one of a pixel loss function, a perceptual loss function, a feature matching loss function, an adversarial loss function, a temporal loss function, and an optical flow loss function.

[0024] In addition, to achieve the above-mentioned object, the present invention further provides a video generation device, comprising:

[0025] an acquisition module, configured to acquire a current frame control signal and a historical frame control signal, inject parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, and determine a preliminary generated image based on the parameter injection results;

[0026] a determination module, configured to determine a historical image corresponding to the historical frame control signal, and determine an optical flow and a mask based on the historical image;

[0027] A construction module is used to determine a current frame image according to the preliminary generated image, the optical flow and the mask, and to construct a video according to the current frame image.

[0028] In addition, to achieve the above-mentioned purpose, the present invention also provides a video generation device, which includes a memory, a processor, and a video generation program stored in the memory and runnable on the processor. When the video generation program is executed by the processor, the steps of the video generation method as described above are implemented.

[0029] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a video generation program is stored. When the video generation program is executed by a processor, the steps of the above-mentioned video generation method are implemented.

[0030] The present invention injects parameters into all network layers of the target video generation unit according to the current frame control signal and the historical frame control signal, thereby effectively preventing the signal attenuation of the control signal, making the generated preliminary generated image higher in quality, and determining the optical flow and mask according to the historical image corresponding to the historical frame control signal, and then determining the current frame image according to the preliminary generated image, the optical flow and the mask, and constructing the video according to the current frame image, thereby achieving no reliance on CG technology and only using the target video generation network to generate the video, avoiding the current reliance on CG technology that makes the video generation process too complicated, improving the overall efficiency of the video generation process, and thus also improving the rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the terminal / device structure of the hardware operating environment involved in the embodiment of the present invention;

[0032] Figure 2 This is a flow chart of a first embodiment of a video generation method according to the present invention;

[0033] Figure 3 Schematic diagram of the device unit of the video generating device of the present invention;

[0034] Figure 4 Schematic diagram of the training and deployment process in the video generation method of the present invention;

[0035] Figure 5 Schematic diagram of the architecture of the target video generation unit network in the video generation method of the present invention.

[0036] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0037] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0039] The terminal in the embodiment of the present invention is a video generating device.

[0040] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0041] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, the sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the terminal device is moved to the ear. Of course, the terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0042] Those skilled in the art will understand that Figure 1The terminal structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0043] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a video generation program.

[0044] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the video generation program stored in the memory 1005 and perform the following operations:

[0045] Reference Figure 2 The present invention provides a video generation method. In a first embodiment of the video generation method, the video generation method includes the following steps:

[0046] Step S10, obtaining a current frame control signal and a historical frame control signal, injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, and determining a preliminary generated image based on the parameter injection result;

[0047] Since the current CG rendering process is cumbersome and high image quality leads to a significant increase in rendering time, it is unable to cope with situations with high real-time requirements. If you want to reduce rendering time, you can only sacrifice image quality. In addition, during the deployment stage, the CG process relies on many components, such as 3D models, maps, scenes, renderers, etc., which makes it more complicated to deploy. Therefore, in this embodiment, a neural network is used, and the forward calculation amount is constant to ensure the efficiency of video output. At the same time, due to the addition of a multi-layer parameter injection structure and an explicit timing constraint mechanism, the generation from control signal to video can be completed directly, and the timing is coherent and the picture flow is smooth. In this embodiment, the video generation unit network needs to be trained first, and then deployed after the trained target video generation unit network is obtained.

[0048] Therefore, in this embodiment, a batch of control signals can be processed by the data generation unit to generate a series of high-quality image sequences, that is, high-quality video sequences. The data generation unit can process the control signals by processing high-precision CG models, materials, lighting and scene data together with the control signals through a high-quality renderer to obtain high-quality image sequences corresponding to the control signals. The control signals include but are not limited to parameter signals such as character posture, expression, gesture and position. Matching data is then determined based on the video sequence and the control signals, and the pre-set video generation unit network is trained and optimized based on the matching data and a pre-set loss function to obtain a trained target video generation unit network.

[0049] In this embodiment, when the training of the video generation unit network is completed and the trained target video generation unit network is obtained, the deployment phase can be entered, that is, the control signal sequence (i.e., matching data <control signal, video sequence>) can be obtained through the driving module, and the current frame control signal and the historical frame control signal in the control signal sequence can be obtained. Among them, the current frame control signal is the control signal of the current frame. The historical frame control signal is the control signal of the historical frame, and the historical frame can be any number of frames before the current frame, such as the previous frame, the previous two frames, etc. Then, according to the current frame control signal and the historical frame control signal, all network layers in the trained target video generation unit network are injected with parameters (i.e., the control signal is injected into each network layer), that is, a multi-layer injection operation is performed. Multi-layer injection is a special network structure that transmits the control signal separately to each network layer, making the effect of the control signal more effective and the control signal less likely to attenuate during network transmission. In addition, the control signal is projected into the implicit space by a multi-layer perceptron MLP to display the control parameters. The control parameters projected into the implicit space are then transmitted to the AdaIN (Adaptive Instance Normalization) layer of each network module in the form of mean and variance. The mean and variance of the current instance are calculated and normalized in the AdaIN layer. And by converting the control signal into the form of mean and variance and injecting it into the AdaIN layer of each network block, each network block is effectively integrated with the control signal. And the use of multi-layer injection can uniformly control the matching of high-level semantic features and low-level local features with the control signal. In addition, in another scenario, a cascade method can be used to incorporate the control signal into the input of the target video generation unit network.

[0050] And the target video generation unit network in this embodiment may include a generation network branch and an optical flow network branch. The input part of the target video generation unit network may include a control signal P t-2 , P t-1 , P t , and Pt-2 , P t-1 , P t I is the control signal of the current frame and the previous two frames (that is, the current frame control signal and the historical frame control signal); t-2 , I t-1 is the generated result of the previous two frames, that is, the historical image corresponding to the control signal of the historical frame; constant tensor, where the dimension of the constant tensor can be 1x1x512. Then, according to the control signal, the parameter injection is implemented, and in the generation network branch, the constant tensor is trained through a series of convolution, upsampling layers and parameter injection results in the AdaIN layer to obtain the preliminary result I′ of the current frame. t , that is, the image is initially generated.

[0051] Step S20, determining a historical image corresponding to the historical frame control signal, and determining an optical flow and a mask based on the historical image;

[0052] In the optical flow network branch of the target video generation unit network, the historical image corresponding to the historical frame control signal can be determined first, such as I t-2 , I t-1 . Then I t-2 , I t-1 The input passes through a series of downsampling and upsampling layers, and a control signal is injected, i.e., the parameter injection result, to obtain the optical flow O and mask M. The optical flow O describes the offset relationship between each pixel in the current frame and the previous frame. The mask M describes the occlusion of each pixel.

[0053] Step S30 , determining a current frame image according to the preliminary generated image, the optical flow, and the mask, and constructing a video according to the current frame image.

[0054] After obtaining the preliminary generated image, optical flow and mask, the final image synthesis can be performed. That is, according to the formula To determine the current frame image. Among them, ξ(I t-1 ,0) is the deformation operation, that is, the deformation result is obtained by adding the optical flow to the content of the previous frame. ☉ is the bitwise multiplication operation. t 'To generate the initial image. t-1 is the history graph. M is the mask. O is the optical flow. I t The current frame image. Simply put, the final current frame is determined by the result of fusing the generated image with the previous frame. The current frame image is determined by combining the historical frame image with the current frame image to construct a video.

[0055] In addition, to assist in understanding the video generation principle in this embodiment, an example is given below.

[0056] For example, Figure 4As shown, it includes two stages: training and deployment. In the training stage, the control signal can be used as the input of the data generation unit and the video generation unit, and in the data generation unit, the high-precision CG model, material & texture, lighting and scene are processed by a high-quality renderer to obtain a high-quality image sequence, that is, the paired data of <control signal, video sequence> can be determined. And in the video generation unit, a neural network is set up, and the paired data and loss function are used to optimize the neural network to obtain a trained target video generation unit network. In the deployment stage, the control signal can be obtained through the driving module, and the control signal is input into the video generation unit (that is, the trained target video generation unit network) for training to obtain high-quality video. Among them, the architecture of the target video generation unit network can be as follows Figure 5 If the current frame control signal is parameter P t , the historical frame control signal is parameter P t-1 and parameter P t-2 , frame I t-1 and frame I t-2 is the parameter P t-1 and parameter P t-2 The two-frame generation result is the corresponding historical image. Therefore, the input variable of the target video generation unit network can be parameter P t , parameter P t-1 and parameter P t-2 , frame I t-1 and frame I t-2 , constant tensor 1x1x512. And parameter P t , parameter P t-1 and parameter P t-2 It will enter a multi-layer perceptron MLP to obtain the control variables of the implicit space, and the control variables will be transmitted to the AdaIN layer of each network layer to achieve parameter injection. And because the target video generation unit network includes a generation network branch and an optical flow network branch, on the generation network branch side, the constant tensor 1x1x512 passes through a series of convolution 3*3, parameter-injected AdaIN layers and upsampling layers to obtain the preliminary generated image I of the current frame. t '. On the optical flow network branch side, frame I t-1 and frame I t-2 As input, a series of downsampling layers (such as the AdaIN layer with parameter injection, convolution 3*3, downsampling layer and upsampling layer) are used to obtain the optical flow O and mask M. Then, according to the mask M, optical flow O, frame I t-1 and the initial generated image I t 'To determine the final image I t , that is, the current frame image, and then combine the current frame image and the historical frame image to obtain the video.

[0057] In this embodiment, parameters are injected into all network layers in the target video generation unit network according to the current frame control signal and the historical frame control signal, thereby effectively preventing the signal attenuation of the control signal, making the generated preliminary generated image of higher quality, and determining the optical flow and mask according to the historical image corresponding to the historical frame control signal, and then determining the current frame image according to the preliminary generated image, the optical flow and the mask, and constructing the video according to the current frame image, thereby achieving no reliance on CG technology, and only using the target video generation network to generate the video, thereby improving the overall efficiency of the video generation process, and thus also improving the rendering efficiency.

[0058] Furthermore, based on the first embodiment of the present invention, a second embodiment of the video generation method of the present invention is proposed. In this embodiment, the refinement of step S30 of the above embodiment, the step of determining the current frame image based on the preliminary generated image, the optical flow, and the mask, includes:

[0059] Step a, calculating a difference between a preset value and the mask, and performing bitwise multiplication on the preliminary generated image and the difference to obtain a first bitwise multiplication result;

[0060] Step b: determining a deformation result based on the historical image and the optical flow, and performing bitwise multiplication on the deformation result and the mask to obtain a second bitwise multiplication result;

[0061] Step c: determining the current frame image based on the first bitwise multiplication result and the second bitwise multiplication result.

[0062] In this embodiment, when determining the current frame image, since the preliminary generated image I has been generated according to the target video generation unit network, t ', optical flow O and mask M, at this time, the first bitwise multiplication result can be calculated first. That is, determine the preset value (such as the natural number 1), and calculate the difference between 1 and M, that is, 1-M, and then calculate the first bitwise multiplication result, that is Then, deformation operation is performed based on the historical image and optical flow to obtain the deformation result, namely ξ(I t-1 , O). And determine the second bitwise multiplication result according to the deformation result and the mask, calculate the sum of the first bitwise multiplication result and the second bitwise multiplication result, and use this sum as the current frame image I t .Right now Among them, ξ(I t-1 , O) is the deformation operation, that is, the deformation result is obtained by adding the optical flow to the content of the previous frame. ☉ is the bitwise multiplication operation. t 'To generate the initial image. t-1 is the history graph. M is the mask. O is the optical flow. I t is the current frame image.

[0063] In this embodiment, the first bitwise multiplication result is calculated based on the mask and the preliminary generated image, the second bitwise multiplication result is determined based on the historical image, optical flow and mask, and the current frame image is determined based on the first bitwise multiplication result and the second bitwise multiplication result, thereby ensuring the accuracy of the acquired current frame image.

[0064] Furthermore, the step of injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal includes:

[0065] Step d, using the current frame control signal and the historical frame control signal as control parameters, and projecting the control parameters into an implicit space according to a preset multi-layer perceptron;

[0066] In this embodiment, when injecting parameters into each network layer in the target video generation unit network, that is, injecting control signals. First, the input current frame control signal and historical frame control signal are determined and used as control parameters, and then the control parameters are projected into the implicit space through a pre-set multilayer perceptron (MLP). The multilayer perceptron is an artificial neural network with a forward structure that maps a set of input vectors to a set of output vectors. The MLP can be regarded as a directed graph consisting of multiple node layers, and each layer is fully connected to the next layer. Among them, the implicit space is the hidden layer in the multilayer perceptron.

[0067] Step e: performing dimensionality-upgrading processing on the control parameters in the implicit space to obtain control variables, and injecting parameters into all network layers in the trained target video generation unit network according to the control variables.

[0068] The control parameters in the implicit space are then upgraded from one-dimensional data to multidimensional data. This allows them to be transferred to the AdaIN layers of all network layers in the target video generation unit network for parameter injection, or control signal injection. This effectively integrates the control signal into the AdaIN layers of every network layer in the target video generation unit network. The dimensionality increase process converts the control parameters into mean and variance form.

[0069] In this embodiment, the current frame control signal and the historical frame control signal are projected into the implicit space according to the multi-layer perceptron, and dimensionality is increased. Then, parameters are injected into all network layers in the target video generation unit network, thereby achieving parameter injection into multiple network layers. This can more uniformly control the matching of high-level semantic features and low-level local features with the control signal.

[0070] Furthermore, the step of determining the preliminary generation of an image based on the parameter injection result includes:

[0071] Step f, obtaining the input constant tensor, and performing model training on the constant tensor according to the parameter injection result and the upsampling layer in the target video generation unit network to obtain a preliminary generated image.

[0072] In this embodiment, when determining the preliminary generated image, the input variable of the target video generation unit network also requires a constant tensor. Therefore, a constant tensor input by the user can be obtained, where the dimension of the constant tensor is 1x1x512. Then, based on the parameter injection results, the AdaIN layer that has completed parameter injection is determined. A series of convolution and upsampling layers are performed on the constant tensor based on the AdaIN layer that has completed parameter injection to obtain the preliminary result of the current frame, that is, the preliminary generated image.

[0073] In this embodiment, a preliminary generated image is obtained by performing model training on the constant tensor according to the parameter injection result and the upsampling layer in the target video generation unit network, thereby ensuring the accuracy and effectiveness of the obtained preliminary generated image.

[0074] Furthermore, the step of determining a historical image corresponding to the historical frame control signal and determining an optical flow and a mask based on the historical image includes:

[0075] Step h: if the historical images corresponding to the historical frame control signal are the previous image and the previous two images, determining the pixel offset between the previous two images and the previous image, and determining the optical flow based on the pixel offset;

[0076] In this embodiment, in order to make the generated current frame image have temporal continuity, "display timing constraints" are introduced to enhance inter-frame stability, and the inter-frame temporal continuity is reflected in the form of optical flow, so that the final result generated by each frame (i.e., the current frame image) depends on the historical frame results (i.e., historical images). For example, the current frame image I t by I t-1 After optical flow distortion (warp) and superposition, the result I is generated t 'And so I t with I t-1 It has strong continuity, which makes the overall video have strong continuity.

[0077] Therefore, in this embodiment, the optical flow and mask can be determined directly based on the historical image corresponding to the historical frame control signal. If the historical image is the previous frame and the previous two frames, the pixel offset from the previous two frames to the previous frame is statistically calculated, and the pixel flow direction is determined based on the pixel offset and used as the optical flow.

[0078] Step i: determining an occlusion area between the first two frames of image and the first frame of image, and determining a mask according to the occlusion area.

[0079] When determining the mask, the occlusion region between the first two frames of image and the first frame of image may be determined, and the mask may be created based on the occlusion region.

[0080] In this embodiment, when the historical images are the previous frame and the two previous frames, the pixel offset is calculated and then the optical flow is determined based on the pixel offset. A mask is then determined based on the occlusion area between the two previous frames and the previous frame. This ensures that the current frame, subsequently generated based on the optical flow and mask, is strongly correlated with the historical images, ensuring continuity in the subsequently generated video.

[0081] Furthermore, based on the first or second embodiment of the present invention, a third embodiment of the video generation method of the present invention is proposed. In this embodiment, before the step of injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, the method includes:

[0082] Step m, obtaining a control signal to be trained, and determining a video sequence corresponding to the control signal to be trained;

[0083] In this embodiment, a pre-set control signal to be trained is first obtained and input into a data generation unit. A high-quality renderer then generates a video sequence corresponding to the control signal. When generating the video sequence, the high-quality renderer also comprehensively considers data such as high-precision CG models, materials, lighting, and scene data.

[0084] Step n: determining matching data based on the control signal to be trained and the video sequence, and optimizing the preset video generation unit network based on the matching data and a preset loss function to obtain a trained target video generation unit network.

[0085] Each control signal to be trained and its corresponding video sequence are determined to form matching data, such as <control signal, video sequence>. The control signal in the matching data is then input into a pre-set video generation unit network, which is configured with at least one loss function. Training is performed based on the loss function and the control signal to obtain a result. The video generation unit network is then trained and optimized based on the result and the video sequence corresponding to the control signal to obtain a trained video generation unit network.

[0086] Specifically, the loss function includes at least one of a pixel loss function, a perceptual loss function, a feature matching loss function, an adversarial loss function, a temporal loss function, and an optical flow loss function.

[0087] For example, when the loss function is a pixel loss function, assuming is the value of the jth channel of the i-th pixel (RGB three channels), and y ij is the true value of the jth channel of the i-th pixel, then the pixel loss is:

[0088]

[0089] Where N is the number of pixels in an image. The pixel loss function can be used to constrain the generated image to match the target image at the pixel level.

[0090] When the loss function is a perceptual loss function, the perceptual loss is to use the trained VGG model to supervise the generated image at the feature level. Let Φ(x) be a VGG subnetwork used to extract features. I is the target image, is the reconstructed image. Then the perceptual loss is expressed as:

[0091]

[0092] Where C, H, and W represent the number of channels, length, and width of the feature, respectively. The perceptual loss function can further achieve feature-level matching between the generated image and the target image under pixel-level constraints.

[0093] When the loss function is an adversarial loss function, the adversarial loss uses a discriminator D to compete with the image generator to achieve the goal of improving image details. For real images, the discriminator will get 1, while the generated image will get 0. The video generation unit network is simply represented by G, and the adversarial loss is expressed as:

[0094] l gan =E[log(D(I))]+E[log(1-D(G(P)))];

[0095]

[0096] Where I is the real image, P is the input parameter, and D attempts to maximize the loss, that is, to distinguish the real image from the generated image. G attempts to minimize the loss, that is, to make D unable to distinguish the generated image from the real image. Through adversarial loss function, more realistic generated images can be obtained. By using the adversarial loss function, the generated images can be made more realistic and more consistent with the distribution of real images.

[0097] When the loss function is a feature matching loss function, it needs to be used in conjunction with the adversarial loss function. The output of the discriminator is used for loss calculation, and the discriminator output of the generated image is strived to be similar to the real image. The specific expression is:

[0098] l fm =E[|D(I)-D(G(P))|];

[0099] The feature matching loss function can make the distribution of generated images and real images more similar.

[0100] When the loss function is a temporal loss function, the temporal loss is mainly used to make the generated two adjacent frames conform to the distribution of the real adjacent frames. temp Implementation. The discriminator outputs 1 for the two adjacent real frames and 0 for the two generated frames. The temporal loss is then expressed as:

[0101] l temp =E[log(D temp (I t-1 , I t ))]+E[log(1-D temp (G(P t-1 ,P t )))];

[0102]

[0103] D temp Try to maximize the loss, that is, distinguish between the continuous frames of real images and the continuous frames of generated images. G tries to minimize the loss, that is, let D temp It is impossible to distinguish the generated image sequence from the real image sequence. The temporal loss function can constrain the network to generate results that are more consistent with temporal continuity.

[0104] In addition, refer to Figure 3 , an embodiment of the present invention further provides a video generating device, comprising:

[0105] an acquisition module A10, configured to acquire a current frame control signal and a historical frame control signal, inject parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, and determine a preliminary generated image based on the parameter injection results;

[0106] a determination module A20, configured to determine a historical image corresponding to the historical frame control signal, and determine an optical flow and a mask based on the historical image;

[0107] The construction module A30 is configured to determine a current frame image according to the preliminary generated image, the optical flow, and the mask, and construct a video according to the current frame image.

[0108] Optionally, building block A30 is configured to:

[0109] Calculating a difference between a preset value and the mask, and performing bitwise multiplication on the preliminary generated image and the difference to obtain a first bitwise multiplication result;

[0110] determining a deformation result according to the historical image and the optical flow, and performing bitwise multiplication on the deformation result and the mask to obtain a second bitwise multiplication result;

[0111] A current frame image is determined based on the first bitwise multiplication result and the second bitwise multiplication result.

[0112] Optionally, the acquisition module A10 is used to:

[0113] Taking the current frame control signal and the historical frame control signal as control parameters, projecting the control parameters into an implicit space according to a preset multi-layer perceptron;

[0114] The control parameters in the implicit space are subjected to dimensionality increase processing to obtain control variables, and parameters are injected into all network layers in the trained target video generation unit network according to the control variables.

[0115] Optionally, the acquisition module A10 is used to:

[0116] An input constant tensor is obtained, and model training is performed on the constant tensor according to the parameter injection result and the upsampling layer in the target video generation unit network to obtain a preliminary generated image.

[0117] Optionally, the determination module A20 is configured to:

[0118] If the historical images corresponding to the historical frame control signal are a previous image and two previous images, determining a pixel offset between the two previous images and the previous image, and determining an optical flow based on the pixel offset;

[0119] An occlusion region between the first two frames of image and the first frame of image is determined, and a mask is determined according to the occlusion region.

[0120] Optionally, obtain module A10, for:

[0121] Acquire a control signal to be trained, and determine a video sequence corresponding to the control signal to be trained;

[0122] Matching data is determined based on the control signal to be trained and the video sequence, and a preset video generation unit network is optimized based on the matching data and a preset loss function to obtain a trained target video generation unit network. The loss function includes at least one of a pixel loss function, a perceptual loss function, a feature matching loss function, an adversarial loss function, a temporal loss function, and an optical flow loss function.

[0123] The steps implemented by the functional modules of the video generating device may refer to the various embodiments of the video generating method of the present invention, and will not be described in detail here.

[0124] In addition, the present invention also provides a video generation device, which includes: a memory, a processor, and a video generation program stored in the memory; the processor is used to execute the video generation program to implement the steps of each embodiment of the above-mentioned video generation method.

[0125] The present invention also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can also be executed by one or more processors to implement the steps of each embodiment of the above-mentioned video generation method.

[0126] The specific implementation of the computer-readable storage medium of the present invention is basically the same as the embodiments of the above-mentioned video generation method, and will not be repeated here.

[0127] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0128] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0130] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video generation method, characterized in that: The video generation method comprises the following steps: Obtaining a current frame control signal and a historical frame control signal, injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, and determining a preliminary generated image based on the parameter injection results; Determining a historical image corresponding to the historical frame control signal, and determining an optical flow and a mask based on the historical image; Determining a current frame image according to the preliminary generated image, the optical flow, and the mask, and constructing a video according to the current frame image; The step of injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal includes: Taking the current frame control signal and the historical frame control signal as control parameters, projecting the control parameters into an implicit space according to a preset multi-layer perceptron; The control parameters in the implicit space are subjected to dimensionality increase processing to obtain control variables, and parameters are injected into all network layers in the trained target video generation unit network according to the control variables.

2. The video generation method according to claim 1, wherein: The step of determining the current frame image according to the preliminary generated image, the optical flow and the mask includes: Calculating a difference between a preset value and the mask, and performing bitwise multiplication on the preliminary generated image and the difference to obtain a first bitwise multiplication result; determining a deformation result according to the historical image and the optical flow, and performing bitwise multiplication on the deformation result and the mask to obtain a second bitwise multiplication result; A current frame image is determined based on the first bitwise multiplication result and the second bitwise multiplication result.

3. The video generation method according to claim 1, wherein: The step of determining a preliminary generated image based on the parameter injection result includes: An input constant tensor is obtained, and model training is performed on the constant tensor according to the parameter injection result and the upsampling layer in the target video generation unit network to obtain a preliminary generated image.

4. The video generation method according to claim 1, wherein: The step of determining the historical image corresponding to the historical frame control signal and determining the optical flow and mask according to the historical image includes: If the historical images corresponding to the historical frame control signal are a previous image and two previous images, determining a pixel offset between the two previous images and the previous image, and determining an optical flow based on the pixel offset; An occlusion region between the first two frames of image and the first frame of image is determined, and a mask is determined according to the occlusion region.

5. The video generation method according to claim 1, wherein: Before the step of injecting parameters into all network layers in the trained target video generation unit network according to the current frame control signal and the historical frame control signal, the method includes: Acquire a control signal to be trained, and determine a video sequence corresponding to the control signal to be trained; Matching data is determined according to the control signal to be trained and the video sequence, and a preset video generation unit network is optimized according to the matching data and a preset loss function to obtain a trained target video generation unit network.

6. The video generation method according to claim 5, wherein: The loss function includes at least one of a pixel loss function, a perceptual loss function, a feature matching loss function, an adversarial loss function, a temporal loss function, and an optical flow loss function.

7. A video generating device, characterized in that: The video generating device comprises: An acquisition module is used to acquire a current frame control signal and a historical frame control signal, perform parameter injection on all network layers in a trained target video generation unit network according to the current frame control signal and the historical frame control signal, and determine a preliminary generated image based on the parameter injection result; wherein, the parameter injection on all network layers in a trained target video generation unit network according to the current frame control signal and the historical frame control signal comprises: using the current frame control signal and the historical frame control signal as control parameters, projecting the control parameters into an implicit space according to a preset multi-layer perceptron; performing dimensionality increase processing on the control parameters in the implicit space to obtain control variables, and injecting parameters into all network layers in a trained target video generation unit network according to the control variables; a determination module, configured to determine a historical image corresponding to the historical frame control signal, and determine an optical flow and a mask based on the historical image; A construction module is used to determine a current frame image according to the preliminary generated image, the optical flow and the mask, and to construct a video according to the current frame image.

8. A video generating device, characterized in that The video generation device includes: a memory, a processor, and a video generation program stored in the memory and executable on the processor. When the video generation program is executed by the processor, the steps of the video generation method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a video generation program, which, when executed by a processor, implements the steps of the video generation method according to any one of claims 1 to 6.

Images