Video generation method, non-volatile storage medium and electronic device

Abstract

The present application discloses a video generation method, a non-volatile storage medium and an electronic device. The method comprises: obtaining a source image and a driving video, wherein the source image is used to describe the appearance information of a target object, and the driving video is used to describe the action information of the target object; and generating a target video based on the source image and the driving video. The present application solves the technical problem that the quality of the generated action video in the prior art is low and there are obvious time domain jitters and jumps.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a video generation method, a non-volatile storage medium and an electronic device. Background Art

[0002] Character action video generation, also known as "static motion", aims to generate new motion videos based on a given static image through gesture-driven editing methods, making the characters in the image move. This problem has application needs in e-commerce promotion, intelligent visual creation, film and television production, etc.

[0003] In the prior art, due to the limitation of algorithm complexity, only human body images with a resolution of 256 can be processed, and the generated video quality is low, the temporal consistency is poor, and there are obvious jitters and flickering.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a video generation method, a non-volatile storage medium, and an electronic device to at least solve the technical problems in the prior art of generating action videos of low quality and having relatively obvious time domain jitter and jump.

[0006] According to one aspect of an embodiment of the present application, a video generation method is provided, comprising: acquiring a source image and a driving video, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object; and generating a target video based on the source image and the driving video.

[0007] According to another aspect of an embodiment of the present application, another video generation method is provided, including: receiving a currently input source image and a driving video, wherein the source image is used to describe the appearance information of a target object, and the driving video is used to describe the action information of the target object; sending the source image and the driving video to a server; receiving a target video from the server, wherein the target video is generated by the server based on the source image and the driving video; and playing the target video locally on a client.

[0008] According to another aspect of an embodiment of the present application, another video generation method is provided, including: receiving a source image and a driving video from a client, wherein the source image is used to describe the appearance information of a target object, and the driving video is used to describe the action information of the target object; generating a target video based on the source image and the driving video; returning the target video to the client, and playing the target video locally on the client.

[0009] According to another aspect of an embodiment of the present application, there is provided another video generation method, comprising: obtaining a source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; inputting the source image and the driving video into a first granularity network model to obtain target motion flow data, wherein the first granularity network model is used to perform multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video; inputting the source image and the target motion flow data into a second granularity network model to obtain a target video, wherein the second granularity network model comprises: at least one encoding layer and at least one reconstruction layer, wherein the at least one encoding layer is used to obtain multi-scale image features of the source image, and the at least one reconstruction layer is used to obtain multiple target video frames using the target motion flow data and the multi-scale image features.

[0010] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned video generation methods.

[0011] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a processor; and a memory, connected to the above-mentioned processor, for providing the above-mentioned processor with instructions for processing the following processing steps: obtaining a source image and a driving video, wherein the above-mentioned source image is used to describe the appearance information of the target object, and the above-mentioned driving video is used to describe the action information of the above-mentioned target object; generating a target video based on the above-mentioned source image and the above-mentioned driving video.

[0012] In an embodiment of the present application, a source image and a driving video are obtained, wherein the source image is used to describe the appearance information of the target object and the driving video is used to describe the action information of the target object; a target video is generated based on the source image and the driving video.

[0013] It is easy to notice that the embodiment of the present application is based on a given source image and a driving video, and uses a deep neural network to process and generate a target video according to the appearance information of the target object carried in the source image and the motion information of the target object carried in the driving video.

[0014] Therefore, the embodiment of the present application achieves the purpose of generating an action video with relatively coherent time domain motion, high resolution and realistic details, thereby achieving the technical effect of improving the quality of the generated action video, and further solving the technical problem of low quality of generated action video and obvious time domain jitter and jump in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method according to an embodiment of the present application;

[0017] Figure 2 is a flow chart of a video generation method according to an embodiment of the present application;

[0018] Figure 3 is a scene schematic diagram of a video generation method according to an embodiment of the present application;

[0019] Figure 4 is a flowchart of another video generation method according to an embodiment of the present application;

[0020] Figure 5 is a flowchart of another video generation method according to an embodiment of the present application;

[0021] Figure 6 is a flowchart of another video generation method according to an embodiment of the present application;

[0022] Figure 7 is a structural schematic diagram of a video generating device according to an embodiment of the present application;

[0023] Figure 8 is a structural schematic diagram of another video generating device according to an embodiment of the present application;

[0024] Fig. 9 is a structural schematic diagram of another video generating device according to an embodiment of the present application;

[0025] Fig.10 It is a structural block diagram of another computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] According to an embodiment of the present application, an embodiment of a video generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiment provided in Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0031] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the video generation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned video generation method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0035] Under the above operating environment, this application provides Figure 2 A video generation method is shown. Figure 2 is a flow chart of a video generation method according to an embodiment of the present application, such as Figure 2 As shown, the above video generation method includes:

[0036] Step S202, obtaining a source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object;

[0037] Step S204: generating a target video based on the source image and the driving video.

[0038] In an embodiment of the present application, a source image and a driving video are obtained, wherein the source image is used to describe the appearance information of the target object and the driving video is used to describe the action information of the target object; a target video is generated based on the source image and the driving video.

[0039] It is easy to notice that the embodiment of the present application is based on a given source image and a driving video, and uses a deep neural network to process and generate a target video according to the appearance information of the target object carried in the source image and the motion information of the target object carried in the driving video.

[0040] Therefore, the embodiment of the present application achieves the purpose of generating an action video with relatively coherent time domain motion, high resolution and realistic details, thereby achieving the technical effect of improving the quality of the generated action video, and further solving the technical problem of low quality of generated action video and obvious time domain jitter and jump in the prior art.

[0041] Optionally, in an embodiment of the present application, the target object may be a person, an animal, a robot, etc., and the target video may be an action video; for example, taking the target object as a person, the source image may be a person image, the driving video may be a person driving video, and the target video may be a person action video; taking the target object as an animal, the source image may be an animal image, the driving video may be an animal driving video, and the target video may be an animal action video.

[0042] It should be noted that the embodiments of the present application can be applied to, but not limited to, online shopping platforms, live broadcast platforms, and other scenarios where action videos need to be generated, and can significantly enhance the resolution and continuity of the generated videos.

[0043] The video generation method provided in the embodiment of the present application actually includes a novel action video generation framework, which combines multi-scale feature migration and intrinsic time domain consistency constraints to generate high-resolution character action videos that are continuous in time domain, rich in details and realistic (for example, 512*512 character action videos, while the video generation level of the prior art can only reach 256*256), and has certain practical and economic value.

[0044] Optionally, the high-resolution character video generation framework proposed in the embodiment of the present application can support the generation of 512*512 high-quality action videos; and, based on the novel intrinsic time-domain consistency constraint scheme provided in the embodiment of the present application, it can effectively solve the technical problem of inaccurate high-resolution motion estimation in action videos generated by the prior art, and effectively reduce or avoid the occurrence of time-domain jitter, flicker and other phenomena in the generated action videos to a large extent.

[0045] In an optional embodiment, generating the target video based on the source image and the driving video includes:

[0046] Step S302, performing multi-scale motion estimation frame by frame based on the source image and the driving frames included in the driving video to obtain target motion flow data;

[0047] Step S304: using the source image and the target motion flow data to determine a plurality of target video frames frame by frame, and synthesizing the plurality of target video frames into the target video.

[0048] For example, the target object is a person, and the target video is a person action video. Figure 3 As shown, an embodiment of the present application provides a character action video generation framework, which generates a target video O based on a given source image S and a driving video D. The character appearance information in the target video O is consistent with the source image S, and the action information is consistent with the driving video D.

[0049] It should be noted that the action video generation algorithm in the embodiment of the present application adopts a frame-by-frame generation method to synthesize the target video based on the source image and the driving video. Figure 3 The flowchart of the video generation method when generating the tth frame and the corresponding model training strategy are shown in Figure 1. It is divided into the following two steps:

[0050] Step S302, performing multi-scale motion estimation frame by frame based on the source image and the driving frames included in the driving video to obtain target motion flow data;

[0051] Step S304: using the source image and the target motion flow data to determine a plurality of target video frames frame by frame, and synthesizing the plurality of target video frames into the target video.

[0052] In an optional embodiment, the step S302, performing multi-scale motion estimation frame by frame based on the source image and the driving frames included in the driving video, to obtain the target motion flow data includes:

[0053] Step S402, performing multi-scale motion estimation frame by frame based on the source image and the driving frame corresponding to each moment at multiple different moments of the driving video to obtain initial motion flow data;

[0054] Step S404: up-sample the initial motion flow data to obtain the target motion flow data.

[0055] In the embodiment of the present application, Figure 3 As shown, the driving video D (Driving Video) contains multiple driving frames (Driving Frame, ie D_t, D_t-1, etc.) at different times.

[0056] As an optional embodiment, multi-scale motion estimation (Motion Filed Refiner) refers to using a deep neural network to process a given source image S and a driving video D containing a plurality of driving frames at different times at time t. Figure 3 The Base Motion Estimator shown in the figure is used to obtain the coarsest scale (smallest resolution) initial motion flow F_t(0), and then the initial motion flow F_t(0) is upsampled to obtain several layers of more refined target motion flows F_t(1), F_t(2)... and so on (it should be noted that Figure 3 Four layers are drawn to schematically illustrate the above-mentioned layers, but any other number of layers may be used, so the number of layers should be included in the scope of the embodiments of the present application).

[0057] In an optional embodiment, the step S304 of determining the plurality of target video frames frame by frame using the source image and the target motion flow data, and synthesizing the plurality of target video frames into the target video comprises:

[0058] Step S502, inputting the source image into at least one coding layer, and outputting multi-scale image features;

[0059] Step S504, performing spatial domain transformation on the data stream corresponding to each scale image feature in the multi-scale image feature in the target motion stream data, and outputting the transformation result;

[0060] Step S506: input the multi-scale image features and the transformation results into at least one reconstruction layer, output the multiple target video frames, and synthesize the multiple target video frames into the target video.

[0061] In the embodiment of the present application, Figure 3As shown, in the process of synthesizing a target video based on multiple target video frames (Progressive Fusion Generator) in the embodiment of the present application, the target video frame is generated in a progressive manner, for example, by taking a given source image S, obtaining multi-scale image features through several coding layer operations, and then obtaining an output frame O_t through several reconstruction layers. In addition, the image features of each scale also need to be spatially transformed with the estimated target motion flow data and the data flow at the corresponding scale (i.e., as Figure 3 w), and input the multi-scale image features and the output transformation results into the corresponding reconstruction layer (i.e. Figure 3 Inverse Warping as shown), output the above-mentioned multiple target video frames, and synthesize the above-mentioned multiple target video frames into the above-mentioned target video.

[0062] It should be noted that in the embodiment of the present application, the above multi-scale motion estimation and video frame generation network can have any number of layers and is not subject to the following conditions: Figure 3 Schematic diagram of the limitations of the training phase shown.

[0063] In an optional embodiment, the above method further includes:

[0064] Step S602, performing multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video to obtain a weight matrix, wherein the weight matrix is ​​used to calculate the temporal consistency constraint loss during the training process of the deep neural network.

[0065] In addition, based on the multi-scale motion estimation performed frame by frame based on the source image and the driving frames contained in the driving video, the deep neural network model also outputs a weight matrix A with a value in [0,1] for calculating the temporal consistency constraint loss in the training phase (e.g. Figure 3 Intrinsic Attention Map shown in ).

[0066] In an optional embodiment, the above method further includes:

[0067] Step S702, inputting adjacent first and second video frames in the video to be trained into a pre-trained optical flow estimation model, and outputting an optical flow from the second video frame to the first video frame, wherein the video to be trained is the same as the target video, the first video frame is a video frame at a current moment, and the second video frame is a video frame at a previous moment;

[0068] Step S704, using the above optical flow to perform spatial domain transformation on the target video frame at the current moment to obtain a transformed image frame;

[0069] Step S706, calculating the weighted distance between the transformed image frame and the second video frame using the weight matrix;

[0070] Step S708: adjusting the transformed image frame using the weighted distance.

[0071] In an embodiment of the present application, in the training stage of the deep neural network model for video generation, the generation algorithm adopts supervised training, assuming that the target video (Generated Video) to be generated exists, denoted as I. In practice, the input source image S is taken as the first frame of the target video, that is, S=D_0.

[0072] like Figure 3 The right side part shown is the training function calculation part, which may include but is not limited to the following steps (it should be noted that the following steps do not need to be performed after the deep neural network model training is completed): the current frame I_t (i.e., the first video frame) and the previous frame I_{t-1} (i.e., the second video frame) of the target video are input into a pre-trained optical flow estimation model to extract the optical flow f_t from the t-1 frame to the t frame.

[0073] The optical flow f_t is used to perform spatial domain transformation on the current generated frame O_t to obtain the transformed image frame W_{t-1}; the weight matrix A obtained in the above multi-scale motion estimation process is multiplied by the weighted distance between the transformed image frames W_{t-1} and I_{t-1} (that is, Figure 3 Intrinsic Temporal loss in

[0074] Weighted distance L_{intrinsic}=||(W_{t-1}-I_{t-1})*A||;

[0075] In the embodiment of the present application, after the weighted distance is obtained, the transformed image frame W_{t-1} is adjusted using the weighted distance.

[0076] Optionally, the weighted distance obtained by the above multi-scale motion estimation process can be extended to multiple scales. Accordingly, the weight matrix A used in the embodiment of the present application can also perform motion estimation processing at multiple scales.

[0077] It should be noted that in the embodiments of the present application, the type of the optical flow estimation model is not limited, and can be either a traditional algorithm or a deep neural network. If it is a deep neural network, it does not need to be pre-trained in advance, but can be end-to-end trained together with the deep neural network generation model in the embodiments of the present application, which does not affect the applicability of the embodiments of the present application.

[0078] The multi-scale motion estimation processing proposed in the embodiment of the present application improves the problems of the current algorithm in estimating high-resolution optical flow (the problematic parts can be reduced by lowering the weights), and on the other hand, provides a method directly connected to the front-end motion estimation module, so that the gradient can be directly obtained without the need for back propagation through the entire generation network, thereby improving the stability of the model training. The video generation embodiment proposed in this solution can generate high-resolution, realistic action videos with relatively coherent time domain actions.

[0079] According to the embodiment of the present application, there is provided Figure 4 Another video generation method shown. Figure 4 is a flowchart of another video generation method according to an embodiment of the present application. Figure 4 As shown, the above video generation method includes:

[0080] Step S802, receiving a currently input source image and a driving video, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object;

[0081] Step S804, sending the source image and the driving video to the server;

[0082] Step S806, receiving a target video from the server, wherein the target video is generated by the server based on the source image and the driving video;

[0083] Step S808: Play the target video locally on the client.

[0084] In an embodiment of the present application, by receiving the currently input source image and driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; sending the source image and the driving video to the server; receiving the target video from the server, wherein the target video is generated by the server based on the source image and the driving video; and playing the target video locally on the client.

[0085] It should be noted that the executor of the embodiment of the present application is a SaaS client. It is easy to notice that the client of the embodiment of the present application will send the above-mentioned source image and the above-mentioned driving video to the server. The server uses a deep neural network to process and generate a target video based on the given source image and driving video, according to the appearance information of the target object carried in the source image, and the action information of the target object carried in the driving video. The client plays the target video received from the above-mentioned server locally on the client.

[0086] Therefore, the embodiment of the present application achieves the purpose of generating an action video with relatively coherent time domain motion, high resolution and realistic details, thereby achieving the technical effect of improving the quality of the generated action video, and further solving the technical problem of low quality of generated action video and obvious time domain jitter and jump in the prior art.

[0087] According to the embodiment of the present application, there is provided Figure 5 Another video generation method shown. Figure 5 is a flowchart of another video generation method according to an embodiment of the present application. Figure 5 As shown, the above video generation method includes:

[0088] Step S902, receiving a source image and a driving video from a client, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object;

[0089] Step S902, generating a target video based on the source image and the driving video;

[0090] Step S902: Return the target video to the client, and play the target video locally on the client.

[0091] In an embodiment of the present application, a source image and a driving video are received from a client, wherein the source image is used to describe the appearance information of the target object and the driving video is used to describe the action information of the target object; a target video is generated based on the source image and the driving video; the target video is returned to the client, and the target video is played locally on the client.

[0092] It should be noted that the executor of the embodiment of the present application is the SaaS server. It is easy to notice that the client of the embodiment of the present application will send the above-mentioned source image and the above-mentioned driving video to the server. The server uses a deep neural network to process and generate a target video based on the given source image and driving video, according to the appearance information of the target object carried in the source image, and the action information of the target object carried in the driving video. The client plays the target video received from the above-mentioned server locally on the client.

[0093] Therefore, the embodiment of the present application achieves the purpose of generating an action video with relatively coherent time domain motion, high resolution and realistic details, thereby achieving the technical effect of improving the quality of the generated action video, and further solving the technical problem of low quality of generated action video and obvious time domain jitter and jump in the prior art.

[0094] According to the embodiment of the present application, there is provided Figure 6 Another video generation method shown. Figure 6 is a flowchart of another video generation method according to an embodiment of the present application. Figure 6 As shown, the above video generation method includes:

[0095] Step S1002, obtaining a source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object;

[0096] Step S1004, inputting the source image and the driving video into a first granularity network model to obtain target motion flow data, wherein the first granularity network model is used to perform multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video;

[0097] Step S1006: input the source image and the target motion flow data into a second granularity network model to obtain a target video, wherein the second granularity network model includes: at least one encoding layer and at least one reconstruction layer, the at least one encoding layer is used to obtain multi-scale image features of the source image, and the at least one reconstruction layer is used to obtain multiple target video frames using the target motion flow data and the multi-scale image features.

[0098] In an embodiment of the present application, a source image and a driving video are obtained, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; the source image and the driving video are input into a first granularity network model to obtain target motion flow data, wherein the first granularity network model is used to perform multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video; the source image and the target motion flow data are input into a second granularity network model to obtain a target video, wherein the second granularity network model includes: at least one encoding layer and at least one reconstruction layer, the at least one encoding layer is used to obtain multi-scale image features of the source image, and the at least one reconstruction layer is used to obtain multiple target video frames using the target motion flow data and the multi-scale image features.

[0099] It is easy to notice that the embodiment of the present application is based on a given source image and a driving video, and according to the appearance information of the target object carried in the source image and the motion information of the target object carried in the driving video, the source image and the driving video are input into a first granularity network model to obtain target motion flow data, and the source image and the target motion flow data are input into a second granularity network model to obtain a target video.

[0100] Therefore, the embodiment of the present application achieves the purpose of generating an action video with relatively coherent time domain motion, high resolution and realistic details, thereby achieving the technical effect of improving the quality of the generated action video, and further solving the technical problem of low quality of generated action video and obvious time domain jitter and jump in the prior art.

[0101] Optionally, in an embodiment of the present application, the target object may be a person, an animal, a robot, etc., and the target video may be an action video; for example, taking the target object as a person, the source image may be a person image, the driving video may be a person driving video, and the target video may be a person action video; taking the target object as an animal, the source image may be an animal image, the driving video may be an animal driving video, and the target video may be an animal action video.

[0102] Optionally, the first granularity network model is used to perform multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video; the second granularity network model includes: at least one encoding layer and at least one reconstruction layer, the at least one encoding layer is used to obtain multi-scale image features of the source image, and the at least one reconstruction layer is used to obtain multiple target video frames using the target motion flow data and the multi-scale image features.

[0103] It should be noted that the embodiments of the present application can be applied to, but not limited to, online shopping platforms, live broadcast platforms, and other scenarios where action videos need to be generated, and can significantly enhance the resolution and continuity of the generated videos.

[0104] The video generation method provided in the embodiment of the present application actually includes a novel action video generation framework, which combines multi-scale feature migration and intrinsic time domain consistency constraints to generate high-resolution character action videos that are continuous in time domain, rich in details and realistic (for example, 512*512 character action videos, while the video generation level of the prior art can only reach 256*256), and has certain practical and economic value.

[0105] As an optional embodiment, the high-resolution character video generation framework proposed in the embodiment of the present application can support the generation of 512*512 high-quality action videos; and, based on the novel intrinsic time domain consistency constraint scheme provided in the embodiment of the present application, it can effectively solve the technical problem of inaccurate high-resolution motion estimation in action videos generated by the prior art, and effectively reduce or avoid the occurrence of time domain jitter, flicker and other phenomena in the generated action videos to a large extent.

[0106] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the above methods of each embodiment of the present application.

[0108] Example 2

[0109] According to an embodiment of the present application, a device embodiment for implementing the above-mentioned video generation method is also provided. Figure 7 is a structural diagram of a video generating device according to an embodiment of the present application, such as Figure 7 As shown, the device includes: an acquisition module 600 and a first generation module 602, wherein:

[0110] The acquisition module 600 is used to acquire a source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; the first generation module 602 is used to generate a target video based on the source image and the driving video.

[0111] It should be noted that the acquisition module 600 and the first generation module 602 correspond to steps S202 to S204 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0112] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned video generation method is also provided. Figure 8 is a structural diagram of another video generating device according to an embodiment of the present application, such as Figure 8As shown, the device includes: a first receiving module 700, a sending module 702, a second receiving module 704 and a playing module 706, wherein:

[0113] The first receiving module 700 is used to receive the currently input source image and driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; the sending module 702 is used to send the source image and the driving video to the server; the second receiving module 704 is used to receive the target video from the server, wherein the target video is generated by the server based on the source image and the driving video; the playing module 706 is used to play the target video locally on the client.

[0114] It should be noted that the first receiving module 700, the sending module 702, the second receiving module 704 and the playing module 706 correspond to steps S802 to S908 in Example 1, and the examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0115] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned video generation method is also provided. Fig. 9 is a structural diagram of another video generating device according to an embodiment of the present application, such as Fig. 9 As shown, the device includes: a third receiving module 800, a second generating module 802 and a processing module 804, wherein:

[0116] The third receiving module 800 is used to receive the source image and the driving video from the client, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; the second generating module 802 is used to generate the target video based on the source image and the driving video; the processing module 804 is used to return the target video to the client and play the target video locally on the client.

[0117] It should be noted that the third receiving module 800, the second generating module 802 and the processing module 804 correspond to steps S902 to S904 in Embodiment 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Embodiment 1.

[0118] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0119] Example 3

[0120] According to an embodiment of the present application, an embodiment of an electronic device is also provided, and the electronic device can be any computing device in a computing device group. The electronic device includes: a processor and a memory, wherein:

[0121] A processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: acquiring a source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; and generating a target video based on the source image and the driving video.

[0122] In an embodiment of the present application, a source image and a driving video are obtained, wherein the source image is used to describe the appearance information of the target object and the driving video is used to describe the action information of the target object; a target video is generated based on the source image and the driving video.

[0123] It is easy to notice that the embodiment of the present application is based on a given source image and a driving video, and uses a deep neural network to process and generate a target video according to the appearance information of the target object carried in the source image and the motion information of the target object carried in the driving video.

[0124] Therefore, the embodiment of the present application achieves the purpose of generating an action video with relatively coherent time domain motion, high resolution and realistic details, thereby achieving the technical effect of improving the quality of the generated action video, and further solving the technical problem of low quality of generated action video and obvious time domain jitter and jump in the prior art.

[0125] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0126] Example 4

[0127] According to an embodiment of the present application, an embodiment of a computer terminal is also provided, and the computer terminal can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above-mentioned computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0128] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0129] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the video generation method of the application: obtaining a source image and a driving video, wherein the above-mentioned source image is used to describe the appearance information of the target object, and the above-mentioned driving video is used to describe the action information of the above-mentioned target object; generating a target video based on the above-mentioned source image and the above-mentioned driving video.

[0130] Optionally, Fig.10 is a structural block diagram of another computer terminal according to an embodiment of the present application, such as Fig.10 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 902 , a memory 904 , and a peripheral interface 906 .

[0131] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned video generation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; generate a target video based on the source image and the driving video.

[0133] Optionally, the processor may also execute program code of the following steps: performing multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video to obtain target motion flow data; determining multiple target video frames frame by frame using the source image and the target motion flow data, and synthesizing the multiple target video frames into the target video.

[0134] Optionally, the processor may also execute the program code of the following steps: performing multi-scale motion estimation frame by frame based on the source image and the drive frame corresponding to each moment in multiple different moments of the drive video to obtain initial motion flow data; upsampling the initial motion flow data to obtain the target motion flow data.

[0135] Optionally, the processor may also execute program codes of the following steps: inputting the source image into at least one encoding layer and outputting multi-scale image features; performing spatial domain transformation on the data stream corresponding to each scale image feature in the multi-scale image features in the target motion stream data, and outputting the transformation result; inputting the multi-scale image features and the transformation result into at least one reconstruction layer, outputting the multiple target video frames, and synthesizing the multiple target video frames into the target video.

[0136] Optionally, the processor may also execute the program code of the following steps: performing multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video to obtain a weight matrix, wherein the weight matrix is ​​used to calculate the time domain consistency constraint loss during the training process of the deep neural network.

[0137] Optionally, the processor may also execute the program code of the following steps: inputting adjacent first and second video frames in the video to be trained into a pre-trained optical flow estimation model, and outputting the optical flow from the second video frame to the first video frame, wherein the video to be trained is the same as the target video, the first video frame is the video frame at the current moment, and the second video frame is the video frame at the previous moment; using the optical flow to perform spatial domain transformation on the target video frame at the current moment to obtain a transformed image frame; using the weight matrix to calculate the weighted distance between the transformed image frame and the second video frame; and using the weighted distance to adjust the transformed image frame.

[0138] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive the currently input source image and driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; send the source image and the driving video to the server; receive the target video from the server, wherein the target video is generated by the server based on the source image and the driving video; and play the target video locally on the client.

[0139] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive the source image and driving video from the client, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; generate the target video based on the source image and the driving video; return the target video to the client, and play the target video locally on the client.

[0140] According to an embodiment of the present application, a video generation scheme is provided. A source image and a driving video are obtained, wherein the source image is used to describe the appearance information of a target object, and the driving video is used to describe the action information of the target object; and a target video is generated based on the source image and the driving video.

[0141] It is easy to notice that the embodiment of the present application is based on a given source image and a driving video, and uses a deep neural network to process and generate a target video according to the appearance information of the target object carried in the source image and the motion information of the target object carried in the driving video.

[0142] Therefore, the embodiment of the present application achieves the purpose of generating an action video with relatively coherent time domain motion, high resolution and realistic details, thereby achieving the technical effect of improving the quality of the generated action video, and further solving the technical problem of low quality of generated action video and obvious time domain jitter and jump in the prior art.

[0143] It can be understood by those skilled in the art that Fig.10 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.10 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Fig.10 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.10 Different configurations shown.

[0144] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0145] Example 5

[0146] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the video generation method provided in the above embodiment 1.

[0147] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0148] Optionally, in this embodiment, the storage medium is configured to store program codes for performing the following steps: obtaining a source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; and generating a target video based on the source image and the driving video.

[0149] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video to obtain target motion flow data; determining multiple target video frames frame by frame using the source image and the target motion flow data, and synthesizing the multiple target video frames into the target video.

[0150] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing multi-scale motion estimation frame by frame based on the driving frames corresponding to each of multiple different moments of the above-mentioned source image and the above-mentioned driving video to obtain initial motion flow data; upsampling the above-mentioned initial motion flow data to obtain the above-mentioned target motion flow data.

[0151] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: inputting the above-mentioned source image into at least one encoding layer and outputting multi-scale image features; performing spatial domain transformation on the data stream corresponding to each scale image feature in the above-mentioned multi-scale image features in the above-mentioned target motion flow data, and outputting the transformation result; inputting the above-mentioned multi-scale image features and the above-mentioned transformation result into at least one reconstruction layer, outputting the above-mentioned multiple target video frames, and synthesizing the above-mentioned multiple target video frames into the above-mentioned target video.

[0152] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video to obtain a weight matrix, wherein the weight matrix is ​​used to calculate the time domain consistency constraint loss during the training process of the deep neural network.

[0153] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: inputting adjacent first and second video frames in the video to be trained into a pre-trained optical flow estimation model, and outputting the optical flow from the second video frame to the first video frame, wherein the video to be trained is the same as the target video, the first video frame is the video frame at the current moment, and the second video frame is the video frame at the previous moment; using the optical flow to perform spatial domain transformation on the target video frame at the current moment to obtain a transformed image frame; using the weight matrix to calculate the weighted distance between the transformed image frame and the second video frame; and using the weighted distance to adjust the transformed image frame.

[0154] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: receiving a currently input source image and a driving video, wherein the source image is used to describe the appearance information of the target object, and the driving video is used to describe the action information of the target object; sending the source image and the driving video to a server; receiving a target video from the server, wherein the target video is generated by the server based on the source image and the driving video; and playing the target video locally on the client.

[0155] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: receiving a source image and a driving video from a client, wherein the source image is used to describe the appearance information of a target object, and the driving video is used to describe the action information of the target object; generating a target video based on the source image and the driving video; returning the target video to the client, and playing the target video locally on the client.

[0156] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0157] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program code.

[0162] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video generation method, characterized in that: include: Acquire a source image and a driving video, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object; Performing multi-scale motion estimation frame by frame based on the source image and the driving frames included in the driving video to obtain target motion flow data; Inputting the source image into at least one encoding layer and outputting multi-scale image features; Determine a plurality of target video frames using the multi-scale image features and a data stream in the target motion stream data corresponding to each scale image feature in the multi-scale image features; The multiple target video frames are synthesized into the target video.

2. The video generation method according to claim 1, characterized in that: The target motion flow data is obtained by performing multi-scale motion estimation frame by frame based on the source image and the driving frames included in the driving video, including: Performing multi-scale motion estimation frame by frame based on the source image and the driving frame corresponding to each moment at multiple different moments of the driving video to obtain initial motion flow data; The initial motion flow data is up-sampled to obtain the target motion flow data.

3. The video generation method according to claim 1, characterized in that: Determining the plurality of target video frames using the multi-scale image features and a data stream corresponding to each scale image feature in the multi-scale image features in the target motion stream data comprises: Performing spatial domain transformation on the data stream corresponding to each scale image feature in the multi-scale image features in the target motion flow data, and outputting the transformation result; The multi-scale image features and the transformation results are input into at least one reconstruction layer, and the multiple target video frames are output.

4. The video generation method according to claim 3, characterized in that: The method further comprises: Multi-scale motion estimation is performed frame by frame based on the source image and the driving frames contained in the driving video to obtain a weight matrix, wherein the weight matrix is ​​used to calculate the temporal consistency constraint loss during the training process of the deep neural network.

5. The video generation method according to claim 4, characterized in that: The method further comprises: Inputting adjacent first and second video frames in the video to be trained into a pre-trained optical flow estimation model, and outputting an optical flow from the second video frame to the first video frame, wherein the video to be trained is the same as the target video, the first video frame is a video frame at a current moment, and the second video frame is a video frame at a previous moment; Using the optical flow to perform spatial domain transformation on the target video frame at the current moment to obtain a transformed image frame; Calculating a weighted distance between the transformed image frame and the second video frame using the weight matrix; The transformed image frame is adjusted using the weighted distance.

6. A video generation method, characterized in that: include: Receiving a currently input source image and a driving video, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object; Sending the source image and the driving video to a server; Receive a target video from the server, wherein the target video is synthesized by a plurality of target video frames, the plurality of target video frames are determined by using a multi-scale image feature and a data stream corresponding to each scale image feature in the multi-scale image feature in target motion stream data, the multi-scale image feature is obtained by inputting the source image into at least one coding layer output, and the target stream data is obtained by performing multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video; The target video is played locally on the client.

7. A video generation method, characterized in that: include: Receiving a source image and a driving video from a client, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object; Performing multi-scale motion estimation frame by frame based on the source image and the driving frames included in the driving video to obtain target motion flow data; Inputting the source image into at least one encoding layer and outputting multi-scale image features; Determine a plurality of target video frames using the multi-scale image features and a data stream in the target motion stream data corresponding to each scale image feature in the multi-scale image features; Combining the multiple target video frames into the target video; The target video is returned to the client, and the target video is played locally on the client.

8. A video generation method, characterized in that: include: Acquire a source image and a driving video, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object; Inputting the source image and the driving video into a first granularity network model to obtain target motion flow data, wherein the first granularity network model is used to perform multi-scale motion estimation frame by frame based on the source image and the driving frames contained in the driving video; The source image and the target motion flow data are input into a second granularity network model to obtain a target video, wherein the second granularity network model includes: at least one encoding layer and at least one reconstruction layer, the at least one encoding layer is used to obtain multi-scale image features of the source image, and the at least one reconstruction layer is used to obtain multiple target video frames using the target motion flow data and the data stream corresponding to each scale image feature in the multi-scale image features.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the video generation method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Acquire a source image and a driving video, wherein the source image is used to describe appearance information of a target object, and the driving video is used to describe action information of the target object; Performing multi-scale motion estimation frame by frame based on the source image and the driving frames included in the driving video to obtain target motion flow data; Inputting the source image into at least one encoding layer and outputting multi-scale image features; Determine a plurality of target video frames using the multi-scale image features and a data stream in the target motion stream data corresponding to each scale image feature in the multi-scale image features; The multiple target video frames are synthesized into the target video.

Images