Video generation method and device

Abstract

The present application discloses a video generation method and device. A specific implementation of the method includes: obtaining a target image and a target continuous posture sequence; inputting the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence; inputting the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, thereby improving the coherence of the video to be processed through the coherence model on the basis of each video frame of the high-quality video to be processed generated by the video generation model, obtaining a high-quality and relatively coherent video to be processed, and improving the quality and coherence of the generated video.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video generation method and device. Background Art

[0002] Portrait video synthesis is an important topic in the field of computer vision. It can be used as a data enhancement method for some video analysis tasks and is used in many scenarios, such as film production and interactive applications.

[0003] The current portrait video synthesis technology is mainly divided into two types: the first is to synthesize a video that maintains the appearance of the portrait based on a single portrait image and additional conditions, wherein the video needs to obtain the corresponding action according to the additional conditions, and the additional conditions can be the action label of the person, the continuous posture information of the person, etc.; the second is to synthesize a video of the portrait with the same action based on a portrait video and additional conditions, wherein the video needs to replace the attributes of the person according to the additional conditions, and the additional conditions can be another portrait image, a top image, etc. Summary of the invention

[0004] The embodiments of the present application provide a video generation method and device.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, comprising: acquiring a target image and a target continuous posture sequence; inputting the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that an object included in the target image presents posture information represented by the target continuous posture sequence, and the video generation model is used to represent the correspondence between the target image, the target continuous posture sequence and the video to be processed; inputting the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, wherein the coherence model is used to represent the correspondence between the video to be processed and the coherent video.

[0006] In some embodiments, the above-mentioned input of the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed includes: splitting the target continuous posture sequence into multiple single-frame posture information; for each single-frame posture information in the multiple single-frame posture information, generating a single-frame image matching the single-frame posture information based on the target image; and synthesizing the video to be processed including each single-frame image according to the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence.

[0007] In some embodiments, the above-mentioned inputting the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video includes: inputting the video to be processed into the coherence model, and obtaining each video frame in the coherent video in the following manner: based on a preset number of video frames before a target video frame in the video to be processed, the target video frame, and a preset number of video frames before the video frame in the coherent video, determining optical flow information between a previous video frame in the coherent video and the video frame and a predicted video frame of the video frame, wherein the target video frame in the video to be processed corresponds to the video frame in the coherent video; and obtaining the video frame according to the predicted image and the optical flow information.

[0008] In some embodiments, the video generation model and the coherence model are trained in the following manner: obtaining a training sample set, wherein the training samples in the training sample set include: sample images, sample continuous gesture sequences, and sample videos; obtaining an initial video model, wherein the initial video model includes a generation network and a discriminant network, the generation network includes an initial video generation model and an initial coherence model, and is used to generate videos using sample images and sample continuous gesture sequences, and the discriminant network is used to distinguish between videos generated by the generation network and sample videos; using a machine learning method, taking sample images and sample continuous gesture sequences in the training samples as inputs to the generation network, taking videos generated by the generation network and sample videos in the training samples as inputs to the discriminant network, training the initial video model, determining the trained initial video generation model as the video generation model, and determining the trained initial coherence model as the coherence model.

[0009] In some embodiments, the discriminant network includes a video frame discriminant network and a video discriminant network, the video frame discriminant network is used to distinguish between video frames of a video generated by a generation network and video frames of a sample video, and the video discriminant network is used to distinguish between a video generated by a generation network and a sample video.

[0010] In a second aspect, an embodiment of the present application provides a video generating device, comprising: an acquiring unit, configured to acquire a target image and a target continuous posture sequence; a generating unit, configured to input the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents posture information represented by the target continuous posture sequence, and the video generation model is used to represent the correspondence between the target image, the target continuous posture sequence and the video to be processed; a obtaining unit, configured to input the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, wherein the coherence model is used to represent the correspondence between the video to be processed and the coherent video.

[0011] In some embodiments, the generation unit is further configured to: split the target continuous posture sequence into multiple single-frame posture information; for each single-frame posture information in the multiple single-frame posture information, generate a single-frame image matching the single-frame posture information based on the target image; and synthesize a video to be processed including each single-frame image according to the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence.

[0012] In some embodiments, the obtaining unit is further configured to: input the video to be processed into a coherence model, and obtain each video frame in the coherent video in the following manner: based on a preset number of video frames before the target video frame in the video to be processed, the target video frame, and a preset number of video frames before the video frame in the coherent video, determine the optical flow information between the previous video frame in the coherent video and the video frame, and the predicted video frame of the video frame, wherein the target video frame in the video to be processed corresponds to the video frame in the coherent video; and obtain the video frame according to the predicted image and the optical flow information.

[0013] In some embodiments, the video generation model and the coherence model are trained in the following manner: obtaining a training sample set, wherein the training samples in the training sample set include: sample images, sample continuous gesture sequences, and sample videos; obtaining an initial video model, wherein the initial video model includes a generation network and a discriminant network, the generation network includes an initial video generation model and an initial coherence model, and is used to generate videos using sample images and sample continuous gesture sequences, and the discriminant network is used to distinguish between videos generated by the generation network and sample videos; using a machine learning method, taking sample images and sample continuous gesture sequences in the training samples as inputs to the generation network, taking videos generated by the generation network and sample videos in the training samples as inputs to the discriminant network, training the initial video model, determining the trained initial video generation model as the video generation model, and determining the trained initial coherence model as the coherence model.

[0014] In some embodiments, the discriminant network includes a video frame discriminant network and a video discriminant network, the video frame discriminant network is used to distinguish between video frames of a video generated by a generation network and video frames of a sample video, and the video discriminant network is used to distinguish between a video generated by a generation network and a sample video.

[0015] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0017] The video generation method and device provided in the embodiment of the present application obtain a target image and a target continuous posture sequence; input the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence; input the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, thereby improving the coherence of the video to be processed through the coherence model on the basis of each video frame of the high-quality video to be processed generated by the video generation model, obtaining a high-quality and well-coherent video to be processed, thereby improving the quality and coherence of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0020] Figure 2 is a flow chart of an embodiment of a video generation method according to the present application;

[0021] Figure 3 is a schematic diagram of an application scenario of the video generation method according to this embodiment;

[0022] Figure 4 is a flowchart of another embodiment of a video generation method according to the present application;

[0023] Figure 5 is a structural diagram of an embodiment of a video generating device according to the present application;

[0024] Figure 6 It is a structural diagram of a computer system suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0025] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] Figure 1 An exemplary architecture 100 to which the video generation method and apparatus of the present application can be applied is shown.

[0028] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0029] Terminal devices 101, 102, 103 may be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, 103 are hardware, they may be various electronic devices that support network connection, information interaction, display, processing and other functions, including but not limited to smart phones, tablet computers, e-book readers, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules for providing distributed services, or they may be implemented as a single software or software module. No specific limitation is made here.

[0030] Server 105 can be a server that provides various services, such as a background processing server for a coherent video based on the target image and target continuous posture sequence sent by terminal devices 101, 102, and 103. The background processing server can input the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence; the video to be processed is input into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video. Optionally, the background processing server can also feed back the coherent video to the terminal device for display by the terminal device. As an example, server 105 can be a cloud server.

[0031] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0032] It should also be noted that the video generation method provided in the embodiments of the present disclosure can be executed by a server, or by a terminal device, or by a server and a terminal device in cooperation with each other. Accordingly, the various parts (such as various units, sub-units, modules, and sub-modules) included in the video generation device can be all arranged in the server, or all arranged in the terminal device, or respectively arranged in the server and the terminal device.

[0033] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system architecture is only illustrative. Depending on the implementation requirements, there may be any number of terminal devices, networks and servers. When the electronic device on which the video generation method is running does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (e.g., server or terminal device) on which the video generation method is running.

[0034] Continue to refer Figure 2 , a process 200 of an embodiment of a video generation method is shown, comprising the following steps:

[0035] Step 201, obtaining a target image and a target continuous posture sequence.

[0036] In this embodiment, the execution subject of the video generation method (for example Figure 1 The terminal device or server in the system can obtain the target image and the target continuous posture sequence remotely or locally through a wired connection or a wireless connection.

[0037] The object included in the target image may be various objects, including but not limited to human objects, animal objects, cartoon objects, etc. The target continuous posture sequence is used to represent the motion posture information to be presented by the object included in the target image.

[0038] As an example, the object included in the target image is a human object, and the target continuous posture sequence represents the motion posture information of dancing. When a video is generated based on the target image and the target continuous posture sequence, in the generated video, the human object presents the motion posture information of dancing represented by the target continuous posture sequence.

[0039] In this embodiment, the target continuous gesture sequence can be a gesture sequence extracted from the target video. As an example, for each video frame in the target video, the execution subject extracts the gesture information in the video frame. For all gesture information extracted from the target video, the execution subject generates a target continuous gesture sequence including all gesture information according to the playback order of the video frames of the target video.

[0040] Step 202: input the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed.

[0041] In this embodiment, the above-mentioned execution subject can input the target image and the target continuous posture sequence obtained in step 201 into a pre-trained video generation model to generate a video to be processed. The video to be processed represents the posture information represented by the target continuous posture sequence presented by the object included in the target image, and the video generation model is used to represent the corresponding relationship between the target image, the target continuous posture sequence and the video to be processed.

[0042] The above video generation model can be trained as follows:

[0043] First, a training sample set is obtained, wherein the training samples in the training sample set include: sample images, sample continuous posture sequences and sample videos.

[0044] Then, a machine learning algorithm is used to input sample images and sample continuous posture sequences into the initial video generation model, and the sample images in the input training samples are used as the expected output to train a video output model.

[0045] In some optional implementations of this embodiment, the execution subject may perform step 202 in the following manner:

[0046] First, the target continuous posture sequence is split into multiple single-frame posture information.

[0047] As an example, each split single-frame pose information may correspond to a video frame.

[0048] Then, for each single-frame posture information in the plurality of single-frame posture information, a single-frame image matching the single-frame posture information is generated based on the target image.

[0049] In a single-frame image, the included object presents the action represented by the single-frame posture information.

[0050] Finally, according to the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence, a video to be processed including each single-frame image is synthesized.

[0051] For each single-frame image, the object therein presents an action in the single-frame posture information. All the single-frame images are arranged in the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence to synthesize the video to be processed including each single-frame image.

[0052] It can be understood that the video to be processed is synthesized from single-frame images and may have continuity issues, and its continuity needs to be improved according to subsequent steps.

[0053] Step 203: input the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video.

[0054] In this embodiment, the execution subject may input the to-be-processed video obtained in step 202 into a pre-trained coherence model to improve the coherence of the to-be-processed video and obtain a coherent video. The coherence model is used to characterize the corresponding relationship between the to-be-processed video and the coherent video.

[0055] The coherence model may be any network model that improves video coherence, such as a Video-to-Video Synthesis model.

[0056] In some optional implementations of this embodiment, the execution subject may perform step 203 in the following manner:

[0057] The video to be processed is input into the coherence model, and each video frame in the coherent video is obtained in the following way:

[0058] First, based on a preset number of video frames before a target video frame in a to-be-processed video, a target video frame, and a preset number of video frames before the video frame in a coherent video, optical flow information between a previous video frame and the video frame in the coherent video and a predicted video frame of the video frame are determined, wherein the target video frame in the to-be-processed video corresponds to the video frame in the coherent video.

[0059] The preset number can be set according to actual conditions. For example, the preset number can be set to 2.

[0060] Then, the video frame is obtained according to the predicted image and optical flow information.

[0061] Specifically, the execution subject can obtain the optical flow information between the previous video frame and the current video frame, the predicted video frame and weight mask information of the current video frame through the coherence model, and obtain the current video frame through the following formula:

[0062] x t =(1-m t )⊙w t-1 (x t-1 )+m t ⊙h t

[0063] Among them, x t Characterize the frame video, m t Represents weight mask information, w t-1 Represents the optical flow information between the previous video frame and this video frame, x t-1 Represents the previous video frame of the video frame in the coherent video, ht Represent the predicted image.

[0064] In some optional implementations of this embodiment, the video generation model and the coherence model are trained together. Specifically, the video generation model and the coherence model are trained in the following manner:

[0065] First, a training sample set is obtained, wherein the training samples in the training sample set include: sample images, sample continuous posture sequences and sample videos.

[0066] Then, an initial video model is obtained, wherein the initial video model includes a generation network and a discriminant network. The generation network includes an initial video generation model and an initial coherence model, which is used to generate videos using sample images and sample continuous posture sequences. The discriminant network is used to distinguish between videos generated by the generation network and sample videos.

[0067] Finally, using machine learning methods, the sample images and sample continuous posture sequences in the training samples are used as the input of the generation network, and the videos generated by the generation network and the sample videos in the training samples are used as the input of the discriminative network to train the initial video model. The trained initial video generation model is determined as the video generation model, and the trained initial coherence model is determined as the coherence model.

[0068] In some optional implementations, the discriminant network includes a video frame discriminant network and a video discriminant network, the video frame discriminant network is used to distinguish between video frames of the video generated by the generation network and video frames of the sample video, and the video discriminant network is used to distinguish between the video generated by the generation network and the sample video. Based on the dual discrimination of video frames and videos, the coherence of the coherent video obtained by the coherence model can be further improved.

[0069] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the video generation method according to this embodiment. Figure 3 In the application scenario of , during the process of video processing by user 301, a target image 303 and a target continuous gesture sequence 304 are selected in the terminal device 302, and the selected target image 303 and the target continuous gesture sequence 304 are fed back to the server 305 through the terminal device 302. The server 305 inputs the target image 303 and the target continuous gesture sequence 304 into a pre-trained video generation model 306 to generate a video to be processed 307. Among them, the video to be processed 307 represents the gesture information represented by the target continuous gesture sequence 304 presented by the object included in the target image 303. Then, the video to be processed 307 is input into the pre-trained coherence model 308 to improve the coherence of the video to be processed, obtain a coherent video 309, and feed the coherent video 309 back to the terminal device 302.

[0070] The method provided by the above-mentioned embodiments of the present disclosure obtains a target image and a target continuous posture sequence; inputs the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence; inputs the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed to obtain a coherent video, thereby improving the coherence of the video to be processed through the coherence model on the basis of each video frame of the high-quality video to be processed generated by the video generation model, obtaining a high-quality and well-coherent video to be processed, thereby improving the quality and coherence of the generated video.

[0071] Continue to refer Figure 4 , shows a schematic process 400 of another embodiment of a video generation method according to the present application, comprising the following steps:

[0072] Step 401: Acquire a target image and a target continuous posture sequence.

[0073] Step 402: split the target continuous posture sequence into multiple single-frame posture information.

[0074] Step 403 : for each single frame of posture information in the plurality of single frame of posture information, a single frame image matching the single frame of posture information is generated based on the target image.

[0075] Step 404 , synthesizing a to-be-processed video including each single-frame image according to the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence.

[0076] Step 405: Input the video to be processed into the coherence model, and obtain each video frame in the coherent video in the following manner:

[0077] Step 4051, based on a preset number of video frames before the target video frame in the to-be-processed video, the target video frame, and a preset number of video frames before the video frame in the continuous video, determine the optical flow information between the previous video frame in the continuous video and the video frame, and the predicted video frame of the video frame, wherein the target video frame in the to-be-processed video corresponds to the video frame in the continuous video.

[0078] Step 4052, obtaining the video frame according to the predicted image and the optical flow information.

[0079] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiment, the process 400 of the video generation method in this embodiment specifically describes the generation process of the video to be processed and the process of improving the coherence of the video to be processed. In this way, the quality and coherence of the generated video are further improved in this embodiment.

[0080] Continue to refer Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a video generation device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0081] like Figure 5 As shown, the video generation device includes: including: an acquisition unit 501 is configured to acquire a target image and a target continuous posture sequence; a generation unit 502 is configured to input the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence, and the video generation model is used to characterize the correspondence between the target image, the target continuous posture sequence and the video to be processed; a obtaining unit 503 is configured to input the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, wherein the coherence model is used to characterize the correspondence between the video to be processed and the coherent video.

[0082] In some embodiments, the generation unit 502 is further configured to: split the target continuous posture sequence into multiple single-frame posture information; for each single-frame posture information in the multiple single-frame posture information, generate a single-frame image matching the single-frame posture information based on the target image; and synthesize a video to be processed including each single-frame image according to the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence.

[0083] In some embodiments, the obtaining unit 503 is further configured to: input the video to be processed into a coherence model, and obtain each video frame in the coherent video in the following manner: based on a preset number of video frames before the target video frame in the video to be processed, the target video frame, and a preset number of video frames before the video frame in the coherent video, determine the optical flow information between the previous video frame in the coherent video and the video frame, and the predicted video frame of the video frame, wherein the target video frame in the video to be processed corresponds to the video frame in the coherent video; and obtain the video frame according to the predicted image and the optical flow information.

[0084] In some embodiments, the video generation model and the coherence model are trained in the following manner: obtaining a training sample set, wherein the training samples in the training sample set include: sample images, sample continuous gesture sequences, and sample videos; obtaining an initial video model, wherein the initial video model includes a generation network and a discriminant network, the generation network includes an initial video generation model and an initial coherence model, and is used to generate videos using sample images and sample continuous gesture sequences, and the discriminant network is used to distinguish between videos generated by the generation network and sample videos; using a machine learning method, taking sample images and sample continuous gesture sequences in the training samples as inputs to the generation network, taking videos generated by the generation network and sample videos in the training samples as inputs to the discriminant network, training the initial video model, determining the trained initial video generation model as the video generation model, and determining the trained initial coherence model as the coherence model.

[0085] In some embodiments, the discriminant network includes a video frame discriminant network and a video discriminant network, the video frame discriminant network is used to distinguish between video frames of a video generated by a generation network and video frames of a sample video, and the video discriminant network is used to distinguish between a video generated by a generation network and a sample video.

[0086] In this embodiment, the acquisition unit in the video generation device acquires the target image and the target continuous posture sequence; the generation unit inputs the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence; the acquisition unit inputs the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, thereby improving the coherence of the video to be processed through the coherence model on the basis of each high-quality video frame of the video to be processed generated by the video generation model, obtaining a high-quality and well-coherent video to be processed, thereby improving the quality and coherence of the generated video.

[0087] Reference below Figure 6 , which shows a device suitable for implementing the embodiments of the present application (eg Figure 1 Schematic diagram of the structure of a computer system 600 of the devices 101, 102, 103, 105 shown. Figure 6 The device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0088] like Figure 6As shown, the computer system 600 includes a processor (e.g., CPU, central processing unit) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the system 600 are also stored in the RAM 603. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0089] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0090] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from a removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the method of the present application are executed.

[0091] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0092] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the client computer, partially on the client computer, as a separate software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the client computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0093] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0094] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor comprising an acquisition unit, a generation unit and a obtaining unit. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the obtaining unit may also be described as a unit of "inputting the video to be processed into a pre-trained coherence model, improving the coherence of the video to be processed, and obtaining a coherent video".

[0095] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the computer device: obtains a target image and a target continuous posture sequence; inputs the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence, and the video generation model is used to represent the corresponding relationship between the target image, the target continuous posture sequence and the video to be processed; inputs the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, wherein the coherence model is used to represent the corresponding relationship between the video to be processed and the coherent video.

[0096] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A video generation method, comprising: Acquire target images and target continuous posture sequences; Inputting the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents posture information represented by the target continuous posture sequence, and the video generation model is used to represent the corresponding relationship between the target image, the target continuous posture sequence and the video to be processed; The video to be processed is input into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, wherein the coherence model is used to characterize the corresponding relationship between the video to be processed and the coherent video.

2. The method according to claim 1, wherein: The step of inputting the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed includes: Splitting the target continuous posture sequence into multiple single-frame posture information; For each single-frame posture information in the plurality of single-frame posture information, generating a single-frame image matching the single-frame posture information based on the target image; The video to be processed including each single-frame image is synthesized according to the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence.

3. The method according to claim 1, wherein: The step of inputting the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed to obtain a coherent video includes: The video to be processed is input into the coherence model, and each video frame in the coherent video is obtained in the following manner: Determining optical flow information between a previous video frame in the continuous video and the video frame and a predicted video frame of the video frame based on a preset number of video frames before a target video frame in the video to be processed, the target video frame, and a preset number of video frames before the video frame in the continuous video, wherein the target video frame in the video to be processed corresponds to the video frame in the continuous video; The video frame is obtained according to the predicted image and the optical flow information.

4. The method according to claim 1, wherein: The video generation model and the coherence model are trained in the following manner: Acquire a training sample set, wherein the training samples in the training sample set include: sample images, sample continuous posture sequences and sample videos; Acquire an initial video model, wherein the initial video model includes a generation network and a discriminant network, the generation network includes an initial video generation model and an initial coherence model, and is used to generate a video using a sample image and a sample continuous posture sequence, and the discriminant network is used to distinguish between the video generated by the generation network and the sample video; By using a machine learning method, sample images and sample continuous posture sequences in training samples are used as inputs of a generative network, and videos generated by the generative network and sample videos in training samples are used as inputs of a discriminative network. The initial video generation model after training is determined as the video generation model, and the initial coherence model after training is determined as the coherence model.

5. The method according to claim 4, wherein: The discriminant network includes a video frame discriminant network and a video discriminant network. The video frame discriminant network is used to distinguish the video frames of the video generated by the generation network and the video frames of the sample video. The video discriminant network is used to distinguish the video generated by the generation network and the sample video.

6. A video generating device, comprising: An acquisition unit is configured to acquire a target image and a target continuous posture sequence; A generating unit is configured to input the target image and the target continuous posture sequence into a pre-trained video generation model to generate a video to be processed, wherein the video to be processed represents that the object included in the target image presents the posture information represented by the target continuous posture sequence, and the video generation model is used to represent the corresponding relationship between the target image, the target continuous posture sequence and the video to be processed; The obtaining unit is configured to input the video to be processed into a pre-trained coherence model to improve the coherence of the video to be processed and obtain a coherent video, wherein the coherence model is used to characterize the corresponding relationship between the video to be processed and the coherent video.

7. The device according to claim 6, wherein: The generating unit is further configured to: The target continuous posture sequence is split into multiple single-frame posture information; for each single-frame posture information in the multiple single-frame posture information, a single-frame image matching the single-frame posture information is generated based on the target image; according to the order of the single-frame posture information corresponding to each single-frame image in the target continuous posture sequence, the video to be processed including each single-frame image is synthesized.

8. The device according to claim 6, wherein: The obtaining unit is further configured to: The video to be processed is input into the coherence model, and each video frame in the coherent video is obtained in the following manner: based on a preset number of video frames before a target video frame in the video to be processed, the target video frame, and a preset number of video frames before the video frame in the coherent video, optical flow information between a previous video frame in the coherent video and the video frame and a predicted video frame of the video frame are determined, wherein the target video frame in the video to be processed corresponds to the video frame in the coherent video; and the video frame is obtained according to the predicted image and the optical flow information.

9. The device according to claim 6, wherein: The video generation model and the coherence model are trained in the following manner: A training sample set is obtained, wherein the training samples in the training sample set include: sample images, sample continuous posture sequences and sample videos; an initial video model is obtained, wherein the initial video model includes a generative network and a discriminative network, the generative network includes an initial video generative model and an initial coherence model, and is used to generate videos using sample images and sample continuous posture sequences, and the discriminative network is used to distinguish between videos generated by the generative network and sample videos; a machine learning method is used to train the initial video model by taking the sample images and sample continuous posture sequences in the training samples as inputs of the generative network, and taking the videos generated by the generative network and the sample videos in the training samples as inputs of the discriminative network, and determining the trained initial video generative model as the video generative model, and determining the trained initial coherence model as the coherence model.

10. The device according to claim 9, wherein: The discriminant network includes a video frame discriminant network and a video discriminant network. The video frame discriminant network is used to distinguish the video frames of the video generated by the generation network and the video frames of the sample video. The video discriminant network is used to distinguish the video generated by the generation network and the sample video.

11. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

12. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

Images