2d digital human action video generation method and device, cluster and storage medium
Patent Information
- Application Number
- CN202311099873.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-05
- Filing Date
- 2023-08-29
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, it takes a lot of time and manpower to generate 2D digital human action videos, and requires professional skills and experience, making it difficult to efficiently generate high-quality action videos.
By obtaining driver voice and image files, using voice driver body model to generate matching 2D digital human action videos, and using the reference action library to edit edited action clips, automatically adjust the action specifications, and improve generation efficiency.
It realizes automatic generation and editing of 2D digital human action videos, improves generation efficiency, reduces the time and professional skills requirements for manual adjustment, and the generated video actions are more standardized and smooth.
Smart Images

Figure CN119091014A_ABST
Abstract
Description
[0001] This disclosure claims priority to Chinese patent application No. 202310658027.9 filed on June 5, 2023, with invention name “Action editing method, device, cluster and storage medium”, the entire contents of which are incorporated by reference in this disclosure. Technical Field
[0002] The present disclosure relates to the technical field of 2D digital human, and in particular to a method, device, equipment and storage medium for generating a 2D digital human action video. Background Art
[0003] With the development of two-dimensional (2D) digital human technology, 2D digital human action videos are used in more and more scenes, such as speech scenes and news scenes.
[0004] In the related art, 2D digital human motion videos are generated through manual editing technology. Specifically, the joints and postures of the 2D digital human are manually adjusted to achieve the modification and optimization of the 2D digital human motion. Although manual editing can adopt an intuitive method to enable users to finely control the 2D digital human motion and achieve high-quality 2D digital human motion, manual editing requires a lot of time and manpower, and requires professional skills and experience. Summary of the invention
[0005] The present invention provides a method for generating a 2D digital human action video, which can automatically generate a 2D digital human action video, and the actions in the 2D digital human action video are standardized, thereby improving the generation efficiency of high-quality 2D digital human action videos.
[0006] In a first aspect, the present disclosure provides a method for generating a 2D digital human action video, the method comprising: obtaining a driving voice and an image file, wherein the image file comprises a 2D real-life photo of a first image or a 2D real-life action video of the first image, generating a 2D digital human action video matching the driving voice based on the driving voice and the image file, obtaining an action segment to be edited of the 2D digital human action video, editing the action segment to be edited in the 2D digital human action video using a reference action library of the first image, and obtaining the edited 2D digital human action video.
[0007] In the solution disclosed in the present invention, a 2D digital human action video matching the driving voice is automatically generated through the driving voice and the image file of the first image. In addition, the action clip to be edited in the 2D digital human action video can be obtained for re-editing, so that the action in the edited 2D digital human action video is standardized. Therefore, a 2D digital human action video with standardized action can be automatically obtained, thereby improving the generation efficiency of high-quality 2D digital human action videos.
[0008] In an optional manner, the two-dimensional 2D digital human action video matching the driving voice is generated based on the driving voice and the image file, including: inputting the driving voice and the image file of the first image into a voice-driven limb model to generate a 2D digital human action video matching the driving voice, wherein the voice-driven limb model is a neural network model, and the voice-driven limb model is obtained by training based on multiple action videos with voice. In this way, the 2D digital human action video can be generated using the voice-driven limb model.
[0009] In an optional manner, after obtaining the edited 2D digital human motion video, the edited 2D digital human motion video is used to optimize the voice-driven limb model, and the voice-driven limb model can be continuously updated, so that the 2D digital human motion video generated by the voice-driven limb model becomes more and more standardized.
[0010] In an optional manner, the reference action library of the first image is used to edit the action segment to be edited in the 2D digital human action video to obtain the edited 2D digital human action video, including: determining an action of the same action type as the action segment to be edited in the reference action library of the first image, and replacing the action segment to be edited in the 2D digital human action video with the determined action segment to obtain the edited 2D digital human action video. In this way, using the standardized actions in the reference action library to replace the action segment to be edited can make the actions in the 2D digital human action video standardized.
[0011] In an optional manner, when obtaining the action segment to be edited of the 2D digital human action video, an action annotation interface is provided for the user, wherein the action annotation interface is used for the user to annotate the action segment to be edited of the 2D digital human action video, and obtain the action type of the action segment to be edited and the time period in the 2D digital human action video fed back by the user through the action annotation interface. In this way, the user can annotate the action segment to be edited.
[0012] In an optional manner, when obtaining the reference work library of the first image, the reference action library of the second image is borrowed, and based on the reference action library of the second image and the image file of the first image, a reference action library of the first image is generated, wherein the actions in the reference action library of the first image are the same as the actions in the reference action library of the second image. In this way, the reference action library can be obtained without recording an action video for the first image according to the scene requirements, simplifying the implementation process.
[0013] In a second aspect, the present disclosure provides a device for generating a 2D digital human action video, the device comprising at least one module, and the at least one module is used to implement the method for generating a 2D digital human action video provided by the above-mentioned first aspect or any optional method in the first aspect.
[0014] In some embodiments, the modules in the device for generating 2D digital human motion video are implemented by software, and the modules in the device for generating 2D digital human motion video are program modules. In other embodiments, the modules in the device for generating 2D digital human motion video are implemented by hardware or firmware.
[0015] In a third aspect, the present disclosure provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method for generating 2D digital human action video provided by the above-mentioned first aspect or any optional method of the first aspect.
[0016] In a fourth aspect, the present disclosure provides a computer-readable storage medium, which includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method for generating a 2D digital human action video provided by the above-mentioned first aspect or any optional method in the first aspect.
[0017] In a fifth aspect, the present disclosure provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the method for generating a 2D digital human action video provided in the first aspect or any optional method in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present disclosure;
[0019] Figure 2 is a schematic diagram of a system architecture provided by an exemplary embodiment of the present disclosure;
[0020] Figure 3 is a flow chart of a method for generating a 2D digital human action video provided by an exemplary embodiment of the present disclosure;
[0021] Figure 4 is a flow chart of a method for generating a 2D digital human action video provided by an exemplary embodiment of the present disclosure;
[0022] Figure 5 is a flowchart of obtaining a reference action library of a first image provided by an exemplary embodiment of the present disclosure;
[0023] Figure 6 is a flow chart of a method for generating a 2D digital human action video provided by an exemplary embodiment of the present disclosure;
[0024] Figure 7 is a flow chart of a method for generating a 2D digital human action video provided by an exemplary embodiment of the present disclosure;
[0025] Figure 8 It is a structural schematic diagram of a device for generating a 2D digital human action video provided by an exemplary embodiment of the present disclosure;
[0026] Fig. 9 is a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure;
[0027] Fig.10 is a schematic diagram of the structure of a computing device cluster provided by an exemplary embodiment of the present disclosure;
[0028] Fig.11 It is a connection diagram of a computing device cluster provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0029] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.
[0030] Some terminology concepts involved in the embodiments of the present disclosure are explained below.
[0031] 1. 2D digital human: is a 2D image generated by computer. 2D digital human has human appearance, with specific features such as appearance, gender and personality, and has human behavior, the ability to express with language, facial expressions and body movements, and has human thoughts, the ability to recognize the external environment, and the ability to communicate and interact with people. 2D digital human represents a virtual character, also known as 2D virtual digital human. For 2D digital human, from the perspective of presentation, users can only browse 2D digital human from a single perspective. The perspective has been determined when it was made, and users cannot change the perspective by themselves.
[0032] 2. Deep learning algorithm: It is a branch of machine learning. It is an algorithm that uses artificial neural networks as the architecture to represent and learn data.
[0033] 3. 2D digital human action library: It is a database of multiple virtual character actions, which can be called and applied to the virtual character when needed to obtain a 2D digital human. The action library usually includes multiple actions, such as walking, running and gestures.
[0034] 4. Digital Human Motion Editing: Modify and edit the digital human motion, including but not limited to motion splicing, adjustment, editing and other operations, to achieve specific motion performance effects. Digital Human Motion Editing can make the digital human motion more natural and smooth, meeting specific needs.
[0035] The following describes the background, execution subject, system architecture, method flow and device of the embodiments of the present disclosure in sequence. First, the background is described.
[0036] In 2D digital human technology, more and more scenes use 2D digital human action videos, and manual editing technology is currently used to generate 2D digital human action videos. Specifically, manual editing refers to manually adjusting the joints and postures of the digital human to achieve the modification and optimization of the digital human's actions. Although manual editing can use an intuitive method to enable users to finely control the 2D digital human's actions and achieve high-quality 2D digital human actions, manual editing requires a lot of time and manpower, and requires professional skills and experience.
[0037] Based on this, the present disclosure provides a method for generating a 2D digital human action video, which can automatically generate a 2D digital human action video and can also re-edit necessary action clips in the automatically generated 2D digital human action video to obtain a high-quality 2D digital human action video.
[0038] The following describes the execution subject of the method for generating 2D digital human action video.
[0039] The execution subject of the method is a generating device. Optionally, the generating device is a hardware device, which is a computing device such as a server or a terminal device. Optionally, the generating device is a software device, such as a set of software programs running on the computing device.
[0040] The system architecture of the embodiment of the present disclosure is described below.
[0041] In an optional manner, the present disclosure embodiment provides a system architecture 100. Figure 1As shown, the system architecture 100 includes a terminal device 101 and a computing device 102. The terminal device 101 and the computing device 102 are connected via a wired or wireless network. The terminal device 101 is a device used by a user, such as a desktop computer, a laptop computer, a tablet computer or a mobile phone, etc. The terminal device 101 is used for the user to interact with the computing device 102. For example, the user can input driving voice and image files, etc. through the terminal device 101. The computing device 102 can be a server, etc., and the computing device 102 is used to generate a 2D digital human action video using the driving voice and image files, and edit the action segments to be edited in the 2D digital human action video. The computing device 102 sends the edited 2D digital human action video to the terminal device 101.
[0042] In another optional manner, the embodiment of the present disclosure provides another system architecture 200. Figure 2 As shown, the system architecture 200 includes a terminal device 101 and a public cloud 201. The terminal device 101 is connected to the public cloud 201 via a wired or wireless network. The terminal device 101 is a device used by the user, and the terminal device 101 is used for the user to interact with the public cloud 201. The public cloud 201 is an entity that uses basic resources to provide cloud services to users in a cloud computing model. The public cloud 201 can also be considered as a cloud environment. The public cloud 201 includes a cloud data center, which includes a large number of basic resources owned by a cloud service provider. The large number of basic resources include computing resources, storage resources, and network resources. The computing resources included in the cloud data center can be a computing device cluster, and the computing device cluster includes at least one computing device 102. The computing device 102 can be a server, etc. When a user uses the cloud service, the user can input the driving voice and image files to the public cloud 201 through an API or an interactive interface (the interactive interface can be a graphical user interface (GUI)). The computing device cluster in the public cloud 201 receives the driving voice and image files input by the user, generates a 2D digital human action video based on the driving voice and image files, and edits the action segments to be edited in the 2D digital human action video. The public cloud 201 sends the edited 2D digital human action video to the terminal device 101.
[0043] In the system architecture 200, the method for generating 2D digital human action videos can be abstracted into a cloud service by the cloud service provider in the public cloud 201 and provided to users. After the user obtains the permission to use the cloud service, the cloud service can be used to generate 2D digital human action videos. The cloud service is a 2D digital human action video generation service.
[0044] In another optional manner, the embodiment of the present disclosure further provides a system architecture. The system architecture includes a terminal device, which is a device used by a user. The terminal device executes the method for generating a 2D digital human action video in the embodiment of the present disclosure.
[0045] The following describes the process of the method for generating 2D digital human action video in the embodiment of the present disclosure.
[0046] Figure 3 The generation method of 2D digital human action video is provided, see Figure 1 Steps 301 to 306, Figure 3 In the previous text Figure 2 The system architecture shown in the figure is used as an example. Figure 3 In the process shown, the terminal device is the terminal device 101 mentioned above, and the public cloud is the public cloud 201 mentioned above.
[0047] Step 301, obtaining a driving voice and an image file, wherein the image file includes a 2D real-life photo of a first image or a 2D real-life action video of the first image.
[0048] In this embodiment, the first image is a digital human image. When generating a 2D digital human action video of the first image, the user operates the terminal device so that the terminal device displays an input interface. The user operates in the input interface to input a driving voice and an image file. The driving voice is the voice in the 2D digital human action video that is finally generated. For example, in a speech scene, the driving voice is the speech content of the speaker. The image file includes a 2D real-person photo of the first image or a 2D real-person action video of the first image. The 2D real-person photo is a full-body photo or a half-body photo, which matches the application scenario of the 2D digital human action video. For example, in a scene where the first image is standing and giving a speech, the 2D real-person photo is a full-body photo of the first image. In a scene where the first image is sitting and reporting news, the 2D real-person photo is a half-body photo of the first image. The 2D real-person action video is a video recorded with actions of the first image. The actions in the 2D real-person action video may match the application scenario or may not match the application scenario. For example, in a scene where the first image is standing and giving a speech, the actions may be hand actions, etc. The terminal device sends the driving voice and image file to the public cloud. The public cloud obtains the driving voice and image files.
[0049] It should be noted that the driving voice may be a voice recorded by a real person of the first image, or may be a voice not recorded by a real person of the first image. When the action in the 2D real person action video matches the application scenario, the finally generated 2D digital human action video may be closer to the first image.
[0050] Step 302: Based on the driving voice and the image file, generate a 2D digital human action video matching the driving voice.
[0051] Among them, in the 2D digital human action video, the body movements of the first image basically match the rhythm of the driving voice. For example, at a certain time point in the driving voice, the first image introduces an object, and the body movement of the first image is to extend the right hand forward.
[0052] In an optional manner, the public cloud obtains a voice-driven body model, which is a deep learning model, such as a convolutional neural network model. The public cloud inputs the driving voice and the image file of the first image into the voice-driven body model to obtain an output, which is a 2D digital human action video.
[0053] In the disclosed embodiment, the specific structure of the voice-driven limb model is not limited, and any model that can generate a 2D digital human action video based on driving voice and image files can be used as the voice-driven limb model.
[0054] Step 303: Obtain the action segment to be edited of the 2D digital human action video.
[0055] In this embodiment, after generating the 2D digital human action video, the public cloud provides the 2D digital human action video to the user. The public cloud marks the action clips to be edited in the 2D digital human action video. The action clips to be edited are action clips in the 2D digital human action video that need to be adjusted. For example, the action clips that need to be adjusted are action clips with irregular movements or unsmooth movements.
[0056] Alternatively, an action segment annotation algorithm is stored in the public cloud, and the public cloud inputs the 2D digital human action video into the action segment annotation algorithm to obtain the action segment to be edited.
[0057] In an optional manner, the process of the user marking the action segment to be edited is as follows:
[0058] The public cloud provides the user with an action annotation interface through the terminal device. The action annotation interface is used for the user to annotate the action clips to be edited in the 2D digital human action video. The terminal device displays the action annotation interface, which displays the 2D digital human action video and the action clip selection option. The user can select the non-standard action clips in the 2D digital human action video through the selection option. The non-standard action clips can be marked by the time period in the 2D digital human action video, and the time period includes the start time point and the end time point. And after selecting the action clip, it will trigger the display of the action type edit box, and the user can enter the action type of the action clip, such as the action type is waving. Or, after selecting the action clip, the public cloud identifies the action type of the action clip and displays the identified action type so that the user can confirm the action type. After the user confirms the action type or modifies the action type, the public cloud obtains the action type entered by the user and the time period of the selected action clip. The selected action clip is the action clip to be edited.
[0059] Exemplarily, the format of the clip to be edited obtained by the public cloud is {"17-19":"rhythm gesture","19-23":"waving"}, where "17-19" and "19-23" are the time periods of the action clip to be edited, and "rhythm gesture" and "waving" are action types.
[0060] Step 304: Use the reference action library of the first image to edit the action segment to be edited in the 2D digital human action video to obtain the edited 2D digital human action video.
[0061] In this embodiment, the public cloud obtains a reference action library of the first image, which is a collection of action videos of the first image. The action specifications in the reference action library are different for different scenes. For example, in a speech scene, the reference action library covers action videos of common actions in the speech scene, and in a recording scene, the reference action library covers action videos of common actions in the recording scene.
[0062] The public cloud uses the reference action library of the first image to edit the action clip to be edited so that the edited action clip meets the requirements. After editing the action clip to be edited, the edited 2D digital human action video is obtained. In the edited 2D digital human action video, the action is standardized and meets the requirements.
[0063] It should be noted that the reference action library of the first image can also be uploaded to the public cloud through the input interface mentioned above.
[0064] In an optional manner, the process of editing the action clip to be edited is as follows:
[0065] Each action in the reference action library of the first image is set with an action type. The public cloud determines the action of the same action type as the action segment to be edited in the reference action library of the first image. For action segment A in the action segment to be edited, action B in the reference action library has the same action type as action segment A. In the 2D digital human action video, the public cloud uses action B in the action reference library to replace action segment A. Since the starting posture and ending posture of action B and action segment A may be different, action B also needs to be adjusted so that action B can be inserted into the 2D digital human action video with smooth connection. In this way, after replacing each action segment in the action segment to be edited, the edited 2D digital human action video is obtained.
[0066] It should be noted that the duration of action B and action clip A may be different. If the duration of action B is greater than that of action clip A, the playback speed of action B is increased or video frames are deleted to make the duration of action B the same as that of action clip A. If the duration of action B is less than that of action clip A, the playback speed of action B is reduced or video frames are inserted to make the duration of action B the same as that of action clip A.
[0067] Exemplarily, a method for achieving smooth connection is: obtaining an action insertion model in an action rearrangement system, wherein the action insertion model can insert action B into a 2D digital human action video and enable smooth connection between action B and adjacent video frames in the 2D digital human action video.
[0068] Among them, the action insertion model is a deep learning algorithm model, and the training process is: obtain multiple samples and an initial action insertion model, each sample includes video data with some action clips deleted, clips of standard actions and deleted action clips, the standard action clips and the deleted action clips have the same action type and the same duration, input the clips of standard actions and video data in the first sample into the initial action insertion model, and obtain an output, which is the video data of inserting the standard action clip into the video data, compare the inserted action clip with the deleted action clip, determine the loss, and update the parameters of the initial action insertion model based on the loss. Input the clips of standard actions and video data in the second sample into the updated action insertion model, and obtain an output, which is the video data of inserting the standard action clip into the video data, compare the inserted action clip with the deleted action clip, determine the loss, and update the parameters of the updated action insertion model based on the loss. If the loss is greater than a certain value, continue to update the parameters and train until the loss is less than a certain value to obtain the action insertion model. Here, a single sample is used for each parameter update. In other optional methods, multiple samples can also be used for each parameter update.
[0069] Step 305: Use the edited 2D digital human action video to optimize the voice-driven body model.
[0070] In this embodiment, the loss of the edited 2D digital human action video and the 2D digital human action video obtained in step 302 is calculated, and based on the loss, the parameters of the voice-driven limb model are adjusted to optimize the voice-driven limb model, thereby realizing a closed data loop. The optimization process is the same as the process of training the voice-driven limb model, which will not be repeated here. In this way, each time the edited 2D digital human action video is obtained, the voice-driven limb model is optimized. After optimizing the voice-driven limb model for many times, the 2D digital human action video generated by the voice-driven limb model has relatively standardized movements, and the driving voice limb model can no longer be optimized. For example, when optimizing the voice-driven limb model for many times in a row, if the loss obtained many times is less than a certain value, the optimization of the voice-driven limb model can be stopped.
[0071] It should be noted that step 305 is an optional step. After stopping the optimization of the voice-driven body model, steps 303 to 305 may not be performed every time.
[0072] Step 306: output the edited 2D digital human action video.
[0073] In this embodiment, the public cloud feeds back the edited 2D digital human action video to the terminal device.
[0074] use Figure 3 The process shown can not only automatically obtain the 2D digital human action video, but also edit the necessary action clips in the generated 2D digital human action video based on the reference action library, so that the final 2D digital human action video has a better effect.
[0075] exist Figure 3 In the process shown, there are multiple ways to obtain the reference action library of the first image, and three optional ways are given below.
[0076] In an optional manner, when uploading the driving voice, uploading the reference action library of the first image through the input interface is equivalent to obtaining the reference action library of the first image in step 301, see Figure 4 .
[0077] In another optional way, when uploading the driving voice, an action video is uploaded through the input interface. The action video is an action video recorded by the first image according to the current application scenario, and the action video includes common actions in the application scenario. The public cloud captures multiple actions in the action video and displays the multiple actions to the user. The user can add an action type for each action, thereby building a reference action library for the first image. For example, the current application scenario is a speech scenario, and the recorded action video includes common actions in the speech scenario. For another example, the current application scenario is a news broadcast scenario, and the recorded action video includes common actions in the news broadcast scenario.
[0078] In yet another alternative, see Figure 5 In the process, step 501, a reference action library of a second image and an image file of a first image are obtained.
[0079] The second image and the first image are different digital human images. For example, the first image is a digital human image of real person A, and the second image is a digital human image of real person B.
[0080] In this embodiment, the public cloud determines the application scenario of the embodiment of the present disclosure, which may be input by the user, and the public cloud determines the reference action library of the second image corresponding to the application scenario, which includes common actions in the application scenario. Alternatively, the public cloud obtains the reference action library of the second image uploaded by the user.
[0081] Step 502: Generate a reference action library of the first image based on the reference action library of the second image and the image file of the first image, wherein the actions in the reference action library of the first image are the same as the actions in the reference action library of the second image.
[0082] In this embodiment, the public cloud obtains a 2D real-person action generation algorithm, inputs the reference action library of the second image and the 2D real-person photo of the first image into the 2D real-person action generation algorithm, and obtains multiple actions of the first image, which constitute the reference action library of the first image, and the actions in the reference action library of the first image are the same as the actions in the reference action library of the second image. Alternatively, the public cloud inputs the reference action library of the second image and a frame of an image containing a 2D real-person image in the 2D real-person action video of the first image into the 2D real-person action generation algorithm, and obtains multiple actions of the first image, which constitute the basic action library of the first image. The actions in the basic action library of the first image are the same as the actions in the reference action library of the second image. If the first image's exclusive action does not exist in the 2D real-person action video of the first image, the basic action library of the first image is the reference action library of the first image. If the first image's exclusive action exists in the 2D real-person action video of the first image, the exclusive action is added to the basic action library of the first image to obtain the reference action library of the first image. When determining whether there is a proprietary action of the first image in the 2D real-life action video of the first image, the public cloud provides a marking interface to the user. If the user selects a proprietary action in the 2D real-life action video through the marking interface, it is determined that there is a proprietary action of the first image in the 2D real-life action video; otherwise, it is determined that there is no proprietary action of the first image.
[0083] In addition, the starting and ending postures of each action in the reference action library are the same, which is convenient for inserting into the 2D digital human action video.
[0084] Optionally, when using the 2D real-life action video of the first image to generate the reference action library of the first image, the 2D real-life action video can be used to update the 2D real-life action generation algorithm, and then the updated 2D real-life action generation algorithm is used to generate the reference library of the first image. In this way, since the updated 2D real-life action generation algorithm has learned the image of the first image in the 2D real-life action video, the first image in the reference action library of the first image can be made more similar to the real first image.
[0085] The following describes the method flow of generating a 2D digital human action video by taking a 2D real person photo and a driving voice of a given first image as an example. Figure 6 Steps 601 to 606.
[0086] Step 601, obtaining a driving voice and a 2D real-person photo of a first image.
[0087] Step 602: Based on the driving voice and the 2D real-person photo of the first image, a 2D digital human action video matching the driving voice is generated.
[0088] In an optional manner, the public cloud obtains a voice-driven body model, which is a deep learning model, such as a convolutional neural network model. The public cloud inputs the driving voice and the 2D real person photo of the first image into the voice-driven body model to obtain an output, which is the 2D digital human action video of the first image.
[0089] The speech-driven limb model is obtained through training, and the training process is: obtain multiple video data and an initial speech-driven limb model, each action video is a video data with speech and action, and the speech and action match. For the first video data, the speech and video image of each video frame are input into the initial speech-driven limb model, and the video frame with speech and action corresponding to each video frame is obtained. The difference between the obtained video frame and the original video frame is calculated to obtain the loss corresponding to the first video data, and the parameters in the initial speech-driven limb model are updated based on the loss. Continue to update the parameters based on the second video data and the updated speech-driven limb model until the loss is less than a certain value when a certain video data is used, or the number of training times reaches a certain value, and the speech-driven limb model is obtained.
[0090] Step 603: Obtain the action segment to be edited of the 2D digital human action video.
[0091] Step 604: Use the reference action library of the first image to edit the action segment to be edited in the 2D digital human action video to obtain the edited 2D digital human action video.
[0092] Step 605: Use the edited 2D digital human action video to optimize the voice-driven body model.
[0093] Step 606: output the edited 2D digital human action video.
[0094] The following describes the method flow of generating 2D digital human action video by taking a given 2D real-person action video and driving voice of the first image as an example. Figure 7 Steps 701 to 706.
[0095] Step 701, obtaining a driving voice and a 2D real-life action video of a first image.
[0096] Step 702: Based on the driving voice and the 2D real-person action video of the first image, generate a 2D digital human action video matching the driving voice.
[0097] In an optional manner, the public cloud obtains a voice-driven body model, which is a deep learning model, such as a convolutional neural network model. In addition, the public cloud obtains a video frame including a 2D real person of the first image in a 2D real person action video of the first image. The public cloud inputs the driving voice and the video frame including the 2D real person of the first image into the voice-driven body model to obtain an output, which is the 2D digital human action video of the first image.
[0098] The voice-driven body language model is obtained through training. The training process can be found in Figure 6 The description in the process shown will not be repeated here.
[0099] Step 703: Obtain the action segment to be edited of the 2D digital human action video.
[0100] Exemplarily, the format of the clip to be edited obtained by the public cloud is {"7-13":"rhythm gesture","13-20":"waving hand",,,}, where "7-13" and "13-20" are the time periods of the action clip to be edited, and "rhythm gesture" and "waving hand" are action types.
[0101] Step 704: Use the reference action library of the first image to edit the action segment to be edited in the 2D digital human action video to obtain the edited 2D digital human action video.
[0102] Step 705: Use the edited 2D digital human action video to optimize the voice-driven body model.
[0103] Step 706: output the edited 2D digital human action video.
[0104] Figure 6 and Figure 7 For the processing of each step in the process shown, see Figure 3 The process shown will not be repeated here.
[0105] In the disclosed embodiment, by editing the action clips to be edited, the 2D digital human action video generated by voice drive can be adjusted to achieve the effect improvement and personalized optimization of the voice-driven digital human action, and finally obtain the 2D hyper-realistic digital human action video, solving the problem that the voice-driven 2D digital human action cannot be adjusted. In addition, the edited 2D digital human action video is used to continuously optimize the voice-driven limb model, increase the correctness and certainty of the action in important actions, realize data closed loop, and continuously optimize the voice-driven limb model as the user uses it.
[0106] The device provided by the embodiment of the present disclosure is described below. Figure 8The structure diagram of the device for generating 2D digital human action video provided by the embodiment of the present disclosure is shown in FIG. The device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided by the embodiment of the present disclosure can implement the embodiment of the present disclosure. Figures 3 to 7 The process shown in the figure, the device includes: an acquisition module 810, a generation module 820 and an editing module 830, wherein:
[0107] The acquisition module 810 is used to acquire the driving voice and image files, wherein the image files include a 2D real-life photo of the first image or a 2D real-life action video of the first image, which can be specifically used to implement the acquisition function of step 301 and execute the implicit steps included in step 301;
[0108] A generation module 820 is used to generate a 2D digital human action video matching the driving voice based on the driving voice and the image file, and can be specifically used to implement the generation function of step 302 and execute the implicit steps included in step 302;
[0109] The acquisition module 810 is also used to acquire the action segment to be edited of the 2D digital human action video, and can be specifically used to implement the acquisition function of step 303 and execute the implicit steps included in step 303;
[0110] The editing module 830 is used to use the reference action library of the first image to edit the action segment to be edited in the 2D digital human action video to obtain the edited 2D digital human action video, which can be specifically used to implement the editing function of step 304 and execute the implicit steps included in step 304.
[0111] In an optional manner, the generating module 810 is used to:
[0112] The driving voice and the image file of the first image are input into a voice-driven limb model to generate a 2D digital human action video matching the driving voice, wherein the voice-driven limb model is a neural network model, and the voice-driven limb model is obtained based on training of multiple action videos with voice.
[0113] In an optional manner, the device further includes:
[0114] The updating module is used to optimize the voice-driven limb model by using the edited 2D digital human action video after obtaining the edited 2D digital human action video.
[0115] In an optional manner, the editing module 830 is used to:
[0116] Determine, in the reference action library of the first image, an action of the same type as that of the action segment to be edited;
[0117] In the 2D digital human action video, the determined action segment is used to replace the action segment to be edited, so as to obtain the edited 2D digital human action video.
[0118] In an optional manner, the acquisition module 810 is further used to:
[0119] Providing an action annotation interface, wherein the action annotation interface is used for the user to annotate the action segment to be edited of the 2D digital human action video;
[0120] The action type of the action segment to be edited and the time period in the 2D digital human action video fed back by the user through the action annotation interface are obtained.
[0121] In an optional manner, the acquisition module 810 is further used to:
[0122] Obtaining a reference action library of the second image;
[0123] A reference action library of the first image is generated based on the reference action library of the second image and the image file of the first image, wherein the actions in the reference action library of the first image are the same as the actions in the reference action library of the second image.
[0124] Among them, the acquisition module 810, the generation module 820 and the editing module 830 can all be implemented by software, or can be implemented by hardware. Exemplarily, the generation module 820 is taken as an example to introduce the implementation of the generation module 820. Similarly, the implementation of the acquisition module 810 and the editing module 830 can refer to the implementation of the generation module 820.
[0125] As an example of a software functional unit, the generation module 820 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the generation module 820 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.
[0126] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0127] As an example of a hardware functional unit, the generation module 820 may include at least one computing device, such as a server, etc. Alternatively, the generation module 820 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0128] The multiple computing devices included in the generation module 820 can be distributed in the same region or in different regions. The multiple computing devices included in the generation module 820 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the generation module 820 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0129] It should be noted that, in other embodiments, the generation module 820 can be used to execute any step in the method for generating a 2D digital human action video, the acquisition module 810 can be used to execute any step in the method for generating a 2D digital human action video, and the editing module 830 can be used to execute any step in the method for generating a 2D digital human action video.
[0130] The present disclosure also provides a computing device 900. Fig. 9As shown, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the present disclosure does not limit the number of processors and memories in the computing device 900.
[0131] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 902 may include a path for transmitting information between various components of the computing device 900 (eg, the memory 906, the processor 904, and the communication interface 908).
[0132] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0133] The memory 906 may include a volatile memory, such as a random access memory (RAM). The memory 906 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0134] The memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module 810, generation module 820, and editing module 830, thereby implementing the method for generating a 2D digital human action video. That is, the memory 906 stores instructions for executing the method for generating a 2D digital human action video.
[0135] The present disclosure also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.
[0136] like Fig.10 As shown, the computing device cluster includes at least one computing device 900. The memory 906 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the method for generating 2D digital human action video.
[0137] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the method for generating 2D digital human action videos. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing the method for generating 2D digital human action videos.
[0138] It should be noted that the memory 906 in different computing devices 900 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the device for generating 2D digital human action videos. That is, the instructions stored in the memory 906 in different computing devices 900 can implement the functions of one or more modules in the acquisition module 810, the generation module 820 and the editing module 830.
[0139] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11 As shown, the two computing devices include a first computing device 900A and a second computing device 900B, and the first computing device 900A and the second computing device 900B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 906 in the first computing device 900A stores instructions for executing the functions of the acquisition module 810. At the same time, the memory 906 in the second computing device 900B stores instructions for executing the functions of the generation module 820 and the editing module 830.
[0140] Fig.11 The connection mode between the computing device clusters shown may be that considering that the method for generating 2D digital human action video provided by the present disclosure needs to interact with the user, the function of the acquisition module 810 is considered to be executed by the first computing device 900A.
[0141] It should be understood that Fig.11 The functions of the first computing device 900A shown in FIG. 9A may also be completed by multiple computing devices 900. Similarly, the functions of the second computing device 900B may also be completed by multiple computing devices 900.
[0142] The present disclosure also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.10 and Fig.11 The connection mode of the computing device cluster is different in that the memory 906 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the method for generating 2D digital human action video.
[0143] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the method for generating 2D digital human action videos. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing the method for generating 2D digital human action videos.
[0144] The present disclosure also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a method for generating a 2D digital human action video.
[0145] The present disclosure also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute a method for generating a 2D digital human action video.
[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A method for generating a two-dimensional 2D digital human action video, It is characterized in that The method comprises: Acquire a driving voice and an image file, wherein the image file includes a 2D real-life photo of the first image or a 2D real-life action video of the first image; Based on the driving voice and the image file, generating a 2D digital human action video matching the driving voice; Obtaining the action segment to be edited of the 2D digital human action video; Using the reference action library of the first image, the action segment to be edited in the 2D digital human action video is edited to obtain the edited 2D digital human action video.
2. The method according to claim 1, It is characterized in that The step of generating a 2D digital human action video matching the driving voice based on the driving voice and the image file comprises: The driving voice and the image file are input into a voice-driven limb model to generate a 2D digital human action video matching the driving voice, wherein the voice-driven limb model is a neural network model, and the voice-driven limb model is obtained based on training of multiple action videos with voice.
3. The method according to claim 2, It is characterized in that After obtaining the edited 2D digital human action video, the method further includes: The edited 2D digital human action video is used to optimize the voice-driven limb model.
4. The method according to any one of claims 1 to 3, It is characterized in that The step of using the reference action library of the first image to edit the action segment to be edited in the 2D digital human action video to obtain the edited 2D digital human action video comprises: Determine, in the reference action library of the first image, an action of the same type as that of the action segment to be edited; In the 2D digital human action video, the determined action segment is used to replace the action segment to be edited, so as to obtain the edited 2D digital human action video.
5. The method according to any one of claims 1 to 4, It is characterized in that The step of obtaining the action segment to be edited of the 2D digital human action video comprises: Providing an action annotation interface, wherein the action annotation interface is used for the user to annotate the action segment to be edited of the 2D digital human action video; The action type of the action segment to be edited and the time period in the 2D digital human action video fed back by the user through the action annotation interface are obtained.
6. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: Obtaining a reference action library of the second image; A reference action library of the first image is generated based on the reference action library of the second image and the image file of the first image, wherein the actions in the reference action library of the first image are the same as the actions in the reference action library of the second image.
7. A device for generating 2D digital human action video, It is characterized in that The device comprises: An acquisition module, used to acquire a driving voice and an image file, wherein the image file includes a 2D real-life photo of the first image or a 2D real-life action video of the first image; A generating module, for generating a 2D digital human action video matching the driving voice based on the driving voice and the image file; The acquisition module is further used to acquire the action segment to be edited of the 2D digital human action video; The editing module is used to edit the action segment to be edited in the 2D digital human action video using the reference action library of the first image to obtain the edited 2D digital human action video.
8. The device according to claim 7, It is characterized in that The generating module is used for: The driving voice and the image file are input into a voice-driven limb model to generate a 2D digital human action video matching the driving voice, wherein the voice-driven limb model is a neural network model, and the voice-driven limb model is obtained based on training of multiple action videos with voice.
9. The device according to claim 8, It is characterized in that The device also includes: The updating module is used to optimize the voice-driven limb model by using the edited 2D digital human action video after obtaining the edited 2D digital human action video.
10. The device according to any one of claims 7 to 9, It is characterized in that The editing module is used to: Determine, in the reference action library of the first image, an action of the same type as that of the action segment to be edited; In the 2D digital human action video, the determined action segment is used to replace the action segment to be edited, so as to obtain the edited 2D digital human action video.
11. The device according to any one of claims 7 to 10, It is characterized in that The acquisition module is further used for: Providing an action annotation interface, wherein the action annotation interface is used for the user to annotate the action segment to be edited of the 2D digital human action video; The action type of the action segment to be edited and the time period in the 2D digital human action video fed back by the user through the action annotation interface are obtained.
12. The device according to any one of claims 7 to 11, It is characterized in that The acquisition module is further used for: Obtaining a reference action library of the second image; A reference action library of the first image is generated based on the reference action library of the second image and the image file of the first image, wherein the actions in the reference action library of the first image are the same as the actions in the reference action library of the second image.
13. A computing device cluster, It is characterized in that comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.
14. A computer-readable storage medium, It is characterized in that The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.
