A digital human video generation method, system, device and medium

Abstract

Embodiments of the present application provide a digital person video generation method, system, device and medium, which are applied to a digital person video generation system. A video editor acquires video content configuration information and sends the video content configuration information to a video factory. The video factory converts the video content configuration information into video parameters, generates a rendering task according to the video parameters, and sends a video generation request containing the video parameters to a distributed rendering engine according to a remote procedure call framework. The distributed rendering engine acquires the rendering task from a rendering task queue and processes the rendering task according to the video generation request to generate a corresponding digital person video. Embodiments of the present application generate high-quality rendering tasks according to video parameters by the video factory, quickly transfer and process video generation requests according to the remote procedure call framework, and perform parallel processing on the rendering task by the distributed rendering engine, thereby realizing efficient, high-quality and real-time digital person video generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital humans, in particular to a digital human video generation method, a digital human video generation system, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of metaverse and artificial intelligence technology, virtual digital human videos have gradually become an important application form in various industries and are widely used in information reporting, advertising, virtual live streaming and other scenarios. However, although the existing digital human video generation and processing methods have made some progress, there are still many limitations, which are difficult to meet the growing market demand.

[0003] The existing technology generates video content through text description and then uses a video model to render a digital human video. However, the process of generating a digital human video is time-consuming and has poor real-time performance, and there are problems such as difficulty in guaranteeing video quality and low efficiency, which makes it difficult for users to obtain high-quality video output in a short time. SUMMARY

[0004] The embodiments of the present application provide a digital human video generation method, system, device and medium to solve or partially solve the problems of low efficiency and poor quality of existing digital human video generation.

[0005] The embodiments of the present application disclose a digital human video generation method applied to a digital human video generation system, wherein the system at least includes a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory, and the video factory includes a rendering task queue. The method comprises the following steps:

[0006] Obtaining video content configuration information through the video editor and sending the video content configuration information to the video factory;

[0007] Receiving the video content configuration information through the video factory, converting the video content configuration information into video parameters, generating a rendering task according to the video parameters, sending the rendering task to the rendering task queue, and sending a video generation request containing the video parameters to the distributed rendering engine according to a pre-set remote procedure call framework;

[0008] Obtaining a rendering task from the rendering task queue through the distributed rendering engine, receiving the video generation request through the remote procedure call framework, processing the rendering task according to the video generation request, and generating a digital human video corresponding to the rendering task.

[0009] In some possible implementation manners, the distributed rendering engine comprises a plurality of rendering sub-nodes, and before the video factory sends the rendering task to the rendering task queue, the method further comprises:

[0010] The video factory monitors the load status of the rendering sub-nodes in real time, obtains an idle rendering sub-node, and generates a video task number as an identifier of the rendering task according to the idle rendering sub-node;

[0011] The rendering task is obtained from the rendering task queue by the distributed rendering engine, comprising:

[0012] The rendering task corresponding to the rendering sub-node is obtained from the rendering queue by the rendering sub-node according to the video task number.

[0013] In some possible implementation manners, the rendering task queue comprises a waiting queue and a working queue, and the rendering task is sent to the rendering task queue by the video factory, comprising:

[0014] The rendering task is sent to the waiting queue by the video factory, and the rendering task is transferred to the working queue according to preset task allocation information;

[0015] The rendering task is obtained from the rendering task queue by the distributed rendering engine, comprising:

[0016] The rendering task is obtained from the working queue by the distributed rendering engine.

[0017] In some possible implementation manners, the video factory comprises a video asset database, the video asset database comprises a plurality of video asset tables, the video asset tables are used to save video production data, and the method further comprises:

[0018] The video parameter is saved to the video asset database by the video factory, and the video parameter is associated with the video asset table;

[0019] The video generation request containing the video parameter is sent to the distributed rendering engine according to a preset remote procedure call framework by the video factory, comprising:

[0020] The video production data corresponding to the video parameter is obtained from the video asset database according to the video parameter by the video factory, and the video generation request containing the video parameter and the video production data is sent to the distributed rendering engine;

[0021] The processing of the rendering task by the distributed rendering engine according to the video generation request comprises:

[0022] The processing of the rendering task by the distributed rendering engine according to the video parameter and the video production data comprises generating the digital human video.

[0023] In some possible implementation manners, the distributed rendering engine further comprises an internal rendering module, and the processing of the rendering task according to the video parameter and the generation of the digital human video corresponding to the rendering task comprise:

[0024] The processing of the rendering task by the internal rendering module according to the video parameter comprises lip shape driving processing, text-to-speech processing, and body driving and expression generation processing, and the digital human video is obtained.

[0025] In some possible implementation manners, the video factory further comprises a task detection component, and the method further comprises:

[0026] The processing state of the rendering task is detected by the task detection component in real time, the state information of the rendering task is obtained, and the state information is sent to the video editor;

[0027] The state information is received by the video editor, and the state information is visually displayed.

[0028] In some possible implementation manners, the system further comprises a distributed storage service, and the video factory further comprises a watermark component, and the method further comprises:

[0029] The digital human video is uploaded to the distributed storage service by the distributed rendering engine, the original storage link of the digital human video is obtained, and the original storage link of the digital human video is sent to the video factory;

[0030] The original storage link is received by the video factory, the digital human video is downloaded to the local memory of the video factory according to the original storage link, the watermark component is called to add a watermark to the digital human video, a target digital human video is obtained, the target digital human video is uploaded to the distributed storage service, and a target storage link of the target digital human video is obtained.

[0031] The embodiment of the application further discloses a digital human video generation system, the system at least comprising a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory, the video factory comprising a rendering task queue; wherein,

[0032] The video editor is configured to acquire video content configuration information and send the video content configuration information to the video factory.

[0033] The video factory is configured to receive the video content configuration information, convert the video content configuration information into video parameters, generate a rendering task according to the video parameters, send the rendering task to the rendering task queue, and send a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework.

[0034] The distributed rendering engine is configured to acquire a rendering task from the rendering task queue, receive the video generation request through the remote procedure call framework, and process the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task.

[0035] In some possible implementation manners, the distributed rendering engine includes a plurality of rendering sub-nodes.

[0036] The video factory is further configured to monitor the load status of the rendering sub-nodes in real time, obtain an idle rendering sub-node, and generate a video task number as an identifier of the rendering task according to the idle rendering sub-node.

[0037] The rendering sub-node is configured to acquire a rendering task corresponding to the rendering sub-node from the rendering queue according to the video task number.

[0038] In some possible implementation manners, the rendering task queue includes a waiting queue and a working queue.

[0039] The video factory is specifically configured to send the rendering task to the waiting queue and transfer the rendering task to the working queue according to preset task allocation information.

[0040] The distributed rendering engine is specifically configured to acquire the rendering task from the working queue.

[0041] In some possible implementation manners, the video factory includes a video asset database, and the video asset database includes a plurality of video asset tables configured to save video production data.

[0042] The video factory is specifically configured to save the video parameters in the video asset database, associate the video parameters with the video asset tables, acquire video production data corresponding to the video parameters from the video asset database according to the video parameters, and send a video generation request containing the video parameters and the video production data to the distributed rendering engine.

[0043] The distributed rendering engine is specifically configured to process the rendering task according to the video parameter and the video production data, and generate the digital human video.

[0044] In some possible implementation manners, the distributed rendering engine further includes an internal rendering module; wherein,

[0045] The internal rendering module is configured to perform lip shape driving processing, text-to-speech processing, and limb driving and expression generation processing on the rendering task according to the video parameter, and obtain the digital human video.

[0046] In some possible implementation manners, the video factory further includes a task detection component; wherein,

[0047] The task detection component is configured to detect a processing state of the rendering task in real time, obtain state information of the rendering task, and send the state information to the video editor.

[0048] The video editor is further configured to receive the state information and visually display the state information.

[0049] In some possible implementation manners, the system further includes a distributed storage service, and the video factory further includes a watermark component; wherein,

[0050] The distributed rendering engine is further configured to upload the digital human video to the distributed storage service, obtain an original storage link of the digital human video, and send the original storage link of the digital human video to the video factory.

[0051] The video factory is further configured to receive the original storage link, download the digital human video to a local memory of the video factory according to the original storage link, call the watermark component to add a watermark to the digital human video, obtain a target digital human video, and upload the target digital human video to the distributed storage service to obtain a target storage link of the target digital human video.

[0052] An electronic device is also disclosed, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0053] The memory is configured to store a computer program.

[0054] The processor is configured to execute the program stored on the memory, and implement the method according to the embodiments of the present application.

[0055] The embodiment of the present application also discloses a computer readable storage medium, which stores instructions, and when executed by one or more processors, causes the processors to perform the method according to the embodiment of the present application.

[0056] The embodiment of the present application has the following advantages:

[0057] The embodiment of the present application is applied to a digital human video generation system, and the digital human video generation system at least includes a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory. The video factory includes a rendering task queue. The video editor acquires video content configuration information and sends the video content configuration information to the video factory. The video factory converts the video content configuration information into video parameters, generates a rendering task according to the video parameters, and sends a video generation request containing the video parameters to the distributed rendering engine according to a remote procedure call framework. The distributed rendering engine acquires the rendering task from the rendering task queue and processes the rendering task to generate a corresponding digital human video according to the video generation request. The embodiment of the present application generates a high-quality rendering task according to the video parameters through the video factory, quickly transmits and processes the video generation request according to the remote procedure call framework, and processes the rendering task in parallel through the distributed rendering engine, thereby realizing efficient, high-quality and real-time digital human video generation. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a step flowchart of a digital human video generation method provided in the embodiment of the present application;

[0059] Figure 2 is a flowchart of a digital human video generation provided in the embodiment of the present application;

[0060] Figure 3 is a structural block diagram of a digital human video generation system provided in the embodiment of the present application;

[0061] Figure 4 is a block diagram of an electronic device provided in the embodiment of the present application;

[0062] Figure 5 is a schematic diagram of a computer readable storage medium provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the above objectives, features and advantages of the present application more apparent and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] As an example, the prior art generates video content through a text description, and then uses a video model to render to generate a digital human video, but the generation process from text to video requires a large amount of computing resources and time, and there are problems of long generation time of digital human video, poor real-time performance, and difficulty in guaranteeing the quality of output video. The prior art also involves using GPT (Generative Pre-trained Transformer) technology to process real-time information and generate text, but when applied to video generation, it still faces the above problems, making it difficult for users to obtain high-quality video output in a short time.

[0065] To this end, in the present application, a digital human video generation system is applied, which at least includes a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory. The video factory includes a rendering task queue. The video editor obtains video content configuration information and sends it to the video factory. The video factory converts the video content configuration information into video parameters and generates rendering tasks according to the video parameters. According to the remote procedure call framework, the video generation request containing the video parameters is sent to the distributed rendering engine. The distributed rendering engine obtains the rendering task from the rendering task queue and processes the rendering task according to the video generation request to generate the corresponding digital human video. The embodiment of the present application generates high-quality rendering tasks according to the video parameters through the video factory, quickly transmits and processes the video generation request according to the remote procedure call framework, and processes the rendering task in parallel through the distributed rendering engine, thereby realizing efficient, high-quality, and real-time digital human video generation.

[0066] Referring to Figure 1 , a step flowchart of a digital human video generation method provided in an embodiment of the present application is shown, which is applied to a digital human video generation system, and the digital human video generation system at least includes a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory. The video factory includes a rendering task queue. The method can specifically include the following steps:

[0067] Step 101, obtaining video content configuration information through the video editor, and sending the video content configuration information to the video factory;

[0068] For the digital human video generation system, at least a video editor, a video factory and a distributed rendering engine are included, a communication connection is established between the video editor and the video factory through an HTTP (Hypertext Transfer Protocol) protocol, and a communication connection is established between the video factory and the distributed rendering engine through an HTTP / 2 (Hypertext Transfer Protocol Version 2) protocol. Specifically, the video editor is used to obtain video content configuration information input by a user, the user configures video content through the video editor, for example, selects a digital human image, sets a scene, adjusts an action, etc.; the video factory is used to convert the video content configured by the user into an executable rendering task, and distributes the rendering task to the distributed rendering engine for processing according to a rendering task queue, that is, the video factory realizes management and scheduling of the rendering task through the rendering task queue, and the rendering task queue can be a Redis (Remote Dictionary Server) queue; the distributed rendering engine is used to utilize distributed computing resources to process the rendering task in parallel to generate a digital human video, and the digital human video generation system realizes automatic, efficient and high-quality digital human video generation through cooperative work of the video editor, the video factory and the distributed rendering engine.

[0069] In the process of digital human video generation, first, video content configuration information configured by a user is obtained through a video editor and sent to a video factory, the video content configuration information is used to define specific content and forms of generated digital human videos, and the video content configuration information can include information such as a digital human image, subtitles, scene settings, action sequences, special effect parameters and audio configurations.

[0070] Specifically, the video editor is configured with functions such as digital human image setting, text broadcast content input, subtitle style selection and track editor operation, a user can flexibly configure video content through the video editor, and then the video editor packs video content configuration information configured by the user and sends it to the video factory. The video editor enables the user to conveniently configure video content without the need to write complex codes or scripts, and the video editor can collect detailed configuration information of the user on the video content, provide necessary data support for the subsequent video generation process, and ensure that the generated video meets the needs of the user.

[0071] Step 102, receiving the video content configuration information through the video factory, converting the video content configuration information into video parameters, generating a rendering task according to the video parameters, sending the rendering task to the rendering task queue, and sending a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework.

[0072] In this embodiment of the invention, after receiving the video content configuration information sent by the video editor, the video factory converts the video content configuration information into video parameters used internally by the system, such as digital human image, scene settings, action sequences, special effects parameters, audio configuration, etc., and further converts the video parameters into rendering task parameters. A rendering task is created based on the rendering task parameters, and the video factory sends the generated rendering task to the rendering task queue. The rendering task queue is a first-in-first-out (FIFO) data structure used to store and manage rendering tasks to be processed. The video factory ensures that rendering tasks can be processed by the distributed rendering engine in sequence through the rendering task queue, avoiding task loss or confusion. The video factory also sends a video generation request containing video parameters to the distributed rendering engine through a preset remote procedure call framework. The remote procedure call framework is pre-built based on the HTTP / 2 protocol between the video factory and the distributed rendering engine. In this embodiment of the invention, the remote procedure call framework uses the gRPC framework, an open-source, high-performance remote procedure call (RPC, Google Remote Procedure Call) framework. The gRPC framework is responsible for efficient and reliable communication between the video factory and the distributed rendering engine, enabling rapid transmission of video generation requests and related video parameters. This invention does not impose specific limitations on this aspect.

[0073] In some embodiments, the distributed rendering engine includes a plurality of rendering sub-nodes. Before step 102, which sends the rendering task to the rendering task queue through the video factory, the method further includes: monitoring the load status of the rendering sub-nodes in real time through the video factory, obtaining the rendering sub-nodes in an idle state, and generating a video task number based on the rendering sub-nodes in an idle state as an identifier for the rendering task; obtaining the rendering task from the rendering task queue through the distributed rendering engine includes: obtaining the rendering task corresponding to the rendering sub-node from the rendering queue through the rendering sub-node based on the video task number.

[0074] In this embodiment of the invention, the distributed rendering engine consists of multiple rendering sub-nodes. The rendering tasks are processed in parallel by the multiple rendering sub-nodes. The video factory also involves real-time monitoring of the load of each rendering sub-node to understand the current working status of each rendering sub-node, determining whether the rendering sub-node is idle, and selecting the rendering sub-node in an idle state as the execution node for generating rendering tasks.

[0075] Specifically, the video factory identifies the idle rendering sub-node according to the monitored load of the rendering sub-node, and then generates a unique video task number for the rendering task according to the idle rendering sub-node, the video task number is associated with the idle rendering sub-node, the video task number is used as an identifier of the rendering task, and then the rendering sub-node obtains the corresponding rendering task from the rendering task queue according to the video task number allocated to itself, so that the rendering task can be efficiently allocated to the appropriate node, and the rendering efficiency is improved.

[0076] In some embodiments, the rendering task queue includes a waiting queue and a working queue, and the sending of the rendering task to the rendering task queue by the video factory includes: sending the rendering task to the waiting queue by the video factory, and transferring the rendering task to the working queue according to preset task allocation information; and the obtaining of the rendering task from the rendering task queue by the distributed rendering engine includes: obtaining the rendering task from the working queue by the distributed rendering engine.

[0077] In the embodiment of the application, the rendering task queue is divided into two sub-queues, i.e., a waiting queue and a working queue, the waiting queue is used to store rendering tasks that have not been allocated to rendering sub-nodes, and the working queue is used to store rendering tasks that have been allocated to rendering sub-nodes.

[0078] Specifically, the video factory allocates the rendering task in the waiting queue to the corresponding rendering sub-node and transfers it to the working queue according to the preset task allocation information, the task allocation information can include the idle state of the rendering sub-node, the task priority, the task type, etc., and then the rendering sub-node obtains the rendering task allocated to itself from the working queue. According to the division of the waiting queue and the working queue, the rendering task can be better managed and dispatched, the rendering task can be processed in order, and according to the preset task allocation information, the system can efficiently allocate the rendering task to the appropriate rendering sub-node, thereby improving the rendering efficiency.

[0079] In some embodiments, the video factory includes a video asset database, the video asset database includes a plurality of video asset tables, the video asset tables are used to save video production data, and the method further includes:

[0080] The video factory saves the video parameters to the video asset database and associates the video parameters with the video asset table;

[0081] The video factory sends a video generation request containing the video parameters to the distributed rendering engine according to the preset remote procedure call framework, including:

[0082] The video factory retrieves video production data corresponding to the video parameters from the video asset database based on the video parameters, and sends a video generation request containing the video parameters and the video production data to the distributed rendering engine.

[0083] The step of processing the rendering task according to the video generation request through the distributed rendering engine to generate the digital human video corresponding to the rendering task includes:

[0084] The distributed rendering engine processes the rendering task based on the video parameters and the video production data to generate the digital human video.

[0085] In this embodiment of the invention, the video factory also includes a video asset database for storing and managing various data required during the video production process. The video asset database consists of multiple video asset tables, each of which stores specific types of video production data, such as a digital human 3D image asset table, a digital human video record table, a digital human video and image association table, and a digital human video basic asset table. Through multiple video asset tables, the system can classify, store, and manage different types of video production data, facilitating subsequent querying and use.

[0086] Specifically, the video factory saves the generated video parameters to a video asset database and associates these parameters with corresponding video asset tables. For example, digital human image parameters are associated with a digital human image table, scene setting parameters are associated with a scene table, and so on. Based on the generated video parameters, the video factory can retrieve the corresponding video production data from the video asset database. Then, the video factory sends a video generation request containing the video parameters and corresponding video production data to a distributed rendering engine. The distributed rendering engine processes the rendering task based on the received video parameters and video production data to generate the digital human video. In this embodiment of the invention, the video factory can effectively manage and store various data required during the video production process through the video asset database, ensuring that all data required during video generation can be accurately obtained.

[0087] Step 103: Obtain a rendering task from the rendering task queue through the distributed rendering engine, receive the video generation request through the remote procedure call framework, and process the rendering task according to the video generation request to generate the digital human video corresponding to the rendering task.

[0088] In the embodiment of the present application, the distributed rendering engine periodically obtains a rendering task to be processed from the rendering task queue, receives a video generation request corresponding to the rendering task through the gRPC remote procedure call framework, and then the distributed rendering engine performs specific video rendering operations such as loading a digital human image, setting subtitles, rendering action sequences, synthesizing audio, etc. according to the received video generation request and the rendering task to generate a digital human video. The distributed rendering engine of the embodiment of the present application utilizes multiple computer computing resources to process rendering tasks in parallel, thereby improving the generation efficiency of digital human videos.

[0089] In some embodiments, the distributed rendering engine further comprises an internal rendering module, and the processing of the rendering task according to the video parameters to generate the digital human video corresponding to the rendering task comprises: performing lip driving processing, text-to-speech processing, and body driving and expression generation processing on the rendering task according to the video parameters through the internal rendering module to obtain the digital human video.

[0090] In the embodiment of the present application, the distributed rendering engine further comprises a plurality of internal rendering modules, and the internal rendering modules are used to perform video rendering operations required by the rendering task to ensure that the generated video content is not only efficient but also has high natural expressiveness. The internal rendering modules at least include a lip driving submodule, a TTS (text-to-speech) submodule, a body driving submodule, and an expression generation submodule, etc. By calling the internal rendering modules to perform video rendering operations such as lip driving, text-to-speech, body driving, and expression generation, a digital human video is obtained. The distributed rendering engine of the embodiment of the present application can be flexibly extended through the internal rendering modules, supports more rendering functions and effects, meets the changing rendering requirements, and improves the quality of the generated digital human video.

[0091] In some embodiments, the video factory further comprises a task detection component, which detects the processing state of the rendering task in real time, obtains state information of the rendering task such as the progress of the task or error information, etc., and sends the state information to the video editor for visual display, so as to inform the user about the processing state of the rendering task through the video editor, thereby ensuring that the user can learn about the progress of the rendering task in a timely manner.

[0092] In some embodiments, the system further comprises a distributed storage service, the video factory further comprises a watermark component, and the method further comprises: uploading, by the distributed rendering engine, the digital human video to the distributed storage service, obtaining an original storage link of the digital human video, and sending the original storage link of the digital human video to the video factory; receiving, by the video factory, the original storage link, downloading the digital human video into a local memory of the video factory according to the original storage link, calling the watermark component to add a watermark to the digital human video, obtaining a target digital human video, and uploading the target digital human video to the distributed storage service to obtain a target storage link of the target digital human video.

[0093] In the embodiment of the present application, the digital human video generation system further comprises a distributed storage service, which respectively establishes a communication connection with the video factory and the distributed rendering engine through the HTTP protocol. Optionally, the distributed storage service can adopt a Ceph (Ceph Distributed Storage System) distributed storage system. The video factory further comprises a watermark component, which is used to add pre-set watermark information to the digital human video to ensure the copyright and source information of the video. Optionally, the watermark component can adopt an FFMPEG (Fast Forward Moving Picture Experts Group) component, and the present application does not limit this.

[0094] In the specific implementation process, after the distributed rendering engine generates the digital human video, the digital human video is uploaded to the distributed storage service to obtain an original storage link of the digital human video, and then the original storage link is sent to the video factory. After receiving the original storage link, the video factory accesses the distributed storage service and downloads the digital human video into a local memory according to the original storage link. Further, the video factory calls the watermark component to add watermark information to the digital human video to generate a digital human video with watermark information as a target digital human video, and uploads the target digital human video to the distributed storage service to obtain a target storage link corresponding to the target digital human video. Further, the video factory stores the original video link and the target video link into the video asset database for saving. In the embodiment of the present application, the digital human video generation system can efficiently store and manage the digital human video through the distributed storage service to ensure the safety and reliability of the video file, and can add a watermark to the digital human video through the watermark component to ensure that the copyright and source information of the video are protected, and can store the video link persistently through the video asset database to facilitate the subsequent query and use of video production.

[0095] In order for those skilled in the art to better understand the technical solutions in the embodiments of the present application, the following will be exemplarily described through corresponding examples:

[0096] Referring to Figure 2 , a flowchart of a digital human video generation provided in the embodiments of the present application is shown. In the process of digital human video generation, the user first configures video content configuration information such as digital human image, text and subtitle through a video editor, and then the video editor sends the video content configuration information input by the user to a video factory, the video factory creates a rendering task according to the video content configuration information and puts the rendering task into a waiting queue of a rendering task queue for waiting for allocation, the video factory detects the distributed rendering engine in real time to obtain a rendering engine sub-node in an idle state, and then allocates the rendering task to the rendering engine sub-node in the idle state and transfers it to a working queue of the rendering task queue, at the same time, the video factory sends a video generation request to the distributed rendering engine through a gRPC framework, further the distributed rendering engine obtains the rendering task from the working queue and allocates it to the corresponding rendering engine sub-node in the idle state, the rendering engine sub-node receives the rendering task and performs video rendering to obtain a digital human video. In addition, the video factory also involves detecting the processing state of the rendering task in real time, and sending the processing state information of the rendering task to the video editor to inform the user of the processing state of the rendering task through the video editor.

[0097] It should be noted that for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0098] It should be noted that the embodiments of the present application include but are not limited to the above examples, and it can be understood that those skilled in the art can also set according to actual needs under the guidance of the idea of the embodiments of the present application, and the present application does not limit this.

[0099] In the embodiment of the present application, the digital human video generation system at least includes a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory. The video factory includes a rendering task queue. The video editor obtains video content configuration information and sends it to the video factory. The video factory converts the video content configuration information into video parameters and generates rendering tasks according to the video parameters. The video factory sends a video generation request containing the video parameters to the distributed rendering engine according to a remote procedure call framework. The distributed rendering engine obtains the rendering tasks from the rendering task queue and processes the rendering tasks according to the video generation request to generate corresponding digital human videos. In the embodiment of the present application, the video factory generates high-quality rendering tasks according to the video parameters, quickly transmits and processes the video generation request according to the remote procedure call framework, and processes the rendering tasks in parallel through the distributed rendering engine, thereby realizing efficient, high-quality, and real-time digital human video generation.

[0100] Referring to Figure 3 , a structural block diagram of a digital human video generation system provided in the embodiment of the present application is shown. The system at least includes a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory. The video factory includes a rendering task queue. Wherein,

[0101] The video editor is configured to obtain video content configuration information and send the video content configuration information to the video factory.

[0102] The video factory is configured to receive the video content configuration information, convert the video content configuration information into video parameters, generate rendering tasks according to the video parameters, send the rendering tasks to the rendering task queue, and send a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework.

[0103] The distributed rendering engine is configured to obtain rendering tasks from the rendering task queue, receive the video generation request through the remote procedure call framework, and process the rendering tasks according to the video generation request to generate digital human videos corresponding to the rendering tasks.

[0104] In some possible implementation manners, the distributed rendering engine includes a plurality of rendering sub-nodes. Wherein,

[0105] The video factory is further configured to monitor the load conditions of the rendering sub-nodes in real time, obtain idle rendering sub-nodes, and generate a video task number as an identifier of the rendering task according to the idle rendering sub-nodes.

[0106] The rendering sub-node is configured to obtain a rendering task corresponding to the rendering sub-node from the rendering queue according to the video task number.

[0107] In some possible implementation manners, the rendering task queue comprises a waiting queue and a working queue; wherein,

[0108] The video factory is specifically configured to send the rendering task to the waiting queue, and transfer the rendering task to the working queue according to preset task allocation information.

[0109] The distributed rendering engine is specifically configured to obtain the rendering task from the working queue.

[0110] In some possible implementation manners, the video factory comprises a video asset database, and the video asset database comprises a plurality of video asset tables configured to save video production data; wherein,

[0111] The video factory is specifically configured to save the video parameter in the video asset database, and associate the video parameter with the video asset table; obtain video production data corresponding to the video parameter from the video asset database according to the video parameter, and send a video generation request comprising the video parameter and the video production data to the distributed rendering engine.

[0112] The distributed rendering engine is specifically configured to process the rendering task according to the video parameter and the video production data, and generate the digital human video.

[0113] In some possible implementation manners, the distributed rendering engine further comprises an internal rendering module; wherein,

[0114] The internal rendering module is configured to perform lip shape driving processing, text-to-speech processing, and limb driving and expression generation processing on the rendering task according to the video parameter, and obtain the digital human video.

[0115] In some possible implementation manners, the video factory further comprises a task detection component; wherein,

[0116] The task detection component is configured to detect a processing state of the rendering task in real time, obtain state information of the rendering task, and send the state information to the video editor.

[0117] The video editor is further configured to receive the state information, and visually display the state information.

[0118] In some possible implementation manners, the system further comprises a distributed storage service, and the video factory further comprises a watermark component; wherein,

[0119] The distributed rendering engine is further configured to upload the digital human video to the distributed storage service, obtain an original storage link of the digital human video, and send the original storage link of the digital human video to the video factory.

[0120] The video factory is further configured to receive the original storage link, download the digital human video into a local memory of the video factory according to the original storage link, call the watermark component to add a watermark to the digital human video, obtain a target digital human video, and upload the target digital human video to the distributed storage service to obtain a target storage link of the target digital human video.

[0121] For the system embodiment, it is basically similar to the method embodiment, so the description is relatively simple, and the relevant parts are described in the system embodiment.

[0122] In addition, the embodiment of the present application also provides an electronic device, such as Figure 4 As shown in the figure, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, the memory 403 complete the communication among each other through the communication bus 404,

[0123] The memory 403 is used to store computer programs.

[0124] The processor 401 is used to execute the programs stored in the memory 403, realize the processes of the above-mentioned method embodiments, and achieve the same technical effects. To avoid repetition, it will not be repeated here.

[0125] The communication bus mentioned in the above terminal can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0126] The communication interface is used for communication between the above-mentioned terminal and other devices.

[0127] The memory can include a random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0128] The aforementioned processor can be a general-purpose processor including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components.

[0129] As shown in FIG. 1, in an embodiment, the application provides a computer readable storage medium 101, which stores instructions, when executed by one or more processors, cause the processors to perform the processes of the above method embodiments and achieve the same technical effects. For the sake of brevity, details are not repeated here. Figure 5 As shown in FIG. 1, in an embodiment, the application provides a computer readable storage medium 101, which stores instructions, when executed by one or more processors, cause the processors to perform the processes of the above method embodiments and achieve the same technical effects. For the sake of brevity, details are not repeated here.

[0130] It should be noted that in this document, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or device that includes the element.

[0131] Those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device) execute the method described in each embodiment of the present application.

[0132] The embodiments of the present application are described above with reference to the drawings; however, the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative rather than restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

[0133] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the embodiments of the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solutions. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0134] Those of ordinary skill in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0135] In the embodiments provided by the present application, it should be understood that the disclosed method and system can be implemented by other ways. For example, the system embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0137] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0138] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application or the part of the prior art that contributes essentially or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the application. The foregoing storage medium includes various storage media that can store program codes, such as U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc.

[0139] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A digital human video generation method, characterized by, The method is applied to a digital human video generation system, the system at least comprising a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory, the video factory comprising a rendering task queue, the method comprising: obtaining video content configuration information through the video editor, and sending the video content configuration information to the video factory; receiving the video content configuration information through the video factory, converting the video content configuration information into video parameters, generating a rendering task according to the video parameters, sending the rendering task to the rendering task queue, and sending a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework; obtaining a rendering task from the rendering task queue through the distributed rendering engine, receiving the video generation request through the remote procedure call framework, and processing the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task; wherein the video factory comprises a video asset database, the video asset database comprises a plurality of video asset tables, the video asset tables are used to save video production data, and the method further comprises: saving the video parameters to the video asset database through the video factory, and associating the video parameters with the video asset tables; the video generation request containing the video parameters is sent to the distributed rendering engine according to the preset remote procedure call framework, comprising: obtaining video production data corresponding to the video parameters from the video asset database through the video factory according to the video parameters, and sending a video generation request containing the video parameters and the video production data to the distributed rendering engine; the rendering task is processed according to the video generation request to generate a digital human video corresponding to the rendering task, comprising: the rendering task is processed according to the video parameters and the video production data through the distributed rendering engine to generate the digital human video.

2. The method of claim 1, wherein, The distributed rendering engine comprises a plurality of rendering subnodes, and before the rendering task is sent to the rendering task queue, it further comprises: real-time monitoring the load condition of the rendering subnodes through the video factory, obtaining an idle state rendering subnode, and generating a video task number as an identifier of the rendering task according to the idle state rendering subnode; the rendering task is obtained from the rendering task queue through the distributed rendering engine, comprising: the rendering task corresponding to the rendering subnode is obtained from the rendering task queue through the rendering subnode according to the video task number.

3. The method of claim 1, wherein, The rendering task queue comprises a waiting queue and a working queue, and the rendering task is sent to the rendering task queue, comprising: the rendering task is sent to the waiting queue through the video factory, and the rendering task is transferred to the working queue according to preset task allocation information; The obtaining, by the distributed rendering engine, the rendering task from the rendering task queue comprises: The distributed rendering engine further comprises an internal rendering module, and the generating the digital human video corresponding to the rendering task further comprises:

4. The method of claim 1, wherein, The distributed rendering engine further comprises an internal rendering module, and the generating the digital human video corresponding to the rendering task further comprises: The distributed rendering engine further comprises an internal rendering module, and the generating the digital human video corresponding to the rendering task further comprises:

5. The method of claim 1, wherein, The video factory further comprises a task detection component, and the method further comprises: The video factory further comprises a task detection component, and the method further comprises: The video factory further comprises a task detection component, and the method further comprises:

6. The method of claim 1, wherein, The system further comprises a distributed storage service, and the video factory further comprises a watermark component, and the method further comprises: The system further comprises a distributed storage service, and the video factory further comprises a watermark component, and the method further comprises: The system further comprises a distributed storage service, and the video factory further comprises a watermark component, and the method further comprises:

7. A digital human video generation system characterized by, The system at least comprises a video editor, a video factory connected with the video editor, and a distributed rendering engine connected with the video factory, and the video factory comprises a rendering task queue; wherein, The video editor is configured to obtain video content configuration information and send the video content configuration information to the video factory; The video factory is configured to receive the video content configuration information, convert the video content configuration information into video parameters, generate a rendering task according to the video parameters, send the rendering task to the rendering task queue, and send a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework; The distributed rendering engine is configured to obtain a rendering task from the rendering task queue, receive the video generation request through the remote procedure call framework, and process the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task; The video factory comprises a video asset database, the video asset database comprises a plurality of video asset tables, and the video asset tables are configured to save video production data; wherein, The video factory is specifically configured to save the video parameter to the video asset database, and associate the video parameter with the video asset table; acquire video production data corresponding to the video parameter from the video asset database according to the video parameter, and send a video generation request containing the video parameter and the video production data to the distributed rendering engine; The distributed rendering engine is specifically configured to process the rendering task according to the video parameter and the video production data, and generate the digital human video.

8. An electronic device, comprising: The computer device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory is used for storing a computer program; The processor is used for executing the program stored on the memory, and realizes the method in any one of claims 1-6. 9.A computer readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method of any one of claims 1-6.

Images