Digital human video generation method, system, equipment and medium

Abstract

The embodiment of the invention provides a digital human video generation method, system and device and a medium, applied to a digital human video generation system, a video editor obtains video content configuration information and sends the information to a video factory, the video factory converts the video content configuration information into video parameters and generates a rendering task according to the video parameters, and the rendering task is used for rendering the video parameters. And sending a video generation request containing the video parameters to a distributed rendering engine according to the remote procedure call framework, obtaining the rendering task from the rendering task queue by the distributed rendering engine, and processing the rendering task according to the video generation request to generate a corresponding digital human video. According to the embodiment of the invention, the video factory generates the high-quality rendering task according to the video parameters, the video generation request is quickly transmitted and processed according to the remote procedure call framework, and the rendering task is processed in parallel through the distributed rendering engine, so that efficient, high-quality and real-time digital human video generation is realized.

Description

Technical Field

[0001] The present invention relates to the field of digital human technology, and in particular to a digital human video generation method, a digital human video generation system, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of the metaverse and artificial intelligence technologies, virtual digital human videos have gradually become an important application form in various industries and are widely used in news broadcasting, advertising, virtual live broadcasting, etc. However, although the digital human video generation and processing methods in the existing technology have made some progress, they still have many limitations and are difficult to meet the growing market demand.

[0003] The existing technology generates video content through text description, and then uses the video model to render and generate digital human video. However, the process of generating digital human video takes too long and has poor real-time performance. There are problems such as difficulty in ensuring video quality and low efficiency, which makes it difficult for users to obtain high-quality video output in a short time. Summary of the invention

[0004] The embodiments of the present invention provide a method, system, device and medium for generating digital human videos, so as to solve or partially solve the problems of low efficiency and poor quality of the existing digital human video generation.

[0005] The embodiment of the present invention discloses a digital human video generation method, which is applied to a digital human video generation system. The system at least includes a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory. The video factory includes a rendering task queue. The method includes:

[0006] Acquire video content configuration information through the video editor, and send the video content configuration information to the video factory;

[0007] Receiving the video content configuration information through the video factory, converting the video content configuration information into video parameters, generating a rendering task according to the video parameters, sending the rendering task to the rendering task queue, and sending a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework;

[0008] The distributed rendering engine obtains the rendering task from the rendering task queue, receives the video generation request through the remote procedure call framework, and processes the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task.

[0009] In some feasible implementations, the distributed rendering engine includes a plurality of rendering sub-nodes, and before sending the rendering task to the rendering task queue through the video factory, further includes:

[0010] The video factory monitors the load of the rendering sub-node in real time, obtains the rendering sub-node in an idle state, and generates a video task number according to the rendering sub-node in the idle state as an identifier of the rendering task;

[0011] The obtaining the rendering task from the rendering task queue by the distributed rendering engine includes:

[0012] The rendering task corresponding to the rendering sub-node is obtained from the rendering queue according to the video task number by the rendering sub-node.

[0013] In some feasible implementations, the rendering task queue includes a waiting queue and a working queue, and sending the rendering task to the rendering task queue through the video factory includes:

[0014] Sending the rendering task to the waiting queue through the video factory, and transferring the rendering task to the working queue according to preset task allocation information;

[0015] The obtaining the rendering task from the rendering task queue by the distributed rendering engine includes:

[0016] The rendering task is obtained from the work queue by the distributed rendering engine.

[0017] In some feasible implementations, the video factory includes a video asset database, the video asset database includes a plurality of video asset tables, the video asset tables are used to store video production data, and the method further includes:

[0018] Saving the video parameters to the video asset database through the video factory, and associating the video parameters with the video asset table;

[0019] The step of sending a video generation request including the video parameters to the distributed rendering engine through the video factory according to a preset remote procedure call framework includes:

[0020] Acquire, by the video factory, video production data corresponding to the video parameters from the video asset database according to the video parameters, and send a video generation request including the video parameters and the video production data to the distributed rendering engine;

[0021] The step of processing the rendering task according to the video generation request by the distributed rendering engine to generate a digital human video corresponding to the rendering task includes:

[0022] The distributed rendering engine processes the rendering task according to the video parameters and the video production data to generate the digital human video.

[0023] In some feasible implementations, the distributed rendering engine further includes an internal rendering module, and the processing of the rendering task according to the video parameters to generate a digital human video corresponding to the rendering task includes:

[0024] The internal rendering module performs lip shape driving processing, text-to-speech processing, and body driving and expression generation processing on the rendering task according to the video parameters to obtain the digital human video.

[0025] In some feasible implementations, the video factory further includes a task detection component, and the method further includes:

[0026] Detecting the processing status of the rendering task in real time through the task detection component, obtaining status information of the rendering task, and sending the status information to the video editor;

[0027] The status information is received through the video editor, and the status information is visually displayed.

[0028] In some feasible implementations, the system further includes a distributed storage service, the video factory further includes a watermark component, and the method further includes:

[0029] Uploading the digital human video to the distributed storage service through the distributed rendering engine, obtaining the original storage link of the digital human video, and sending the original storage link of the digital human video to the video factory;

[0030] The original storage link is received through the video factory, the digital human video is downloaded to the local memory of the video factory according to the original storage link, and the watermark component is called to add a watermark to the digital human video to obtain the target digital human video, and the target digital human video is uploaded to the distributed storage service to obtain the target storage link of the target digital human video.

[0031] The embodiment of the present invention also discloses a digital human video generation system, the system at least comprising a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory, the video factory comprising a rendering task queue; wherein,

[0032] The video editor is used to obtain video content configuration information and send the video content configuration information to the video factory;

[0033] The video factory is used to receive the video content configuration information, convert the video content configuration information into video parameters, generate a rendering task according to the video parameters, send the rendering task to the rendering task queue, and send a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework;

[0034] The distributed rendering engine is used to obtain the rendering task from the rendering task queue, receive the video generation request through the remote procedure call framework, and process the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task.

[0035] In some feasible implementations, the distributed rendering engine includes a plurality of rendering sub-nodes; wherein,

[0036] The video factory is also used to monitor the load of the rendering sub-node in real time, obtain the rendering sub-node in an idle state, and generate a video task number according to the rendering sub-node in the idle state as an identifier of the rendering task;

[0037] The rendering sub-node is used to obtain the rendering task corresponding to the rendering sub-node from the rendering queue according to the video task number.

[0038] In some feasible implementations, the rendering task queue includes a waiting queue and a working queue; wherein,

[0039] The video factory is specifically used to send the rendering task to the waiting queue, and transfer the rendering task to the working queue according to preset task allocation information;

[0040] The distributed rendering engine is specifically configured to obtain the rendering task from the work queue.

[0041] In some feasible implementations, the video factory includes a video asset database, the video asset database includes a plurality of video asset tables, and the video asset tables are used to store video production data; wherein,

[0042] The video factory is specifically used to save the video parameters to the video asset database and associate the video parameters with the video asset table; obtain the video production data corresponding to the video parameters from the video asset database according to the video parameters, and send a video generation request containing the video parameters and the video production data to the distributed rendering engine;

[0043] The distributed rendering engine is specifically used to process the rendering task according to the video parameters and the video production data to generate the digital human video.

[0044] In some feasible implementations, the distributed rendering engine further includes an internal rendering module; wherein,

[0045] The internal rendering module is used to perform lip shape driving processing, text-to-speech processing, and body driving and expression generation processing on the rendering task according to the video parameters to obtain the digital human video.

[0046] In some feasible implementations, the video factory further includes a task detection component; wherein,

[0047] The task detection component is used to detect the processing status of the rendering task in real time, obtain the status information of the rendering task, and send the status information to the video editor;

[0048] The video editor is also used to receive the status information and visually display the status information.

[0049] In some feasible implementations, the system further includes a distributed storage service, and the video factory further includes a watermark component; wherein,

[0050] The distributed rendering engine is further used to upload the digital human video to the distributed storage service, obtain the original storage link of the digital human video, and send the original storage link of the digital human video to the video factory;

[0051] The video factory is also used to receive the original storage link, download the digital human video to the local memory of the video factory according to the original storage link, call the watermark component to add a watermark to the digital human video, obtain the target digital human video, and upload the target digital human video to the distributed storage service to obtain the target storage link of the target digital human video.

[0052] The embodiment of the present invention further discloses an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0053] The memory is used to store computer programs;

[0054] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0055] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0056] The embodiments of the present invention include the following advantages:

[0057] The embodiment of the present invention is applied to a digital human video generation system, which at least includes a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory. The video factory includes a rendering task queue. The video editor obtains video content configuration information and sends it to the video factory. The video factory converts the video content configuration information into video parameters and generates rendering tasks according to the video parameters, and sends a video generation request containing video parameters to the distributed rendering engine according to a remote procedure call framework. The distributed rendering engine obtains the rendering task from the rendering task queue, and processes the rendering task according to the video generation request to generate a corresponding digital human video. The embodiment of the present invention generates high-quality rendering tasks according to video parameters through the video factory, quickly transmits and processes video generation requests according to the remote procedure call framework, and processes rendering tasks in parallel through the distributed rendering engine, thereby achieving efficient, high-quality, and real-time digital human video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flowchart of the steps of a method for generating a digital human video provided in an embodiment of the present invention;

[0059] Figure 2 is a schematic diagram of a process of generating a digital human video provided in an embodiment of the present invention;

[0060] Figure 3 is a structural block diagram of a digital human video generation system provided in an embodiment of the present invention;

[0061] Figure 4 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of a computer-readable storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] As an example, the prior art generates video content through text description, and then uses a video model for rendering to generate a digital human video. However, the generation process from text to video requires a lot of computing resources and time. There are problems such as the process of generating digital human videos is too long, the real-time performance is poor, and the output video quality is difficult to guarantee. The prior art also involves the use of GPT (Generative Pre-trained Transformer) technology to process real-time information and generate text, but when it is applied to video generation, it still faces the above problems, making it difficult for users to obtain high-quality video output in a short time.

[0065] In this regard, in the present invention, it is applied to a digital human video generation system, which at least includes a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory. The video factory includes a rendering task queue. The video editor obtains video content configuration information and sends it to the video factory. The video factory converts the video content configuration information into video parameters and generates rendering tasks according to the video parameters, and sends a video generation request containing video parameters to the distributed rendering engine according to a remote procedure call framework. The distributed rendering engine obtains the rendering task from the rendering task queue, and processes the rendering task according to the video generation request to generate a corresponding digital human video. The embodiment of the present invention generates high-quality rendering tasks according to video parameters through the video factory, and quickly transmits and processes video generation requests according to the remote procedure call framework, and processes rendering tasks in parallel through the distributed rendering engine, thereby achieving efficient, high-quality, and real-time digital human video generation.

[0066] Reference Figure 1 , shows a flowchart of the steps of a digital human video generation method provided in an embodiment of the present invention, which is applied to a digital human video generation system. The digital human video generation system at least includes a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory. The video factory includes a rendering task queue. The method may specifically include the following steps:

[0067] Step 101: Obtain video content configuration information through the video editor, and send the video content configuration information to the video factory;

[0068] The digital human video generation system at least includes a video editor, a video factory and a distributed rendering engine. The video editor and the video factory establish a communication connection through the HTTP (Hypertext Transfer Protocol) protocol, and the video factory and the distributed rendering engine establish a communication connection through HTTP / 2 (Hypertext Transfer Protocol Version 2). Specifically, the video editor is used to obtain the video content configuration information input by the user. The user configures the video content through the video editor, such as selecting the digital human image, setting the scene, adjusting the action, etc.; the video factory is used to convert the video content configured by the user into an executable rendering task, and distribute the rendering task to the distributed rendering engine for processing according to the rendering task queue, that is, the video factory manages and schedules the rendering task through the rendering task queue, and the rendering task queue can be a Redis (Remote Dictionary Server) queue; the distributed rendering engine is used to use distributed computing resources to parallelly process the rendering tasks to generate digital human videos. The digital human video generation system realizes automatic, efficient and high-quality digital human video generation through the collaborative work of the video editor, the video factory and the distributed rendering engine.

[0069] In the process of digital human video generation, the video content configuration information configured by the user is first obtained through the video editor and sent to the video factory. The video content configuration information is used to define the specific content and form of expression of the generated digital human video. The video content configuration information may include the digital human image, subtitles, scene settings, action sequences, special effects parameters, audio configuration and other information.

[0070] Specifically, the video editor is configured with functions such as digital human image setting, text broadcast content input, subtitle style selection, and track editor operation. Users can flexibly configure video content through the video editor, and then the video editor packages the video content configuration information configured by the user and sends it to the video factory. The embodiment of the present invention enables users to configure video content conveniently through the video editor without writing complex codes or scripts, and the video editor can collect detailed configuration information of the user on the video content, provide necessary data support for the subsequent video generation process, and ensure that the generated video meets the needs of the user.

[0071] Step 102: receiving the video content configuration information through the video factory, converting the video content configuration information into video parameters, generating a rendering task according to the video parameters, sending the rendering task to the rendering task queue, and sending a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework;

[0072] In an embodiment of the present invention, after receiving the video content configuration information sent by the video editor, the video factory converts the video content configuration information into video parameters used within the system, such as digital human image, scene setting, action sequence, special effect parameters, audio configuration, etc., and further converts the video parameters into rendering task parameters, creates a rendering task according to the rendering task parameters, and the video factory sends the generated rendering task to the rendering task queue. The rendering task queue is a first-in-first-out (FIFO) data structure used to store and manage rendering tasks to be processed. The video factory ensures that the rendering tasks can be processed by the distributed rendering engine in sequence through the rendering task queue to avoid task loss or confusion, and the video factory sends a video generation request containing video parameters to the distributed rendering engine through a preset remote procedure call framework. The remote procedure call framework is pre-built based on the HTTP / 2 protocol between the video factory and the distributed rendering engine. In the embodiment of the present invention, the remote procedure call framework adopts the gRPC framework, which is an open source high-performance remote procedure call (RPC, Google Remote Procedure Call) framework. The gRPC framework is responsible for efficient and reliable communication between the video factory and the distributed rendering engine to achieve rapid transmission of video generation requests and related video parameters. The present invention does not make specific restrictions on this.

[0073] In some embodiments, the distributed rendering engine includes a plurality of rendering sub-nodes. Before sending the rendering task to the rendering task queue through the video factory in step 102, it also includes: monitoring the load of the rendering sub-node in real time through the video factory, obtaining an idle rendering sub-node, and generating a video task number according to the idle rendering sub-node as an identifier of the rendering task; obtaining the rendering task from the rendering task queue through the distributed rendering engine includes: obtaining the rendering task corresponding to the rendering sub-node from the rendering queue through the rendering sub-node according to the video task number.

[0074] In an embodiment of the present invention, the distributed rendering engine is composed of multiple rendering sub-nodes, and rendering tasks are processed in parallel by multiple rendering sub-nodes. The video factory also involves real-time monitoring of the load status of each rendering sub-node to understand the current working status of each rendering sub-node, determine whether the rendering sub-node is idle, and select the idle rendering sub-node as the execution node for generating the rendering task.

[0075] Specifically, the video factory identifies the idle rendering sub-nodes according to the load of the monitored rendering sub-nodes, and then generates a unique video task number for the rendering task based on the idle rendering sub-nodes. The video task number is associated with the idle rendering sub-nodes, and the video task number is used as the identifier of the rendering task. Then, the rendering sub-node obtains the corresponding rendering task from the rendering task queue according to the video task number assigned to it, ensuring that the rendering task can be efficiently allocated to the appropriate node, thereby improving rendering efficiency.

[0076] In some embodiments, the rendering task queue includes a waiting queue and a working queue, and sending the rendering task to the rendering task queue through the video factory includes: sending the rendering task to the waiting queue through the video factory, and transferring the rendering task to the working queue according to preset task allocation information; obtaining the rendering task from the rendering task queue through the distributed rendering engine includes: obtaining the rendering task from the working queue through the distributed rendering engine.

[0077] In the embodiment of the present invention, the rendering task queue is divided into two sub-queues: a waiting queue and a working queue. The waiting queue is used to store rendering tasks that have not been assigned to the rendering sub-nodes, and the working queue is used to store rendering tasks that have been assigned to the rendering sub-nodes.

[0078] Specifically, the video factory assigns the rendering tasks in the waiting queue to the corresponding rendering subnodes and transfers them to the working queue according to the preset task allocation information. The task allocation information may include the idle state, task priority, task type, etc. of the rendering subnodes, and then the rendering subnodes obtain the rendering tasks assigned to them from the working queue. The embodiments of the present invention can better manage and schedule rendering tasks by dividing the waiting queue and the working queue, ensuring that the rendering tasks can be processed in order, and according to the preset task allocation information, the system can efficiently allocate rendering tasks to appropriate rendering subnodes to improve rendering efficiency.

[0079] In some embodiments, the video factory includes a video asset database, the video asset database includes a plurality of video asset tables, the video asset tables are used to store video production data, and the method further includes:

[0080] Saving the video parameters to the video asset database through the video factory, and associating the video parameters with the video asset table;

[0081] The step of sending a video generation request including the video parameters to the distributed rendering engine through the video factory according to a preset remote procedure call framework includes:

[0082] Acquire, by the video factory, video production data corresponding to the video parameters from the video asset database according to the video parameters, and send a video generation request including the video parameters and the video production data to the distributed rendering engine;

[0083] The step of processing the rendering task according to the video generation request by the distributed rendering engine to generate a digital human video corresponding to the rendering task includes:

[0084] The distributed rendering engine processes the rendering task according to the video parameters and the video production data to generate the digital human video.

[0085] In an embodiment of the present invention, the video factory also includes a video asset database for storing and managing various data required in the video production process. The video asset database is composed of multiple video asset tables, each of which is used to store a specific type of video production data, such as a digital human 3D image asset table, a digital human video record table, a digital human video and image association table, and a digital human video basic asset table. Through multiple video asset tables, the system can classify, store and manage different types of video production data, which is convenient for subsequent query and use.

[0086] Specifically, the video factory saves the generated video parameters to the video asset database, and associates these parameters with the corresponding video asset table, for example, the digital human image parameters are associated with the digital human image table, the scene setting parameters are associated with the scene table, etc. According to the generated video parameters, the corresponding video production data can be obtained from the video asset database, and then the video factory sends the video generation request containing the video parameters and the corresponding video production data to the distributed rendering engine, and the distributed rendering engine processes the rendering task according to the received video parameters and video production data to generate the digital human video. The video factory of the embodiment of the present invention can effectively manage and store various data required in the video production process through the video asset database, ensuring that various data required in the video generation process can be accurately obtained.

[0087] Step 103: Obtain a rendering task from the rendering task queue through the distributed rendering engine, receive the video generation request through the remote procedure call framework, and process the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task.

[0088] In an embodiment of the present invention, the distributed rendering engine periodically obtains the rendering tasks to be processed from the rendering task queue, and receives the video generation request corresponding to the rendering task through the gRPC remote procedure call framework, and then the distributed rendering engine performs specific video rendering operations such as loading digital human images, setting subtitles, rendering action sequences, synthesizing audio, etc. according to the received video generation request and rendering task to generate digital human videos. The distributed rendering engine in the embodiment of the present invention uses the computing resources of multiple computers to process rendering tasks in parallel, thereby improving the efficiency of generating digital human videos.

[0089] In some embodiments, the distributed rendering engine also includes an internal rendering module, which processes the rendering task according to the video parameters to generate a digital human video corresponding to the rendering task, including: performing lip driving processing, text-to-speech processing, and body driving and expression generation processing on the rendering task according to the video parameters by the internal rendering module to obtain the digital human video.

[0090] In an embodiment of the present invention, the distributed rendering engine also includes multiple internal rendering modules, which are used to perform video rendering operations required by the rendering task to ensure that the generated video content is not only efficient but also highly natural. The internal rendering module includes at least a lip shape driving submodule, a TTS (text-to-speech) submodule, a body driving submodule, and an expression generation submodule, etc. By calling the internal rendering module to perform video rendering operations such as lip shape driving, text-to-speech, body driving, and expression generation, a digital human video is obtained. The distributed rendering engine of the embodiment of the present invention can be flexibly expanded through the internal rendering module, supports more rendering functions and effects, meets the ever-changing rendering requirements, and improves the quality of the generated digital human video.

[0091] In some embodiments, the video factory also includes a task detection component, which detects the processing status of the rendering task in real time, obtains status information of the rendering task such as the progress or error information of the task, and sends the status information to the video editor for visual display, and then notifies the user about the processing status of the rendering task through the video editor, ensuring that the user can understand the progress of the rendering task in a timely manner.

[0092] In some embodiments, the system also includes a distributed storage service, the video factory also includes a watermark component, and the method also includes: uploading the digital human video to the distributed storage service through the distributed rendering engine, obtaining the original storage link of the digital human video, and sending the original storage link of the digital human video to the video factory; receiving the original storage link through the video factory, downloading the digital human video to the local memory of the video factory according to the original storage link, and calling the watermark component to add a watermark to the digital human video to obtain a target digital human video, and uploading the target digital human video to the distributed storage service to obtain a target storage link of the target digital human video.

[0093] In an embodiment of the present invention, the digital human video generation system also includes a distributed storage service, which establishes communication connections with the video factory and the distributed rendering engine through the HTTP protocol respectively. Optionally, the distributed storage service can adopt the Ceph (Ceph Distributed Storage System) distributed storage system; the video factory also includes a watermark component, which is used to add pre-set watermark information to the digital human video to ensure the copyright and source information of the video. Optionally, the watermark component can adopt the FFMPEG (Fast Forward Moving Picture Experts Group) component, and the present invention does not limit this.

[0094] In the specific implementation process, after the distributed rendering engine generates the digital human video, it uploads the digital human video to the distributed storage service and obtains the original storage link of the digital human video, and then sends the original storage link to the video factory. After receiving the original storage link, the video factory accesses the distributed storage service and downloads the digital human video to the local memory according to the original storage link. The video factory further calls the watermark component to add watermark information to the digital human video, generates a digital human video with watermark information as the target digital human video, and uploads the target digital human video to the distributed storage service to obtain the target storage link corresponding to the target digital human video. The video factory further stores the original video link and the target video link in the video asset database for preservation. In the embodiment of the present invention, the digital human video generation system can efficiently store and manage digital human videos through the distributed storage service to ensure the security and reliability of video files, and can add watermarks to digital human videos through the watermark component to ensure that the copyright and source information of the video are protected, and the video link is persistently stored through the video asset database to facilitate the query and use of subsequent video production.

[0095] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0096] Reference Figure 2 , shows a schematic flow chart of a digital human video generation process provided in an embodiment of the present invention. In the process of digital human video generation, the user first configures video content configuration information such as digital human image, text and subtitles through a video editor, and then the video editor sends the video content configuration information input by the user to the video factory. The video factory creates a rendering task according to the video content configuration information and puts the rendering task into the waiting queue of the rendering task queue for allocation. The video factory detects the distributed rendering engine in real time to obtain the rendering engine subnode in the idle state, and then allocates the rendering task to the rendering engine subnode in the idle state and transfers it to the working queue of the rendering task queue. At the same time, the video factory sends a video generation request to the distributed rendering engine through the gRPC framework. The distributed rendering engine further obtains the rendering task from the working queue and allocates it to the corresponding rendering engine subnode in the idle state. After receiving the rendering task, the rendering engine subnode performs video rendering to obtain the digital human video. In addition, the video factory also involves real-time detection of the processing status of the rendering task, and sends the processing status information of the rendering task to the video editor, and notifies the user of the processing status of the rendering task through the video editor.

[0097] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0098] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0099] In an embodiment of the present invention, it is applied to a digital human video generation system, which at least includes a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory. The video factory includes a rendering task queue. The video editor obtains video content configuration information and sends it to the video factory. The video factory converts the video content configuration information into video parameters and generates rendering tasks according to the video parameters, and sends a video generation request containing video parameters to the distributed rendering engine according to a remote procedure call framework. The distributed rendering engine obtains the rendering task from the rendering task queue, and processes the rendering task according to the video generation request to generate a corresponding digital human video. The video factory in the embodiment of the present invention generates high-quality rendering tasks according to the video parameters, and quickly transmits and processes the video generation request according to the remote procedure call framework, and processes the rendering tasks in parallel through the distributed rendering engine, thereby realizing efficient, high-quality, and real-time digital human video generation.

[0100] Reference Figure 3 , shows a structural block diagram of a digital human video generation system provided in an embodiment of the present invention, the system at least includes a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory, the video factory includes a rendering task queue; wherein,

[0101] The video editor is used to obtain video content configuration information and send the video content configuration information to the video factory;

[0102] The video factory is used to receive the video content configuration information, convert the video content configuration information into video parameters, generate a rendering task according to the video parameters, send the rendering task to the rendering task queue, and send a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework;

[0103] The distributed rendering engine is used to obtain the rendering task from the rendering task queue, receive the video generation request through the remote procedure call framework, and process the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task.

[0104] In some feasible implementations, the distributed rendering engine includes a plurality of rendering sub-nodes; wherein,

[0105] The video factory is also used to monitor the load of the rendering sub-node in real time, obtain the rendering sub-node in an idle state, and generate a video task number according to the rendering sub-node in the idle state as an identifier of the rendering task;

[0106] The rendering sub-node is used to obtain the rendering task corresponding to the rendering sub-node from the rendering queue according to the video task number.

[0107] In some feasible implementations, the rendering task queue includes a waiting queue and a working queue; wherein,

[0108] The video factory is specifically used to send the rendering task to the waiting queue, and transfer the rendering task to the working queue according to preset task allocation information;

[0109] The distributed rendering engine is specifically configured to obtain the rendering task from the work queue.

[0110] In some feasible implementations, the video factory includes a video asset database, the video asset database includes a plurality of video asset tables, and the video asset tables are used to store video production data; wherein,

[0111] The video factory is specifically used to save the video parameters to the video asset database and associate the video parameters with the video asset table; obtain the video production data corresponding to the video parameters from the video asset database according to the video parameters, and send a video generation request containing the video parameters and the video production data to the distributed rendering engine;

[0112] The distributed rendering engine is specifically used to process the rendering task according to the video parameters and the video production data to generate the digital human video.

[0113] In some feasible implementations, the distributed rendering engine further includes an internal rendering module; wherein,

[0114] The internal rendering module is used to perform lip shape driving processing, text-to-speech processing, and body driving and expression generation processing on the rendering task according to the video parameters to obtain the digital human video.

[0115] In some feasible implementations, the video factory further includes a task detection component; wherein,

[0116] The task detection component is used to detect the processing status of the rendering task in real time, obtain the status information of the rendering task, and send the status information to the video editor;

[0117] The video editor is also used to receive the status information and visually display the status information.

[0118] In some feasible implementations, the system further includes a distributed storage service, and the video factory further includes a watermark component; wherein,

[0119] The distributed rendering engine is further used to upload the digital human video to the distributed storage service, obtain the original storage link of the digital human video, and send the original storage link of the digital human video to the video factory;

[0120] The video factory is also used to receive the original storage link, download the digital human video to the local memory of the video factory according to the original storage link, call the watermark component to add a watermark to the digital human video, obtain the target digital human video, and upload the target digital human video to the distributed storage service to obtain the target storage link of the target digital human video.

[0121] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.

[0122] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0123] Memory 403, used for storing computer programs;

[0124] The processor 401 is used to execute the program stored in the memory 403 to implement the various processes of the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0125] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0126] The communication interface is used for communication between the above terminal and other devices.

[0127] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0128] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0129] like Figure 5 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 501 is also provided, in which instructions are stored. When executed by one or more processors, the processors execute the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, they are not repeated here.

[0130] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0132] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0133] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] In the embodiments provided by the present invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0139] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating a digital human video, characterized in that: Applied to a digital human video generation system, the system at least comprises a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory, the video factory comprises a rendering task queue, and the method comprises: Acquire video content configuration information through the video editor, and send the video content configuration information to the video factory; Receiving the video content configuration information through the video factory, converting the video content configuration information into video parameters, generating a rendering task according to the video parameters, sending the rendering task to the rendering task queue, and sending a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework; The distributed rendering engine obtains the rendering task from the rendering task queue, receives the video generation request through the remote procedure call framework, and processes the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task.

2. The method according to claim 1, characterized in that: The distributed rendering engine includes a plurality of rendering sub-nodes, and before sending the rendering task to the rendering task queue through the video factory, further includes: The video factory monitors the load of the rendering sub-node in real time, obtains the rendering sub-node in an idle state, and generates a video task number according to the rendering sub-node in the idle state as an identifier of the rendering task; The obtaining the rendering task from the rendering task queue by the distributed rendering engine includes: The rendering task corresponding to the rendering sub-node is obtained from the rendering queue according to the video task number by the rendering sub-node.

3. The method according to claim 1, characterized in that: The rendering task queue includes a waiting queue and a working queue, and sending the rendering task to the rendering task queue through the video factory includes: Sending the rendering task to the waiting queue through the video factory, and transferring the rendering task to the working queue according to preset task allocation information; The obtaining the rendering task from the rendering task queue by the distributed rendering engine includes: The rendering task is obtained from the work queue by the distributed rendering engine.

4. The method according to claim 1, characterized in that: The video factory includes a video asset database, the video asset database includes a plurality of video asset tables, the video asset tables are used to store video production data, and the method further includes: Saving the video parameters to the video asset database through the video factory, and associating the video parameters with the video asset table; The step of sending a video generation request including the video parameters to the distributed rendering engine through the video factory according to a preset remote procedure call framework includes: Acquire, by the video factory, video production data corresponding to the video parameters from the video asset database according to the video parameters, and send a video generation request including the video parameters and the video production data to the distributed rendering engine; The step of processing the rendering task according to the video generation request by the distributed rendering engine to generate a digital human video corresponding to the rendering task includes: The distributed rendering engine processes the rendering task according to the video parameters and the video production data to generate the digital human video.

5. The method according to claim 1, characterized in that: The distributed rendering engine also includes an internal rendering module, which processes the rendering task according to the video parameters to generate a digital human video corresponding to the rendering task, including: The internal rendering module performs lip shape driving processing, text-to-speech processing, and body driving and expression generation processing on the rendering task according to the video parameters to obtain the digital human video.

6. The method according to claim 1, characterized in that: The video factory also includes a task detection component, and the method further includes: Detecting the processing status of the rendering task in real time through the task detection component, obtaining status information of the rendering task, and sending the status information to the video editor; The status information is received through the video editor, and the status information is visually displayed.

7. The method according to claim 1, characterized in that: The system further includes a distributed storage service, the video factory further includes a watermark component, and the method further includes: Uploading the digital human video to the distributed storage service through the distributed rendering engine, obtaining the original storage link of the digital human video, and sending the original storage link of the digital human video to the video factory; The original storage link is received through the video factory, the digital human video is downloaded to the local memory of the video factory according to the original storage link, and the watermark component is called to add a watermark to the digital human video to obtain the target digital human video, and the target digital human video is uploaded to the distributed storage service to obtain the target storage link of the target digital human video.

8. A digital human video generation system, characterized in that: The system at least includes a video editor, a video factory connected to the video editor, and a distributed rendering engine connected to the video factory, wherein the video factory includes a rendering task queue; The video editor is used to obtain video content configuration information and send the video content configuration information to the video factory; The video factory is used to receive the video content configuration information, convert the video content configuration information into video parameters, generate a rendering task according to the video parameters, send the rendering task to the rendering task queue, and send a video generation request containing the video parameters to the distributed rendering engine according to a preset remote procedure call framework; The distributed rendering engine is used to obtain the rendering task from the rendering task queue, receive the video generation request through the remote procedure call framework, and process the rendering task according to the video generation request to generate a digital human video corresponding to the rendering task.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images