Virtual digital human rendering method, device and system and electronic equipment

Abstract

The invention relates to the technical field of computer vision, and particularly provides a virtual digital human rendering method, device and system and electronic device.The method can comprise the steps that feedback data matched with input data of a client side are obtained; wherein the input data comprises user voice, user characters, instruction information or scene configuration information; the feedback data comprises feedback audio, text subtitles, scene configuration and emotion labels; processing the feedback audio to obtain expression data and a timestamp; rendering and generating a video frame containing a virtual digital human based on the expression data, the timestamp, the scene configuration and the emotion label; encoding the video frame and the feedback audio to generate a video stream; wherein the video stream is pushed to the client for playing. According to the embodiment of the invention, personalized customization and rendering of the virtual digital human can be improved, and the simulation effect is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method, apparatus, system, and electronic device for rendering virtual digital humans. Background Technology

[0002] Virtual digital humans are virtual characters created using information technology, modeled after human appearance and intelligence. They feature artificial intelligence, realistic visual presentation, and personalization, and have wide applications in fields such as medicine, the metaverse, business, education, and cultural tourism.

[0003] Currently, most exhibition halls, pavilions, and service organizations use human receptionists, resulting in excessively high labor costs. Furthermore, the quality of service is significantly affected by factors such as emotions and working hours. To improve service quality, virtual digital humans are being widely promoted. However, traditional virtual digital human rendering solutions can only play pre-set images or videos, failing to provide users with location-based services.

[0004] Therefore, how to provide a technical solution for personalized virtual digital human rendering has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of some embodiments of this application is to provide a method, apparatus, system and electronic device for rendering virtual digital humans. The technical solutions of the embodiments of this application can realize the customized design of virtual digital humans and improve the user experience.

[0006] In a first aspect, some embodiments of this application provide a method for rendering a virtual digital human, comprising: acquiring feedback data matching input data from a client; wherein the input data includes user voice, user text, command information, or scene configuration information of the virtual digital human; the feedback data includes feedback audio, text subtitles, scene configuration, and emotion tags; processing the feedback audio to acquire facial expression data and a timestamp; rendering and generating a video frame containing the virtual digital human based on the facial expression data, the timestamp, the scene configuration, and the emotion tags; encoding the video frame and the feedback audio to generate a video stream; wherein the video stream is pushed to the client for playback.

[0007] Some embodiments of this application determine feedback data through user input data, process the feedback audio in the feedback data to obtain facial expression data and timestamps, then combine it with other relevant data to render and generate video frames of the virtual digital human, and finally encode the feedback audio to generate a video stream. Embodiments of this application can realize customized design of virtual digital humans, improve user experience and simulation effect, and through customization, make the virtual digital human applicable to various scenarios.

[0008] In some embodiments, obtaining feedback data that matches the client's input data includes: receiving the feedback data sent by a large model server; wherein the feedback data is generated by inputting the input data into the large model server.

[0009] Some embodiments of this application automatically generate feedback data from large models in a large model server, thereby improving the accuracy of subsequent simulation rendering.

[0010] In some embodiments, processing the feedback audio to obtain facial expression data and a timestamp includes: segmenting and decomposing the feedback audio using an audio-to-lip-sync algorithm to obtain the facial expression data, the timestamp, and the audio segmentation results.

[0011] Some embodiments of this application process the feedback audio using an audio-to-lip-sync algorithm to obtain facial expression data, timestamps, and audio segmentation results, thereby achieving effective generation of digital human emotions.

[0012] In some embodiments, the step of rendering a video frame containing a virtual digital human based on the facial expression data, the timestamp, the scene configuration, and the emotion tag includes: sending the facial expression data, the timestamp, the scene configuration, and the emotion tag to a rendering module via a memory-sharing method; and rendering the facial expression data, the timestamp, the scene configuration, and the emotion tag in real time by the rendering module to obtain the video frame; wherein the video frame includes multiple frames; and each frame in the multiple frames carries a corresponding timestamp.

[0013] Some embodiments of this application send the data required for rendering to the rendering module for real-time rendering through memory sharing to obtain video frames, which can reduce the interactive latency of real-time rendering of virtual digital humans and improve rendering efficiency.

[0014] In some embodiments, the step of rendering the facial expression data, the timestamp, the scene configuration, and the emotion tag in real time through the rendering module to obtain the video frame includes: using the facial expression data and the timestamp to drive the lip movements of the character model, and customizing the character image and background of the character model based on the scene configuration and the emotion tag to obtain the video frame.

[0015] Some embodiments of this application can achieve customized rendering of video frames by combining relevant data through a rendering module.

[0016] In some embodiments, encoding the video frame and the feedback audio to generate a video stream includes: sending the video frame and the audio block results to the encoding module via memory sharing; comparing the timestamps of the video frame and the audio block results and encoding them in real time through the encoding module; and adding the text subtitles to obtain the video stream.

[0017] Some embodiments of this application use a memory-sharing method to send video frames and feedback audio to an encoding module for real-time encoding to obtain a video stream, thereby achieving audio-video alignment and generating a high-quality video stream.

[0018] In some embodiments, the method further includes: if the encoding module is found to contain a pre-stored cached video that matches the feedback data, then the cached video is used as the video stream.

[0019] Some embodiments of this application can reduce rendering pressure in certain situations by pre-storing videos.

[0020] Secondly, some embodiments of this application provide an apparatus for rendering a virtual digital human, comprising: an acquisition module for acquiring feedback data matching input data from a client; wherein the input data includes user voice, user text, instruction information, or scene configuration information; the feedback data includes feedback audio, text subtitles, scene configuration, and emotion tags; a processing module for processing the feedback audio to acquire facial expression data and a timestamp; a rendering module for rendering a video frame containing a virtual digital human based on the facial expression data, the timestamp, the scene configuration, and the emotion tags; and an encoding module for encoding the video frame and the feedback audio to generate a video stream; wherein the video stream is pushed to the client for playback.

[0021] In some embodiments, the acquisition module is configured to: receive the feedback data sent by the large model server; wherein the feedback data is generated by inputting the input data into the large model in the large model server.

[0022] Thirdly, some embodiments of this application provide a system for rendering a virtual digital human, comprising: a client, configured to acquire input data in response to user operation commands; the input data including user voice, user text, command information, or scene configuration information of the virtual digital human; a large model server, configured to input the input data into a large model to generate feedback data; the feedback data including feedback audio, text subtitles, scene configuration, and emotion tags; a processing server, configured to process the feedback audio to acquire facial expression data and a timestamp; render and generate video frames containing the virtual digital human based on the facial expression data, the timestamp, the scene configuration, and the emotion tags; encode the video frames and the feedback audio to generate a video stream; wherein the video stream is pushed to the client for playback.

[0023] Fourthly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0024] Fifthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0025] Sixthly, some embodiments of this application provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 System diagrams for rendering virtual digital humans provided for some embodiments of this application; Figure 2 One of the flowcharts for a method of rendering a virtual digital human provided for some embodiments of this application; Figure 3 A second flowchart illustrating a method for rendering a virtual digital human, provided for some embodiments of this application; Figure 4Block diagram of a device for rendering virtual digital humans provided for some embodiments of this application; Figure 5 Architecture diagrams of virtual digital human rendering provided for some embodiments of this application; Figure 6 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0028] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0029] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] In related technologies, traditional video-fusion digital human rendering solutions suffer from high costs for customized character creation, generally low simulation accuracy, and high latency; traditional 3D model digital human rendering solutions exhibit stiff lip movements, high latency, heavy rendering load, and poor interactivity. In the field of intelligent customer service, current intelligent customer service systems can only provide text or image-based messages, resulting in a poor user experience; in some reception settings, the quality of service is severely affected by factors such as human interaction with staff, including emotions and working hours.

[0031] In view of this, some embodiments of this application provide a method for rendering a virtual digital human, which can be applied in intelligent customer service or various reception occasions (such as exhibition halls, exhibition centers, service institutions, etc.). This method first matches feedback data based on the client's input data; after processing the feedback data, it obtains the virtual digital human's facial expression data and timestamp; then, it combines this data with other data to render video frames in real time; finally, it encodes the video to generate a video stream for playback on the client. This application can achieve personalized customization of virtual digital humans, reduce data transmission latency, and improve the simulation realism and user experience of virtual digital humans.

[0032] The following is in conjunction with the appendix Figure 1 The overall composition and structure of a virtual digital human rendering system provided by some embodiments of this application are illustrated by way of example.

[0033] like Figure 1As shown, some embodiments of this application provide a system diagram for rendering a virtual digital human. This virtual digital human rendering system may include: a client 100, an AI large model terminal 200 (as a specific example of a large model server), and a processing server 300. The AI ​​large model terminal 200 deploys a pre-trained large model. The processing server 300 includes lip-sync algorithm services, rendering services, and encoding services. The client 100, the AI ​​large model terminal 200, and the processing server 300 can communicate sequentially, and the processing server 300 and the client 100 can communicate bidirectionally.

[0034] In some embodiments of this application, the client 100 can be a mobile terminal or a non-portable computer terminal. For example, the client 100 can be a mobile phone, laptop, iPad, glasses (such as AR glasses or VR glasses), a web page on a PC, an IoT device, etc. The embodiments of this application do not specifically limit the scope of the application.

[0035] The functions of each structure within the above system are illustrated below.

[0036] In some embodiments of this application, the client 100 is configured to: obtain the input data in response to a user's operation instruction; the input data includes user voice, user text, instruction information, or scene configuration information of a virtual digital human; In some embodiments of this application, the large model server 200 is used to: input the input data into the large model to generate feedback data; the feedback data includes feedback audio, text subtitles, scene configurations, and emotion tags; In some embodiments of this application, the processing server 300 is configured to: process the feedback audio to obtain facial expression data and a timestamp; render and generate a video frame containing a virtual digital human based on the facial expression data, the timestamp, the scene configuration, and the emotion tag; encode the video frame and the feedback audio to generate a video stream; wherein the video stream is pushed to the client for playback.

[0037] The following is in conjunction with the appendix Figure 2 The present application provides an exemplary embodiment of the virtual digital human rendering process performed by a processing server 300.

[0038] Please see the appendix Figure 2 , Figure 2 A flowchart of a method for rendering a virtual digital human is provided for some embodiments of this application. The method for rendering a virtual digital human may include: S210, Obtain feedback data that matches the input data from the client; wherein the input data includes user voice, user text, command information, or scene configuration information of a virtual digital human; the feedback data includes feedback audio, text subtitles, scene configuration, and emotion tags.

[0039] For example, in a specific embodiment of this application, a user can input their customization requirements for the virtual digital human through client 100 (as a specific example of input data). These customization requirements may include user voice and / or user text, as well as scene configuration information, such as reception scenarios, customer service scenarios, etc. Client 100 sends these customization requirements to the AI ​​large model terminal 200 to obtain feedback data. It is understood that customization requirements can contain one or more types of data, including user voice, user text, instruction information, and scene configuration information, and can be flexibly adjusted. When only one type of data is included, the system can subsequently automatically identify the context, background, semantics, and other information corresponding to that type of data, and generate a video stream corresponding to the matching virtual digital human.

[0040] In some embodiments of this application, S210 may include: receiving the feedback data sent by the large model server; wherein the feedback data is generated by inputting the input data into the large model in the large model server.

[0041] For example, in a specific embodiment of this application, the above-mentioned customized requirements are input into a pre-trained large model. The large model analyzes the user's current emotions by analyzing the user's voice and text, and outputs matching response content (as a specific example of feedback voice and text subtitles), the virtual digital human's behavioral tone (as a specific example of scenario configuration), and emotion labels (e.g., happy, angry, furious, etc.) in combination with the actual content.

[0042] In some embodiments of this application, S210 may include: confirming that the encoding module contains a pre-stored cached video that matches the feedback data, and then using the cached video as the video stream.

[0043] For example, in a specific embodiment of this application, the processing server 300 can also determine whether there is a pre-cached video stream matching the current encoding service based on the feedback data. If it exists, it is directly pushed to the client 100; otherwise, the implementation process of S220~S240 described below is executed. It is understood that the encoding service pre-caches commonly used video streams such as digital human standby video streams, greeting video streams, closing remarks video streams, keyword video streams, or fixed answer video streams. When needed, these streams can be seamlessly switched and pushed, greatly reducing the rendering pressure on the rendering server.

[0044] S220, process the feedback audio to obtain facial expression data and timestamp.

[0045] For example, in a specific embodiment of this application, the feedback audio is processed in detail to determine the BS data (as a specific example of facial expression data) and timestamp contained therein.

[0046] Blend Shape (BS) is a technique in 3D computer graphics used to create facial expressions or other deformation effects. It achieves smooth transitions by interpolating between a base mesh and a target shape. For example, in facial animation, each Blend Shape corresponds to a specific expression (such as a smile or a blink), and by adjusting weight values, different shapes are blended to generate complex expressions.

[0047] In some embodiments of this application, S220 may include: segmenting and processing the feedback audio using an audio-to-lip-sync algorithm to obtain the facial expression data, the timestamp, and the audio block results.

[0048] For example, in a specific embodiment of this application, the lip-sync algorithm service includes an audio-to-lip-sync algorithm. By inputting the feedback audio into this algorithm, the feedback audio is segmented into blocks, and the corresponding BS data and timestamps are calculated. At the same time, the segmented audio block results are maintained (for example, 10 seconds of audio is segmented into 5 blocks with a period of 2 seconds).

[0049] S230, based on the expression data, the timestamp, the scene configuration, and the emotion tag, render and generate a video frame containing a virtual digital human.

[0050] For example, in a specific embodiment of this application, video frames are generated in real time using the BS data obtained above, along with its corresponding timestamp, the behavior tone and emotion tags of the virtual digital human.

[0051] In some embodiments of this application, S230 may include: sending the expression data, the timestamp, the scene configuration, and the emotion tag to the rendering module via memory sharing; rendering the expression data, the timestamp, the scene configuration, and the emotion tag in real time through the rendering module to obtain the video frame; wherein, the video frame includes multiple frames; each frame in the multiple frames carries the corresponding timestamp.

[0052] For example, in a specific embodiment of this application, the audio segmentation result obtained by cutting the feedback audio into small blocks, along with BS data and timestamps, is transmitted to the UE5 rendering server (as a specific example of a rendering service) via shared memory for real-time rendering to obtain video frames. Each frame in the video frame carries a timestamp. This method can reduce the rendering latency caused by the large audio stream and transmission process while ensuring audio-visual synchronization. It is understandable that by pre-setting a large library of virtual digital human images, backgrounds, actions, expressions, etc., the rendering server can render more realistic and interactive virtual digital humans in real time according to the requirements of the scenario configuration in the dialogue interaction needs, which can greatly improve the user's interactive experience.

[0053] Shared memory is a mechanism that allows multiple processes to access the same physical memory region. This is a highly efficient IPC method because data does not need to be copied between processes; it can be read and written directly to the shared region. In user scenarios, UE5 renders large amounts of visual and audio data; using shared memory can reduce copying overhead and improve real-time performance.

[0054] In some embodiments of this application, S230 may include: using the facial expression data and the timestamp to drive the lip movements of the character model, and customizing the character image and background of the character model based on the scenario configuration and the emotion tag to obtain the video frame.

[0055] For example, in a specific embodiment of this application, the UE5 rendering server can drive the lip movements of the virtual digital human's character model through BS and timestamps, and configure the character's appearance and background through scene settings. A rendering frame is a video frame; the rendered image of the rendering frame is placed in shared memory and the encoding service is notified via a semaphore, that is, the rendering frame is provided to the encoding service through memory sharing.

[0056] S240, the video frame and the feedback audio are encoded to generate a video stream; wherein the video stream is pushed to the client for playback.

[0057] For example, in a specific embodiment of this application, the timestamps in each frame of the video frame and the audio block results are time-aligned to determine that the audio is synchronously encoded into a video stream and pushed to the client 100 for playback.

[0058] In some embodiments of this application, S240 may include: sending the video frame and the audio block result to the encoding module via memory sharing; comparing the timestamps of the video frame and the audio block result and encoding them in real time via the encoding module, and adding the text subtitles to obtain the video stream.

[0059] For example, in a specific embodiment of this application, each rendered frame and audio block result is sent to the encoding service via shared memory. The encoding service can compare the audio within each frame and audio block result using timestamps, encode them in real time, and generate a video stream. The subtitles in the video stream are text subtitles output by the AI ​​large model terminal 200. This real-time encoding and streaming of the rendered image to the encoding server via shared memory significantly reduces latency during transmission and encoding. Client 100 subscribes to the video stream and plays the video image (i.e., the video stream) rendered by the processing server 300, thus completing the interaction with the user.

[0060] The embodiments of this application can be applied to intelligent digital human customer service, digital human standing screen reception in exhibition halls, customized corporate virtual image, VR / AR / MR virtual digital human explanation and companionship, and virtual digital human live streaming or explanation video rendering platforms, etc. Personalized customization of different virtual digital humans can be achieved according to different platforms. The embodiments of this application are not specifically limited here.

[0061] The following is in conjunction with the appendix Figure 3 and Figure 4 The present application provides an exemplary description of the specific process of rendering a virtual digital human, as illustrated in some embodiments thereof.

[0062] Please see the appendix Figure 3 , Figure 3 A flowchart of a method for rendering a virtual digital human is provided for some embodiments of this application; Figure 4 An architecture diagram of a virtual digital human rendered for some embodiments of this application. Figure 4 As can be seen from this, the architecture of virtual digital human rendering includes a front-end (equivalent to client 100), an AI large model (i.e., AI large model client 200), lip-sync algorithm service, UE5 rendering server, and encoding service; the front-end can be a mobile terminal, a glasses terminal, a web terminal, or an IoT device, etc.

[0063] The above process is illustrated below by example.

[0064] The S310 front end responds to user commands and obtains input data.

[0065] The input data includes, for example: Figure 4 The content shown includes voice, text, events, commands, and scenario configurations.

[0066] The S320 inputs the data into the AI ​​model and receives the feedback audio stream and text.

[0067] The AI ​​large-scale model is internally deployed using a knowledge base, FAQs, and scenario-related command terms. The audio stream is the feedback audio, and the text is the subtitles.

[0068] S330 uses a lip-sync algorithm service to process the audio stream and obtain BS data and timestamps.

[0069] The lip-sync algorithm service supports 51BS or 114BS speech-to-lip-sync algorithms.

[0070] S340, the UE5 rendering server receives BS data, timestamps, sentiment predictions, and scene configuration content, and then performs real-time rendering to obtain rendering frames.

[0071] In the UE5 rendering server, the browser window (BS) can drive the lip movements of the virtual digital human character model, and configure the character's appearance and background according to the scene. The rendered frame is a video frame. The rendered image of the frame is placed in shared memory and the encoding service is notified through a semaphore. That is, the rendered frame is given to the encoding service through memory sharing.

[0072] It's important to note that a semaphore is a synchronization mechanism used to coordinate access to shared resources by multiple processes. Semaphores can control the number of processes accessing a resource simultaneously, preventing race conditions. Based on shared memory, semaphores are used to synchronize the operations of UE5 and the encoding service. For example, after UE5 writes data, it notifies the encoding service to read it via a semaphore, avoiding data inconsistency.

[0073] The S350 encoding service encodes the rendered frames, audio chunks, and timestamps in real time to generate a video stream.

[0074] The audio segmentation results and timestamps can be directly provided to the encoding service by the lip-sync algorithm service. The encoding service is a unified video encoding service, which also caches rendering frames for common actions such as standby and greetings.

[0075] S360, the encoding service pushes the video stream to the front end so that the front end can play the video stream.

[0076] It is understood that the specific implementation process of S310~S360 can be referred to the method implementation examples provided above. To avoid repetition, detailed descriptions are omitted here.

[0077] As can be seen from the above embodiments of this application, this application can customize more suitable virtual digital humans according to different business needs and scenarios, realize personalized customization, and be suitable for more scenarios; by combining the character's lip movements, expressions, actions, and tone of voice, the simulation level and interactive experience are improved; by pre-storing images, the rendering pressure on the rendering server can be reduced by 10%-30%; by optimizing the process of this framework, the response time of a single dialogue interaction can be reduced, and the interaction latency can be reduced; at the same time, it can also reduce the human resource costs of customer service, reception, live streaming, etc.

[0078] Please refer to Figure 5 , Figure 5 The diagram illustrates the composition of an apparatus for rendering virtual digital humans according to some embodiments of this application. It should be understood that this apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0079] Figure 5 The virtual digital human rendering apparatus includes at least one software functional module that can be stored in a memory or embedded in the virtual digital human rendering apparatus in the form of software or firmware. The virtual digital human rendering apparatus includes: an acquisition module 510, used to acquire feedback data matching input data from a client; wherein the input data includes user voice, user text, command information, or scene configuration information; the feedback data includes feedback audio, text subtitles, scene configuration, and emotion tags; a processing module 520, used to process the feedback audio to acquire facial expression data and a timestamp; a rendering module 530, used to render and generate video frames containing a virtual digital human based on the facial expression data, the timestamp, the scene configuration, and the emotion tags; and an encoding module 540, used to encode the video frames and the feedback audio to generate a video stream; wherein the video stream is pushed to the client for playback.

[0080] In some embodiments of this application, the acquisition module 510 is used to: receive the feedback data sent by the large model server; wherein the feedback data is generated by inputting the input data into the large model server.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0082] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0083] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0084] like Figure 6As shown, some embodiments of this application provide an electronic device 600, which includes a memory 610, a processor 620, and a computer program stored in the memory 610 and executable on the processor 620. When the processor 620 reads the program from the memory 610 via a bus 630 and executes the program, it can implement the methods of any of the above embodiments.

[0085] Processor 620 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 620 can be a microprocessor.

[0086] The memory 610 can be used to store instructions executed by the processor 620 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 620 of this disclosure embodiment can be used to execute the instructions in the memory 610 to implement the methods shown above. The memory 610 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0087] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0089] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for rendering a virtual digital human, characterized in that, include: Obtain feedback data that matches the input data from the client; wherein the input data includes: user voice, user text, command information, or scene configuration information of a virtual digital human; the feedback data includes feedback audio, text subtitles, scene configuration, and emotion tags; The feedback audio is processed to obtain facial expression data and timestamps; Based on the facial expression data, the timestamp, the scene configuration, and the emotion tag, a video frame containing a virtual digital human is rendered and generated. The video frames and the feedback audio are encoded to generate a video stream; the video stream is then pushed to the client for playback.

2. The method as described in claim 1, characterized in that, The acquisition of feedback data that matches the client's input data includes: The system receives feedback data sent by a large model server; wherein the feedback data is generated by inputting the input data into a large model in the large model server.

3. The method as described in claim 1 or 2, characterized in that, The process of processing the feedback audio to obtain facial expression data and a timestamp includes: The feedback audio is segmented and processed using an audio-to-lip-sync algorithm to obtain the facial expression data, the timestamp, and the audio block results.

4. The method as described in claim 1 or 2, characterized in that, The step of rendering and generating video frames containing a virtual digital human based on the facial expression data, the timestamp, the scene configuration, and the emotion tags includes: The facial expression data, the timestamp, the scene configuration, and the emotion tag are sent to the rendering module via memory sharing. The video frame is obtained by rendering the expression data, the timestamp, the scene configuration, and the emotion tag in real time through the rendering module; wherein, the video frame includes multiple frames; each frame in the multiple frames carries the corresponding timestamp.

5. The method as described in claim 4, characterized in that, The process of rendering the facial expression data, timestamp, scene configuration, and emotion tags in real time through the rendering module to obtain the video frame includes: The facial expression data and timestamps are used to drive the lip movements of the character model, and the character image and background of the character model are customized based on the scene configuration and emotion tags to obtain the video frame.

6. The method as described in claim 3, characterized in that, The step of encoding the video frame and the feedback audio to generate a video stream includes: The video frames and audio block results are sent to the encoding module via memory sharing. The video stream is obtained by comparing the timestamps of the video frames and the audio block results using the encoding module, encoding them in real time, and adding the text subtitles.

7. The method as described in claim 1 or 2, characterized in that, The method further includes: If the encoding module confirms that it contains a pre-stored cached video that matches the feedback data, then the cached video is used as the video stream.

8. A device for rendering a virtual digital human, characterized in that, include: The acquisition module is used to acquire feedback data that matches the input data of the client; wherein, the input data includes user voice, user text, command information or scene configuration information; the feedback data includes feedback audio, text subtitles, scene configuration and emotion tags; The processing module is used to process the feedback audio and obtain facial expression data and timestamps; The rendering module is used to render and generate video frames containing virtual digital humans based on the facial expression data, the timestamp, the scene configuration, and the emotion tags. An encoding module is used to encode the video frames and the feedback audio to generate a video stream; wherein the video stream is pushed to the client for playback.

9. A system for rendering virtual digital humans, characterized in that, include: The client is used to respond to user operation commands and obtain input data; the input data includes user voice, user text, command information, or scene configuration information of virtual digital humans; A large model server is used to input the input data into a large model to generate feedback data; the feedback data includes feedback audio, text subtitles, scene configuration, and emotion tags. A processing server is used to process the feedback audio and obtain facial expression data and timestamps; Based on the facial expression data, the timestamp, the scene configuration, and the emotion tag, a video frame containing a virtual digital human is rendered and generated. The video frames and the feedback audio are encoded to generate a video stream; the video stream is then pushed to the client for playback.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-5.

Images