Digital human driving method, device, system, equipment, program product and medium

Abstract

The invention provides a digital human driving method, device, system and equipment, a program product and a medium, and relates to the technical field of digital human. The method comprises the steps that according to an obtained instruction, a driving mode corresponding to a target digital human is switched into a target driving mode, different driving modes support different driving modes, and the driving modes comprise a face driving mode and / or a limb driving mode; and driving the target digital person according to the target driving mode. According to the scheme, the problem that a digital human driving method in the related technology does not support digital human driving mode switching is solved.

Description

Technical Field

[0001] The present invention relates to the field of digital human technology, and in particular to a digital human driving method, device, system, equipment, program product and medium. Background Art

[0002] At present, the scenarios of the digital human system are mainly oriented to digital human generation and production, digital human live broadcast and customer service, etc. In the social scenario, Meta has created an open virtual reality (VR) social platform HorizonWorlds. With the help of complete three-dimensional (3D) motion capture technology and handheld motion controllers, it allows Oculus users to roam in the open world in the form of their avatars. Horizon players can use the world generation system in the game to develop their own virtual works. However, the social system can only complete motion capture and handle operation, and does not support facial or voice-driven functions.

[0003] In a social scenario with multiple people connected, the digital human driving method in the related art does not support digital human driving mode switching. Summary of the invention

[0004] The purpose of the present invention is to provide a digital human driving method, device, system, equipment, program product and medium, which solves the problem that the digital human driving method in the related art does not support the switching of digital human driving modes.

[0005] To achieve the above object, an embodiment of the present invention provides a digital human driving method, comprising:

[0006] According to the acquired instruction, the driving mode corresponding to the target digital human is switched to the target driving mode, wherein different driving modes support different driving modes, and the driving modes include: a facial driving mode and / or a limb driving mode;

[0007] According to the target driving mode, the target digital human is driven.

[0008] To achieve the above-mentioned purpose, an embodiment of the present invention provides a digital human driving device, comprising:

[0009] A first processing module is used to switch the driving mode corresponding to the target digital human to the target driving mode according to the acquired instruction, wherein different driving modes support different driving modes, and the driving modes include: facial driving mode and / or limb driving mode;

[0010] The second processing module is used to drive the target digital human according to the target driving mode.

[0011] To achieve the above objective, an embodiment of the present invention provides a device, including a processor and a transceiver, wherein the processor is configured to:

[0012] According to the acquired instruction, the driving mode corresponding to the target digital human is switched to the target driving mode, wherein different driving modes support different driving modes, and the driving modes include: a facial driving mode and / or a limb driving mode;

[0013] According to the target driving mode, the target digital human is driven.

[0014] To achieve the above-mentioned purpose, an embodiment of the present invention provides a device, including a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the digital human driving method as described above is implemented.

[0015] In order to achieve the above-mentioned object, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the above-mentioned digital human driving method.

[0016] To achieve the above-mentioned purpose, an embodiment of the present invention provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the above-mentioned digital human driving method are implemented.

[0017] The beneficial effects of the above technical solution of the present invention are as follows:

[0018] The method of the embodiment of the present invention supports different digital human driving modes. Different driving modes support different driving methods. The driving mode corresponding to the target digital human can be switched according to the acquired instructions. In this way, different social relationships can be distinguished in a virtual scene with multiple people connected, and differentiated driving can be achieved, thereby better simulating real social scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a digital human driving method according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the architecture of a digital human driving method according to an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a media processing module according to an embodiment of the present invention;

[0022] Figure 4 is a structural diagram of a media interaction module according to an embodiment of the present invention;

[0023] Figure 5 A flowchart of a digital human driving method according to another embodiment of the present invention;

[0024] Figure 6 A flowchart of a digital human driving method according to another embodiment of the present invention;

[0025] Figure 7 A schematic diagram of the architecture of a digital human driving method according to another embodiment of the present invention;

[0026] Figure 8 A structural diagram of a digital human driving device according to an embodiment of the present invention;

[0027] Fig. 9 is a structural diagram of a device according to an embodiment of the present invention;

[0028] Fig.10 This is a structural diagram of a device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0030] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0031] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0032] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0033] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0034] like Figure 1 As shown, a digital human driving method according to an embodiment of the present invention includes:

[0035] Step 101: according to the acquired instruction, switch the driving mode corresponding to the target digital human to the target driving mode, wherein different driving modes support different driving modes, and the driving modes include: facial driving mode and / or limb driving mode.

[0036] Here, the target digital person is the virtual digital person image corresponding to the target user, and the instruction is used to instruct the driving mode to be switched to the target driving mode. The instruction can be triggered by the user through the terminal. For example, a default driving mode can be set, and the user can switch the driving mode through the terminal (for example, by clicking or sliding the controls on the terminal interface); it can also be automatically triggered as the virtual scene where the target digital person is located changes (using map anchor points, plot nodes, etc.). For example, in a certain social virtual scene, the instruction is automatically triggered when the digital person walks to one or some map anchor points in the scene.

[0037] Exemplarily, the driving mode may include a first driving mode and a second driving mode, wherein the first driving mode supports a facial driving mode, or supports a limb driving mode and a facial driving mode, and the second driving mode supports a limb driving mode.

[0038] It should be noted that in order to better simulate the real social mode in social scenes, the corresponding driving mode can be adopted according to the intimacy. For example, social relationships can be divided into at least friends and passers-by according to the intimacy, and the two social relationships correspond to two different driving modes. Among them, in order to be closer to the real social experience, when the social relationship is friends, the first driving mode that supports the facial driving mode, or supports the body driving mode and the facial driving mode can be adopted; when the social relationship is a passer-by, the second driving mode that supports the body driving mode can be adopted.

[0039] Step 102: driving the target digital human according to the target driving mode.

[0040] Here, according to the target-driven mode, the target digital person can be driven to conduct social communication in a virtual scene. The virtual scene can specifically include social scenes with multiple people accessing, such as meeting scenes and community scenes.

[0041] It should be noted that in the embodiment of the present invention, different driving modes support different driving methods, that is, the embodiment of the present invention can support the selection and switching of expression animation mode (corresponding to facial driving mode) and action animation mode (corresponding to limb driving mode), and can distribute related animations and render digital humans according to different paths / ports. Specifically, media data (that is, social interaction data) can be generated based on the input information (images, audio, etc.) of the target user, the target digital human (that is, the digital human model), and the digital scene, and distributed to the social objects in the virtual scene to achieve social communication.

[0042] It should also be noted that different driving modes are implemented in completely different technical ways, and the driving data that needs to be transmitted is also different. The embodiments of the present invention distinguish different driving modes at the application level, which is conducive to improving the utilization of computing resources and optimizing network transmission efficiency.

[0043] In this embodiment, different digital human driving modes are supported. Different driving modes support different driving methods. The driving mode corresponding to the target digital human can be switched according to the acquired instructions. In this way, different social relationships can be distinguished in a virtual scene with multiple people connected, and differentiated driving can be achieved, thereby better simulating real social scenes.

[0044] In some embodiments, step 102, driving the target digital human according to the target driving mode, may specifically include the following steps:

[0045] Step 1021, obtaining input information of a target user corresponding to the target digital human.

[0046] Here, the input may support multimodal methods, and the input information may include voice, image, text, preset information, etc. The input information may be obtained through a camera, microphone or other input device.

[0047] Step 1022: According to the input information of the target user, the driving information corresponding to the target digital human in the target driving mode is obtained, where the driving information includes facial driving information and / or limb driving information.

[0048] For example, if the target driving mode is the first driving mode, then in step 1022, facial driving information and limb driving information need to be generated to drive the face and limbs of the target digital person for social communication; if the target driving mode is the second driving mode, then in step 1022, limb driving information needs to be generated to drive the limbs of the target digital person for social communication.

[0049] It should be noted that, when generating the driving information in step 1022, the driving information may be generated directly according to the input information, or reasoning may be performed according to the input information, and then the driving information may be generated according to the obtained reasoning result.

[0050] Step 1023: driving the target digital human according to the driving information.

[0051] In this embodiment, after obtaining the input information of the target user, corresponding driving information in the target driving mode can be generated, so that the corresponding parts of the target digital human can be driven according to the generated driving information to realize social communication.

[0052] In some specific embodiments, the application server executes the digital human driving method provided by the embodiment of the present invention, or, Figure 2 As shown, the digital human media service platform is deployed on a server to provide digital human media services, and when the digital human media service platform is running, the server executes the digital human driving method provided by the embodiment of the present invention. Here, the server can be a cloud server, or it can include a central cloud server and an edge cloud server.

[0053] Specifically, the application server (or digital human media service platform) includes a digital human content management module, a media processing module and a media interaction module. Here, the main functions of each module are described as follows:

[0054] (1) Content management module: supports content management related affairs of the digital human platform and solves the unified data structure (the digital human model follows A=A facial +A body structure, that is, facial data and limb data are stored separately), and manage the content reception, provision and storage / caching of different nodes, as well as the customization and credibility processing of digital human models.

[0055] (2) Media processing module: Figure 3 As shown, this module provides media processing functions for digital human services and applications. For example, non-real-time media processing can be provided on the central cloud node (i.e., the central cloud server), and real-time media processing can be provided on the edge node (i.e., the edge cloud server), such as digital human generation, driving, rendering, encoding and compression. It should be noted that these functions provide corresponding 3D model animations based on different driving modes, and the driving modes can be selected and switched, rendering a 3D model with a user view for display.

[0056] (3) Media interaction module: Figure 4 As shown, this module can handle the sharing of digital human models and / or media data between different central cloud servers or edge cloud servers, as well as recognize driving mode features to complete reasoning before driving in different modes. Functions such as multimodal data ingestion, multimodal data reasoning, and multimodal data encapsulation can be deployed on this module. These functions provide media collection and instruction recognition for driving modes to enable corresponding reasoning and driving engines, and encapsulate videos, images, texts, and audios after real-time animation and rendering to achieve media sharing.

[0057] It should be noted that the above-mentioned content management module, media processing module, media interaction module and other modules may also be provided in the client, and are not limited to the application server (or digital human media service platform).

[0058] The following describes the application of the embodiments of the present invention in combination with specific scenarios:

[0059] like Figure 5As shown, in a specific embodiment, after the user accesses in the second driving mode (supporting the limb driving mode) by default, the user can switch to the first driving mode (supporting the facial driving mode, or supporting the limb driving mode and the facial driving mode, involving the facial driving information I facial and limb drive information I body 、Facial AnimationA facial and Body Animation A body , and scene S1) to conduct social activities such as speeches and exchanges in virtual scenes. Figure 2 The specific process is as follows:

[0060] S501, application loading: the user sends a social request through the client, and obtains the digital human model from the content management module through the media interaction module (see Figure 2 The process shown in M-3 in the figure) is selected to determine the digital human model corresponding to the user, and the digital human model is loaded synchronously (see Figure 2 The process shown in M-1).

[0061] It should be noted that the digital human model in the embodiment of the present invention supports facial animation A facial and Body Animation A body In addition, the application server (or digital human media service platform) needs to obtain the management authority of the client's camera and microphone to obtain the user's input information.

[0062] S502, default setting: the client accesses the social scene, and the initial state is set to the second driving mode by default (supporting body driving mode, involving body driving information I body 、Body Animation A body and scene SN), according to the social scene coordinate system, the media processing module body drive engine is started through the media interaction module (see Figure 2 The media interaction module can set the codec configuration of the media processing module according to the network resources and the client report, and forward the media or instructions to the media processing module (see Figure 2 The media processing module extracts the limb drive information I according to the media or instructions. body , generate body animation A body , and, according to the scene SN rendering, the digital human media is encoded and returned to the media interaction module for packaging and streaming.

[0063] S503, user setting: The user can set the driving mode in the application through the client (for example, set it to the first driving mode). After the client sends the driving mode instruction (see Figure 2In the process shown in I-0, the media interaction module identifies the driving mode instruction: if the driving mode instruction indicates that the driving mode is switched to the first driving mode, and the first driving mode supports the facial driving mode and the limb driving mode, the facial + limb driving engine and scene rendering of the media processing module are started; if the driving mode instruction indicates that the driving mode is switched to the first driving mode, and the first driving mode supports the facial driving mode, the facial driving engine and scene rendering of the media processing module are started.

[0064] S504, driver processing: the media interaction module and the media processing module obtain microphone\camera\other inputs according to the settings (see Figure 2 The process shown in I-0 in FIG. 1 is to extract information through multimodal data reasoning and digital human model driving to drive the face and / or limbs of the digital human. Taking the first driving mode as an example, the media interaction module receives the audio, video, text, and gesture instructions from the client. If question-answering or content generation is required, the media interaction module can choose to make inference decisions on the user's input information, generate corresponding text or audio, and then forward it to the media processing module. The media processing module extracts facial driving information I according to the media or instructions. facial and limb drive information I body , generate facial animation A facial and Body Animation A body , and synthesize and render the digital human media according to the scene S1, encode it and return it to the media interaction module. If the media package needs to be transmitted, it needs to be streamed.

[0065] S505, display output: the media interaction module can output the multimodal digital human animation and publish it to the client for display.

[0066] like Figure 6 As shown, in a specific embodiment, the virtual scene is a conference scene. When the target user, as a speaker, is giving a speech and communicating, he can choose to use the first driving mode to drive the target digital human (i.e., drive the face and limbs of the target digital human) and distribute it to other participants (i.e., send the media data generated by the driving, rendering, etc. processes to other participants), while the participants can choose the second driving mode to drive their corresponding digital human and distribute it to others. After the target user finishes his speech, he can choose to switch the driving mode to the second driving mode (i.e., drive the limbs of the target digital human). In this way, the target user, as a passerby or participant, no longer drives the face of the target digital human, but only drives the limbs.

[0067] S601, exit the original mode: the user can choose to actively exit or passively exit the first driving mode, triggering the media interaction module to instruct the media processing module to exit the original driving and rendering engine, that is, the facial animation A facialSet to default, only keep body animation A body .

[0068] S602, select a new mode: the user enters the second driving mode, updates the media interaction module settings, and starts the media processing module limb driving engine and scene rendering through the media interaction module.

[0069] S603, update drive processing: the media interaction module and the media processing module can drive the digital human limbs according to the microphone, camera input or other input, through AI reasoning (not necessary) and posture information extraction. The media interaction module receives the client audio, video, text, and posture instructions. If question-answering or content generation is required, the media interaction module can choose to make reasoning decisions on the input, generate corresponding text or audio, and forward it to the media processing module. The media processing module extracts limb driving information according to the media or instructions. body , generate body animation A body , and according to the scenario S N The digital human media is synthesized and rendered, and returned to the media interaction module for packaging. If transmission is required, the media needs to be streamed.

[0070] S604, display output: the media interaction module outputs the multimodal digital human animation for display on the client.

[0071] In the above embodiments, the digital human driving method generally meets the processes of triggering-pre-processing-driving-post-processing-rendering and display. Each node processes according to the input media and instructions and outputs corresponding media and instructions, thereby improving the versatility of the technology.

[0072] In some embodiments, the method further includes: in a target virtual scene, determining a target communication object of the target digital human, the target communication object being a digital human other than the target digital human in the target virtual scene; respectively acquiring first media data and second media data of the target digital human, sending the first media data to a terminal device corresponding to the target communication object, and sending the second media data to a terminal device corresponding to at least one first digital human, the first digital human being being a digital human other than the target digital human and the target communication object in the target virtual scene; wherein the first media data includes a digital human image generated by the target digital human in a first driving mode, the first driving mode supporting the facial driving mode, or supporting the facial driving mode and the limb driving mode; the second media data includes a digital human image generated by the target digital human in a second driving mode, the second driving mode supporting the limb driving mode.

[0073] It should be noted that the target virtual scene here can be a social scene. In this scene, the target digital human and the target communication object may use the first driving mode for social communication in face-to-face communication, that is, the body and face driving mode, while the target digital human can use the second driving mode for passers-by (that is, the first digital human mentioned above), that is, the body driving mode only. In this way, in the same scene, a differentiated digital human-driven immersive experience can be achieved, which is more in line with the actual social scene and can simulate a real multi-person social experience.

[0074] In this embodiment, in a target virtual scene accessed by multiple people, a driving mode can be set for a social object (target communication object, first digital human), driving information and digital human media data can be generated according to the driving methods of different driving modes, and the media data (including the rendered digital human image) can be distributed according to different paths / ports (for example, the first media data is sent to the terminal device corresponding to the target communication object, and the second media data is sent to the terminal device corresponding to at least one first digital human). In this way, the first driving mode (corresponding to the facial driving mode and the body driving mode) is adopted in a one-to-one social scene with the target communication object, and the second driving mode (corresponding to the body driving mode) is adopted in a passerby social scene with at least one first digital human, which can better distinguish different social relationships, make the social communication in the target virtual scene more in line with the actual social scene, and help improve the utilization efficiency of platform computing resources and optimize network transmission efficiency.

[0075] It should be noted that different driving modes are technically implemented in completely different ways, and the driving data that needs to be transmitted is also different. The digital human driving method of the embodiment of the present invention distinguishes the driving modes at the application level, solves the contradiction between the different technical paths of digital human full-body driving and facial driving, improves the technical versatility, and lowers the threshold for technical application.

[0076] like Figure 7 As shown in another specific embodiment, in a digital human multi-access application based on a central cloud and an edge cloud, the three modules of the system (digital human content management module, media processing module, and media interaction module) can be deployed on different central cloud and edge cloud servers according to computing resources and data management to realize proprietary functions. N , the content management module of the central cloud provides the digital human model (A0=A facial0 +A body0 Structure), the content management of the edge cloud can obtain content management rights from the central cloud through the steps shown in M-0. The following is a description of the various processes involved in this embodiment:

[0077] I-0: The user sends a drive mode instruction on the client through the Hypertext Transfer Protocol (HTTP), indicating that the drive mode is set to the first drive mode, sends user video and audio, text and operation data, and receives the digital human media output by the system for display on the client.

[0078] I-1, I-2: The central cloud and edge cloud media interaction modules obtain platform or user media data, perform pre-drive processing, and forward instructions and media.

[0079] M-0: The digital human model is stored or cached in the central cloud node and edge cloud node, and the digital human model follows the unified A=A facial +A body structure.

[0080] P-1: The edge cloud forwards the client's instructions and forwards the client media and the inference results processed by the media interaction module; obtains the digital human media (A = A facial +A body ), and scene rendering S+A, for encapsulation and streaming, supporting protocols such as Dynamic Adaptive Streaming over HTTP (DASH) protocol, HTTP Live Streaming (HLS) protocol, and Real-Time Transport Protocol (RTP) protocol.

[0081] P-2: The central cloud obtains the trusted digital human rendered by the media processing module for sharing with other central clouds or edge cloud nodes.

[0082] M-1: The central cloud content management module sends the digital human model to the media processing for non-real-time digital human generation, editing, rendering and trust processing. The generation can support image-based and model-based digital human generation technology, and can further facial0 +A body0 The structure stores the generated digital human model A0, renders the fused model, and adds some details and effects to make the generated digital human more realistic and natural.

[0083] There are many rendering methods, such as texture mapping, lighting model, shadow effect, reflection effect, etc., which can be implemented using existing engines such as UE. Trusted processing can be achieved through technologies such as digital watermarking, DRM, and biometric matching.

[0084] M-2: The edge cloud caches the digital human model (A0 = Afacial0 +A body0 ) is sent to the media processing module for driving, rendering, encoding and other operations. Among them, the driving method supports internal driving and external driving. Internal driving refers to the use of the digital human's own intelligent system to automatically read and parse and identify external input information, and decide the subsequent output text of the digital human based on the analysis results, and drive the character model to generate corresponding voice and action to enable the digital human to interact with the user, such as voice, text or other sensor driving. At this time, the input media needs to be deep learned to predict and extract facial I facial and Limb I body Driving information or coefficients; external drive refers to collecting real-life action and expression data through external devices or manual operation, and then converting these data into control signals for the digital human model: Facial I facial and Limb I body Driving information, thereby driving the digital human to perform corresponding actions and expressions, including actions, faces, sounds, etc., such as based on BlendShape and bones, supported in engines such as Unity and Unreal. In addition, if the rendered digital human media needs to be transmitted, it is compressed and encoded, such as using 3D to 2D High Efficiency Video Codec (HEVC) encoding or video based point cloud compression (VPCC) codec.

[0085] M-3: Obtain the digital human model from the content management module for application loading.

[0086] In some embodiments, before switching the drive mode corresponding to the target digital human to the target drive mode according to the acquired instruction, the method further includes: acquiring the target digital human according to user indication information; wherein the user indication information is used to indicate the digital human selected by the target user, and different digital humans are stored in the form of a combined data structure of facial data and limb data.

[0087] It should be noted that the digital human model can be structured according to the face and limbs. For example, when storing a digital human (i.e., a digital human model), it is stored in the form of a combined data structure of facial data and limb data. In this way, in different driving modes, different driving engines can be called according to the driving methods supported by the driving modes to process and transmit different parts of the digital human model structure (facial data and / or limb data), thereby realizing differentiated digital human driving, which is beneficial to the optimization of computing resource utilization and network transmission efficiency.

[0088] In this embodiment, the digital human can be stored as a unified data structure, that is, facial data and corresponding limb data are stored separately, and the data can be processed on demand according to the driving mode. For example, in the second driving mode, the limb data is acquired and processed separately, thereby reducing the transmission of unnecessary data and improving the utilization rate of computing resources and network transmission efficiency.

[0089] In application development, as a metaverse entrance for multiple people to access, this method can be used to optimize business capability deployment and business processes, providing users with a better immersive experience.

[0090] The digital human driving method of this embodiment supports setting or switching different digital human driving modes. Different driving modes support different driving methods. The driving mode corresponding to the target digital human can be switched according to the acquired instructions. In this way, different social relationships can be distinguished in a virtual scene with multiple people connected, and differentiated driving can be achieved, thereby better simulating real social scenes.

[0091] A digital human driving system according to an embodiment of the present invention includes:

[0092] Content management module, media processing module and media interaction module;

[0093] Wherein, the content management module, the media processing module and the media interaction module are deployed on at least one server;

[0094] The content management module is used to: structure the data of the digital human, and manage and store the data related to the digital human;

[0095] The media processing module is used to: switch the driving mode corresponding to the target digital human, and according to the switched target driving mode, drive and render the target digital human corresponding to the target user from the user perspective of the target user to obtain target media data, wherein different driving modes support different driving methods, and the driving methods include: facial driving method and / or limb driving method;

[0096] The media interaction module is used to encapsulate the target media data and send the target media data to the target terminal according to the target driving mode.

[0097] Here, the media interaction module can encapsulate and stream the target media data.

[0098] It should be noted that the content management module, the media processing module and the media interaction module are deployed on at least one server, for example, Figure 7 As shown, it can be deployed on different central cloud servers and edge cloud servers.

[0099] The above-mentioned content management module can be specifically responsible for content management related matters in the digital human driving system, managing the content reception, provision and storage / caching of different nodes, as well as the customization and trustworthiness processing of the digital human model.

[0100] The media processing module can be used to provide media processing functions for digital human services and applications. For example, it can provide non-real-time media processing on the central cloud server and real-time media processing on the edge cloud server, such as digital human generation, driving, rendering, encoding and compression.

[0101] Media Interaction Module This module can be used to process digital human model sharing and / or media data sharing between different central cloud servers or edge cloud servers, as well as identify driving mode features to complete reasoning before driving in different modes. This module can be deployed with functions such as multimodal data ingestion, multimodal data reasoning, and multimodal data encapsulation, which provide media collection and instruction recognition for driving modes to enable corresponding reasoning and driving engines, and encapsulate video, images, text, and audio after real-time animation and rendering, thereby achieving media sharing.

[0102] In some embodiments, the media interaction module is further used to: obtain multimodal media data corresponding to the target user, and perform reasoning on the multimodal media data to obtain a reasoning result;

[0103] The media processing module is also used to: switch the driving mode corresponding to the target digital human and the inference result, and according to the switched target driving mode, drive and render the target digital human corresponding to the target user from the user perspective of the target user to obtain target media data.

[0104] The digital human driving system of this embodiment supports different digital human driving modes. Different driving modes support different driving methods. The driving mode corresponding to the target digital human can be switched according to the acquired instructions. In this way, different social relationships can be distinguished in a virtual scene with multiple people connected, and differentiated driving can be achieved, thereby better simulating real social scenes.

[0105] like Figure 8 As shown, a digital human driving device according to an embodiment of the present invention includes:

[0106] The first processing module 810 is used to switch the driving mode corresponding to the target digital human to the target driving mode according to the acquired instruction, wherein different driving modes support different driving modes, and the driving modes include: facial driving mode and / or limb driving mode;

[0107] The second processing module 820 is used to drive the target digital human according to the target driving mode.

[0108] Optionally, the second processing module 820 includes:

[0109] An information acquisition unit, used to acquire input information of a target user corresponding to the target digital human;

[0110] A drive extraction unit, configured to obtain drive information corresponding to the target digital human in the target drive mode according to the input information of the target user, wherein the drive information includes facial drive information and / or limb drive information;

[0111] The digital human driving unit is used to drive the target digital human according to the driving information.

[0112] Optionally, the device further comprises:

[0113] A third processing module is used to determine a target communication object of the target digital human in a target virtual scene, wherein the target communication object is a digital human other than the target digital human in the target virtual scene;

[0114] A data distribution module, used to respectively obtain the first media data and the second media data of the target digital person, send the first media data to the terminal device corresponding to the target communication object, and send the second media data to the terminal device corresponding to at least one first digital person, the first digital person being a digital person other than the target digital person and the target communication object in the target virtual scene;

[0115] The first media data includes a digital human image generated by the target digital human in a first driving mode, and the first driving mode supports the facial driving mode, or supports the facial driving mode and the limb driving mode;

[0116] The second media data includes a digital human image generated by the target digital human in a second driving mode, and the second driving mode supports the limb driving method.

[0117] Optionally, the device further comprises:

[0118] A first acquisition module, used to acquire the target digital human according to user instruction information;

[0119] The user indication information is used to indicate the digital human selected by the target user, and different digital humans are stored in the form of a combined data structure of facial data and body data.

[0120] It should be noted here that the above-mentioned digital human driving device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned digital human driving method embodiment, and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0121] like Fig. 9 As shown, a device 900 according to an embodiment of the present invention includes a processor 910 and a transceiver 920, wherein the processor 910 is configured to:

[0122] According to the acquired instruction, the driving mode corresponding to the target digital human is switched to the target driving mode, wherein different driving modes support different driving modes, and the driving modes include: a facial driving mode and / or a limb driving mode;

[0123] According to the target driving mode, the target digital human is driven.

[0124] Optionally, when the processor 910 drives the target digital human according to the target driving mode, it is specifically configured to:

[0125] Obtaining input information of a target user corresponding to the target digital human;

[0126] According to the input information of the target user, obtaining the driving information corresponding to the target digital human in the target driving mode, wherein the driving information includes facial driving information and / or limb driving information;

[0127] The target digital human is driven according to the driving information.

[0128] Optionally, the processor 910 is further configured to:

[0129] In a target virtual scene, determining a target communication partner of the target digital person, wherein the target communication partner is a digital person other than the target digital person in the target virtual scene;

[0130] Respectively acquiring first media data and second media data of the target digital human, sending the first media data to a terminal device corresponding to the target communication object, and sending the second media data to a terminal device corresponding to at least one first digital human, the first digital human being being a digital human other than the target digital human and the target communication object in the target virtual scene;

[0131] The first media data includes a digital human image generated by the target digital human in a first driving mode, and the first driving mode supports the facial driving mode, or supports the facial driving mode and the limb driving mode;

[0132] The second media data includes a digital human image generated by the target digital human in a second driving mode, and the second driving mode supports the limb driving method.

[0133] Optionally, the processor 910 is further configured to:

[0134] According to the user's instruction information, obtain the target digital human;

[0135] The user indication information is used to indicate the digital human selected by the target user, and different digital humans are stored in the form of a combined data structure of facial data and body data.

[0136] It should be noted here that the above-mentioned device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned digital human driving method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0137] A device according to another embodiment of the present invention, such as Fig.10 As shown, it includes a transceiver 1010, a processor 1000, a memory 1020, and a program or instruction stored in the memory 1020 and executable on the processor 1000; when the processor 1000 executes the program or instruction, the above-mentioned digital human driving method is implemented.

[0138] The transceiver 1010 is used to receive and send data under the control of the processor 1000.

[0139] Among them, Fig.10 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1000 and memory represented by memory 1020. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1010 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 1000 is responsible for managing the bus architecture and general processing, and the memory 1020 may store data used by the processor 1000 when performing operations.

[0140] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0141] A computer-readable storage medium according to an embodiment of the present invention stores a program or instruction thereon, and when the program or instruction is executed by a processor, the steps in the digital human driving method described above are implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0143] In the embodiment of the present invention, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0144] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0145] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0146] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present invention. Therefore, the present invention should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present invention will be perfect and complete, and the scope of the present invention will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "including" and / or "comprising" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0147] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A digital human driving method, characterized in that: include: According to the acquired instruction, the driving mode corresponding to the target digital human is switched to the target driving mode, wherein different driving modes support different driving modes, and the driving modes include: a facial driving mode and / or a limb driving mode; According to the target driving mode, the target digital human is driven.

2. The method according to claim 1, characterized in that The step of driving the target digital human according to the target driving mode includes: Obtaining input information of a target user corresponding to the target digital human; According to the input information of the target user, obtaining the driving information corresponding to the target digital human in the target driving mode, wherein the driving information includes facial driving information and / or limb driving information; The target digital human is driven according to the driving information.

3. The method according to claim 1, characterized in that The method further comprises: In a target virtual scene, determining a target communication partner of the target digital person, wherein the target communication partner is a digital person other than the target digital person in the target virtual scene; Respectively acquiring first media data and second media data of the target digital human, sending the first media data to a terminal device corresponding to the target communication object, and sending the second media data to a terminal device corresponding to at least one first digital human, the first digital human being being a digital human other than the target digital human and the target communication object in the target virtual scene; The first media data includes a digital human image generated by the target digital human in a first driving mode, and the first driving mode supports the facial driving mode, or supports the facial driving mode and the limb driving mode; The second media data includes a digital human image generated by the target digital human in a second driving mode, and the second driving mode supports the limb driving method.

4. The method according to claim 1, characterized in that Before switching the driving mode corresponding to the target digital human to the target driving mode according to the acquired instruction, the method further includes: According to the user's instruction information, obtain the target digital human; The user indication information is used to indicate the digital human selected by the target user, and different digital humans are stored in the form of a combined data structure of facial data and body data.

5. A digital human driving system, characterized in that: include: Content management module, media processing module and media interaction module; Wherein, the content management module, the media processing module and the media interaction module are deployed on at least one server; The content management module is used to: structure the data of the digital human, and manage and store the data related to the digital human; The media processing module is used to: switch the driving mode corresponding to the target digital human, and according to the switched target driving mode, drive and render the target digital human corresponding to the target user from the user perspective of the target user to obtain target media data, wherein different driving modes support different driving methods, and the driving methods include: facial driving method and / or limb driving method; The media interaction module is used to encapsulate the target media data and send the target media data to the target terminal according to the target driving mode.

6. A digital human driving device, characterized in that: include: A first processing module is used to switch the driving mode corresponding to the target digital human to the target driving mode according to the acquired instruction, wherein different driving modes support different driving modes, and the driving modes include: facial driving mode and / or limb driving mode; The second processing module is used to drive the target digital human according to the target driving mode.

7. A device, characterized in that: include: transceivers and processors; The processor is used to: According to the acquired instruction, the driving mode corresponding to the target digital human is switched to the target driving mode, wherein different driving modes support different driving modes, and the driving modes include: a facial driving mode and / or a limb driving mode; According to the target driving mode, the target digital human is driven.

8. A device comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; characterized in that when the processor executes the program or instruction, the digital human driving method as described in any one of claims 1-4 is implemented.

9. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the digital human driving method as described in any one of claims 1 to 4.

10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by the processor, the steps in the digital human driving method as described in any one of claims 1 to 4 are implemented.

Images