Method for providing digital human, system, and computing device cluster

Abstract

The present application relates to the technical field of cloud computing, and provides a method for providing a digital human, a system, and a computing device cluster. The method can provide users with clear and smooth real-time digital human images in a media scene based on the real-time audio and video technology, especially can provide users with high-image-quality and smooth real-time digital humans even under the conditions of limited network bandwidth and a large quality difference between an uplink transmission network and a downlink transmission network.

Description

A method, system and computing device cluster for providing digital human

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 6, 2023, with application number 202310827688.X and invention name “A method, apparatus and computing device cluster for providing real-time digital humans”, the entire contents of which are incorporated by reference into this application.

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 28, 2023, with application number 202311616005.2 and invention name “A method, system and computing device cluster for providing digital humans”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of cloud computing technology, and in particular to a method, system, computing device cluster, computer-readable storage medium, and computer program product for providing a digital human. Background Art

[0004] A digital human is an entity that exists in the non-physical world and can be driven by a real person or computer. This entity is created based on artificial intelligence technologies such as computer graphics, image rendering, motion capture, deep learning, speech synthesis, and portrait modeling. It possesses the ability to perceive, recognize, and express the physical world, and can achieve functions such as human-computer interaction through electronic screens, VR, and AR devices. Furthermore, a real person's voice, expression, and movements in the real world can be used to drive the digital human to produce the same voice, expression, and movement in real time, allowing the driven digital human to reproduce the real person's performance in the real world in real time. Such a digital human can also be called a real-time digital human.

[0005] With the development of digital human applications, how to provide users with clear and smooth real-time digital humans in media scenarios based on real-time audio and video technology, such as online meetings, live broadcasts, online classes, and real-time audio and video calls, has become a technical problem that needs to be solved urgently.

[0006] Summary of the Invention

[0007] This application proposes a method for providing a digital human, capable of providing users with a clear, smooth, real-time digital human image in media scenarios based on real-time audio and video technology, such as online conferences, live broadcasts, online classrooms, and real-time audio and video calls. In particular, this method can provide high-quality, smooth real-time digital humans even in unstable, weak, or bandwidth-constrained network conditions. This application also provides a corresponding system, computing device cluster, computer-readable storage medium, and computer program product.

[0008] In a first aspect, the present application proposes a method for providing a digital human. The method includes: a source terminal acquiring media data of a real person, the media data of the real person including video and / or audio of the real person; the source terminal extracting feature data describing the real person from the media data of the real person, the feature data including one or more of facial features, morphological features, and voice features of the real person, the facial features and morphological features of the real person being extracted from the video of the real person, and the voice features of the real person being extracted from the audio of the real person; the source terminal sending the feature data to a cloud server; the cloud server generating media data of the digital human based on the feature data and a multimodal AI-driven model, the media data of the digital human including video and / or audio of the digital human; and the cloud server sending the media data of the digital human to a destination terminal.

[0009] In the above method, the bandwidth required for uplink transmission is reduced by using feature data instead of real-person video during uplink transmission, so that even in the case of poor uplink network quality, the feature data can still reach the cloud quickly and losslessly, thereby providing users with high-definition real-time digital humans.

[0010] In one possible implementation, the source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person according to a certain period; the source terminal extracts feature data for describing the real person from the media data of the real person, including: the source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in the facial features of the real person and changes in the morphological features of the real person.

[0011] In the above implementation, by extracting only feature data containing feature changes from live video based on the acquisition cycle, the bandwidth required for uplink transmission of the feature data is further reduced.

[0012] In one possible implementation, before the cloud server generates media data of the digital human based on the feature data and the multimodal AI-driven model, the method further includes: the cloud server obtains input information on the destination terminal, and the input information indicates that the media data of the digital human is generated by the cloud server.

[0013] In one possible implementation, the method further includes: the source terminal receives first input information, the first input information indicates a feature type, and the feature type includes one or more facial features, morphological features, and sound features; the source terminal extracts the feature data used to describe the real person from the media data of the real person, including: the source terminal extracts target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0014] In one possible implementation, the method further includes: the cloud server obtaining second input information on the source terminal, where the second input information indicates the style of the digital human; the cloud server generating media data of the digital human based on the feature data and the multimodal AI-driven model, including: the cloud server generating media data of the digital human of the corresponding style based on the feature data, the multimodal AI-driven model and the second input information.

[0015] In one possible implementation, the cloud server sends the video of the digital human to a destination terminal, including: the cloud server determines the transmission bit rate of the video of the digital human based on the network status of the destination terminal; the cloud server encodes the video of the digital human based on the transmission bit rate of the video of the digital human, wherein the clarity of the first area and the clarity of the second area in the picture of the encoded video of the digital human are different; and the cloud server sends the encoded video of the digital human to the destination terminal.

[0016] In the above implementation, through the generative coding capability of the cloud, the clarity of the digital human image can be guaranteed as much as possible when the downlink transmission network status is poor, so that users can have a better digital human media service experience.

[0017] In a second aspect, the present application proposes another method for providing a digital human. The method includes: a source terminal obtains media data of a real person, the media data of the real person including the video and / or audio of the real person; the source terminal extracts feature data describing the real person from the media data of the real person, the feature data including one or more of the facial features, morphological features, and voice features of the real person, the facial features and morphological features of the real person being extracted from the video of the real person, and the voice features of the real person being extracted from the audio of the real person; the source terminal sends the feature data to a cloud server; the cloud server sends the feature data to a destination terminal; the destination terminal generates media data of the digital human based on the feature data and a multimodal AI-driven model, the media data of the digital human including the video and / or audio of the digital human.

[0018] In the above method, after receiving the feature data from the source terminal, the cloud directly transmits it to the destination terminal, further reducing the bandwidth required for downlink transmission. The destination terminal then drives the digital human locally to generate the digital human's media data, ultimately achieving the goal of providing users with high-definition real-time digital humans even when the uplink and downlink transmission network conditions are poor.

[0019] In one possible implementation, the source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person according to a certain period; the source terminal extracts feature data for describing the real person from the media data of the real person, including: the source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in the facial features of the real person and changes in the morphological features of the real person.

[0020] In one possible implementation, before the cloud server sends the feature data to the destination terminal, the method further includes: the cloud server obtaining input information on the destination terminal, the input information indicating that the media data of the digital human is generated by the destination terminal; and the cloud server determining that the destination terminal has the ability to generate a digital human.

[0021] In one possible implementation, the method further includes: the source terminal receives first input information, the first input information indicates a feature type, and the feature type includes one or more facial features, morphological features, and sound features; the source terminal extracts the feature data used to describe the real person from the media data of the real person, including: the source terminal extracts target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0022] In a third aspect, the present application proposes a system for providing a digital human, which includes a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and voice features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to generate media data of the digital person based on the feature data and a multimodal AI-driven model, and the media data of the digital person includes the video and / or audio of the digital person; the cloud server is also used to send the media data of the digital person to the destination terminal.

[0023] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person according to a certain period; extract feature data used to describe the real person from the video of the real person, and the feature data includes one or more of the changes in the facial features of the real person and the changes in the morphological features of the real person.

[0024] In one possible implementation, before the cloud server is used to generate media data of the digital human based on the feature data and the multimodal AI-driven model, the cloud server is also used to: obtain input information on the destination terminal, where the input information indicates that the media data of the digital human is generated by the cloud server.

[0025] In one possible implementation, the source terminal is also used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0026] In one possible implementation, the cloud server is further used to: obtain second input information on the source terminal, where the second input information indicates the style of the digital human; the cloud server is used to generate media data of the digital human based on the feature data and the multimodal AI-driven model, specifically including: the cloud server is used to generate media data of the digital human of the corresponding style based on the feature data, the multimodal AI-driven model and the second input information.

[0027] In one possible implementation, the cloud server is specifically used to: determine the transmission bit rate of the digital human video according to the network status of the destination terminal; encode the digital human video according to the transmission bit rate of the digital human video, wherein the clarity of the first area and the clarity of the second area in the picture of the encoded digital human video are different; and send the encoded digital human video to the destination terminal.

[0028] In a fourth aspect, the present application proposes another system for providing a digital human, which includes a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and voice features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to send the feature data to the destination terminal; the destination terminal is used to generate media data of the digital human based on the feature data and a multimodal AI-driven model, and the media data of the digital human includes the video and / or audio of the digital human.

[0029] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person according to a certain period; extract feature data used to describe the real person from the video of the real person, and the feature data includes one or more of the changes in the facial features of the real person and the changes in the morphological features of the real person.

[0030] In one possible implementation, before the cloud server is used to send the feature data to the destination terminal, the cloud server is further used to: obtain input information on the destination terminal, the input information indicating that the media data of the digital human is generated by the destination terminal; and determine whether the destination terminal has the ability to generate a digital human.

[0031] In one possible implementation, the source terminal is also used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0032] In the fifth aspect, the present application proposes a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes a method for providing a digital human corresponding to the above-mentioned first aspect or any possible implementation of the first aspect.

[0033] In the sixth aspect, the present application proposes another computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method of providing a digital human corresponding to the above-mentioned second aspect or any possible implementation of the second aspect.

[0034] In the seventh aspect, the present application proposes a computer program product containing instructions. When the instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human corresponding to the above-mentioned first aspect or any possible implementation of the first aspect.

[0035] In an eighth aspect, the present application proposes another computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute a method for providing a digital human corresponding to the above-mentioned second aspect or any possible implementation of the second aspect.

[0036] In the ninth aspect, the present application proposes a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes a method for providing a digital human corresponding to the above-mentioned first aspect or any possible implementation of the first aspect.

[0037] In the tenth aspect, the present application proposes another computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes a method for providing a digital human corresponding to the above-mentioned second aspect or any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic diagram of a media transmission network architecture provided in an embodiment of the present application.

[0039] FIG2 is a schematic diagram of the architecture of a digital human media system provided in an embodiment of the present application.

[0040] FIG3 is a flow chart of the method for providing a digital human proposed in this application.

[0041] FIG4 is an online conference scenario provided in an embodiment of the present application.

[0042] FIG5 is one of the UI interaction interfaces of a digital human media system provided in an embodiment of the present application.

[0043] FIG6 is a second UI interaction interface of a digital human media system provided in an embodiment of the present application.

[0044] FIG7 is another flowchart of the method for providing a digital human proposed in this application.

[0045] FIG8 is another online conference scenario provided in an embodiment of the present application.

[0046] FIG9 is a schematic diagram of a system for providing a digital human proposed in this application.

[0047] FIG10 is a schematic diagram of a computing device provided in an embodiment of the present application.

[0048] FIG11 is a schematic diagram of a computing device cluster provided in an embodiment of the present application.

[0049] FIG12 is a schematic diagram of an implementation method of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions provided by this application will be described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0051] In the description of the embodiments of this application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc. can refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0052] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0053] A digital human is an entity that exists in the non-physical world and can be driven by a real person or computer. This entity is created using artificial intelligence technologies such as computer graphics, image rendering, motion capture, deep learning, speech synthesis, and portrait modeling. It possesses the ability to perceive, understand, and express the physical world, and can achieve functions such as human-computer interaction through electronic screens, VR, and AR devices. Furthermore, a real person's voice, expression, and movements in the real world can be used to drive the digital human to produce the same voice, expression, and movement in real time, allowing the digital human to reproduce the real person's performance in the real world in real time. Such a digital human can also be called a real-time digital human.

[0054] With the development of digital human technology, providing users with real-time digital humans in real-time audio and video-based media scenarios such as online conferences, live broadcasts, online classrooms, and real-time audio and video calls has become a hot application direction for digital humans. Providing real-time digital humans in these media scenarios requires first capturing a live video of the sending user. Then, the digital human is driven based on this live video to generate a digital human video. Finally, the receiving end displays this digital human video, allowing the receiving end to view a digital human image that reflects the sending user's real-time expressions, movements, and other characteristics.

[0055] Real-time audio and video technology enables users to conduct real-time audio and video conversations through an end-to-end network acquisition, encoding, transmission, decoding, and rendering process. It is widely used in scenarios such as online conferencing, live broadcasts, online classrooms, real-time calls, and cloud mobile phones, cloud desktops, and cloud gaming. Figure 1 shows a schematic diagram of the end-to-end logical architecture of a media scenario based on real-time audio and video technology. The following describes the application scenarios of this application in conjunction with Figure 1.

[0056] As shown in Figure 1, the entire media transmission network architecture includes a transmitter, a cloud, and a receiver. The transmitter is responsible for capturing the image and audio of the transmitter user in real time, while the cloud forwards the captured audio and video to the receiver. The receiver then displays the image and audio of the transmitter user to the receiver. For ease of description, audio data and video data are collectively referred to as media data in this application.

[0057] After performing analog-to-digital signal conversion, audio noise reduction, echo cancellation, and video sharpening, the media acquisition and pre-processing module passes the media data to the sender-side SDK (Software Development Kit). The sender-side SDK then compresses and encodes the media data into a media stream format that takes up less memory and is easier to transmit over the network, and sends it to the cloud.

[0058] To ensure that media data packets reach the receiving end on time and without loss, the sending SDK typically includes a low-latency sending module and a network adaptation module. The low-latency sending module efficiently packages media data packets and periodically sends them to the network, avoiding network congestion and data loss caused by instantaneous large-scale transmission. The network adaptation module dynamically adjusts the amount of data sent by analyzing the current network capacity to prevent service interruptions and reduced user experience caused by transmission data loss.

[0059] The cloud media service is responsible for receiving media data from the sender and forwarding it to the receiver.

[0060] The receiving SDK will continuously attempt to receive the media data stream transmitted from the cloud, decode the media data, and display it, allowing the receiving user to see the image or hear the voice of the sending user. However, due to network fluctuations and the fact that audio and video data are transmitted independently on the network, it is often the case that a frame of audio data has been collected, but the video data for that frame has not yet been collected. In this case, a smooth frame output module is required to cache a portion of the received data stream so that the media data to be decoded can be sent to the decoding module at a relatively stable period. At the same time, an audio and video synchronization module is also required to synchronize the audio and video so that the images seen and sounds heard by the receiving user are logical.

[0061] A key fundamental characteristic of real-time audio and video technology is that media data must be transmitted over public networks. The higher the quality of the media data, the higher the transmission bitrate, and the greater the network bandwidth required to transmit it. For example, a video with a resolution of 720p and a frame rate of 30 fps has a transmission bitrate of approximately 3 Mbps, while a video with a resolution of 1080p and a frame rate of 30 fps has a transmission bitrate of approximately 6 Mbps. Public networks are complex and volatile environments. During media data transmission, issues such as limited bandwidth, poor network quality (i.e., weak connections), and network fluctuations may occur. These issues can limit the quality of the transmitted and received media data, resulting in degradation of image quality, reduced frame rates, audio stuttering, even frozen images, and disconnected connections, impacting the user's experience in media scenarios based on real-time audio and video technology.

[0062] Furthermore, in the media scenario shown in Figure 1, the transmission of media data in the public network includes uplink transmission from the sender to the cloud and downlink transmission from the cloud to the receiver. With the development of network technology, user devices can access the public network in different ways, such as 4G, 5G, Wi-Fi, or wired networks. When the network access method used by the sender and the network access method used by the receiver are different, the quality of the media data received by the receiver will be affected by the status of both the uplink and downlink transmission networks. For example, when the uplink transmission network bandwidth is limited and the supported transmission bit rate is low, the video quality provided by the sender will be low. In this case, even if the quality of the downlink transmission network is good, the receiver will not be able to watch the high-quality video images of the sender user; even if the quality of the downlink transmission network is poor, the video quality received by the receiver will be further reduced.

[0063] Therefore, faced with the complex and ever-changing media transmission network, how to provide users with clear and smooth real-time digital humans in media scenarios based on real-time audio and video technology has become a technical problem that needs to be solved urgently.

[0064] This application proposes a method for providing a digital human, which can provide users with a clear and smooth real-time digital human image in a media scene based on real-time audio and video technology. In particular, even when the network bandwidth is limited and the quality of the uplink transmission network and the downlink transmission network differ greatly, high-definition and smooth real-time digital human can be provided.

[0065] Figure 2 shows a schematic diagram of the architecture of a digital human media system 100 provided in this application. This system can be used to implement the method for providing a digital human proposed in this application. To facilitate a better understanding of the method for providing a digital human proposed in this application, the system architecture shown in Figure 2 is first introduced below.

[0066] Digital human media system 100 includes a source terminal 101, a cloud server 102, and a destination terminal 103. In this application, cloud server 102 is also referred to as cloud 102. Source terminal 101, acting as a transmitter, collects user A's live media data, while the destination terminal, acting as a receiver, displays user A's digital human media data to user B. For ease of description, user A's live media data will be referred to as user A's media data or live media data, and user A's digital human media data will be referred to as digital human media data.

[0067] The source terminal 101 includes a media acquisition and pre-processing module, a real-time feature recognition SDK, and a transmitter-side SDK.

[0068] The media acquisition and pre-processing module is used to continuously acquire real-person images and sounds to obtain real-person media data.

[0069] The Real-Time Feature Recognition SDK is used to extract feature data describing real people from media data. Specifically, the SDK analyzes and identifies information such as facial expressions, eye distance, and face width in live video to obtain feature data including real-person facial features. It also analyzes and identifies information such as standing posture, height, and body movements in live video to obtain feature data including real-person morphological features. It also analyzes and identifies information such as the text, emotion, and tone of voice in real-person audio to obtain feature data including real-person vocal features.

[0070] After receiving the feature data extracted by the real-time feature recognition SDK, the sender-side SDK can invoke real-time encoding and transmission capabilities to encode and transmit the feature data to the cloud 102 for subsequent digital human activation. The sender-side SDK of the source terminal 101 also has the same functions as the sender-side SDK in the traditional media system shown in Figure 1, such as the ability to encode and compress received audio and video before sending it to the network, as well as network adaptive capabilities to collect and adjust configurations such as encoding bitrate and encoding parameters based on network conditions. However, unlike the sender-side SDK functions of traditional media systems, the sender-side SDK of the source terminal 101 can identify the type of feature data used to activate the digital human and perform specialized transmission optimizations. For example, it implements redundant protection for feature data against weak networks to ensure that a single transmission ensures lossless arrival at the cloud, and prioritizes the transmission of more critical feature data required for digital human activation. Furthermore, when both feature data and audio are used to activate the digital human, the sender-side SDK of the source terminal 101 can timestamp the frames of the feature data and audio to be sent to the cloud before packaging and transmitting them.

[0071] The cloud 102 includes a real-time interactive cloud service module and a multimodal digital human driving module.

[0072] On the one hand, the real-time interactive cloud service module is used to receive feature data and audio sent by the source terminal 101, and synchronize different types of data at the frame level according to the frame timestamp to ensure that the multi-type data sent to the digital human driver are clock-aligned.

[0073] The real-time interactive cloud service module is also used to transmit the generated digital human media data to the destination terminal 103. The real-time interactive cloud service module features high-definition encoding, intelligent forwarding, and low-latency framing. The high-definition encoding capability enables high-definition encoding of the digital human's audio and video generated in the cloud for transmission. The low-latency framing capability allows the audio and video data to be efficiently packaged and transmitted in frames according to latency requirements when transmitting the digital human's media data to the destination terminal 103. The intelligent forwarding capabilities of the real-time interactive cloud service module include: when the destination terminal 103 does not have the ability to generate a digital human, the media data of the digital human generated by the multimodal digital human driving module is sent to the destination terminal 103; when the destination terminal 103 has complete real-time digital human generation capabilities, the real-time interactive cloud service module can directly forward the feature data and compressed audio sent by the source terminal 101 to the destination terminal 103 stably and quickly through transparent transmission; and when the destination terminal 103 has partial digital human generation capabilities, the real-time interactive cloud service module can combine the receiving end digital human capability identification, the cloud-side digital human drive configuration and the actual data type to send to the destination terminal 103 some feature data that can be used for digital human drive processing by the destination terminal 103, as well as some processing results generated by the cloud-side digital human driving module.

[0074] The multimodal digital human driver module provides a multimodal AI-driven model that uses feature data to drive the digital human and generate video and audio of the digital human. Specifically, the multimodal AI-driven model uses feature data to perform voice-driven, posture-predicted, and stylized processing on the digital human, enabling the driven digital human to perform like a real person in the real world.

[0075] The receiving-side SDK of the destination terminal 103 completely has the function of processing media data by the receiving-side SDK in the traditional media system of FIG. 1 . The only difference is that the receiving-side SDK of the destination terminal 103 processes the media data of the digital human, which will not be described in detail here.

[0076] When the destination terminal 103 has the ability to generate a digital human, it can also include a cloud-based multimodal digital human driver module, which is not repeated in Figure 2. Specifically, when the destination terminal has partial digital human generation capabilities, the cloud-based multimodal digital human driver module can only be responsible for the initial processing of the digital human, such as data initialization and skeletal posture correction. It then sends the cloud-based initial processing results and the feature data required for subsequent processing by the destination terminal 103 to the destination terminal 103 to ultimately generate the digital human's media data.

[0077] FIG3 shows a first implementation flow of the method for providing a digital human proposed in this application, including the following steps 201 to 205 .

[0078] Step 201: A source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person.

[0079] After user A enters the digital human media service through the source terminal 101, the media acquisition and pre-processing module continuously acquires the image and sound of user A and obtains the video data and audio data of user A by converting the analog signal into a digital signal.

[0080] In actual applications, the type of real-person media data acquired by the media acquisition and pre-processing module depends on the selection made by user A. When user A chooses to activate the camera to access the digital human media service, the real-person media data acquired by source terminal 101 includes the real-person video; when user A chooses to activate the microphone to access the digital human media service, the real-person media data acquired by source terminal 101 includes the real-person audio; and when user A chooses to activate both the camera and microphone to access the digital human media service, the real-person media data acquired by source terminal 101 includes both the real-person video and audio.

[0081] Furthermore, the type of media data acquired by the real-time feature recognition SDK also depends on the choices made by user A. When user A chooses to present their original voice and the digital human's image to user B, the media acquisition and pre-processing module can send only the video data to the real-time feature recognition SDK, while the audio data is directly sent to the sending SDK for transmission to the cloud. When user A chooses to present the digital human's voice and image to user B, the media acquisition and pre-processing module can send both the video and audio data to the real-time feature recognition SDK. When user A chooses to present their original voice and the digital human's image to user B, but also wants the digital human's mouth shape in the digital human image to more closely resemble a real person speaking, the media acquisition and pre-processing module can send both the video and audio data to the real-time feature recognition SDK, while also sending the audio data to the sending SDK for transmission to the cloud.

[0082] Step 202: The source terminal extracts feature data describing the real person from the real person's media data. The feature data includes one or more of the real person's facial features, morphological features, and voice features. The real person's facial features and morphological features are extracted from the real person's video, and the real person's voice features are extracted from the real person's audio.

[0083] A person's facial features may include one or more of the following: facial expression, face shape, eye distance, face width, face height, shape of facial features, and distribution of facial features. A person's morphological features may include one or more of the following: height, shoulder width, posture, and body movements. A person's voice features may include one or more of the following: the words, emotion, tone, timbre, and intonation of the person's speech. The information included in a person's facial, morphological, or voice features is also referred to as feature information.

[0084] In one possible embodiment, the real-time feature recognition SDK can use visual AI technology to analyze videos of real people, identify the facial feature information and morphological feature information of the real people, and record them as facial feature data and morphological feature data.

[0085] In a possible embodiment, the real-time feature recognition SDK can use voiceprint recognition AI technology to analyze the audio of a real person, identify the real person's voice feature information, and record it as voice feature data.

[0086] Specifically, the system tracks multiple facial, body, and voice features of a person to identify their facial, morphological, and voice features. For example, a person's mouth shape varies with different expressions, and the position of points along the mouth's outline can reflect that shape.

[0087] In actual applications, the number of feature points can be dynamically configured by the user based on the hardware capabilities of the terminal device, the data accuracy required for digital human driving, etc.; and regarding the selection of feature points, any representative points that help to identify real-person behavioral characteristics can be used. This application does not limit the specific locations on the real-person body or face as feature points.

[0088] When user A initially enters the digital human media service, the source terminal 101 needs to extract all feature point information. The corresponding feature data is static feature data. After user A enters the digital human media service, the media acquisition and pre-processing module often acquires user A's video according to a certain acquisition cycle, that is, capturing one frame at a certain interval. Therefore, during the service, the source terminal only needs to extract dynamic feature data from the real person's video to reflect changes in feature information. Dynamic feature data can include one or more changes in the real person's facial features and morphological features.

[0089] For example, when a user raises their hand, the position of feature points on their palm, arm, joints, and other parts of their body will change. Based on the acquisition cycle, the real-time feature recognition SDK only needs to identify the change in the position of the feature points in the current frame compared to the previous frame. Specifically, dynamic feature data can use displacement vectors to record the position changes of feature points in the reference frame. When implementing the feature recognition SDK for dynamic feature data extraction, the recognition accuracy can be configured to filter out subtle feature point changes and recognition errors.

[0090] Step 203: The source terminal sends the feature data to the cloud server.

[0091] After extracting the feature data, the real-time feature recognition SDK will pass it to the sending-side SDK of the source terminal. The sending-side SDK will then call the real-time encoding compression capability to transmit the feature data to the cloud server.

[0092] In the above method, after extracting feature data from real-person media data, the source terminal only needs to transmit the feature data to the cloud to drive the digital human. The data volume of the feature data is much smaller than that of the original media data, and the network bandwidth required to transmit the feature data is also much smaller than that of transmitting the original media data. Therefore, even if the uplink network bandwidth of the source terminal 101 is limited, the uplink data can still reach the cloud quickly and losslessly. For example, the network bandwidth required to transmit a video with a resolution of 720p and a frame rate of 30fps is approximately 3Mbps, while the network bandwidth required to transmit the feature data extracted from the video is only 60-80Kbps. In addition, when the feature data is dynamic feature data that only contains changes in feature information, the data volume of the feature data is further reduced, thereby further reducing the bandwidth required for uplink transmission.

[0093] Step 204: The cloud server generates media data of the digital human based on the feature data and the multimodal AI-driven model. The media data of the digital human includes video and / or audio of the digital human.

[0094] After receiving the feature data from the SDK on the sending end, the multimodal digital human driving module will input the feature data and the digital human in the digital human asset library into the multimodal AI driving model. The AI ​​driving model can use the feature data to drive the digital human to make it look and sound like a real user, and call the real-time audio and video generation capability to obtain the video and audio of the driven digital human.

[0095] Specifically, the multimodal AI-driven model has voice-driven capabilities, that is, it can drive the digital human's mouth shape and expression to match the corresponding speech content and tone based on sound feature data or audio, and generate the digital human's audio based on sound feature data; it has posture prediction capabilities, that is, it can drive the digital human to perform the same movements as a real person based on morphological feature data; it has stylization capabilities, that is, it can drive the digital human's appearance to be like a real person based on facial feature data and morphological feature data.

[0096] In one possible embodiment, the multimodal AI-driven model can be obtained by offline training using a deep learning algorithm based on a large amount of real-life media data.

[0097] When user A chooses to show his or her original voice and the image of the digital human to user B, the media data of the digital human generated by the cloud server only includes the video of the digital human; when user A chooses to show the voice and image of the digital human to user B, the media data of the digital human generated by the cloud server includes the video and audio of the digital human; when the user chooses to show only the voice of the digital human to user B, the media data of the digital human generated by the cloud server only includes the audio of the digital human.

[0098] The cloud server generates digital human videos in the following application scenarios:

[0099] In one possible scenario, the media data acquired by the real-time feature recognition SDK in step 201 includes video, and the feature data extracted in step 202 includes one or more facial features or morphological features. Then, in step 204, the multimodal AI-driven model will generate a video of the digital human based on one or more facial features or morphological features.

[0100] In another possible scenario, the media data acquired by the real-time feature recognition SDK in step 201 includes video and audio, and the feature data extracted in step 202 includes one or more facial features, morphological features, and voice features. In step 204, the multimodal AI-driven model generates a video of the digital human based on one or more of the facial features, morphological features, and voice features. Specifically, the multimodal AI-driven model uses voice features to make the digital human's mouth shape in the digital human video resemble that of a real person speaking, and uses one or more of the facial features and morphological features to make the digital human's appearance in the digital human video resemble that of a real person.

[0101] In another possible scenario, the media data acquired by the real-time feature recognition SDK in step 201 includes audio, and the feature data acquired in step 202 includes sound features. In step 204, the multimodal AI-driven model generates a video of the digital human based on the sound features. Specifically, the multimodal AI-driven model uses the sound features to make the digital human's mouth shape in the video resemble that of a real person speaking.

[0102] Step 205: The cloud server sends the media data of the digital human to the destination terminal.

[0103] The real-time interactive cloud service module in the cloud uses the high-definition encoding capability to encode the media data of the digital human, and then sends the encoded media data of the digital human to the destination terminal for display through the downlink transmission network.

[0104] Because different network conditions allow for different video transmission bit rates, the cloud first needs to determine the transmission bit rate of the digital human video based on the network status of the destination terminal, then encode the digital human video according to this transmission bit rate, and finally send the encoded digital human video to the destination terminal. For example, when the network bandwidth of the destination terminal is limited, the downlink network can only support the transmission of low-bitrate video. In this case, the cloud will encode the digital human video into a low-definition video, thereby reducing the transmission bit rate of the encoded digital human video, so that the encoded digital human video can be transmitted to the destination terminal via the downlink network. When the network status of the destination terminal improves and the bandwidth is not limited, the downlink network can support the transmission of high-bitrate video. In this case, the cloud can encode the digital human video into a high-definition video, and the encoded high-definition video can still be transmitted to the destination terminal via the downlink network.

[0105] In the above method, although the network bandwidth of the source terminal 101 is limited, since the feature data can still reach the cloud 102 without loss, after the cloud 102 uses the feature data to drive the digital human to generate the digital human's media data, as long as the network status of the destination terminal 103 is good, the cloud 102 can use the high-definition encoding capability to transmit high-definition and high-frame rate video and high-quality audio to the destination terminal 103, thereby ensuring that the digital human image of user A viewed by user B is clear and smooth, and the digital human voice heard by user B is clear and continuous.

[0106] In an embodiment of the present application, the cloud can further perform generative encoding on the video of the digital human, that is, after analyzing the content of different areas in the video screen of the digital human, different areas in the video screen of the digital human can be encoded into different clarity.

[0107] For example, the video image of the digital human generated by the multimodal digital human driver module includes a digital human area and a background area, and the clarity of the digital human area and the background area are the same. When the transmission bit rate of the encoded digital human video needs to be reduced due to limited network bandwidth of the destination terminal 103, the real-time interactive cloud service module can only reduce the clarity of the background area during the encoding process while maintaining the original clarity of the digital human area. This ensures that the encoded digital human video has a low bit rate and can be transmitted on a bandwidth-limited network, while ensuring that the digital human image of user A viewed by user B remains clear. Specifically, the real-time interactive cloud service module can reduce the clarity of the background area by reducing the accuracy of the video data corresponding to the background area or adjusting the resolution of the background area.

[0108] In another possible embodiment, in order to further reduce the transmission bit rate of the encoded digital human video, in addition to reducing the clarity of the background area, the clarity of the digital human body area can also be reduced. In this case, the clarity of the digital human facial area and the body area is also different.

[0109] With traditional video encoding methods, the clarity of different areas of the encoded video remains the same. When the video bitrate needs to be reduced, the clarity of the entire video image is reduced. The aforementioned method, leveraging the generative encoding capabilities of the cloud, can maximize the clarity of the digital human image even in poor downlink network conditions, providing users with a better digital human media service experience.

[0110] In addition, when the media data of the digital human generated by the multimodal digital human driving module includes the digital human's video and audio, the real-time interactive cloud service module can call on the low-latency framing capability to package the digital human's video data and audio data in frames and send them to the destination terminal.

[0111] The following uses the online conference scenario shown in Figure 4 as an example to illustrate the network transmission advantages of the first implementation process. In the scenario shown in Figure 4, user A and user B participate in the same online conference through their respective conference applications. User A accesses the conference using public Wi-Fi, which has poor network conditions and limited uplink bandwidth. User B, on the other hand, accesses the conference using a home wired network, which has good network conditions and ample downlink bandwidth.

[0112] After the meeting begins, user A's conference application continuously extracts feature data containing user A's expressions, movements, and other information from the captured video footage. Because the feature data is small, the bandwidth required for transmission is only about 100Kbps. Therefore, even if user A's uplink network is limited to 1Mbps bandwidth, the feature data can still be quickly and losslessly transmitted by user A's conference application to the cloud-based conference service. After the cloud uses the feature data to drive the digital human and generate the digital human video, because user B's downlink network bandwidth is sufficient, the cloud can adaptively use a 20Mbps bitrate to transmit a 4K high-definition digital human video with a frame rate of 60fbps to user B's conference application. Ultimately, in the scenario shown in Figure 4, even if user A's uplink network bandwidth is limited, the method of this application can still provide user B with a clear and smooth digital human image of user A, ensuring user B's conference experience.

[0113] Taking the online meeting scenario as an example, the following introduces the UI interaction interface provided by this application to users when users use the above-mentioned method of providing digital humans.

[0114] Figure 5 shows an example screen layout for the online conference application after a user enters a digital human conference. As shown in Figure 5, when a user chooses to use a digital human avatar, they can see their own real-time digital human avatar in their own video frame, while simultaneously seeing the real-time digital human avatars of other participants in their corresponding video frames. For example, if participant 1 chooses to display their digital human video, their real-time digital human avatar will be displayed in their video frame, while if participant 3 chooses to display their live video, their live human avatar will be displayed in their video frame. In one possible embodiment, both the user's live video and the real-time digital human avatar can be displayed in their own video frame.

[0115] Figure 6 shows an example page layout for configuring digital human generation options. As shown in Figure 6, after the user enters the meeting settings page and selects the digital human option, the UI will provide options for switching on and off, features, digital human style, and digital human generation method.

[0116] In the switch option column, the user can choose whether to turn on the digital human. When the user chooses to turn on the digital human, other participants will watch the real-time digital human image of the user in the video image frame of the user as shown in Figure 5.

[0117] In the Feature Options column, users can select which features to use to drive the digital human. The more feature types the user selects, the closer the digital human's appearance will be to the user's real-life performance. Therefore, the method for providing a digital human proposed in this application also includes: a source terminal receiving first input information, where the first input information indicates a feature type, including one or more of facial features, morphological features, and voice features. Accordingly, step 202 specifically includes: the source terminal extracting target feature data from the real-life media data, where the target feature data is feature data used to describe the real-life and corresponds to the feature type indicated by the first input information.

[0118] Taking Figure 6 as an example, the first input information may include the user's selection in the Feature Options column. If the user selects both voice and facial features, the source terminal extracts the voice features from the real person's audio and the facial features from the real person's video. This data, including both voice and facial features, is then transmitted to the cloud for driving the digital human. If the user selects only facial features, the source terminal only extracts the facial features from the real person's video and transmits this data to the cloud for driving the digital human.

[0119] In the Digital Human Style option column, users can select the style of the digital human in the generated digital human video. Therefore, the method for providing a digital human in this application also includes: the cloud server obtains second input information from the source terminal, where the second input information indicates the style of the digital human. Accordingly, step 204 specifically includes: the cloud server generates media data of the digital human in the corresponding style based on the feature data, the multimodal AI-driven model, and the second input information.

[0120] Taking Figure 6 as an example, the second input information may include the user's selection in the Digital Human Style option column. The cloud-based Digital Human asset library offers a variety of digital human styles, including realistic digital humans, aesthetic digital humans, and cartoon digital humans, and the digital humans can be 2D or 3D. For example, when the user selects a cartoon digital human in the Digital Human Style option column, the cloud will call the cartoon digital human from the Digital Human asset library and input the cartoon digital human and feature data into the multimodal AI-driven model to generate the cartoon digital human's media data. In addition, after the user selects the cartoon digital human option, the UI interface can further display multiple images of the cartoon digital human image, allowing the user to determine which specific cartoon digital human to select.

[0121] The method for providing a digital human in this application also includes: the cloud server obtaining input information from the destination terminal, wherein the input information indicates that the cloud server or the destination terminal will generate the digital human's media data. Taking Figure 6 as an example, the input information from the destination terminal may include the user's selection in the "Digital Human Generation Method" option column. Corresponding to the intelligent forwarding capabilities of the real-time interactive cloud service module described in the previous system architecture section, when user B selects "Cloud Generation" in the "Digital Human Generation" option column, the cloud will execute the method for providing a digital human according to the first implementation process described above. That is, the cloud will use the feature data sent by the source terminal 101 to generate the digital human's media data and then transmit the digital human's media data to the destination terminal 103. When user B selects "Local Generation" in the "Digital Human Generation" option column, the cloud will execute the method for providing a digital human according to the second implementation process described below.

[0122] FIG7 shows a second implementation flow of the method for providing a digital human proposed in this application, including the following steps 301 to 305 .

[0123] Step 301: A source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person.

[0124] Step 302: The source terminal extracts feature data describing the real person from the real person's media data. The feature data includes one or more of the real person's facial features, morphological features, and voice features. The real person's facial features and morphological features are extracted from the real person's video, and the real person's voice features are extracted from the real person's audio.

[0125] Step 303: The source terminal sends the feature data to the cloud server.

[0126] Step 304: The cloud server sends the feature data to the destination terminal.

[0127] Step 305: The destination terminal generates media data of the digital human based on the feature data and the multimodal AI-driven model. The media data of the digital human includes video and / or audio of the digital human.

[0128] When the cloud server executes the method for providing a digital human according to the second implementation process, before the cloud server sends the feature data to the destination terminal, the method further includes:

[0129] The cloud server obtains input information from the destination terminal, wherein the input information indicates that the destination terminal generates media data of the digital human;

[0130] The cloud server determines that the destination terminal has the ability to generate a digital human.

[0131] Specifically, using Figure 6 as an example, when User B selects Local Generation in the Digital Human Generation Method option, a dialog box will pop up on the UI to verify the target terminal's performance and determine whether it has the ability to generate a digital human. Once the cloud server determines that the target terminal has the ability to generate a digital human, the cloud server will execute the second implementation process to provide the digital human.

[0132] Since the only difference between the second implementation process and the first implementation process is that after the cloud receives the feature data sent by the source terminal, it directly transmits the feature data to the destination terminal, and the destination terminal uses the feature data to drive the digital human, the specific implementation method of each step in the second implementation process can be completely referred to the specific implementation method of the first implementation process, and will not be repeated in this application.

[0133] The following uses the online conference scenario shown in Figure 8 as an example to describe the network transmission advantages of the second implementation process of the method for providing a digital human. As shown in Figure 8, user A accesses the online conference using the same network as Figure 4, but user B accesses the online conference using a 4G network, which also has poor network conditions and limited downlink transmission bandwidth. Furthermore, in the scenario shown in Figure 8, since user B's conference application is deployed on a PC with a high-performance GPU, user B's conference application includes a multimodal digital human driver module, allowing user B to leverage the AI ​​computing power of the local device to generate a video of the digital human based on feature data.

[0134] After the meeting begins, the feature data extracted by the source terminal is quickly and losslessly transmitted to the cloud. Then, considering that User B's conference application is capable of generating a digital human, the cloud forwards the received feature data directly to User B's conference application, reducing the required bandwidth for downlink transmission to approximately 100Kbps. Because the bandwidth required for downlink transmission of feature data is very small, even if User B's network status is poor, the feature data can quickly and losslessly reach the destination terminal. The destination terminal can then locally generate a high-definition digital human video based on the feature data. Ultimately, even if both the uplink and downlink transmission networks are in poor condition, User B can still see a clear digital human image of User A, ensuring User B's meeting experience.

[0135] The present application also proposes a system for providing a digital human as shown in Figure 9, which includes a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and voice features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to generate media data of the digital human based on the feature data and a multimodal AI-driven model, and the media data of the digital human includes the video and / or audio of the digital human; the cloud server is also used to send the media data of the digital human to the destination terminal.

[0136] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person according to a certain period; extract feature data used to describe the real person from the video of the real person, and the feature data includes one or more of the changes in the facial features of the real person and the changes in the morphological features of the real person.

[0137] In one possible implementation, before the cloud server is used to generate media data of the digital human based on the feature data and the multimodal AI-driven model, the cloud server is also used to: obtain input information on the destination terminal, where the input information indicates that the media data of the digital human is generated by the cloud server.

[0138] In one possible implementation, the source terminal is also used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0139] In one possible implementation, the cloud server is further used to: obtain second input information on the source terminal, where the second input information indicates the style of the digital human; the cloud server is used to generate media data of the digital human based on the feature data and the multimodal AI-driven model, specifically including: the cloud server is used to generate media data of the digital human of the corresponding style based on the feature data, the multimodal AI-driven model and the second input information.

[0140] In one possible implementation, the cloud server is specifically used to: determine the transmission bit rate of the digital human video according to the network status of the destination terminal; encode the digital human video according to the transmission bit rate of the digital human video, wherein the clarity of the first area and the clarity of the second area in the picture of the encoded digital human video are different; and send the encoded digital human video to the destination terminal.

[0141] Specifically, the specific implementation of the various operations of the method for providing a digital human performed by the system for providing a digital human can refer to the description of the relevant content of the first implementation process in the above method embodiment, and will not be repeated here.

[0142] The present application also proposes another system for providing a digital human, which includes a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and voice features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to send the feature data to the destination terminal; the destination terminal is used to generate media data of the digital human based on the feature data and a multimodal AI-driven model, and the media data of the digital human includes the video and / or audio of the digital human.

[0143] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person according to a certain period; extract feature data used to describe the real person from the video of the real person, and the feature data includes one or more of the changes in the facial features of the real person and the changes in the morphological features of the real person.

[0144] In one possible implementation, before the cloud server is used to send the feature data to the destination terminal, the cloud server is further used to: obtain input information on the destination terminal, the input information indicating that the media data of the digital human is generated by the destination terminal; and determine whether the destination terminal has the ability to generate a digital human.

[0145] In one possible implementation, the source terminal is also used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0146] Specifically, the specific implementation of the various operations of the method for providing a digital human performed by the system for providing a digital human can refer to the description of the relevant content of the second implementation process in the above method embodiment, and will not be repeated here.

[0147] In the embodiments of the present application, the source terminal, destination terminal, and cloud server in the system for providing a digital human can be implemented through software or hardware. The following is an example of the implementation of the cloud server, which can be used as a reference for the implementation of the source terminal and destination terminal.

[0148] As an example of a software functional unit, a cloud server can include code running on a compute instance. A compute instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, these computing devices can be one or more. For example, a cloud server can include code running on multiple hosts, virtual machines, or containers. It should be noted that the multiple hosts, virtual machines, or containers running the code can be distributed in the same AZ or different AZs, with each AZ comprising a data center or multiple geographically close data centers. The multiple hosts, virtual machines, or containers running the code can be distributed in the same region or different regions. Typically, a region can include multiple AZs, and a VPC is set up within a region. Cross-region communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0149] Similarly, multiple hosts, virtual machines, or containers running the code can be distributed in the same VPC or in multiple VPCs. Typically, a region can include multiple AZs.

[0150] As an example of a hardware functional unit, a cloud server may include at least one computing device, such as a server. Alternatively, the cloud server may be implemented using an ASIC or a PLD. The PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0151] The multiple computing devices included in a cloud server can be distributed in the same AZ or different AZs. They can also be distributed in the same region or different regions. Similarly, they can be distributed in the same VPC or across multiple VPCs. These multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0152] This application also provides a computing device 500. Figure 10 is a schematic diagram of the structure of computing device 500. This computing device 500 can be the source terminal, destination terminal, or cloud server in the above-mentioned embodiments, and includes a bus 502, a processor 504, a memory 506, and a communication interface 508. The processor 504, the memory 506, and the communication interface 508 communicate with each other via bus 502. Computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 500.

[0153] Bus 502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG10 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 502 may include a path for transmitting information between various components of computing device 500 (e.g., memory 506, processor 504, and communication interface 508).

[0154] The processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0155] The memory 506 may include volatile memory, such as random access memory (RAM). The memory 506 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0156] Memory 506 stores executable program code, which processor 504 executes to implement the functions of the aforementioned source terminal, destination terminal, or cloud server, thereby implementing the method for providing a digital human. In other words, memory 506 stores instructions for executing the method for providing a digital human.

[0157] The communication interface 508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0158] This application also provides a computing device cluster. Figure 11 is a schematic diagram of this computing device cluster. This computing device cluster can implement the method for providing a digital human according to the first or second implementation process of the method embodiment. As shown in Figure 11, the computing device cluster includes at least one computing device 500 as shown in Figure 10. This computing device 500 can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0159] Instructions for executing the method for providing a digital human may be stored in the memory 506 of one or more computing devices 500 in the computing device cluster. When at least one computing device in the computing device cluster executes the instructions, the computing device cluster may implement the method for providing a digital human as described in the method embodiment.

[0160] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the method for providing a digital human. In other words, the combination of one or more computing devices 500 can jointly execute the instructions for implementing the method for providing a digital human.

[0161] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster can store different instructions, each used to execute a portion of the functions of the system for providing digital humans. In other words, the instructions stored in the memory 506 in different computing devices 500 can respectively implement the functions of the source terminal, cloud server, or destination terminal.

[0162] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG12 shows a possible implementation. As shown in FIG12 , two computing devices 500A and 500B are connected via a network. Specifically, the connection to the network is made via a communication interface in each computing device. In this type of possible implementation, the memory 506 in the computing device 500A stores instructions for executing the functions of the source terminal and the destination terminal. At the same time, the memory 506 in the computing device 500B stores instructions for executing the functions of the cloud server.

[0163] It should be understood that the functionality of the computing device 500A shown in FIG12 may also be accomplished by multiple computing devices 500. Similarly, the functionality of the computing device 500B may also be accomplished by multiple computing devices 500.

[0164] This application also provides a computer program product containing instructions. This computer program product can be software or a program product containing instructions that can be run on a computing device or stored in any available medium. When this computer program product is run on at least one computing device, it causes at least one computing device to perform the method for providing a digital human according to the first or second implementation flow of the method embodiment.

[0165] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute the method for providing a digital human according to the first or second implementation flow of the method embodiment.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for providing a digital human, characterized in that: The method comprises: The source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal extracts feature data for describing the real person from the media data of the real person, where the feature data includes one or more of facial features, morphological features, and voice features of the real person, where the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal sends the characteristic data to a cloud server; The cloud server generates media data of the digital human according to the feature data and the multimodal AI driving model, wherein the media data of the digital human includes video and / or audio of the digital human; The cloud server sends the media data of the digital human to the destination terminal.

2. The method according to claim 1, characterized in that The source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person according to a certain period; The source terminal extracts feature data for describing the real person from the media data of the real person, including: The source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

3. The method according to claim 1 or 2, characterized in that: Before the cloud server generates media data of the digital human according to the feature data and the multimodal AI driving model, the method further includes: The cloud server obtains input information on the target terminal, where the input information indicates that the media data of the digital human is generated by the cloud server.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The source terminal receives first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal extracts the feature data for describing the real person from the media data of the real person, including: The source terminal extracts target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to a feature type indicated by the first input information.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The cloud server acquires second input information on the source terminal, where the second input information indicates the style of the digital human; The cloud server generates media data of the digital human according to the feature data and the multimodal AI driving model, including: The cloud server generates media data of a digital human of a corresponding style according to the feature data, the multimodal AI driving model and the second input information.

6. The method according to any one of claims 1 to 5, characterized in that The cloud server sends the video of the digital human to the destination terminal, including: The cloud server determines the transmission bit rate of the digital human video according to the network status of the destination terminal; The cloud server encodes the video of the digital human according to the transmission bit rate of the video of the digital human, wherein the clarity of a first area and the clarity of a second area in the picture of the encoded video of the digital human are different; The cloud server sends the encoded video of the digital human to the destination terminal.

7. A method for providing a digital human, characterized in that: The method comprises: The source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal extracts feature data for describing the real person from the media data of the real person, where the feature data includes one or more of facial features, morphological features, and voice features of the real person, where the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal sends the characteristic data to a cloud server; The cloud server sends the characteristic data to the destination terminal; The destination terminal generates media data of the digital human according to the feature data and the multimodal AI driving model, where the media data of the digital human includes video and / or audio of the digital human.

8. The method according to claim 7, characterized in that The source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person according to a certain period; The source terminal extracts feature data for describing the real person from the media data of the real person, including: The source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

9. The method according to claim 7 or 8, characterized in that: Before the cloud server sends the characteristic data to the destination terminal, the method further includes: The cloud server obtains input information on the target terminal, wherein the input information indicates that the media data of the digital human is generated by the target terminal; The cloud server determines that the destination terminal has the capability of generating a digital human.

10. The method according to any one of claims 7 to 9, characterized in that The method further comprises: The source terminal receives first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal extracts the feature data for describing the real person from the media data of the real person, including: The source terminal extracts target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to a feature type indicated by the first input information.

11. A system for providing a digital human, characterized in that: The system includes a source terminal, a cloud server and a destination terminal, wherein: The source terminal is used to obtain media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal is further used to extract feature data for describing the real person from the media data of the real person, wherein the feature data includes one or more of facial features, morphological features, and voice features of the real person, wherein the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal is further used to send the characteristic data to the cloud server; The cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, wherein the media data of the digital human includes video and / or audio of the digital human; The cloud server is also used to send the media data of the digital human to the destination terminal.

12. The system according to claim 11, characterized in that The source terminal is specifically used for: Obtaining the video of the real person at a certain period; Feature data for describing the real person is extracted from the video of the real person, wherein the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

13. The system according to claim 11 or 12, characterized in that Before the cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI-driven model, the cloud server is also used to: Input information on the target terminal is obtained, where the input information indicates that media data of the digital human is generated by the cloud server.

14. The system according to any one of claims 11 to 13, characterized in that The source terminal is further used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: The source terminal is used to extract target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

15. The system according to any one of claims 11 to 14, characterized in that The cloud server is further used to: obtain second input information on the source terminal, where the second input information indicates the style of the digital human; The cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, specifically including: The cloud server is used to generate media data of a digital human of a corresponding style according to the feature data, the multimodal AI driving model and the second input information.

16. The system according to any one of claims 11 to 15, characterized in that The cloud server is specifically used for: Determining the transmission bit rate of the digital human video according to the network status of the destination terminal; Encoding the video of the digital human according to the transmission bit rate of the video of the digital human, wherein the clarity of a first area and the clarity of a second area in a picture of the encoded video of the digital human are different; The encoded video of the digital human is sent to the destination terminal.

17. A system for providing a digital human, characterized in that: The system includes a source terminal, a cloud server and a destination terminal, wherein: The source terminal is used to obtain media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal is further used to extract feature data for describing the real person from the media data of the real person, wherein the feature data includes one or more of facial features, morphological features, and voice features of the real person, wherein the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal is further used to send the characteristic data to the cloud server; The cloud server is used to send the characteristic data to the destination terminal; The destination terminal is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, where the media data of the digital human includes the video and / or audio of the digital human.

18. The system according to claim 17, characterized in that The source terminal is specifically used for: Obtaining the video of the real person at a certain period; Feature data for describing the real person is extracted from the video of the real person, wherein the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

19. The system according to claim 17 or 18, characterized in that Before the cloud server is used to send the characteristic data to the destination terminal, the cloud server is further used to: Acquiring input information on the target terminal, the input information indicating that the media data of the digital human is generated by the target terminal; It is determined that the destination terminal has a digital human generation capability.

20. The system according to any one of claims 17 to 19, characterized in that The source terminal is further used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: The source terminal is used to extract target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method for providing a digital human according to any one of claims 1 to 6.

22. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method for providing a digital human according to any one of claims 7 to 10.

23. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as claimed in any one of claims 1 to 6.

24. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as described in any one of claims 7 to 10.

25. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as claimed in any one of claims 1 to 6.

26. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as claimed in any one of claims 7 to 10.

Images