Method and system for providing digital human and computing device cluster

Abstract

The invention provides a method and system for providing a digital human and a computing device cluster, and belongs to the technical field of cloud computing. According to the method, a clear and smooth real-time digital human image can be provided for a user in a media scene based on a real-time audio and video technology, and particularly, a high-definition and smooth real-time digital human can be provided even under the conditions that the network bandwidth is limited and the quality difference between an uplink transmission network and a downlink transmission network is large.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a method, system, computing device cluster, computer-readable storage medium, and computer program product for providing a digital human. Background Art

[0002] Digital human refers to an image entity that exists in the non-physical world and can be driven by real people or computers based on artificial intelligence technologies such as computer graphics, graphics rendering, motion capture, deep learning, speech synthesis, and portrait modeling. It has the ability to perceive, recognize, and express the physical world, and can use electronic screens, VR, AR and other devices as carriers to achieve functions such as human-computer interaction. Furthermore, the voice, expression, and action of real people in the real world can be used to drive digital humans to complete the same voice, expression, and action in real time, so that the driven digital human can show the performance of real people in the real world in real time. Such digital humans can also be called real-time digital humans.

[0003] With the development of digital human applications, how to provide users with clear and smooth real-time digital humans in media scenarios based on real-time audio and video technologies such as online meetings, live broadcasts, online classes, and real-time audio and video calls has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application proposes a method for providing a digital human, which can provide users with a clear and smooth real-time digital human image in media scenarios based on real-time audio and video technology such as online conferences, live broadcasts, online classes, and real-time audio and video calls, and can provide high-quality and smooth real-time digital humans even in situations such as unstable network conditions, weak networks, and limited network bandwidth. The present application also provides a corresponding system, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0005] In the first aspect, the present application proposes a method for providing a digital human. The method includes: a source terminal acquires media data of a real person, the media data of the real person includes the video and / or audio of the real person; the source terminal extracts feature data for describing the real person from the media data of the real person, the feature data includes one or more of the facial features, morphological features, and sound features of the real person, the facial features and morphological features of the real person are extracted from the video of the real person, and the sound features of the real person are extracted from the audio of the real person; the source terminal sends the feature data to a cloud server; the cloud server generates media data of the digital person based on the feature data and a multimodal AI-driven model, the media data of the digital person includes the video and / or audio of the digital person; the cloud server sends the media data of the digital person to a destination terminal.

[0006] In the above method, the bandwidth required for uplink transmission is reduced by replacing real-person video with feature data during uplink transmission, so that even when the uplink network quality is poor, the feature data can still reach the cloud quickly and losslessly, thereby providing users with high-definition real-time digital humans.

[0007] In a possible implementation, the source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person at a certain period; the source terminal extracts feature data for describing the real person from the media data of the real person, including: the source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

[0008] In the above implementation, by extracting only feature data containing feature changes in the live video based on the acquisition cycle, the bandwidth required for uplink transmission of the feature data is further reduced.

[0009] In a possible implementation, before the cloud server generates media data of the digital human based on the feature data and the multimodal AI-driven model, the method further includes: the cloud server obtains input information on the destination terminal, and the input information indicates that the media data of the digital human is generated by the cloud server.

[0010] In a possible implementation, the method also includes: the source terminal receives first input information, the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; the source terminal extracts the feature data used to describe the real person from the media data of the real person, including: the source terminal extracts target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0011] In a possible implementation, the method further includes: the cloud server obtains second input information on the source terminal, the second input information indicating the style of the digital human; the cloud server generates media data of the digital human according to the feature data and the multimodal AI driving model, including: the cloud server generates media data of the digital human of the corresponding style according to the feature data, the multimodal AI driving model and the second input information.

[0012] In a possible implementation, the cloud server sends the video of the digital human to a destination terminal, including: the cloud server determines the transmission bit rate of the video of the digital human according to the network status of the destination terminal; the cloud server encodes the video of the digital human according to the transmission bit rate of the video of the digital human, wherein the clarity of a first area and a second area in a picture of the encoded video of the digital human are different; and the cloud server sends the encoded video of the digital human to the destination terminal.

[0013] In the above implementation, through the generative coding capability of the cloud, the clarity of the digital human image can be guaranteed as much as possible when the downlink transmission network status is poor, so that users can have a better digital human media service experience.

[0014] In the second aspect, the present application proposes another method for providing a digital human. The method includes: a source terminal obtains media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal extracts feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and sound features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the sound features of the real person are extracted from the audio of the real person; the source terminal sends the feature data to a cloud server; the cloud server sends the feature data to a destination terminal; the destination terminal generates media data of a digital person based on the feature data and a multimodal AI-driven model, and the media data of the digital person includes the video and / or audio of the digital person.

[0015] In the above method, after receiving the feature data from the source terminal, the cloud directly transmits it to the destination terminal, further reducing the bandwidth required for downlink transmission. The destination terminal then drives the digital human locally to generate the digital human's media data, ultimately achieving the goal of providing users with high-definition real-time digital humans even when both the uplink and downlink transmission network conditions are poor.

[0016] In a possible implementation, the source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person at a certain period; the source terminal extracts feature data for describing the real person from the media data of the real person, including: the source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

[0017] In a possible implementation, before the cloud server sends the feature data to the destination terminal, the method further includes: the cloud server obtains input information on the destination terminal, the input information indicates that the media data of the digital human is generated by the destination terminal; and the cloud server determines that the destination terminal has the ability to generate a digital human.

[0018] In a possible implementation, the method also includes: the source terminal receives first input information, the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; the source terminal extracts the feature data used to describe the real person from the media data of the real person, including: the source terminal extracts target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0019] In a third aspect, the present application proposes a system for providing a digital human, the system comprising a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, the media data of the real person including the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, the feature data including one or more of the facial features, morphological features, and sound features of the real person, the facial features and morphological features of the real person are extracted from the video of the real person, and the sound features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to generate media data of the digital person based on the feature data and a multimodal AI-driven model, the media data of the digital person including the video and / or audio of the digital person; the cloud server is also used to send the media data of the digital person to the destination terminal.

[0020] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person at a certain period; extract feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

[0021] In a possible implementation, before the cloud server is used to generate media data of the digital human based on the feature data and the multimodal AI-driven model, the cloud server is also used to: obtain input information on the destination terminal, wherein the input information indicates that the media data of the digital human is generated by the cloud server.

[0022] In one possible implementation, the source terminal is also used to: receive first input information, the first input information indicates a feature type, the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0023] In a possible implementation, the cloud server is further used to: obtain second input information on the source terminal, the second input information indicating the style of the digital human; the cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, specifically including: the cloud server is used to generate media data of the digital human of the corresponding style according to the feature data, the multimodal AI driving model and the second input information.

[0024] In a possible implementation, the cloud server is specifically used to: determine the transmission bit rate of the video of the digital human according to the network status of the destination terminal; encode the video of the digital human according to the transmission bit rate of the video of the digital human, wherein the clarity of the first area and the clarity of the second area in the picture of the encoded video of the digital human are different; and send the encoded video of the digital human to the destination terminal.

[0025] In a fourth aspect, the present application proposes another system for providing a digital human, which includes a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and sound features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the sound features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to send the feature data to the destination terminal; the destination terminal is used to generate media data of the digital human based on the feature data and a multimodal AI-driven model, and the media data of the digital human includes the video and / or audio of the digital human.

[0026] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person at a certain period; extract feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

[0027] In a possible implementation, before the cloud server is used to send the feature data to the destination terminal, the cloud server is also used to: obtain input information on the destination terminal, the input information indicating that the media data of the digital human is generated by the destination terminal; and determine whether the destination terminal has the ability to generate a digital human.

[0028] In one possible implementation, the source terminal is also used to: receive first input information, the first input information indicates a feature type, the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0029] In a fifth aspect, the present application proposes a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes a method for providing a digital human corresponding to the above-mentioned first aspect or any possible implementation of the first aspect.

[0030] In the sixth aspect, the present application proposes another computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes a method for providing a digital human corresponding to the above-mentioned second aspect or any possible implementation of the second aspect.

[0031] In a seventh aspect, the present application proposes a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute a method for providing a digital human corresponding to the above-mentioned first aspect or any possible implementation of the first aspect.

[0032] In an eighth aspect, the present application proposes another computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute a method for providing a digital human corresponding to the above-mentioned second aspect or any possible implementation of the second aspect.

[0033] In a ninth aspect, the present application proposes a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes a method for providing a digital human corresponding to the above-mentioned first aspect or any possible implementation of the first aspect.

[0034] In the tenth aspect, the present application proposes another computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes a method for providing a digital human corresponding to the above-mentioned second aspect or any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of a media transmission network architecture provided in an embodiment of the present application.

[0036] Figure 2 It is a schematic diagram of the architecture of a digital human media system provided in an embodiment of the present application.

[0037] Figure 3 This is one of the flow charts of the method for providing a digital human proposed in this application.

[0038] Figure 4 This is an online conference scenario provided by an embodiment of the present application.

[0039] Figure 5 It is one of the UI interaction interfaces of a digital human media system provided in an embodiment of the present application.

[0040] Figure 6 It is one of the UI interaction interfaces of a digital human media system provided in an embodiment of the present application.

[0041] Figure 7 This is one of the flow charts of the method for providing a digital human proposed in this application.

[0042] Figure 8 This is another online conference scenario provided by an embodiment of the present application.

[0043] Fig. 9 It is a schematic diagram of a system for providing a digital human proposed in this application.

[0044] Fig.10 It is a schematic diagram of a computing device provided in an embodiment of the present application.

[0045] Fig.11 It is a schematic diagram of a computing device cluster provided in an embodiment of the present application.

[0046] Fig.12 It is a schematic diagram of an implementation method of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solution provided by the present application will be described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0048] In the description of the embodiments of the present application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0049] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0050] Digital human refers to an image entity that exists in the non-physical world and can be driven by real people or computers based on artificial intelligence technologies such as computer graphics, graphics rendering, motion capture, deep learning, speech synthesis, and portrait modeling. It has the ability to perceive, recognize, and express the physical world, and can use electronic screens, VR, AR and other devices as carriers to achieve functions such as human-computer interaction. Furthermore, the voice, expression, and action of real people in the real world can be used to drive digital humans to complete the same voice, expression, and action in real time, so that digital humans can show the performance of real people in the real world in real time. Such digital humans can also be called real-time digital humans.

[0051] With the development of digital human technology, providing users with real-time digital humans in media scenarios based on real-time audio and video technology, such as online conferences, live broadcasts, online classes, and real-time audio and video calls, has become a hot application direction for digital humans. To provide users with real-time digital humans in media scenarios based on real-time audio and video technology, it is necessary to first collect the real-life video of the sending user, then drive the digital human based on the real-life video to obtain the digital human video, and finally display the digital human video at the receiving end, so that the receiving end user can see the digital human image with the real-time expression, action and other characteristics of the sending end user.

[0052] Real-time audio and video technology allows users to conduct real-time audio and video conversations through end-to-end network acquisition, encoding, transmission, decoding, rendering and other technical processes. It is widely used in online meetings, live broadcasts, online classrooms, real-time calls, as well as cloud phones, cloud desktops, cloud games and other scenarios. Figure 1 A schematic diagram of the end-to-end logical architecture in a media scenario based on real-time audio and video technology is given below. Figure 1 The application scenario of this application is described.

[0053] like Figure 1 As shown, the entire media transmission network architecture includes a sender, a cloud, and a receiver, wherein the sender is responsible for collecting the images and sounds of the sender user in real time, the cloud is responsible for forwarding the audio and video collected by the sender to the receiver, and the receiver is responsible for displaying the images and sounds of the sender user to the receiver. For the convenience of description, audio data and video data are collectively referred to as media data in this application.

[0054] After performing analog-to-digital signal conversion, audio noise reduction, echo cancellation, video sharpening and other processing, the media acquisition and pre-processing module passes the media data to the sender-side SDK (Software Development Kit). After that, the sender-side SDK compresses and encodes the media data into a media stream format that takes up less memory and is conducive to network transmission, and sends it to the cloud.

[0055] In order to ensure that media data packets can reach the receiving end on time and without loss, the SDK on the sending end usually also has a low-latency sending module and a network adaptation module. The low-latency sending module is used to efficiently package media data packets and periodically send them to the network to avoid network congestion and data loss caused by instantaneous large-scale transmission; the network adaptation module dynamically adjusts the amount of data sent by analyzing the current network carrying capacity to prevent service jams and reduced experience caused by transmission data loss.

[0056] The cloud media service is responsible for receiving media data from the sender and forwarding it to the receiver.

[0057] The receiving SDK will continuously try to receive the media data stream transmitted from the cloud, and decode the media data for display, so that the receiving user can see the image or hear the voice of the sending user. However, since the network is volatile and audio and video data are transmitted independently in the network, there is often a situation where a frame of audio data has been collected, but the video data of that frame has not been collected yet. At this time, a smooth frame output module is required to cache part of the received data stream, so that the media data to be decoded can be sent to the decoding module at a relatively stable period. At the same time, an audio and video synchronization module is also required to synchronize audio and video, so that the images seen and sounds heard by the receiving user are logical.

[0058] The need for media data to be transmitted in the public network is an important basic feature of real-time audio and video technology. The higher the quality of media data, the higher its transmission bit rate, and the larger the network bandwidth required to transmit the media data. For example, the transmission bit rate of a video with a definition of 720p and a frame rate of 30fps is about 3Mbps, while the transmission bit rate of a video with a definition of 1080p and a frame rate of 30fps is about 6Mbps. The public network is a complex and changeable environment. During the transmission of media data, problems such as limited bandwidth of the network where the user device is located, poor network quality (i.e. weak network), and network fluctuations may occur. This will limit the quality of the media data sent and received, such as reducing image quality, reducing frame rate, sound freezes, and even freezing the picture and disconnecting the link, affecting the user's perception experience in media scenarios based on real-time audio and video technology.

[0059] Further, in Figure 1In the media scenario shown, the transmission of media data in the public network includes uplink transmission from the sender to the cloud and downlink transmission from the cloud to the receiver. With the development of network technology, user devices can access the public network in different ways such as 4G, 5G, Wifi or wired network. When the network access method used by the sender and the network access method used by the receiver are different, the quality of the media data received by the receiver will be affected by the status of both the uplink transmission network and the downlink transmission network. For example, when the uplink transmission network bandwidth is limited and the supported transmission bit rate is small, the video quality provided by the sender will be low. At this time, even if the quality of the downlink transmission network is good, the receiver user cannot watch the high-quality video picture of the sender user; even when the quality of the downlink transmission network is poor, the video quality received by the receiver user will be further reduced.

[0060] Therefore, in the face of complex and ever-changing media transmission networks, how to provide users with clear and smooth real-time digital humans in media scenarios based on real-time audio and video technology has become a technical problem that needs to be solved urgently.

[0061] The present application proposes a method for providing a digital human, which can provide users with a clear and smooth real-time digital human image in a media scene based on real-time audio and video technology, especially providing a high-quality and smooth real-time digital human even when the network bandwidth is limited and the quality difference between the uplink transmission network and the downlink transmission network is large.

[0062] Figure 2 A schematic diagram of the architecture of a digital human media system 100 provided in the present application is provided. The system can be used to implement the method for providing a digital human provided in the present application. To facilitate a better understanding of the method for providing a digital human provided in the present application, the following first introduces Figure 2 The system architecture is shown.

[0063] The digital human media system 100 includes a source terminal 101, a cloud server 102, and a destination terminal 103. In this application, the cloud server 102 is also referred to as the cloud 102. The source terminal 101 acts as a sending end to collect the real person media data of user A, and the destination terminal acts as a receiving end to display the digital human media data of user A to user B. For the convenience of description, the real person media data of user A is also referred to as user A's media data or real person's media data; the digital human media data of user A is also referred to as digital human media data.

[0064] The source terminal 101 includes a media acquisition and pre-processing module, a real-time feature recognition SDK and a sending end side SDK.

[0065] The media acquisition and pre-processing module is used to continuously acquire real-person images and sounds to obtain real-person media data.

[0066] The real-time feature recognition SDK is used to extract feature data describing real people from real people's media data. Specifically, the real-time feature recognition SDK obtains feature data including real people's facial features by analyzing and identifying real people's facial expressions, eye distance, face width and other information in real people's videos; obtains feature data including real people's morphological features by analyzing and identifying real people's standing posture, height, body movements and other information in real people's videos; and obtains feature data including real people's voice features by analyzing and identifying real people's spoken text, emotions, tone and other information in real people's audio.

[0067] After receiving the feature data extracted by the real-time feature recognition SDK, the sending end SDK can call the real-time encoding and transmission capability to encode and transmit the feature data to the cloud 102 for subsequent driving of the digital human. Figure 1 The functions of the sending-side SDK in the traditional media system shown in the figure, for example, it has the ability to encode and compress the received audio and video and then send them to the network, and the network adaptive ability to collect and adjust the configurations such as the encoding bit rate and encoding parameters according to the network status. However, unlike the sending-side SDK function of the traditional media system, the sending-side SDK of the source terminal 101 can identify the type of feature data used to drive the digital human, and make special transmission optimization for it, such as redundant protection against weak networks for the feature data so that a single transmission of data can reach the cloud with certainty and losslessness, and priority transmission protection for the feature data with a higher level of importance required for driving the digital human. In addition, when the feature data and audio are used to drive the digital human at the same time, the sending-side SDK of the source terminal 101 can mark the frame timestamps of the feature data and audio to be sent to the cloud before packaging and sending them.

[0068] The cloud 102 includes a real-time interactive cloud service module and a multimodal digital human driving module.

[0069] The real-time interactive cloud service module is used to receive feature data and audio sent from the source terminal 101, and synchronize different types of data at the frame level according to the frame timestamp to ensure that the multi-type data sent to the digital human driver are clock-aligned.

[0070] On the other hand, the real-time interactive cloud service module is used to send the generated digital human media data to the destination terminal 103. The real-time interactive cloud service module has the capabilities of high-definition encoding, intelligent forwarding, and low-latency framing. The high-definition encoding capability can encode the audio and video of the digital human generated in the cloud in high definition before transmission. The low-latency framing capability can efficiently package and send the audio data and video data in frames according to the latency requirements when sending the digital human media data to the destination terminal 103. The intelligent forwarding capabilities of the real-time interactive cloud service module include: when the destination terminal 103 does not have the ability to generate digital humans, the media data of the digital humans generated by the multimodal digital human driving module is sent to the destination terminal 103; when the destination terminal 103 has complete real-time digital human generation capabilities, the real-time interactive cloud service module can directly forward the feature data and compressed audio sent by the source terminal 101 to the destination terminal 103 stably and quickly through transparent transmission; and when the destination terminal 103 has partial digital human generation capabilities, the real-time interactive cloud service module can combine the digital human capability identification of the receiving end, the configuration of the digital human driving down and the actual data type, and send to the destination terminal 103 some feature data that can be used for digital human driving processing by the destination terminal 103, as well as some processing results generated by the digital human driving module on the cloud side.

[0071] The multimodal digital human driving module is used to provide a multimodal AI driving model, which can use feature data to drive the digital human and generate the digital human's video and audio. Specifically, the multimodal AI driving model can perform voice driving, posture prediction, and stylization on the digital human based on the feature data, so that the driven digital human can show the performance of a real person in the real world.

[0072] The receiving end SDK of the target terminal 103 is fully equipped with Figure 1 The function of the receiving end side SDK in the traditional media system to process media data is different only in that the receiving end side SDK of the target terminal 103 processes the media data of the digital human, which will not be described in detail here.

[0073] When the destination terminal 103 has the ability to generate a digital human, the destination terminal 103 may also include a multimodal digital human driving module connected to the cloud. Figure 2 Specifically, when the destination terminal has partial digital human generation capabilities, the multimodal digital human driving module in the cloud can only be responsible for the preliminary processing of the digital human, such as data initialization and skeletal posture correction of the digital human, and then send the preliminary processing results in the cloud and the feature data required for subsequent processing by the destination terminal 103 to the destination terminal 103 to finally generate the media data of the digital human.

[0074] Figure 3The first implementation process of the method for providing a digital human proposed in this application is given, including the following steps 201 to 205.

[0075] Step 201: A source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person.

[0076] After user A enters the digital human media service through the source terminal 101, the media acquisition and pre-processing module continuously acquires the image and sound of user A and obtains the video data and audio data of user A after converting the analog signal into a digital signal.

[0077] In actual applications, the type of real-person media data acquired by the media acquisition and pre-processing module is related to the selection of user A. When user A chooses to turn on the camera to enter the digital human media service, the real-person media data acquired by the source terminal 101 includes the real-person video; when user A chooses to turn on the microphone to enter the digital human media service, the real-person media data acquired by the source terminal 101 includes the real-person audio; when user A chooses to turn on the camera and microphone at the same time to enter the digital human media service, the real-person media data acquired by the source terminal 101 includes the real-person video and audio.

[0078] In addition, the type of media data obtained by the real-time feature recognition SDK is also related to the choice of user A. When user A chooses to show his original voice and digital human image to user B, the media acquisition and pre-processing module can only send the video data to the real-time feature recognition SDK, and the audio data is directly sent to the sending end SDK for transmission to the cloud; when user A chooses to show the voice and image of the digital human to user B, the media acquisition and pre-processing module can send the video data and audio data to the real-time feature recognition SDK at the same time; when user A chooses to show his original voice and digital human image to user B, but at the same time hopes that the mouth shape of the digital human in the digital human image is closer to the performance of a real person speaking, the media acquisition and pre-processing module can send the video data and audio data to the real-time feature recognition SDK, and also send the audio data to the sending end SDK for transmission to the cloud.

[0079] Step 202: The source terminal extracts feature data for describing the real person from the real person's media data, and the feature data includes one or more of the real person's facial features, morphological features, and voice features. The real person's facial features and morphological features are extracted from the real person's video, and the real person's voice features are extracted from the real person's audio.

[0080] The facial features of a real person may include one or more of the following information: the real person's facial expression, face shape, eye distance, face width, face height, shape of the facial features, and distribution of the facial features; the morphological features of a real person may include one or more of the following information: the real person's height, shoulder width, posture, and body movements; the voice features of a real person may include one or more of the following information: the text, emotion, tone, timbre, and intonation of the real person's speech. The information included in the facial features, morphological features, or voice features of a real person is also called feature information.

[0081] In a possible embodiment, the real-time feature recognition SDK can use visual AI technology to analyze videos of real people, identify the facial feature information and morphological feature information of the real people, and record them as facial feature data and morphological feature data.

[0082] In a possible embodiment, the real-time feature recognition SDK can use voiceprint recognition AI technology to analyze the audio of a real person, identify the real person's voice feature information, and record it as voice feature data.

[0083] Specifically, the facial feature information, morphological feature information and voice feature information of a real person are identified by tracking multiple facial feature points, body feature points and voice feature points of the real person. For example, a person's mouth shape is different under different expressions, and the position information of the points on the contour of the real person's mouth can reflect the person's mouth shape.

[0084] In actual applications, the number of feature points can be dynamically configured by the user based on the hardware capabilities of the terminal device, the data accuracy required for the digital human drive, etc.; and regarding the selection of feature points, any representative points that are helpful in identifying real-person behavioral characteristics will do, and this application does not limit the specific locations on the real person's body or face as feature points.

[0085] When user A initially enters the digital human media service, the source terminal 101 needs to extract all feature point information, and the corresponding feature data is static feature data. After user A enters the digital human media service, the media acquisition and pre-processing module often acquires the video screen of user A according to a certain acquisition cycle, that is, it acquires a frame of screen every certain time. In the process of the service, the source terminal only needs to extract dynamic feature data reflecting the changes of feature information from the real person's video. The dynamic feature data may include one or more of the changes of the real person's facial features and morphological features.

[0086] For example, when a user raises his hand, the position of the feature points of the user's palm, arm, joints, etc. will change. Based on the acquisition cycle, the real-time feature recognition SDK only needs to identify the change in the position of the feature point in the current frame compared to the previous frame. Specifically, dynamic feature data can use displacement vectors to record the position change of feature points on the reference system. When implementing the feature recognition SDK to extract dynamic feature data, you can filter out subtle feature point changes and recognition errors by configuring recognition accuracy.

[0087] Step 203: The source terminal sends the feature data to the cloud server.

[0088] After extracting the feature data, the real-time feature recognition SDK will pass it to the sending-side SDK of the source terminal, which will then call the real-time encoding compression capability to transmit the feature data to the cloud server.

[0089] In the above method, after the source terminal extracts the feature data from the real-person media data, it only needs to transmit the feature data to the cloud for driving the digital human. The data volume of the feature data is much smaller than the data volume of the initial media data, and the network bandwidth required to transmit the feature data is also much smaller than the network bandwidth for transmitting the initial media data. Therefore, in the case of the uplink network bandwidth speed limit of the source terminal 101, the uplink data can still reach the cloud quickly and losslessly. For example, the network bandwidth required to transmit a video with a clarity of 720p and a frame rate of 30fps is about 3Mbps, while the network bandwidth required to transmit the feature data extracted from the video is only 60 to 80Kbps. In addition, when the feature data is dynamic feature data that only contains changes in feature information, the data volume of the feature data is further reduced, thereby further reducing the bandwidth required for uplink transmission.

[0090] Step 204: The cloud server generates media data of the digital human according to the feature data and the multimodal AI-driven model, wherein the media data of the digital human includes video and / or audio of the digital human.

[0091] After receiving the feature data from the SDK on the sending end, the multimodal digital human driving module will input the feature data and the digital human in the digital human asset library into the multimodal AI driving model. The AI ​​driving model can use the feature data to drive the digital human to make it look and sound like a real user, and call on the real-time audio and video generation capability to obtain the video and audio of the driven digital human.

[0092] Specifically, the multimodal AI-driven model has voice-driven capabilities, that is, it can drive the digital human's mouth shape and expression to match the corresponding speech content and tone based on sound feature data or audio, and generate the digital human's audio based on sound feature data; it has posture prediction capabilities, that is, it can drive the digital human to show the same movements as a real person based on morphological feature data; it has stylization capabilities, that is, it can drive the digital human's appearance to be like a real person based on facial feature data and morphological feature data.

[0093] In a possible embodiment, the multimodal AI-driven model can be obtained by offline training based on a large amount of real-life media data using a deep learning algorithm.

[0094] When user A chooses to show his or her original voice and the image of the digital human to user B, the media data of the digital human generated by the cloud server only includes the video of the digital human; when user A chooses to show the voice and image of the digital human to user B, the media data of the digital human generated by the cloud server includes the video and audio of the digital human; when the user chooses to show only the voice of the digital human to user B, the media data of the digital human generated by the cloud server only includes the audio of the digital human.

[0095] The cloud server generates digital human videos in the following application scenarios:

[0096] In one possible scenario, the media data acquired by the real-time feature recognition SDK in step 201 includes video, and the feature data extracted in step 202 includes one or more facial features or morphological features. Then in step 204, the multimodal AI-driven model will generate a video of the digital human based on one or more facial features or morphological features.

[0097] In another possible scenario, the media data acquired by the real-time feature recognition SDK in step 201 includes video and audio, and the feature data extracted in step 202 includes one or more of facial features and morphological features and sound features. Then, in step 204, the multimodal AI-driven model will generate a video of the digital human based on one or more of the facial features and morphological features and sound features. Specifically, the multimodal AI-driven model uses sound features to drive the mouth shape of the digital human in the digital human video screen to resemble the performance of a real person speaking, and uses one or more of the facial features and morphological features to drive the appearance of the digital human in the digital human video screen to resemble a real person.

[0098] In another possible scenario, the media data acquired by the real-time feature recognition SDK in step 201 includes audio, and the feature data acquired in step 202 includes sound features, then in step 204, the multimodal AI driving model will generate a video of the digital human based on the sound features. Specifically, the multimodal AI driving model uses the sound features to drive the mouth shape of the digital human in the digital human video screen to resemble the performance of a real person speaking.

[0099] Step 205: The cloud server sends the media data of the digital human to the destination terminal.

[0100] The real-time interactive cloud service module in the cloud uses the high-definition encoding capability to encode the media data of the digital human, and then sends the encoded media data of the digital human to the destination terminal for display through the downlink transmission network.

[0101] Since different network states allow different video transmission bit rates, the cloud first needs to determine the transmission bit rate of the digital human video according to the network state of the destination terminal, then encode the digital human video according to the transmission bit rate, and finally send the encoded digital human video to the destination terminal. For example, when the network bandwidth of the destination terminal is limited, the downlink network can only support the transmission of low-bitrate videos, then the cloud will encode the digital human video into a low-definition video, thereby reducing the transmission bit rate of the encoded digital human video, so that the encoded digital human video can be transmitted to the destination terminal through the downlink network. When the network state of the destination terminal improves and the bandwidth is not limited, the downlink network can support the transmission of high-bitrate videos, then the cloud can encode the digital human video into a high-definition video, and the encoded high-definition video can still be transmitted to the destination terminal through the downlink network.

[0102] In the above method, although the network bandwidth of the source terminal 101 is limited, since the feature data can still reach the cloud 102 without loss, after the cloud 102 uses the feature data to drive the digital human to generate the digital human's media data, as long as the network status of the destination terminal 103 is good, the cloud 102 can call on the high-definition encoding capability to transmit high-definition and high-frame rate video and high-quality audio to the destination terminal 103, thereby ensuring that the digital human image of user A viewed by user B is clear and smooth, and the digital human voice heard is clear and continuous.

[0103] In the embodiment of the present application, the cloud can further perform generative encoding on the video of the digital human, that is, after analyzing the content of different areas in the video screen of the digital human, different areas in the video screen of the digital human can be encoded into different clarity.

[0104] For example, the video screen of the digital human generated by the multimodal digital human driving module includes the digital human area and the background area, and the clarity of the digital human area and the background area is the same. When the transmission bit rate of the encoded digital human video needs to be reduced due to the limited network bandwidth of the destination terminal 103, the real-time interactive cloud service module can only reduce the clarity of the background area during the encoding process and maintain the original clarity of the digital human area, thereby ensuring that the encoded digital human video has a low bit rate and can be transmitted in a network with limited bandwidth, and the digital human image of user A viewed by user B is still clear. Specifically, the real-time interactive cloud service module can reduce the clarity of the background area by reducing the accuracy of the video data corresponding to the background area or adjusting the resolution of the background area.

[0105] In another possible embodiment, in order to further reduce the transmission bit rate of the encoded digital human video, in addition to reducing the clarity of the background area, the clarity of the digital human body area can also be reduced. In this case, the clarity of the digital human face area and body area is also different.

[0106] For traditional video encoding methods, the clarity of different areas in the encoded video screen is the same. When the video transmission bit rate needs to be reduced, the clarity of the entire video screen will be reduced. In the above method, through the generative encoding capability of the cloud, the clarity of the digital human image can be guaranteed as much as possible when the downlink transmission network status is poor, so that users have a better digital human media service experience.

[0107] In addition, when the media data of the digital human generated by the multimodal digital human driving module includes the video and audio of the digital human, the real-time interactive cloud service module can call on the low-latency framing capability to package the video data and audio data of the digital human in frames and send them to the destination terminal.

[0108] Below Figure 4 Taking the online conference scenario shown above as an example, the network transmission advantages of the first implementation process are introduced. Figure 4 In the scenario shown, user A and user B participate in the same online conference through their respective conference applications. User A uses a public Wi-Fi network to access the conference, and its network status is poor and the uplink transmission bandwidth is limited, while user B uses a home wired network to access the conference, and its network status is good and the downlink transmission bandwidth is sufficient.

[0109] After the meeting starts, user A's conference application continuously extracts feature data containing user A's expressions, movements and other information from the captured video. Since the feature data is small in size and the bandwidth required for transmission is only about 100Kbps, even if the upstream network where user A is located is limited to 1Mbps bandwidth, the feature data can still be quickly and losslessly transmitted to the cloud-based conference service by user A's conference terminal. After the cloud uses the feature data to drive the digital human and generate the digital human video, since user B's downstream network bandwidth is sufficient, the cloud can adaptively use a 20Mbps bit rate to transmit a 4K high-definition digital human video with a frame rate of 60fbps to user B's conference application. Finally, Figure 4 In the scenario shown, even if the uplink network bandwidth of user A is limited, the method of the present application can still provide user B with a clear and smooth digital human image of user A, thereby ensuring the conference experience of user B.

[0110] Taking an online conference scenario as an example, the following introduces the UI interaction interface provided to users by this application when users use the above-mentioned method of providing a digital human.

[0111] Figure 5 This example shows the screen layout of the online conference application after the user enters the digital human conference. Figure 5 As shown, when a user chooses to use a digital human image, the user can see his or her real-time digital human image in his or her own video frame, and can also see the real-time digital human images of other participants in the corresponding video frames of other users. For example, if participant 1 chooses to display the digital human video, then the real-time digital human image of participant 1 is displayed in the video frame of participant 1, and if participant 3 chooses to display the real-person video, then the real-person image of participant 3 is displayed in the video frame of participant 3. In a possible embodiment, the real-person image of the user and the real-time digital human image can be displayed simultaneously in the user's own video frame.

[0112] Figure 6 An example page layout is given when the user sets some configuration options for digital human generation. Figure 6 As shown, after the user enters the meeting setting page and selects the digital human column, the UI interface will provide switch options, feature options, digital human style options, and digital human generation method options.

[0113] In the switch option column, users can choose whether to turn on the digital human. When the user chooses to turn on the digital human, other participants will Figure 5 The real-time digital human image of the user can be seen in the video image frame of the user shown.

[0114] In the feature options column, the user can choose which features to use to drive the digital human. The more feature types the user selects, the closer the form of the driven digital human will be to the user's real-life performance. Therefore, the method for providing a digital human proposed in this application also includes: the source terminal receives first input information, the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features. Accordingly, step 202 specifically includes: the source terminal extracts target feature data from the media data of the real person, and the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0115] by Figure 6 For example, the first input information may include the user's selection in the feature option column. When the user selects voice features and facial features, the source terminal will extract voice features from the real person's audio and facial features from the real person's video, and then transmit the feature data containing voice features and facial features to the cloud for driving the digital human; when the user only selects facial features, the source terminal only needs to extract feature data containing facial features from the real person's video, and transmit this part of feature data to the cloud for driving the digital human.

[0116] In the digital human style option column, the user can select the style of the digital human in the generated digital human video. Therefore, the method for providing a digital human in the present application also includes: the cloud server obtains the second input information on the source terminal, the second input information indicates the style of the digital human, and accordingly, step 204 specifically includes: the cloud server generates media data of the digital human of the corresponding style according to the feature data, the multimodal AI driving model and the second input information.

[0117] by Figure 6 For example, the second input information may include the user's selection in the digital human style option column. The digital human asset library on the cloud provides digital humans of various styles, including realistic digital humans, aesthetic digital humans, and cartoon digital humans, and the digital humans can be in 2D or 3D. For example, when the user selects a cartoon digital human in the digital human style option column, the cloud will call the cartoon digital human from the digital human asset library, and input the cartoon digital human and feature data into the multimodal AI-driven model to generate media data for the cartoon digital human. In addition, after the user selects the cartoon digital human option, the UI interface can further display multiple pictures of cartoon digital human images for the user to determine which specific cartoon digital human to select.

[0118] The method for providing a digital human in the present application further includes: the cloud server obtains input information on the target terminal, wherein the input information indicates that the media data of the digital human is generated by the cloud server or the target terminal. Figure 6For example, the input information on the above-mentioned destination terminal may include the user's selection in the digital human generation method option column. Corresponding to the intelligent forwarding capability of the real-time interactive cloud service module introduced in the previous system architecture section, when user B selects cloud generation in the digital human generation option column, the cloud will execute the method of providing a digital human according to the first implementation process described above, that is, the cloud will use the feature data sent by the source terminal 101 to generate the digital human's media data, and then transmit the digital human's media data to the destination terminal 103; when user B selects local generation in the digital human generation option column, the cloud will execute the method of providing a digital human according to the second implementation process described below.

[0119] Figure 7 A second implementation process of the method for providing a digital human proposed in this application is provided, including the following steps 301 to 305.

[0120] Step 301: A source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person.

[0121] Step 302: The source terminal extracts feature data for describing the real person from the real person's media data, wherein the feature data includes one or more of the real person's facial features, morphological features, and voice features. The real person's facial features and morphological features are extracted from the real person's video, and the real person's voice features are extracted from the real person's audio.

[0122] Step 303: The source terminal sends the feature data to the cloud server.

[0123] Step 304: The cloud server sends the feature data to the destination terminal.

[0124] Step 305: The destination terminal generates media data of the digital human according to the feature data and the multimodal AI driving model, where the media data of the digital human includes video and / or audio of the digital human.

[0125] When the cloud server performs the method for providing a digital human according to the second implementation process, before the cloud server sends the feature data to the destination terminal, the method further includes:

[0126] The cloud server obtains input information on the target terminal, where the input information indicates that the media data of the digital human is generated by the target terminal;

[0127] The cloud server determines that the destination terminal has the ability to generate a digital human.

[0128] Specifically, Figure 6For example, when user B selects local generation in the digital human generation method option, a dialog box for performance testing of the target terminal will pop up on the UI interface to determine whether the target terminal has the ability to generate a digital human. When the cloud server determines that the target terminal has the ability to generate a digital human, the cloud will execute the method of providing a digital human according to the second implementation process.

[0129] Since the only difference between the second implementation process and the first implementation process is that after the cloud receives the feature data sent by the source terminal, it directly transmits the feature data to the destination terminal, and the destination terminal uses the feature data to drive the digital human. Therefore, the specific implementation method of each step in the second implementation process can be completely referred to the specific implementation method of the first implementation process, and will not be repeated in this application.

[0130] Below Figure 8 Taking the online conference scenario shown in the figure as an example, the network transmission advantage of the second implementation process of the method of providing digital human is introduced. Figure 8 As shown, the network used by user A to access the online conference is Figure 4 The same, but user B uses 4G network to access the online meeting, the network status is also poor, and the downlink transmission bandwidth is also limited. Figure 8 In the scenario shown, since user B's conference application is deployed on a PC with a high-performance GPU, user B's conference application includes a multimodal digital human driver module, so that user B can use the AI ​​computing power of the local device to generate a video of the digital human based on the feature data.

[0131] After the meeting starts, the feature data extracted by the source terminal can reach the cloud quickly and losslessly. Then, considering that user B's conference application terminal has the ability to generate digital humans, the cloud will directly forward the received feature data to user B's conference application terminal, thereby reducing the bandwidth required for downlink transmission to about 100Kbps. Since the bandwidth required for downlink transmission of feature data is very small, even if user B's network status is poor, the feature data can reach the destination terminal quickly and losslessly. The destination terminal can then generate high-definition digital human videos locally based on the feature data. Ultimately, even if the uplink and downlink transmission network status are poor, user B can still see the clear digital human image of user A, ensuring user B's conference experience.

[0132] This application also proposes a Fig. 9The system for providing a digital human shown in the figure includes a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and sound features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the sound features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to generate media data of the digital person according to the feature data and a multimodal AI-driven model, and the media data of the digital person includes the video and / or audio of the digital person; the cloud server is also used to send the media data of the digital person to the destination terminal.

[0133] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person at a certain period; extract feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

[0134] In a possible implementation, before the cloud server is used to generate media data of the digital human based on the feature data and the multimodal AI-driven model, the cloud server is also used to: obtain input information on the destination terminal, wherein the input information indicates that the media data of the digital human is generated by the cloud server.

[0135] In one possible implementation, the source terminal is also used to: receive first input information, the first input information indicates a feature type, the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0136] In a possible implementation, the cloud server is further used to: obtain second input information on the source terminal, the second input information indicating the style of the digital human; the cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, specifically including: the cloud server is used to generate media data of the digital human of the corresponding style according to the feature data, the multimodal AI driving model and the second input information.

[0137] In a possible implementation, the cloud server is specifically used to: determine the transmission bit rate of the video of the digital human according to the network status of the destination terminal; encode the video of the digital human according to the transmission bit rate of the video of the digital human, wherein the clarity of the first area and the clarity of the second area in the picture of the encoded video of the digital human are different; and send the encoded video of the digital human to the destination terminal.

[0138] Specifically, the specific implementation of the various operations of the method for providing a digital human performed by the system for providing a digital human can refer to the description of the relevant content of the first implementation process in the above method embodiment, which will not be repeated here.

[0139] The present application also proposes another system for providing a digital human, which includes a source terminal, a cloud server and a destination terminal, wherein the source terminal is used to obtain media data of a real person, and the media data of the real person includes the video and / or audio of the real person; the source terminal is also used to extract feature data for describing the real person from the media data of the real person, and the feature data includes one or more of the facial features, morphological features, and sound features of the real person, and the facial features and morphological features of the real person are extracted from the video of the real person, and the sound features of the real person are extracted from the audio of the real person; the source terminal is also used to send the feature data to the cloud server; the cloud server is used to send the feature data to the destination terminal; the destination terminal is used to generate media data of the digital person based on the feature data and a multimodal AI-driven model, and the media data of the digital person includes the video and / or audio of the digital person.

[0140] In one possible implementation, the source terminal is specifically used to: obtain the video of the real person at a certain period; extract feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

[0141] In a possible implementation, before the cloud server is used to send the feature data to the destination terminal, the cloud server is also used to: obtain input information on the destination terminal, the input information indicating that the media data of the digital human is generated by the destination terminal; and determine whether the destination terminal has the ability to generate a digital human.

[0142] In one possible implementation, the source terminal is also used to: receive first input information, the first input information indicates a feature type, the feature type includes one or more of facial features, morphological features, and sound features; the source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: the source terminal is used to extract target feature data from the media data of the real person, the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

[0143] Specifically, the specific implementation of the various operations of the method for providing a digital human performed by the system for providing a digital human can refer to the description of the relevant content of the second implementation process in the above method embodiment, which will not be repeated here.

[0144] In the embodiment of the present application, the source terminal, the destination terminal, and the cloud server in the system for providing a digital human can be implemented by software or hardware. The following is an exemplary introduction to the implementation of the cloud server, and the implementation of the source terminal and the destination terminal can be used as a reference.

[0145] As an example of a software functional unit, a cloud server may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. For example, a cloud server may include code running on multiple hosts, virtual machines, or containers. It should be noted that multiple hosts, virtual machines, or containers for running the code may be distributed in the same AZ or in different AZs, and each AZ includes a data center or multiple data centers with similar geographical locations. Multiple hosts, virtual machines, or containers for running the code may be distributed in the same region or in different regions. Typically, a region may include multiple AZs, and a VPC is set in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC to achieve interconnection between VPCs through the communication gateway.

[0146] Similarly, multiple hosts or virtual machines or containers for running the code may be distributed in the same VPC or in multiple VPCs, wherein usually a region may include multiple AZs.

[0147] As an example of a hardware functional unit, the cloud server may include at least one computing device, such as a server, etc. Alternatively, the cloud server may also be a device implemented using ASIC or PLD, etc. The PLD may be implemented using CPLD, FPGA, GAL or any combination thereof.

[0148] The multiple computing devices included in the cloud server can be distributed in the same AZ or in different AZs. The multiple computing devices included in the cloud server can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the cloud server can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0149] The present application also provides a computing device 500, Fig.10 5 is a schematic diagram of the structure of a computing device 500. The computing device 500 may be a source terminal, a destination terminal or a cloud server in the above embodiment, and includes: a bus 502, a processor 504, a memory 506 and a communication interface 508. The processor 504, the memory 506 and the communication interface 508 communicate with each other via the bus 502. The computing device 500 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 500.

[0150] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The bus 502 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 502 may include a path for transmitting information between various components of the computing device 500 (eg, the memory 506, the processor 504, and the communication interface 508).

[0151] The processor 504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0152] The memory 506 may include a volatile memory, such as a random access memory (RAM). The memory 506 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0153] The memory 506 stores executable program codes, and the processor 504 executes the executable program codes to respectively implement the functions of the aforementioned source terminal, the destination terminal, or the cloud server, thereby implementing the method for providing a digital human. That is, the memory 506 stores instructions for executing the method for providing a digital human.

[0154] The communication interface 508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0155] The present application also provides a computing device cluster, Fig.11 is a schematic diagram of the computing device cluster, which can implement the method of providing a digital human in the present application according to the first or second implementation process in the method embodiment. Fig.11 As shown, the computing device cluster includes at least one Fig.10 The computing device 500 shown may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0156] The memory 506 in one or more computing devices 500 in the computing device cluster may store instructions for executing the method for providing a digital human. When at least one computing device in the computing device cluster executes the instructions, the computing device cluster may implement the method for providing a digital human as described in the method embodiment.

[0157] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the method for providing a digital human. In other words, the combination of one or more computing devices 500 may jointly execute instructions for implementing the method for providing a digital human.

[0158] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the system for providing digital humans. That is, the instructions stored in the memory 506 in different computing devices 500 can respectively implement the functions of the source terminal, the cloud server or the destination terminal.

[0159] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.12 A possible implementation is shown. Fig.12 As shown, two computing devices 500A and 500B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 506 in the computing device 500A stores instructions for executing the functions of the source terminal and the destination terminal. At the same time, the memory 506 in the computing device 500B stores instructions for executing the functions of the cloud server.

[0160] It should be understood that Fig.12 The functions of the computing device 500A shown in FIG. 5A may also be completed by multiple computing devices 500. Similarly, the functions of the computing device 500B may also be completed by multiple computing devices 500.

[0161] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the first or second implementation flow in the method embodiment of at least one computing device executes the method for providing a digital human.

[0162] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to perform the method for providing a digital human according to the first or second implementation process in the method embodiment.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for providing a digital human, characterized in that: The method comprises: The source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal extracts feature data for describing the real person from the media data of the real person, where the feature data includes one or more of facial features, morphological features, and voice features of the real person, where the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal sends the characteristic data to a cloud server; The cloud server generates media data of the digital human according to the feature data and the multimodal AI driving model, wherein the media data of the digital human includes video and / or audio of the digital human; The cloud server sends the media data of the digital human to the destination terminal.

2. The method according to claim 1, characterized in that The source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person according to a certain period; The source terminal extracts feature data for describing the real person from the media data of the real person, including: The source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

3. The method according to claim 1 or 2, characterized in that: Before the cloud server generates media data of the digital human according to the feature data and the multimodal AI driving model, the method further includes: The cloud server obtains input information on the target terminal, where the input information indicates that the media data of the digital human is generated by the cloud server.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The source terminal receives first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal extracts the feature data for describing the real person from the media data of the real person, including: The source terminal extracts target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to a feature type indicated by the first input information.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The cloud server acquires second input information on the source terminal, where the second input information indicates the style of the digital human; The cloud server generates media data of the digital human according to the feature data and the multimodal AI driving model, including: The cloud server generates media data of a digital human of a corresponding style according to the feature data, the multimodal AI driving model and the second input information.

6. The method according to any one of claims 1 to 5, characterized in that The cloud server sends the video of the digital human to the destination terminal, including: The cloud server determines the transmission bit rate of the digital human video according to the network status of the destination terminal; The cloud server encodes the video of the digital human according to the transmission bit rate of the video of the digital human, wherein the clarity of a first area and the clarity of a second area in the picture of the encoded video of the digital human are different; The cloud server sends the encoded video of the digital human to the destination terminal.

7. A method for providing a digital human, characterized in that: The method comprises: The source terminal obtains media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal extracts feature data for describing the real person from the media data of the real person, where the feature data includes one or more of facial features, morphological features, and voice features of the real person, where the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal sends the characteristic data to a cloud server; The cloud server sends the characteristic data to the destination terminal; The destination terminal generates media data of the digital human according to the feature data and the multimodal AI driving model, where the media data of the digital human includes video and / or audio of the digital human.

8. The method according to claim 7, characterized in that The source terminal obtains the media data of the real person, including: the source terminal obtains the video of the real person according to a certain period; The source terminal extracts feature data for describing the real person from the media data of the real person, including: The source terminal extracts feature data for describing the real person from the video of the real person, and the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

9. The method according to claim 7 or 8, characterized in that: Before the cloud server sends the characteristic data to the destination terminal, the method further includes: The cloud server obtains input information on the target terminal, wherein the input information indicates that the media data of the digital human is generated by the target terminal; The cloud server determines that the destination terminal has the capability of generating a digital human.

10. The method according to any one of claims 7 to 9, characterized in that The method further comprises: The source terminal receives first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal extracts the feature data for describing the real person from the media data of the real person, including: The source terminal extracts target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to a feature type indicated by the first input information.

11. A system for providing a digital human, characterized in that: The system includes a source terminal, a cloud server and a destination terminal, wherein: The source terminal is used to obtain media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal is further used to extract feature data for describing the real person from the media data of the real person, wherein the feature data includes one or more of facial features, morphological features, and voice features of the real person, wherein the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal is further used to send the characteristic data to the cloud server; The cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, wherein the media data of the digital human includes video and / or audio of the digital human; The cloud server is also used to send the media data of the digital human to the destination terminal.

12. The system according to claim 11, characterized in that The source terminal is specifically used for: Obtaining the video of the real person at a certain period; Feature data for describing the real person is extracted from the video of the real person, wherein the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

13. The system according to claim 11 or 12, characterized in that Before the cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI-driven model, the cloud server is also used to: Input information on the target terminal is obtained, where the input information indicates that media data of the digital human is generated by the cloud server.

14. The system according to any one of claims 11 to 13, characterized in that The source terminal is further used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: The source terminal is used to extract target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

15. The system according to any one of claims 11 to 14, characterized in that The cloud server is further used to: obtain second input information on the source terminal, where the second input information indicates the style of the digital human; The cloud server is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, specifically including: The cloud server is used to generate media data of a digital human of a corresponding style according to the feature data, the multimodal AI driving model and the second input information.

16. The system according to any one of claims 11 to 15, characterized in that The cloud server is specifically used for: Determining the transmission bit rate of the digital human video according to the network status of the destination terminal; Encoding the video of the digital human according to the transmission bit rate of the video of the digital human, wherein the clarity of a first area and the clarity of a second area in a picture of the encoded video of the digital human are different; The encoded video of the digital human is sent to the destination terminal.

17. A system for providing a digital human, characterized in that: The system includes a source terminal, a cloud server and a destination terminal, wherein: The source terminal is used to obtain media data of a real person, where the media data of the real person includes video and / or audio of the real person; The source terminal is further used to extract feature data for describing the real person from the media data of the real person, wherein the feature data includes one or more of facial features, morphological features, and voice features of the real person, wherein the facial features and morphological features of the real person are extracted from the video of the real person, and the voice features of the real person are extracted from the audio of the real person; The source terminal is further used to send the characteristic data to the cloud server; The cloud server is used to send the characteristic data to the destination terminal; The destination terminal is used to generate media data of the digital human according to the feature data and the multimodal AI driving model, where the media data of the digital human includes the video and / or audio of the digital human.

18. The system according to claim 17, characterized in that The source terminal is specifically used for: Obtaining the video of the real person at a certain period; Feature data for describing the real person is extracted from the video of the real person, wherein the feature data includes one or more of changes in facial features of the real person and changes in morphological features of the real person.

19. The system according to claim 17 or 18, characterized in that Before the cloud server is used to send the characteristic data to the destination terminal, the cloud server is further used to: Acquiring input information on the target terminal, the input information indicating that the media data of the digital human is generated by the target terminal; It is determined that the destination terminal has a digital human generation capability.

20. The system according to any one of claims 17 to 19, characterized in that The source terminal is further used to: receive first input information, where the first input information indicates a feature type, and the feature type includes one or more of facial features, morphological features, and sound features; The source terminal is used to extract the feature data used to describe the real person from the media data of the real person, specifically including: The source terminal is used to extract target feature data from the media data of the real person, where the target feature data is feature data used to describe the real person and corresponds to the feature type indicated by the first input information.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method for providing a digital human according to any one of claims 1 to 6.

22. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method for providing a digital human according to any one of claims 7 to 10.

23. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as claimed in any one of claims 1 to 6.

24. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as described in any one of claims 7 to 10.

25. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as claimed in any one of claims 1 to 6.

26. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method for providing a digital human as claimed in any one of claims 7 to 10.

Images