Digital human interaction apparatus, method, device, storage medium, and program product

Abstract

Embodiments of the present application provide a digital human interaction device, method, equipment, storage medium and program product. The digital human interaction device runs on a terminal device, wherein the playback core layer can provide the ability to generate digital human picture frames according to the audio stream sent by the target server and render the picture frames, thereby migrating the digital human lip driving and picture rendering tasks to the terminal device side for execution. For the terminal device, only the audio stream needs to be obtained from the target server instead of the full amount of digital human video stream, which reduces the dependence on the network quality between the terminal device and the target server and the occupation of network bandwidth. On the one hand, the influence of network quality on the playback stability of digital human video is reduced, the playback smoothness of digital human video stream is significantly improved, and the risk of digital human video stream appearing to be stuck or out of sync with the lip sound is reduced. On the other hand, the network transmission cost is reduced, and the pressure on the network infrastructure is reduced.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a digital human interaction device, method, apparatus, storage medium, and program product. Background Technology

[0002] A "one-on-one (1V1) real-time interactive digital human" is an AI (Artificial Intelligence) virtual human capable of natural dialogue and personalized interaction with a single user. This 1V1 real-time interactive digital human not only understands what the user says but also provides immediate and realistic responses through voice, facial expressions, eye contact, and gestures. Currently, most 1V1 real-time interactive digital human implementations employ a cloud-based processing architecture, where the user's voice or text input is first uploaded to a cloud server. The cloud then performs the entire process of reasoning and video generation, including natural language understanding, speech synthesis, facial expression / lip-syncing, and rendering. The synthesized video stream is then transmitted to the user's terminal for playback via streaming protocols (such as RTC). However, this architecture relies heavily on the network quality between the cloud and the user's terminal, and the stability of the video stream is easily affected by network latency and jitter. Therefore, a new solution is needed. Summary of the Invention

[0003] This application provides a digital human interaction device, method, apparatus, storage medium, and program product to improve the stability of digital human video streams.

[0004] This application provides a digital human interaction device, which runs on a terminal device. The operating system running on the terminal device includes user mode and kernel mode. The digital human interaction device is a runtime driver module located in the user mode. An application in the user mode creates a digital human player instance at runtime through the digital human interaction device. The digital human interaction device includes: a service interface layer, used to: provide the application with a view of the digital human player instance and at least one playback control interface; and output corresponding playback control instructions based on the application's call to the at least one playback control interface, the playback control instructions including: a start playback instruction; and a bridging layer. The system is used for: downloading the background image and digital human model resources corresponding to the digital human player instance from the target server; the playback core layer is used for: generating continuous digital human image frames based on the digital human model resources and the audio stream sent from the target server under the drive of the start playback command; fusing the continuous digital human image frames and the background image to obtain continuous composite frames; calling the rendering module provided by the atomic capability layer to synchronize the audio stream and the continuous composite frames, and rendering the synchronized audio stream and the continuous composite frames in the view; the atomic capability layer is used for: encapsulating the rendering interface provided by the operating system into the rendering module for use.

[0005] This application provides a digital human interaction method applied to a digital human interaction device deployed on a terminal device. The operating system running on the terminal device includes user mode and kernel mode. The digital human interaction device is a runtime driver module located in the user mode. An application in the user mode creates a digital human player instance at runtime through the digital human interaction device. The method includes: providing the application with at least one playback control interface of the digital human player instance, the at least one playback control interface including a start playback interface; if a call to the start playback interface is detected, generating continuous digital human image frames based on digital human model resources and an audio stream sent by a target server; fusing the continuous digital human image frames with a background image to obtain continuous composite frames, wherein the digital human model resources and the background image are pre-downloaded; calling a rendering module to synchronize the audio stream with the continuous composite frames, and rendering the synchronized audio stream and the continuous composite frames in a view of the digital human player instance, wherein the view is embedded in the target interface of the application.

[0006] This application also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to perform the steps in the method provided in this application.

[0007] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the method provided in this application.

[0008] This application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in this application.

[0009] In this embodiment, a digital human interaction device runs on a terminal device. The playback core layer of the digital human interaction device can provide the ability to generate and render digital human image frames based on the audio stream sent by the target server, thereby migrating the lip-syncing and image rendering tasks of the digital human to the terminal device side. For the terminal device, it only needs to obtain the audio stream from the target server instead of the full digital human video stream, reducing the dependence on network quality and network bandwidth usage between the terminal device and the target server. On the one hand, this reduces the impact of network quality on the playback stability of the digital human video, significantly improving the playback smoothness of the digital human video stream and reducing the risk of stuttering or lip-syncing issues. On the other hand, it reduces network transmission costs and alleviates the pressure on network infrastructure, which is conducive to its application in various digital human interaction scenarios of different scales, such as high-concurrency e-commerce scenarios like digital human live-streaming e-commerce and digital human customer service. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a digital human interaction device 100 running on a terminal device, provided by an exemplary embodiment of this application; Figure 2 This is a schematic diagram of an exemplary embodiment of the present application of a digital human interaction device 100 in a live streaming room on a client side; Figure 3 This is a schematic diagram of the internal architecture of a digital human interaction device 100 provided in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of how an MNN is used to distribute the inference task of the lip-driven model to various hardware components on a terminal device, according to an exemplary embodiment of this application. Figure 5 A schematic diagram of the signaling interaction process of each layer in the digital human interaction device 100 provided in this application embodiment; Figure 6 A flowchart illustrating a digital human interaction method provided as an exemplary embodiment of this application; Figure 7 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one. “A plurality” generally includes at least two, but does not exclude the inclusion of at least one.

[0013] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0014] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0015] Some 1-on-1 real-time interactive digital human implementations employ a cloud-based processing architecture, with real-time streaming to the user terminal for decoding and playback. In this architecture, the client uploads the user's input voice or text data to a cloud server. The cloud then sequentially executes a complete inference chain, including Natural Language Understanding (NLU), Text-to-Speech (TTS), voice or text-based facial expression and lip-syncing, and face rendering, generating the final video frames. These frames are then pushed to the client for playback via Real-Time Communication (RTC). This architecture relies heavily on network quality between the cloud and the user terminal, and the stability of the video stream is susceptible to network latency and jitter. For example, in scenarios with poor network quality or high latency, stuttering, distortion, or lip-sync issues can easily occur, severely impacting the interactive experience. Secondly, to ensure low latency (typically <500ms end-to-end latency) and high image quality, each 1V1 session requires dedicated high-performance GPU (Graphics Processing Unit) resources for real-time rendering and AI (Artificial Intelligence) inference, resulting in high computational overhead per session and difficulty in horizontal scaling. Furthermore, high-definition video streams (such as 1080p / 30fps) continuously consume uplink bandwidth, placing enormous pressure on CDN (Content Delivery Network) and network infrastructure under large-scale concurrency, leading to high costs. This high cost makes it difficult to commercialize this technology in cost-sensitive large-scale commercial scenarios such as intelligent customer service, e-commerce, and social companionship.

[0016] To address this technical problem, a solution is provided in some embodiments of this application. The technical solutions provided by the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0017] Figure 1 This is a schematic diagram of a digital human interaction device 100 running on a terminal device, provided by an exemplary embodiment of this application. The operating system running on the terminal device includes user mode and kernel mode, and the digital human interaction device 100 is a runtime driver module located in the user mode.

[0018] The digital human interaction device 100 can be applied in various scenarios requiring human-like interaction, such as digital human shopping guides and virtual anchors in the e-commerce industry, financial customer service in the financial industry, online teaching assistants and homework help in the education industry, and health consultants and psychological counseling in the healthcare industry. This embodiment does not impose any limitations. The digital human interaction device can be implemented as an SDK (Software Development Kit) that encapsulates digital human interaction logic. Applications can integrate digital human interaction functions by calling the SDK's interfaces. In this embodiment, the application in user mode can create a digital human player instance at runtime through the digital human interaction device 100.

[0019] A digital human player instance is an independent and operational digital human interaction unit provided by the digital human interaction device 100. It can use the core capabilities provided by the digital human interaction device 100, such as audio playback, facial animation driving, audio and video rendering, and audio-visual synchronization, to expose control interfaces (such as play, pause, and beauty settings) to the outside world, and manage the lifecycle of resources (such as GPU textures, audio sessions, and model memory) internally.

[0020] When an application (such as a client) needs to display digital human videos, it can call methods provided by the SDK to create a digital human player instance. The SDK can initialize this digital human player instance, loading the digital human base model, initializing audio output, and the graphics rendering context. The SDK can then return an operable player object to the application. Figure 2 As shown, the client's live streaming room can embed the view associated with the digital human player instance into its own UI (User Interface) layer, making the digital human's image visible. This embedding of the view into the UI can be performed at any time after the digital human player instance is created. The application can call the control methods provided by the SDK for the digital human player instance. Under the application's call, the SDK can drive the digital human player instance to call the lip-driven model to render video data and complete processes such as animation generation and audio-visual synchronization rendering, achieving real-time audio-visual interaction with the digital human.

[0021] like Figure 1 As shown, the digital human interactive device 100 includes: a service interface layer 10, a bridging layer 20, a playback core layer 30, and a playback core layer 30.

[0022] Service interface layer 10 serves as the entry point for interaction between the application in user space and the digital human player instance. Service interface layer 10 primarily provides the application with a view of the digital human player instance and at least one playback control interface; and outputs corresponding playback control commands based on the application's calls to the at least one playback control interface. The view of the digital human player instance is a UI (User Interface) container used to display the digital human image within the application's interface.

[0023] The at least one playback control interface provided by the service interface layer 10 can be a standard interface applicable to different applications, ensuring that the digital human interactive device 100 can be quickly integrated into applications in different application scenarios. The service interface layer 10 can display the at least one playback control interface as a UI control object in the view of the digital human player instance for use by application users.

[0024] Optionally, the service interface layer 10 may provide at least one playback control interface to the application, including an interface for a playback control class. The interface for the playback control class may include, for example... Figure 3 The interfaces shown, such as start playback, pause playback, resume playback, and stop playback, may also include interfaces such as jumping to a specified playback node. The at least one playback control interface provided by the service interface layer 10 can be obtained by encapsulating the playback control methods in the internal interfaces of the playback core layer 30, for example... Figure 3 The start, stop, and pause methods shown are not described in detail here. The service interface layer 10 can output different playback control commands based on the application's calls to different control interfaces. For example, the service interface layer 10 can output a start playback command based on the application's call to the "start playback" interface, and can output a pause playback command based on the application's call to the "pause playback" interface. For the application, it can directly trigger the digital human player instance to start playback through at least one of these playback control interfaces and manage the playback behavior of the digital human player instance without needing to be aware of the underlying implementation details of the digital human interactive device 100.

[0025] The bridging layer 20 is primarily used to download the background image and digital human model resources corresponding to the digital human player instance from the target server. The target server can be a cloud server or a regular server, used to perform NLU and TTS (Text-to-Speech) based on user-input interactive text or speech to generate a video stream. The background image is used to render the background of the digital human live stream room, serving as visual content displayed behind the digital human to enhance or customize its presentation environment. The digital human basic model resources refer to the digital human's 3D face or body model, including its geometric structure, material textures, and skeletal rigging data (Rig), used to display the virtual image of the digital human in the live stream room.

[0026] The atomic capability layer 40 is used to encapsulate the rendering interfaces provided by the operating system into rendering modules for use by the playback core layer 30. Optionally, the operating system can provide audio rendering interfaces and video rendering interfaces. Accordingly, the encapsulated rendering modules can include audio rendering submodules and video rendering submodules. Figure 3 As shown, the atomic capability layer 40 can encapsulate the native video rendering interfaces of the iOS operating system (such as Metal) and the native video rendering interfaces of the Android operating system (such as Vulkan) into a unified video rendering submodule; it can also encapsulate the native audio interfaces of the iOS operating system (such as Core Audio) and the native audio interfaces of the Android operating system (such as AAudio) into a unified audio rendering submodule. Based on this encapsulation method, the atomic capability layer 40 can provide a unified rendering module to the playback core layer 30, thereby shielding the differences between different operating systems. In addition to audio and video rendering interfaces, the atomic capability layer can also... Figure 3 The audio decoding interface and video decoding interface shown are uniformly encapsulated for use by the playback core layer 30.

[0027] The playback core layer 30 has the capability to generate digital human images on the terminal device based on the audio stream and digital human model resources. The playback core layer 30 is primarily used to: generate continuous digital human image frames based on the digital human model resources and the audio stream sent by the target server, driven by playback instructions output from the service interface layer 10; and fuse these continuous digital human image frames with the background image to obtain continuous composite frames. The playback core layer 30 can call the rendering module provided by the atomic capability layer 40 to synchronize the audio stream with the continuous composite frames, and render the audio stream and the continuous composite frames in the digital human player instance. The rendering module is encapsulated based on the operating system's native video rendering interface and audio interface. The playback core layer 30 can call the rendering module to use instance methods of the video rendering class and audio rendering class provided by the operating system to complete image rendering and audio playback locally on the terminal device, without relying on the server side.

[0028] In this embodiment, a digital human interaction device runs on a terminal device. The playback core layer of the digital human interaction device 100 can provide the ability to generate and render digital human image frames based on the audio stream sent by the target server, thereby migrating the lip-syncing and image rendering tasks of the digital human to the terminal device side. For the terminal device, it only needs to obtain the audio stream from the target server instead of the full digital human video stream, reducing the dependence on network quality and network bandwidth usage between the terminal device and the target server. On the one hand, this reduces the impact of network quality on the playback stability of the digital human video, significantly improving the playback smoothness of the digital human video stream and reducing the risk of stuttering or lip-syncing issues. On the other hand, it reduces network transmission costs and alleviates the pressure on network infrastructure, which is conducive to its application in various digital human interaction scenarios of different scales, such as high-concurrency e-commerce scenarios like digital human live-streaming e-commerce and digital human customer service.

[0029] Furthermore, in this implementation, the playback core layer 30 generates continuous digital human image frames and merges the continuous digital human image frames with the background image to obtain continuous composite frames. The continuous composite frames can be rendered directly without performing encoding and decoding operations on the continuous composite frames, which saves the computing power overhead caused by encoding and decoding and improves the rendering speed of digital human images.

[0030] In some embodiments, the bridging layer 20 is also used to connect the application and the playback kernel of the digital human player instance. In this embodiment, the service interface layer 10 is further used to: provide a player creation interface to the application; detect the application's call to the player creation interface, and determine the target digital human object to be played based on the call parameters of the player creation interface; wherein, the target digital human object can be a role of the digital human (e.g., salesperson A or salesperson B) or an identifier of a live broadcast room (e.g., the identifier of merchant B1's live broadcast room or the identifier of merchant B2's live broadcast room). The service interface layer 10 can output the initialization instructions corresponding to the target digital human object.

[0031] The call to the player creation interface within an application can be triggered by specific user actions. For example, in the e-commerce industry, when a user enters a product details page or clicks the "virtual shopping guide" button in an online shopping application, the application can initiate a call to the player creation interface. Similarly, in the financial industry, when a user clicks the "AI customer service" entry point within a bank's client application, the application can initiate a call to the player creation interface.

[0032] Initializing the kernel of the digital human player instance refers to configuring a dedicated runtime context for that instance, enabling it to independently handle the digital human's audiovisual output. The playback kernel primarily includes an audio output channel and a graphics rendering context. The audio output channel is responsible for playing the audio stream.

[0033] Different digital human player instances can have independent audio channels to ensure that multiple digital humans do not interfere with each other when speaking simultaneously. The graphics rendering context refers to the independent environment allocated to each digital human player instance for managing and executing image rendering-related operations. This environment may include key components such as GPU resources for image processing and display, a command queue, and rendering targets. The command queue is an ordered list of tasks containing drawing instructions, such as data acquisition instructions and shader usage instructions, which are sent to the GPU for execution in sequence. The rendering target specifies the output location of the final generated image. The graphics rendering context works in conjunction with the lip-driven thread, which generates facial motion parameters (such as FLAME expression coefficients) based on the audio stream. The rendering context then applies these facial motion parameters to the digital human model resources and outputs the synthesized image. Furthermore, the player kernel may include an animation state machine for managing speaking / silent states, a time synchronization controller, and resource reference relationships (such as model and texture holding relationships). Based on initialization operations, the digital human player kernel can independently receive instructions, drive audiovisual synthesis, and manage its own lifecycle.

[0034] In this implementation, the playback kernel of the digital human player instance can be initialized upon explicit request from the application through the interaction between the service interface layer 10 and the bridging layer 20, thus enabling on-demand loading of the playback kernel without occupying related resources during the application startup phase.

[0035] Secondly, this mechanism supports multi-instance concurrency. Each time the application calls the player creation interface, it creates a completely isolated digital human playback instance and initializes the playback kernel, allowing multiple digital humans to run simultaneously without interference between their audio channels, rendering targets, and animation states. When the digital human interactive device 100 creates multiple digital human player instances, the bridge layer 20 manages the lifecycle of these instances and performs resource isolation and scheduling management. Resource isolation refers to isolating the playback kernels of different digital human player instances to ensure that they do not interfere with each other during concurrent operation. Scheduling management refers to selectively sharing some stateless resources, such as lip-driven models or algorithm logic, between the playback kernels of different digital human player instances to balance performance and resource efficiency.

[0036] In some optional embodiments, the service interface layer 10 is further configured to: provide the application with at least one rendering configuration interface for controlling the digital human player instance. The at least one rendering configuration interface may include, but is not limited to, at least one of: a beauty control interface, a frame rate control interface, a background replacement interface, and a virtual human model switching interface. The service interface layer 10 may display the at least one rendering configuration interface as a UI control object in the view of the digital human player instance for use by the application's user.

[0037] The service interface layer 10 can output corresponding rendering configuration instructions based on the application's call to at least one rendering configuration interface. Correspondingly, the bridging layer 20 is also used to adjust the rendering parameters of the digital human player instance according to the rendering configuration instructions. For example, the service interface layer 10 can output a frame rate adjustment instruction based on the application's call to the "frame rate control" interface. The bridging layer 20 can then dynamically adjust the rendering frame rate of the digital human player according to the frame rate adjustment instruction.

[0038] Based on this implementation method, the application can flexibly control the rendering effect of the digital human player instance directly through at least one rendering configuration interface without being aware of the underlying implementation details of the digital human player instance.

[0039] In some alternative embodiments, the playback core layer may include multiple cooperating worker threads. For example... Figure 3As shown, the multiple worker threads may include: audio data receiving thread 301, lip driving thread 302, audio rendering thread 303, and video rendering thread 304.

[0040] The audio data receiving thread 301 is used to: receive the audio stream sent by the target server and store the audio stream in a designated space. Optionally, the audio stream can be received in real time via an RTQ (Real-Time Queue). The RTQ acts as a buffer between the TTS and the digital human player instance in the target server, allowing the target server and the digital human player instance to work at different paces. Furthermore, RTQs typically carry timestamps, ensuring that the audio is played in precise timing and reducing stuttering or abrupt output. The audio data receiving thread 301 can receive an uncompressed, raw audio stream; therefore, the designated space for storing the audio stream can be a PCM (Pulse Code Modulation) buffer.

[0041] like Figure 3 As shown, the lip-driven thread 302 can call instance methods of the video production class, which can be embedded as a component into the mobile live streaming module. The lip-driven thread 302 is mainly used to: call the locally deployed lip-driven model, extract facial motion parameters from the audio stream, and generate continuous digital human image frames based on the facial motion parameters and the digital human model resources. Figure 3 As shown, the digital human model resource can be read by the material loading thread using a reader. In this embodiment, the lip-driven model is a neural network model specifically trained for lip-sync. The lip-driven model is used to predict, in real time, a sequence of facial motion parameters that highly matches the pronunciation based on the input audio segment. These facial motion parameter sequences are used to accurately describe details such as lip opening and closing, corner of the mouth movement, and chin displacement at each moment, thereby achieving a natural speaking effect that is "sound and image in harmony".

[0042] The structure and reasoning process of the lip-driven model will be illustrated below.

[0043] In some optional embodiments, the lip-driven model employs an end-to-end architecture of speech feature extraction, temporal modeling, and action parameter regression. Its model structure includes a speech encoder, a temporal alignment and context modeling module, and a lip-shape decoder. The speech encoder can be implemented using a one-dimensional convolutional network (Conv1D) or a Transformer (a self-attention-based neural network architecture). The input to the speech encoder is raw audio (such as PCM) or audio features (such as Mel-spectrogram, MFCC (Mel-Frequency Cepstral Coefficients) features, etc.). The speech encoder maps the input raw audio or audio features into a high-dimensional semantic feature sequence, and its output is a temporally aligned hidden state sequence. The temporal alignment and context modeling module can be implemented based on a Transformer or Conformer (a speech modeling architecture combining convolution and self-attention mechanisms) to model the non-linear, cross-frame dependency between speech and lip shape; for example, a phoneme may affect lip movements across multiple frames. The temporal alignment and context modeling module takes as input the hidden state sequence output by the speech encoder and outputs an enhanced context feature sequence composed of temporal features. Each temporal feature incorporates global speech context information to accurately reflect the lip shape state that should be driven at that moment. The lip shape decoder can contain fully connected layers or an MLP, and its input is the context feature sequence output by the temporal modeling module. The lip shape decoder can map each temporal feature to specific facial control parameters through fully connected layers or an MLP. These facial action parameters may include: blendshape weight vectors, 3D keypoint coordinates (such as lip contour points), and texture deformation fields.

[0044] Based on this implementation, migrating lip-driven operations to the terminal device significantly reduces the computational, inference, encoding, and bandwidth load on the target server (e.g., a cloud server). For the target server, the communication channel between it and the digital human player instance only needs to transmit a small amount of audio data instead of the full video stream, reducing the dependence on the network quality of the communication channel and improving the stability and playback quality of the digital human video on the terminal device side.

[0045] In some optional embodiments, the terminal device is equipped with an NPU (Neural Processing Unit), a dedicated processor designed specifically for artificial intelligence computing, especially deep learning inference and training. The lip-driven thread 302 is specifically used to: send the inference task of the lip-driven model to the neural processing unit for execution, so that the neural processing unit extracts facial motion parameters from the audio stream according to the inference task, and generates continuous digital human image frames based on the facial motion parameters and the digital human model resources. Figure 4 As shown, in some embodiments, the hardware layer of the terminal device may include hardware such as a CPU (Central Processing Unit), GPU, and NPU. Between the lip-sync driving model and the terminal device's hardware layer, an MNN (Mobile Neural Network) framework can be used as a bridge. The MNN analyzes the operators in the lip-sync driving model and distributes them to the CPU, GPU, or NPU for execution. When the MNN detects that the terminal device supports an NPU and that the operators in the lip-sync driving model are compatible with the NPU, it can offload these computational tasks to the NPU for execution, thereby leveraging the NPU to accelerate inference.

[0046] The NPU is highly optimized for neural network operations such as matrix multiplication and addition, convolution, and activation functions. Compared to processors like GPUs, it offers advantages such as lower power consumption, higher computational throughput per unit of energy, and better real-time performance. Based on this, using the NPU to accelerate lip-sync model inference on terminal devices allows for rapid inference of digital human video frames without a server connection. This reduces server computational load and bandwidth consumption, further lowering the computing power and network bandwidth costs on the server side (e.g., cloud servers) compared to traditional solutions. Furthermore, it ensures low latency and high performance in the digital human video frame inference process. In some embodiments, using the NPU to accelerate lip-sync model inference can reduce lip-sync timing to within 6ms, significantly improving the smoothness of digital human videos.

[0047] like Figure 3 As shown, audio rendering thread 303 can call instance methods of the audio rendering class and interact with the audio queue engine. Specifically, audio rendering thread 303 is used to: retrieve data from the specified space (e.g., ... Figure 3The PCM buffer (shown) reads the audio stream and caches it in the audio queue engine. Driven by the playback command, the audio queue engine calls the audio rendering submodule in the rendering module to play the audio stream and outputs its time base information. Time base information refers to the information in the audio stream used to identify the position of each frame of data on the timeline. This time base information is used to perform audio-visual synchronization.

[0048] like Figure 3 As shown, the video rendering thread 304 can obtain digital human image frames and background images from the lip-driving thread 302 and the material loading thread, respectively, and perform rendering and layer compositing operations by calling instance methods of the video rendering class through the video rendering submodule. Specifically, the video rendering thread 304 is used to: composite the continuous digital human image frames and the background image to obtain continuous composite frames, and, based on the time reference information, call the video rendering submodule provided by the atomic capability layer 40 to render the continuous composite frames. The video rendering thread 304 can call instance methods of the video rendering class through the video rendering submodule. Before rendering each composite frame, the video rendering thread 304 can obtain the actual playback time point T of the audio stream based on the time reference information, determine the composite frame corresponding to time point T in the continuous composite frames based on time point T, and send the composite frame to the video rendering submodule for rendering, thereby achieving complete audio-visual synchronization at time point T. In this process, the dynamic updates of the screen are driven by the audio timing, ensuring that the lip movements of the digital human seen by the user are highly consistent with the voice content heard, thereby achieving a natural and credible digital human interaction experience.

[0049] In this implementation, video rendering, audio rendering, and audio-visual synchronization are all performed locally on the terminal device. Each key processing step does not need to interact with the server side, so it is less affected by the end-to-end network quality. Even under network jitter or weak network conditions, it can still ensure real-time matching of lip movements and speech, ensuring that users have a smooth and natural interactive experience when interacting with the digital human.

[0050] Figure 5 The signaling interaction flow of each layer in the digital human interaction device 100 provided in this application embodiment is illustrated, as follows: Figure 5 As shown, when the client needs to play digital human videos, it can send an initialization command to the service interface layer through the player creation interface. This initialization command is used to trigger the startup of the digital human interactive device. The service interface layer, as the direct interaction point between the digital human interactive device and the application, is responsible for providing standardized control interfaces, enabling the application to seamlessly integrate digital human functions without needing to be aware of the underlying implementation details.

[0051] The service interface layer passes initialization commands to the bridging layer, which is responsible for connecting the application and the playback kernel. For example... Figure 5 As shown, after receiving the initialization command, the bridging layer can perform material download operations, obtain preset background images and digital human model resources, and perform initialization operations on the digital human player instance to ensure that the resources required for subsequent rendering are ready. At the same time, it maintains the lifecycle state of the digital human player instance to support efficient scheduling in multi-instance scenarios.

[0052] The bridging layer can further forward initialization commands to the playback core layer, enabling the playback core layer to initiate the rendering context initialization process. During initialization, the playback core layer is mainly used to configure the necessary rendering environment, including allocating memory resources and initializing the NPU acceleration module, laying the foundation for subsequent real-time lip movement and image compositing.

[0053] After initialization, as follows Figure 5 As shown, the client can embed the digital human view into the client's interface to ensure that the digital human image can be presented directly in the client without relying on external video stream transmission, significantly reducing network bandwidth consumption.

[0054] like Figure 5 As shown, when a user initiates a playback request through the client, the client sends a start playback command to the service interface layer, which then forwards the command to the bridging layer. Upon receiving the command, the bridging layer sets the media path parameters and passes the start playback command to the playback core layer to trigger the construction of the workflow and thread startup, achieving efficient conversion from command to rendering.

[0055] Upon receiving a start playback command, the playback core layer can construct a multi-threaded pipeline, which may include an audio data receiving thread, an audio rendering thread, a lip-driven thread, and a video rendering thread. The audio data receiving thread receives speech input via RTQ and temporarily stores the raw audio stream in a PCM buffer. The lip-driven thread receives speech feature data via RTQ and uses the NPU to accelerate the inference of digital human image data. The video rendering thread acquires and renders the digital human image data, while the audio rendering thread processes the data through an audio queue engine. Both processes are synchronized to ensure accurate matching between lip movements and speech.

[0056] This application also provides a digital human interaction method, applied to a digital human interaction device deployed on a terminal device. The operating system running on the terminal device includes user mode and kernel mode. The digital human interaction device is a runtime driver module located in the user mode. The application in the user mode creates a digital human player instance at runtime through the digital human interaction device. Figure 6 As shown, the method mainly includes: Step 601: Provide the application with at least one playback control interface of the digital human player instance, wherein the at least one playback control interface includes: a start playback interface.

[0057] Step 602: If a call to the start playback interface is detected, then generate continuous digital human image frames based on the digital human model resources and the audio stream sent by the target server.

[0058] Step 603: Fuse the continuous digital human image frames and the background image to obtain continuous composite frames. The digital human model resources and the background image are pre-downloaded.

[0059] Step 604: Invoke the rendering module to synchronize the audio stream with the continuous composite frames, and render the synchronized audio stream and the continuous composite frames in the view of the digital human player instance. The view is embedded in the target interface of the application.

[0060] The rendering module is a wrapper around the rendering interface provided by the operating system to shield the differences between different operating systems. Optionally, the rendering module may include an audio rendering submodule and a video rendering submodule.

[0061] In some optional embodiments, before detecting the application's call to at least one playback control interface of the digital human player instance, the method further includes: providing a player creation interface to the application; detecting the application's call to the player creation interface, and determining the target digital human object to be played based on the call parameters of the player creation interface; creating a player instance corresponding to the target digital human object, and initializing the digital human player instance.

[0062] In some optional embodiments, the method further includes: providing the application with at least one rendering configuration interface for controlling the digital human player instance; and adjusting the rendering parameters of the digital human player instance based on the application's call to the at least one rendering configuration interface.

[0063] In some optional embodiments, one way to generate continuous digital human image frames based on digital human model resources and an audio stream sent by the target server may include: using a lip-driven thread to call a locally deployed lip-driven model, extracting facial motion parameters from the audio stream, and generating continuous digital human image frames based on the facial motion parameters and the digital human model resources.

[0064] In some optional embodiments, the terminal device is equipped with a neural network processing unit; one method of using a lip-driven thread to call a locally deployed lip-driven model to extract facial motion parameters from the audio stream and generate continuous digital human image frames based on the facial motion parameters and the digital human model resources may include: sending the inference task of the lip-driven model to the neural network processing unit for execution, so that the neural network unit extracts facial motion parameters from the audio stream according to the inference task and generates continuous digital human image frames based on the facial motion parameters and the digital human model resources.

[0065] In some optional embodiments, the audio stream sent by the target server is received by an audio data receiving thread and stored in a designated space; a method of calling the rendering module to synchronize the audio stream with the continuous composite frames and rendering the synchronized audio stream and the continuous composite frames in the view of the digital human player instance may include: using an audio rendering thread to read the audio stream from the designated space, caching the audio stream to an audio queue engine, the audio queue engine being used to call the audio rendering submodule in the rendering module to play the audio stream under the drive of the playback instruction, and outputting the time base information of the audio stream; compositing the continuous digital human image frames and the background image to obtain continuous composite frames; and calling the video rendering submodule in the rendering module to render the continuous composite frames in the view of the digital human player instance according to the time base information.

[0066] In this implementation, a digital human interaction device can run on the terminal device. The playback core layer of the digital human interaction device can provide the ability to generate and render digital human image frames based on the audio stream sent by the target server, thereby migrating the tasks of lip-syncing and image rendering of the digital human to the terminal device side. For the terminal device, it only needs to obtain the audio stream from the target server instead of the full digital human video stream, reducing the dependence on network quality and network bandwidth usage between the terminal device and the target server. On the one hand, this reduces the impact of network quality on the playback stability of the digital human video, significantly improving the playback smoothness of the digital human video stream and reducing the risk of stuttering or lip-syncing issues. On the other hand, it reduces network transmission costs and alleviates the pressure on network infrastructure, which is conducive to its application in various digital human interaction scenarios of different scales, such as high-concurrency e-commerce scenarios like digital human live-streaming sales and digital human customer service.

[0067] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 601 to 604 can be device A; or the execution subject of steps 601 and 602 can be device A, and the execution subject of step 603 can be device B; and so on.

[0068] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 601, 602, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0070] Figure 7 This application illustrates a schematic diagram of the structure of an electronic device provided in an exemplary embodiment, as shown below. Figure 7 As shown, the electronic device includes: a memory 701, a processor 702, and a communication component 703.

[0071] Memory 701 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0072] The processor 702, coupled to the memory 701, is used to execute the computer program in the memory 701 to perform the method steps performed by the digital human interaction device in the foregoing embodiments. For details, please refer to the descriptions of the foregoing embodiments, which will not be repeated here.

[0073] Furthermore, such as Figure 7As shown, the electronic device also includes other components such as a power supply component 704, a display component 705, and an audio component 706. Figure 7 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 7 The components shown. Figure 7 In this embodiment, the components within the dashed boxes are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 7 The components within the dashed box; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, etc., it may be omitted. Figure 7 The component within the dashed box.

[0074] The memory 701 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0075] The communication component 703 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0076] The power supply component 704 is used to provide power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0077] The display component includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0078] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0079] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0080] Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0081] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes said element.

[0082] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A digital human interactive device, characterized in that, The digital human interaction device runs on a terminal device, and the operating system running on the terminal device includes user mode and kernel mode. The digital human interaction device is a runtime driver module located in the user mode. The application in the user mode creates a digital human player instance at runtime through the digital human interaction device; The digital human interactive device includes: The service interface layer is used to: provide the application with a view of the digital human player instance and at least one playback control interface; and output corresponding playback control instructions based on the application's call to the at least one playback control interface, the playback control instructions including: a start playback instruction; A bridging layer is used to: download the background image and digital human model resources corresponding to the digital human player instance from the target server; The playback core layer is used to: generate continuous digital human image frames based on the digital human model resources and the audio stream sent from the target server, driven by the start playback command; fuse the continuous digital human image frames with the background image to obtain continuous composite frames; call the rendering module provided by the atomic capability layer to synchronize the audio stream with the continuous composite frames, and render the synchronized audio stream and the continuous composite frames in the view; The atomic capability layer is used to: encapsulate the rendering interface provided by the operating system into the rendering module for use.

2. The apparatus according to claim 1, characterized in that, The service interface layer is also used to: provide a player creation interface to the application; detect the application's call to the player creation interface, and determine the target digital human object to be played based on the call parameters of the player creation interface; Output the initialization command corresponding to the target digital human object; The bridging layer is also used to: create a player instance corresponding to the target digital human object according to the initialization instruction, and initialize the digital human player instance.

3. The apparatus according to claim 1, characterized in that, The service interface layer is also used to: provide the application with at least one rendering configuration interface for controlling the digital human player instance; and output corresponding rendering configuration instructions according to the application's call operation to the at least one rendering configuration interface; The bridging layer is also used to adjust the rendering parameters of the digital human player instance according to the rendering configuration instructions.

4. The apparatus according to any one of claims 1-3, characterized in that, The playback core layer includes: Audio data receiving thread, lip-driven thread, audio rendering thread, and video rendering thread; The audio data receiving thread is used to: receive the audio stream sent by the target server and store the audio stream in a specified space; The lip-driven thread is used to: call the locally deployed lip-driven model, extract facial motion parameters from the audio stream, and generate continuous digital human image frames based on the facial motion parameters and the digital human model resources; The audio rendering thread is used to: read the audio stream from the specified space, cache the audio stream in the audio queue engine, and the audio queue engine is used to call the audio rendering submodule in the rendering module to play the audio stream under the drive of the playback instruction, and output the time base information of the audio stream; The video rendering thread is used to: synthesize the continuous digital human image frames and the background image to obtain continuous composite frames, and, according to the time reference information, call the video rendering submodule in the rendering module to render the continuous composite frames.

5. The apparatus according to claim 4, characterized in that, The terminal device is equipped with a neural network processing unit. The lip-driving thread is used to: send the inference task of the lip-driving model to the neural network processing unit for execution, so that the neural network unit extracts facial motion parameters from the audio stream and generates continuous digital human image frames based on the facial motion parameters and the digital human model resources.

6. A digital human interaction method, characterized in that, A digital human interaction device deployed on a terminal device, wherein the operating system running on the terminal device includes user mode and kernel mode, and the digital human interaction device is a runtime driver module located in the user mode; The application in the user mode creates a digital human player instance at runtime through the digital human interaction device; The method includes: providing the application with at least one playback control interface of the digital human player instance, the at least one playback control interface including: a start playback interface; If a call to the start playback interface is detected, a series of digital human image frames are generated based on the digital human model resources and the audio stream sent by the target server. The continuous digital human image frames and background images are fused together to obtain continuous composite frames. The digital human model resources and the background images are pre-downloaded. The rendering module is invoked to synchronize the audio stream with the continuous composite frames, and the synchronized audio stream and the continuous composite frames are rendered in the view of the digital human player instance. The view is embedded in the target interface of the application.

7. The method according to claim 6, characterized in that, Before detecting the application's call to at least one playback control interface of the digital human player instance, the method further includes: Provide the application with a player creation interface; The application's call to the player creation interface is detected, and the target digital human object to be played is determined based on the call parameters of the player creation interface. Create a player instance corresponding to the target digital human object, and initialize the digital human player instance.

8. The method according to claim 6, characterized in that, Also includes: Provide the application with at least one rendering configuration interface for controlling the digital human player instance; The rendering parameters of the digital human player instance are adjusted according to the application's call to the at least one rendering configuration interface.

9. The method according to any one of claims 6-8, characterized in that, Based on the digital human model resources and the audio stream sent by the target server, a continuous series of digital human image frames are generated, including: The system employs a lip-driven thread to call a locally deployed lip-driven model, extracts facial motion parameters from the audio stream, and generates continuous digital human image frames based on the facial motion parameters and the digital human model resources.

10. The method according to claim 9, characterized in that, The terminal device is equipped with a neural network processing unit; Using a lip-driven thread, a locally deployed lip-driven model is invoked to extract facial motion parameters from the audio stream, and continuous digital human image frames are generated based on the facial motion parameters and the digital human model resources, including: The inference task of the lip-driven model is sent to the neural network processing unit for execution, so that the neural network unit extracts facial motion parameters from the audio stream according to the inference task, and generates continuous digital human image frames according to the facial motion parameters and the digital human model resources.

11. The method according to claim 9, characterized in that, The audio stream sent by the target server is received by the audio data receiving thread and stored in a designated space; the rendering module is invoked to synchronize the audio stream with the continuous composite frames, and the synchronized audio stream and the continuous composite frames are rendered in the view of the digital human player instance, including: An audio rendering thread is used to read the audio stream from the specified space and cache the audio stream in the audio queue engine. The audio queue engine is used to call the audio rendering submodule in the rendering module to play the audio stream under the drive of the playback instruction and output the time base information of the audio stream. A video rendering thread is used to synthesize the continuous digital human image frames and the background image to obtain continuous composite frames; Based on the time reference information, the video rendering submodule in the rendering module is invoked to render the continuous composite frames in the view of the digital human player instance.

12. An electronic device, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 6-11.

13. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method described in any one of claims 6-11.

14. A computer program product, characterized in that, include: A computer program / instruction that, when executed by a processor, enables the implementation of the steps in the method described in any one of claims 6-11.

Images