Digital human interaction method and device

Abstract

The invention discloses a digital human interaction method and device, relates to the field of data processing, and is used for solving the problem of digital human jamming. The method comprises the following steps: monitoring whether an interaction lagging condition exists or not in a process of interacting with a front-end user through a digital human displayed on front-end equipment; if the interaction lagging condition exists, acquiring a first video frame displayed on the front-end equipment and a next to-be-displayed second video frame; and generating at least one mixed frame between the first video frame and the second video frame through the first video frame and the second video frame, and sending the mixed frame to the front-end equipment. Through the scheme, the fluency of the digital human can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a digital human interaction method and apparatus. Background Technology

[0002] Hypertext Markup Language 5 (H5) authentication is a remote identity verification process completed within a browser. Specifically, users access a webpage via a link or QR code, authorize their camera and microphone, and the page then captures their face and voice in real time. The captured audio and video are then transmitted to the cloud, where a server performs liveness detection, face comparison, document verification, and intent recognition, returning the results to the page. However, current digital human interaction scenarios experience lag or stuttering issues.

[0003] Therefore, improving the fluency of digital humans is an urgent problem that needs to be solved. Summary of the Invention

[0004] This invention provides a digital human interaction method and apparatus to improve the fluency of digital humans.

[0005] In a first aspect, the present invention provides a digital human interaction method applied to a backend server. The method includes: monitoring whether there is an interaction lag during the interaction between a digital human displayed on a frontend device and a frontend user; if there is an interaction lag, acquiring a first video frame displayed on the frontend device and a second video frame to be displayed next; generating at least one mixed frame between the first video frame and the second video frame using the first video frame and the second video frame, and sending it to the frontend device.

[0006] By employing the above approach, during the interaction between the digital human displayed on the front-end device and the front-end user, monitoring for interaction stutters allows for real-time determination of discontinuous interaction states, triggering remedial strategies immediately and reducing the abruptness caused by interaction interruptions. If interaction stutters occur, transitional content can be inserted into the original frame sequence, presenting frame changes in smaller steps, reducing the subjective perception of screen tearing, jitter, and lip-sync misalignment, thereby improving the continuous viewing and operation experience for the front-end user. Furthermore, shifting related computational tasks to the back-end server reduces the computational burden on the front-end. This solution can significantly improve the success rate of digital human identity verification, especially in such scenarios.

[0007] Optionally, monitor whether there is any interaction lag, including: determining whether there is a predefined state transition during the interaction process; if there is a state transition, then determine that there is an interaction lag.

[0008] In digital human identity verification scenarios, the normal interaction flow should be smooth and continuous. When a state change is detected, it means the digital human is jumping from one state to another. This jump is often accompanied by rendering interruption, resulting in a perceived stuttering effect for the user. The above solution allows for the triggering of corresponding compensation strategies once stuttering is confirmed, thereby ensuring the smoothness and continuity of digital human interaction.

[0009] Optionally, generating at least one mixed frame between the first video frame and the second video frame using the first video frame and the second video frame includes: determining a first transition duration between the first video frame and the second video frame, and the elapsed duration from the first video frame to the current time; determining a first mixing parameter based on the ratio of the elapsed duration to the first transition duration; and generating at least one mixed frame between the first video frame and the second video frame based on the first mixing parameter, the first video frame, and the second video frame.

[0010] By employing the above approach, and generating intermediate transition frames through time calculation and dynamic blending parameters, a smooth and natural transition between video frames in a digital human can be achieved. In digital human state switching scenarios, the first and second video frames represent two key states before and after the switch. For example, when switching from a speaking state to a silent state, the first video frame might show an open mouth, while the second video frame shows a closed mouth. If the user jumps directly from the first frame to the second frame, they will clearly perceive abrupt changes and a mechanical feel. By determining the transition duration between the two frames and the elapsed duration, the blending ratio of the blended frame to be generated at the current moment can be dynamically calculated. This ratio changes continuously over time, thereby generating a series of transition frames between the two key frames, solving the visual discontinuity problem in digital human rendering.

[0011] Optionally, determining whether a predefined state transition exists during the interaction includes: determining whether a predefined state transition exists based on the digital human's audio information; and / or, determining whether a predefined state transition exists based on the digital human's instruction information.

[0012] The above solutions enable accurate identification of digital human state transitions. Audio-based detection allows for automated state perception by analyzing real-time audio signal parameters such as volume thresholds, audio energy, and speech activity to automatically determine the current state of the digital human. Command-based detection provides proactive control capabilities, triggering predefined state transitions at specific times based on business process needs. Combining these two detection methods enhances the flexibility and adaptability of the digital human.

[0013] Optionally, monitoring for interactive stuttering includes: receiving feedback from the front-end device regarding changes in the digital human's speech rate; if the speech rate change exceeds a set threshold for N consecutive frames, then interactive stuttering is confirmed.

[0014] The above approach uses real-time feedback from the front-end device regarding speech rate changes to monitor interactive stuttering, accurately identifying anomalies during the digital human's broadcast. Under normal circumstances, the digital human's speech rate should remain relatively stable. When the front-end device detects a significant change in speech rate, it usually indicates audio / video transmission delays, network fluctuations, or abnormalities in the front-end playback buffer, directly leading to perceived interactive stuttering by the user. By continuously receiving speech rate data from the front-end, the back-end server can monitor the actual playback status in real time, rather than relying solely on its own sending status, thus more accurately reflecting the user's actual experience.

[0015] Optionally, generating at least one hybrid frame between the first video frame and the second video frame using the first video frame and the second video frame includes: determining a second transition duration based on the current speech rate of the digital human; determining a second mixing parameter based on the second transition duration; and generating at least one hybrid frame between the first video frame and the second video frame based on the second mixing parameter, the first video frame, and the second video frame.

[0016] By using the above approach and dynamically determining the transition duration and mixing parameters based on the digital human's current speech rate as a key parameter, intelligent adaptation between video frame transition effects and the audio playback rhythm can be achieved. During the digital human's playback, the speech rate is not constant but fluctuates depending on the content, the need for emotional expression, and changes in network transmission conditions. When the speech rate is fast, phoneme changes are frequent, and lip movements should be more rapid, requiring a shorter transition duration to match the fast-paced audio content. When the speech rate is slow, phoneme changes are gradual, and lip movements should be more deliberate, requiring a longer transition duration to maintain natural and smooth movements. By sensing the current speech rate in real time and adjusting the transition duration accordingly, it is ensured that the transition rhythm of video frames always remains consistent with the audio playback rhythm, avoiding inconsistencies such as lip movement lag caused by fast speech but slow video transitions, or lip movement jumps caused by slow speech but fast video transitions.

[0017] Optionally, before monitoring for any interaction stuttering, the process may also include: triggering a silent video frame indicating that the digital human is in a silent state after detecting the video screen of the front-end user; triggering the digital human's broadcast audio frame when the digital human enters the broadcasting state, and triggering the digital human's broadcast video frame after detecting the audio information.

[0018] The above approach addresses the issue that in digital human identity verification scenarios, formal interaction cannot begin until the user is ready. If the digital human starts broadcasting too early, issues such as the user not yet appearing on screen, failed face detection, or unauthorized device permissions may arise, leading to a chaotic interaction flow and a fragmented user experience. By detecting the user's video feed as a trigger, it's confirmed that the user has successfully entered the interactive interface and the camera is functioning correctly. At this point, a silent video frame is triggered, allowing the digital human to appear before the user in a static or slightly moving state. This demonstrates the digital human's presence while avoiding information omissions caused by broadcasting before the user is ready, thus establishing a buffer period for subsequent formal interaction.

[0019] Optionally, the method further includes: storing silent video frames representing the silent state through a front-end device; triggering silent video frames representing the digital human being in a silent state, including: triggering the front-end device to retrieve silent video frames locally and play them in a loop.

[0020] By pre-storing silent video frames locally on the front-end device, the above solution significantly reduces network transmission pressure, lowers response latency, and reduces the burden of repeated rendering on the back-end server. The silent state is the most frequent and longest-lasting state in digital human interaction, occurring during multiple stages such as user entry, waiting for face detection, announcement intervals, and waiting for user response. By pre-storing silent video frames locally on the front-end device, the silent state can be read directly from the local storage when needed, eliminating the need for network transmission.

[0021] Optionally, sending to the front-end device includes: for any mixed frame, calculating the timestamp of the mixed frame based on the frame interval and clock frequency; if the timestamp of the mixed frame is greater than the current time, waiting for a set duration; the set duration is determined based on the timestamp of the mixed frame and the current time; after the set duration, sending the mixed frame to the front-end device.

[0022] The above scheme, employing a timestamp-based predictive sleep compensation mechanism, enables precise control over the timing of mixed frame transmission. In real-time digital human interaction scenarios, by calculating the theoretical timestamp of each mixed frame based on frame intervals and clock frequency, it becomes clear when the frame should be sent to the front end. This theoretical timestamp represents the frame's standard time position within the entire audio and video stream. When a mixed frame's timestamp is detected to be greater than the current time, it indicates that the frame's transmission timing has not yet arrived. In this case, a waiting mechanism is used to postpone transmission until the current time catches up with the frame's theoretical timestamp, thus ensuring that each mixed frame is sent at its precise time, achieving a strict match between frame transmission rhythm and theoretical playback rhythm.

[0023] Optionally, it also includes: transmitting audio and video to the front-end device based on WebRTC (Web Real-Time Communication).

[0024] The above solutions, employing the WebRTC protocol for audio and video transmission, enable low-latency, real-time data transmission, meeting the stringent requirements for instant interaction in digital human identity verification scenarios. Furthermore, by offloading computational tasks to backend servers, the current limitations of H5's computational capabilities and resource constraints are addressed.

[0025] Secondly, the present invention provides a digital human interaction device applied to a backend server, the device comprising: The monitoring module is used to monitor for any interaction stuttering during the interaction between the digital human displayed on the front-end device and the front-end user.

[0026] The acquisition module is used to acquire the first video frame displayed on the front-end device and the next second video frame to be displayed if there is an interaction lag.

[0027] The generation module is used to generate at least one mixed frame between the first video frame and the second video frame using the first video frame and the second video frame, and send it to the front-end device.

[0028] In one possible implementation, the monitoring module is specifically used to: determine whether there is a predefined state transition during the interaction process; if there is a state transition, then determine that there is an interaction lag.

[0029] In one possible implementation, the generation module is specifically configured to: determine a first transition duration between a first video frame and a second video frame, and the elapsed duration from the first video frame to the current time; determine a first mixing parameter based on the ratio of the elapsed duration to the first transition duration; and generate at least one mixed frame between the first video frame and the second video frame based on the first mixing parameter, the first video frame, and the second video frame.

[0030] In one possible implementation, the monitoring module is specifically used to: determine whether a predefined state switch exists based on the digital human's audio information; and / or, determine whether a predefined state switch exists based on the instruction information given to the digital human.

[0031] In one possible implementation, the monitoring module is specifically used to: receive the speech rate changes of the digital human reported by the front-end device; if the speech rate changes of N consecutive frames are greater than a set threshold, then it is determined that there is an interactive stuttering situation.

[0032] In one possible implementation, the generation module is specifically used to: determine a second transition duration based on the digital human's current speech rate; determine a second mixing parameter based on the second transition duration; and generate at least one mixed frame between the first video frame and the second video frame based on the second mixing parameter, the first video frame, and the second video frame.

[0033] In one possible implementation, the monitoring module is also used to: trigger a silent video frame indicating that the digital human is in a silent state after detecting the video screen of the front-end user; and trigger the digital human's broadcast audio frame when the digital human enters the broadcasting state, and trigger the digital human's broadcast video frame after detecting the audio information.

[0034] In one possible implementation, the monitoring module is also used to: store silent video frames in a silent state through the front-end device; specifically, the monitoring module is used to: trigger the front-end device to retrieve silent video frames from the local device and play them in a loop.

[0035] In one possible implementation, the generation module is specifically used to: calculate the timestamp of any mixed frame based on the frame interval and clock frequency; if the timestamp of the mixed frame is greater than the current time, wait for a set duration; the set duration is determined based on the timestamp of the mixed frame and the current time; after the set duration, send the mixed frame to the front-end device.

[0036] In one possible implementation, a sending module is also included for transmitting audio and video to the front-end device based on WebRTC (Web Real-Time Communication).

[0037] Thirdly, the present invention also provides a digital human interaction device, the device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method described in various possible designs of the first aspect.

[0038] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the method described in various possible designs of the first aspect.

[0039] Fifthly, the present invention also provides a computer program product that, when run on a computer, causes the computer to perform any of the methods described in the first aspect above.

[0040] These or other implementations of this application will become clearer and easier to understand in the following description of the embodiments. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of a digital human identity verification method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a digital human identity verification process provided in an embodiment of the present invention; Figure 3 A schematic diagram of a system architecture provided for an embodiment of the present invention; Figure 4 A flowchart illustrating a digital human interaction method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a process for generating a hybrid frame according to an embodiment of the present invention; Figure 6 This is a schematic diagram of another process for generating a hybrid frame according to an embodiment of the present invention; Figure 7 This is a schematic diagram of an audio-video synchronization process provided in an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating a real-time transmission process between a front-end device and a back-end server, provided as an embodiment of the present invention. Figure 9 This is a schematic diagram illustrating the state switching of a digital human according to an embodiment of the present invention; Figure 10 This is a schematic diagram illustrating another digital human state switching method provided in an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the rendering process of a digital human based on a dynamic blending algorithm, provided as an embodiment of the present invention. Figure 12 A schematic diagram of a dynamic mixing process provided in an embodiment of the present invention; Figure 13 A flowchart illustrating a hybrid algorithm provided in an embodiment of the present invention; Figure 14 A schematic diagram of a real-time dynamic hybrid transition algorithm provided in an embodiment of the present invention; Figure 15 A schematic diagram of an audio synchronization process provided in an embodiment of the present invention; Figure 16 This is a schematic diagram of another audio synchronization process provided in an embodiment of the present invention; Figure 17 A schematic diagram of a digital human interaction device provided in an embodiment of the present invention; Figure 18 This is a schematic diagram of another digital human interaction device provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] The following provides explanations for some of the terms used in this application. It should be noted that these explanations are for the convenience of those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0045] I. Web Real-Time Communication (WebRTC).

[0046] WebRTC is a browser technology stack for end-to-end real-time audio and video. It has built-in codecs, echo cancellation, bandwidth adaptation, and congestion control to achieve secure real-time transmission, delivering audio and video with extremely low latency, and providing a data channel to transmit low-latency small data.

[0047] 2. WebSocket.

[0048] WebSocket is a full-duplex long-connection message channel that emphasizes reliable and ordered transmission of text or binary messages. Simply put, WebRTC is more suitable for real-time interactive audio and video, while WebSocket is more stable for reliable messages such as commands, status, and chat. The two are often used together: WebRTC carries media streams, and WebSocket carries signaling and business messages.

[0049] Please see Figure 1 This diagram illustrates a digital human identity verification method provided by the present invention. The left side shows an H5 terminal page where the user authorizes the camera and microphone to record a video for identity verification. The video data is uploaded from the front end to the gateway, which is responsible for unified entry and security control, such as authentication, rate limiting, resume upload, format verification, risk control interception, and converting front end requests into internal interfaces usable by the back end. The gateway forwards the compliant video and accompanying metadata (such as session ID, device fingerprint, and timestamp) to the backend identity verification system on the right. Upon receiving the data, the backend identity verification system performs algorithms such as liveness detection, face comparison, and intent recognition to verify and generate the identity verification result. The identity verification result is then sent back to the H5 page via the gateway for presentation to the user.

[0050] Currently, H5 facial recognition primarily uses video recording on the terminal. Specifically, it analyzes video frames to identify user actions and simultaneously collects audio for automatic speech recognition (ASR) to determine user intent. However, traditional solutions guide users through actions and expressions of intent via text or voice prompts, but suffer from the following technical drawbacks: First, existing technologies lack real-time feedback mechanisms, failing to respond to user actions and provide immediate guidance. Second, due to the limited computing power of H5 terminals, data and processes are easily tampered with, posing security risks. Furthermore, facial recognition detection, real-time digital human rendering, and audio / video transmission are all computationally intensive tasks, demanding high CPU and memory requirements. H5 / Web terminal devices have limited computing resources, making it difficult to support such high-load computing demands. Third, it is currently impossible to guide users to focus on key information (such as product demonstrations, correct answer confirmations, etc.) through visual elements like gestures and eye contact. Fourth, it is impossible to adjust the animated expressions of the digital human (such as surprise, smiles, etc.) in real time based on user emotions and voice content, resulting in a mechanical user experience. These issues directly lead to a low H5 facial recognition pass rate. Therefore, improving the pass rate of digital identity verification is an urgent problem to be solved.

[0051] Based on this, the present invention provides a digital human interaction method that can improve the smoothness of the digital human by generating mixed frame video when the digital human is lagging, thereby improving the user experience and increasing the pass rate of digital human identity verification in identity verification scenarios.

[0052] Please see Figure 2 This diagram illustrates a digital human identity verification process. The diagram, from left to right, shows a user's actual interaction during remote identity verification via a mobile browser or H5 page: First, the user enters the page from the business entry page; the H5 page displays a loading screen, then initiates the verification process, with the H5 page activating the camera and entering the face detection framework. After entering the main verification process, a small window in the upper right corner displays the customer service digital human's video stream, while the user's view is displayed in real-time in the center of the page, with step-by-step instructions provided at the bottom or center via text bubbles. Next, the digital human broadcasts questions or instructions, guiding the user to provide voice or verbal responses, while ASR (Automatic Speech Recognition) and intent determination are performed simultaneously. During this process, prompts or dialogue bubbles provide feedback on the recognition status and results. When the answer verification is successful, the page displays a "Verification Completed" or "Process Successful" result page and ends the session.

[0053] After entering the user intent recognition and identification stage, the front-end and back-end complete authentication and authorization. Subsequently, the digital human generates a video stream based on the preset broadcast text using model inference and transmits it to the front-end via WebRTC. Upon receiving the video stream, the H5 page simultaneously plays the digital human's audio and video, guiding the user to complete the business step by step. After the broadcast ends, the microphone is turned on to collect the user's voice, which is then transcribed into text in real time using ASR and compared with the standard answer to determine if the user's intent is consistent. If they are consistent, the intent verification is approved and the session ends; if they are inconsistent, a retry prompt and guidance are provided.

[0054] Please see Figure 3 This diagram illustrates a system architecture provided by an embodiment of the present invention. The system includes an input layer 310, a core processing layer 320, and an output layer 330. The input layer 310 supports multiple modes for driving the digital human, such as audio-driven or text-driven. For example, in audio-driven mode, the user or backend directly provides an audio or speech application programming interface (API). The input layer 310 converts speech to text based on ASR (Automatic Speech Recognition), which is then passed to the core layer 320 to trigger playback and possible special effects animations. In text-driven mode, the user writes text content, which is then streamed and synthesized into the target timbre in real-time via text-to-speech (TTS) and used as the beat source for subsequent rendering. Both types of input can come from a microphone or be sent via the API.

[0055] The core processing layer 320 consists of a state manager, a rendering engine, and a frame synchronizer. The state manager switches between states such as mute, broadcast, and custom business and loads corresponding frame resources. For example, in the mute state, the digital human broadcasts. The rendering engine synthesizes the digital human's posture and lip movements frame by frame according to the current state. The frame synchronizer achieves millisecond-level audio and video alignment through predictive timestamps and sleep compensation, and uses dynamic alpha blending at state transitions to ensure visual continuity from mute to speaking or special actions.

[0056] The output layer 330 presents and distributes content to various front-ends and media carriers. Specifically, it pushes rendered audio and video to the browser via WebRTC to enable real-time playback of H5 pages. It also provides real-time messaging protocol (RTMP) output to connect to content delivery networks and live streaming platforms. This solution can achieve a balance between latency and concurrency in high-concurrency scenarios. In addition, based on APIs and combined with external tools, it can generate "virtual cameras" for meeting software (such as Zoom, Tencent Meeting, and DingTalk) to stream content, thereby maintaining consistent digital human display capabilities across different terminals and scenarios.

[0057] For virtual camera solutions, WebRTC allows the acquisition of video streams via the `navigator.mediaDevices` interface. Specifically, a custom `MediaStream` is generated and injected into the virtual screen, which the conferencing software then recognizes as a "camera device." For example, zero-code access using OBS virtual cameras allows screens, images, or videos to be quickly used as camera sources. After a user accesses the verification link through an application or browser, and the face quality, pose, and occlusion detections pass, the digital human enters the interface, initially in a silent state. Once the face stabilizes at a predetermined frame rate, the system generates audio with the target timbre based on preset text via a text API call to TTS. This audio drives the digital human's real-time broadcast, and WebRTC pushes the video frames to the front end. To ensure smooth transitions from silence to speaking and various actions, dynamic real-time alpha blending transition technology is introduced during the rendering stage to maintain visual continuity. Simultaneously, predictive timestamps and sleep compensation strategies are employed in the frame synchronizer to achieve low-latency, high-precision audio and video synchronization rendering.

[0058] The following is a detailed description of the proposed solution in conjunction with the accompanying drawings.

[0059] Please see Figure 4 The diagram illustrates a flowchart of a digital human interaction method, which includes the following steps: Step 410: During the interaction between the digital human displayed on the front-end device and the front-end user, monitor whether there is any interaction lag.

[0060] Here, the monitoring mechanism determines whether the user's interactive experience is smooth by tracking the transmission timing of audio and video frames in real time, analyzing the fluctuation range of frame intervals, detecting network transmission quality indicators, and identifying latency during digital human state transitions. When it detects that the timestamp deviation of multiple consecutive frames exceeds a preset threshold, or finds that the synchronization error between audio and video exceeds an acceptable range, or identifies playback discontinuity caused by factors such as network fluctuations, device performance limitations, and state transitions, it is determined that there is an interactive stutter, thereby triggering subsequent compensation mechanisms to ensure the smoothness and continuity of the user experience.

[0061] Optionally, before step 410, the method further includes: after detecting the video screen of the front-end user, triggering a silent video frame that indicates the digital human is in a silent state; when the digital human enters the broadcasting state, triggering the digital human's broadcasting audio frame and after detecting audio information, triggering the digital human's broadcasting video frame.

[0062] The aforementioned state-aware mechanism ensures that the digital human can dynamically adjust its behavior according to changes in the interaction scenario, achieving a smooth transition from a silent state to a broadcasting state. Specifically, when a video feed from a user is detected, the digital human immediately enters a standby state, playing pre-prepared silent video frames to present a natural standby posture and avoid abrupt static images. When the digital human needs to convey information to the user, the audio stream is triggered first, followed by the corresponding video frames. Through precise audio-visual synchronization, the digital human's lip movements, facial expressions, and speech content are perfectly matched, providing a more realistic and natural interactive experience.

[0063] The above approach addresses the issue that in digital human identity verification scenarios, formal interaction cannot begin until the user is ready. If the digital human starts broadcasting too early, issues such as the user not yet appearing on screen, failed face detection, or unauthorized device permissions may arise, leading to a chaotic interaction flow and a fragmented user experience. By detecting the user's video feed as a trigger, it's confirmed that the user has successfully entered the interactive interface and the camera is functioning correctly. At this point, a silent video frame is triggered, allowing the digital human to appear before the user in a static or slightly moving state. This demonstrates the digital human's presence while avoiding information omissions caused by broadcasting before the user is ready, thus establishing a buffer period for subsequent formal interaction.

[0064] Furthermore, optionally, it also includes: storing silent video frames in a silent state on the front-end device, triggering the front-end device to retrieve silent video frames from the local device, and playing them in a loop.

[0065] In terms of implementation, during the digital human initialization phase, a sequence of silent video frames is pre-downloaded and stored in the local cache of the front-end device. When the digital human is in silent standby mode, the front-end device directly reads these pre-cached video frames locally for loop playback, without needing to retrieve them from the server in real time. This ensures that even under poor network conditions, users can see a smooth and continuous digital human standby screen. This not only improves the smoothness of the user experience but also effectively saves bandwidth resources and reduces server load. Simultaneously, the local loop playback mechanism can intelligently detect the playback queue status. When a playback command is detected, it can seamlessly switch to the real-time audio and video stream, achieving a zero-latency experience for state transitions.

[0066] By pre-storing silent video frames locally on the front-end device, the above solution significantly reduces network transmission pressure, lowers response latency, and reduces the burden of repeated rendering on the back-end server. The silent state is the most frequent and longest-lasting state in digital human interaction, occurring during multiple stages such as user entry, waiting for face detection, announcement intervals, and waiting for user response. By pre-storing silent video frames locally on the front-end device, the silent state can be read directly from the local storage when needed, eliminating the need for network transmission.

[0067] There are many ways to monitor interactive stuttering, such as real-time tracking of audio and video frame transmission timing, analyzing frame interval fluctuations, detecting network transmission quality indicators (such as packet loss rate and latency jitter), identifying latency during digital human state switching, monitoring the stability of front-end rendering frame rate, and conducting a comprehensive evaluation based on performance indicators fed back by user devices. Two examples are given below: Example 1: Determine if there is a predefined state transition during the interaction. If there is a state transition, then it is determined that there is an interaction lag.

[0068] In digital human identity verification scenarios, the normal interaction flow should be smooth and continuous. When a state change is detected, it means the digital human is jumping from one state to another. This jump is often accompanied by rendering interruption, resulting in a perceived stuttering effect for the user. The above solution allows for the triggering of corresponding compensation strategies once stuttering is confirmed, thereby ensuring the smoothness and continuity of digital human interaction.

[0069] Furthermore, optionally, there are many possible states for switching. For example, it can be determined whether a predefined state switch exists based on the digital human's audio information; and / or, it can be determined whether a predefined state switch exists based on the instruction information given to the digital human.

[0070] The above solutions enable accurate identification of digital human state transitions. Audio-based detection allows for automated state perception by analyzing real-time audio signal parameters such as volume thresholds, audio energy, and speech activity to automatically determine the current state of the digital human. Command-based detection provides proactive control capabilities, triggering predefined state transitions at specific times based on business process needs. Combining these two detection methods enhances the flexibility and adaptability of the digital human.

[0071] Example 2: Receive feedback from the front-end device regarding the speech rate changes of the digital human. If the speech rate change exceeds a set threshold for N consecutive frames, then an interactive stuttering situation is identified.

[0072] For example, the real-time speech rate can be calculated by analyzing parameters such as syllable density, pause intervals, or pronunciation rates in audio frames. This calculation is based on a sliding window mechanism, which comprehensively analyzes several recent audio frames to ensure the accuracy and stability of the speech rate calculation. This threshold can be dynamically adjusted according to actual business scenarios and user experience requirements. The value of parameter N can comprehensively consider detection sensitivity and false positive rate. For example, setting N to 3-5 frames can both quickly respond to real stuttering situations and effectively filter out occasional speech rate fluctuations.

[0073] The above approach uses real-time feedback from the front-end device regarding speech rate changes to monitor interactive stuttering, accurately identifying anomalies during the digital human's broadcast. Under normal circumstances, the digital human's speech rate should remain relatively stable. When the front-end device detects a significant change in speech rate, it usually indicates audio / video transmission delays, network fluctuations, or abnormalities in the front-end playback buffer, directly leading to perceived interactive stuttering by the user. By continuously receiving speech rate data from the front-end, the back-end server can monitor the actual playback status in real time, rather than relying solely on its own sending status, thus more accurately reflecting the user's actual experience.

[0074] Step 420: If there is an interactive lag, obtain the first video frame displayed on the front-end device and the next second video frame to be displayed.

[0075] Through a frame buffer queue management mechanism, the system tracks in real time the video frame currently being displayed on the front-end device (the first video frame) and the next frame to be played in the playback queue (the second video frame). The first video frame represents the user's current visual perception of the digital human's state, while the second video frame represents the target state the digital human is about to transition to. By acquiring complete image data, timestamp information, and corresponding digital human state identifiers (such as lip movements, facial expressions, and posture parameters) for these two frames, accurate input data is provided for the subsequent generation of mixed frames.

[0076] Step 430: Generate at least one mixed frame between the first video frame and the second video frame using the first video frame and the second video frame, and send it to the front-end device.

[0077] There are many schemes for generating hybrid frames. The most suitable hybrid frame generation strategy can be flexibly selected based on the specific cause of the stuttering, network status, device performance, and user experience requirements. Examples include: progressive transition algorithms based on time linear interpolation, dynamic compensation algorithms based on speech rate adaptation, intelligent hybridization algorithms based on keyframe detection, and visual continuity optimization algorithms based on deep learning, etc. Different schemes have their own characteristics in terms of computational complexity, visual effects, and real-time requirements. Two examples are given below: Option 1: Please see Figure 5 The diagram illustrates a process for generating a hybrid frame, which includes the following steps: Step 510: Determine the first transition duration between the first video frame and the second video frame, and the elapsed duration from the first video frame to the current time.

[0078] First, the theoretical transition time between the first video frame (the currently displayed frame) and the second video frame (the next frame to be displayed) is calculated. Optionally, this time is calculated based on the video frame rate and timestamp information. For example, if the video frame rate is 25fps, the standard interval between two adjacent frames is 40ms. Simultaneously, the elapsed time (T_elapsed) from the start of the first video frame's display to the current moment is tracked in real time. This time is obtained through the system clock or a high-precision timer.

[0079] Step 520: Determine the first mixing parameter based on the ratio of the elapsed duration to the first transition duration.

[0080] The first blending parameter (α) represents the weighting coefficient of the first and second video frames during the blending process. Its calculation is based on the linear interpolation principle of time progression, and its value ranges from [0, 1]. Specifically, when α = 0, it means the first video frame is used entirely; when α = 1, it means the second video frame is used entirely; and when α is between 0 and 1, it means the two frames are blended according to their weights. For example, if the first transition duration is 40ms and the elapsed duration is 20ms, then α = 0.5, indicating that the blended frame should be a uniform blend of the two frames.

[0081] Step 530: Based on the first mixing parameters, the first video frame, and the second video frame, generate at least one mixed frame between the first video frame and the second video frame.

[0082] Optionally, the generation of the mixed frames employs a weighted fusion algorithm, performing linear interpolation calculations on each pixel of the first video frame (Frame1) and the second video frame (Frame2). The fusion formula is as follows: MixedFrame = (1-α)×Frame1+α×Frame2 By employing the above approach, and generating intermediate transition frames through time calculation and dynamic blending parameters, a smooth and natural transition between video frames in a digital human can be achieved. In digital human state switching scenarios, the first and second video frames represent two key states before and after the switch. For example, when switching from a speaking state to a silent state, the first video frame might show an open mouth, while the second video frame shows a closed mouth. If the user jumps directly from the first frame to the second frame, they will clearly perceive abrupt changes and a mechanical feel. By determining the transition duration between the two frames and the elapsed duration, the blending ratio of the blended frame to be generated at the current moment can be dynamically calculated. This ratio changes continuously over time, thereby generating a series of transition frames between the two key frames, solving the visual discontinuity problem in digital human rendering.

[0083] Option 2: Please see Figure 6 This illustrates another process for generating hybrid frames, which includes the following steps: Step 610: Determine the second transition duration based on the digital human's current speech rate.

[0084] Here, the backend server monitors the changes in the digital human's speech rate in real time, and calculates the current speech rate parameters based on the phoneme duration information output by the speech synthesis engine and the ASR recognition results.

[0085] Step 620: Determine the second mixing parameters based on the second transition duration.

[0086] In one example, the second transition duration is calculated as follows:

[0087] in, This is the second transition period. This refers to speech rate. Based on the aforementioned transition duration, the calculation method for the second mixing parameter α is as follows:

[0088] Here, k is a parameter used to control the slope and smoothness of the above formula. Different k values ​​correspond to different degrees of curve smoothness; the larger the absolute value of k, the steeper the transition in the transformation stage, and the faster the switching between adjacent target states. This invention does not impose specific restrictions on k and can adaptively adjust it according to the situation. In a specific example, k = -92.

[0089] For a concrete example, when a digital human needs to speak quickly (8 phonemes per second), Approximately 0.13, Approximately 0.9403; when the digital human speaks slowly (3 phonemes / second), Approximately 0.06, It is approximately 0.0247.

[0090] Once the dynamic α is obtained based on the k value, α is used to generate a smoother mixed image sequence between frames and send it to the front end for display in real time. With the above algorithm, the digital human can be adaptively adjusted by the backend service to accommodate different speech rates and broadcast fluctuations caused by network jitter or system lag, so that the overall picture remains smooth and natural without appearing stiff.

[0091] Step 630: Based on the second mixing parameters, the first video frame, and the second video frame, generate at least one mixed frame between the first video frame and the second video frame.

[0092] By using the above approach and dynamically determining the transition duration and mixing parameters based on the digital human's current speech rate as a key parameter, intelligent adaptation between video frame transition effects and the audio playback rhythm can be achieved. During the digital human's playback, the speech rate is not constant but fluctuates depending on the content, the need for emotional expression, and changes in network transmission conditions. When the speech rate is fast, phoneme changes are frequent, and lip movements should be more rapid, requiring a shorter transition duration to match the fast-paced audio content. When the speech rate is slow, phoneme changes are gradual, and lip movements should be more deliberate, requiring a longer transition duration to maintain natural and smooth movements. By sensing the current speech rate in real time and adjusting the transition duration accordingly, it is ensured that the transition rhythm of video frames always remains consistent with the audio playback rhythm, avoiding inconsistencies such as lip movement lag caused by fast speech but slow video transitions, or lip movement jumps caused by slow speech but fast video transitions.

[0093] The choice between the two schemes can be made based on specific circumstances. For example, in Example 1 of step 410, if there is a lag due to the digital human switching states, then Scheme 1 should be applied. In Example 2 of step 410, if there is a lag due to changes in the digital human's speech rate (such as network lag or user adjustment of the digital human's speech rate), then Scheme 2 should be applied. Alternatively, ordinary users can apply Scheme 1, while VIP users can apply Scheme 2. Or, if the backend server performance is average, then Scheme 1 should be applied, while if the backend server performance is good, then Scheme 2 should be applied. And so on, you can set it yourself according to the actual situation.

[0094] Optional, please refer to Figure 7 This diagram illustrates a process for audio and video synchronization, where audio and video can be sent to the front-end device using the following method: Step 710: For any mixed frame, calculate the timestamp of the mixed frame based on the frame interval and clock frequency.

[0095] After each mixed frame is generated, a precise timestamp is assigned to it to ensure synchronized audio and video playback.

[0096] Optionally, timestamp calculation is based on the frame interval and the system clock frequency. The frame interval represents the theoretical time interval between two adjacent frames, and the system clock frequency refers to the precision unit of the system time base, such as 90,000 Hz (i.e., 90,000 clock ticks per second), which is the standard clock frequency in WebRTC and multimedia transport protocols.

[0097] Step 720: If the timestamp of the mixed frame is greater than the current time, wait for the set duration.

[0098] Here, the duration is determined based on the timestamp of the mixed frame and the current time. Before sending the mixed frame to the front-end device, timing control is required to avoid buffer overflows and playback abnormalities caused by data arriving too early. First, the current system time is obtained and converted to the same clock frequency unit as the timestamp. Then, the timestamp of the mixed frame is compared with the current time: if the timestamp > the current time, it means that the playback time of the mixed frame has not yet arrived, and it needs to be delayed in sending.

[0099] Step 730: After the set duration, send the mixed frame to the front-end device.

[0100] It is worth noting that the present invention has a synchronization engine in the backend server. After calculating the timestamp, the synchronization engine sends the timestamp to the digital human renderer. The digital human renderer sends the mixed frame to the front end based on the timestamp sent by the synchronization engine. In this way, compared with the prior art that only uses the renderer's own clock to send the mixed frame to the front end, the present invention can send the video and audio at more accurate times, thereby achieving the effect of audio and video synchronization.

[0101] The above scheme, employing a timestamp-based predictive sleep compensation mechanism, enables precise control over the timing of mixed frame transmission. In real-time digital human interaction scenarios, by calculating the theoretical timestamp of each mixed frame based on frame intervals and clock frequency, it becomes clear when the frame should be sent to the front end. This theoretical timestamp represents the frame's standard time position within the entire audio and video stream. When a mixed frame's timestamp is detected to be greater than the current time, it indicates that the frame's transmission timing has not yet arrived. In this case, a waiting mechanism is used to postpone transmission until the current time catches up with the frame's theoretical timestamp, thus ensuring that each mixed frame is sent at its precise time, achieving a strict match between frame transmission rhythm and theoretical playback rhythm.

[0102] Furthermore, optionally, the backend server transmits audio and video to the frontend device based on WebRTC.

[0103] The above solution, employing the WebRTC protocol for audio and video transmission, enables low-latency, real-time data transmission, meeting the stringent requirements for instant interaction in digital human identity verification scenarios. Furthermore, by uploading computational tasks to a cloud server for processing and returning the results to the terminal device, the powerful computing capabilities of the cloud server can be leveraged to quickly complete computationally intensive tasks, while avoiding dependence on the terminal device's computing resources, thereby improving overall system performance and user experience. Moreover, uploading computational tasks to a cloud server enhances security, preventing H5 terminals from becoming targets for attackers due to their inherent characteristics and limitations, thus avoiding a range of security issues.

[0104] Through steps 410-430 above, during the interaction between the digital human displayed on the front-end device and the front-end user, monitoring for interaction stutters allows for real-time determination of interaction discontinuities, triggering remedial strategies immediately to reduce the abruptness caused by interaction interruptions. If interaction stutters occur, transitional content can be inserted into the original frame sequence, presenting frame changes in smaller steps, reducing the subjective perception of screen tearing, jitter, and lip-sync misalignment, thereby improving the continuous viewing and operation experience for the front-end user. Furthermore, moving related computational tasks to the back-end server reduces the computational burden on the front-end. Especially in digital human identity verification scenarios, this solution can improve the identity verification pass rate.

[0105] The following specific examples further illustrate the solution of the present invention. In the following description, "front-end," "front-end application," etc., refer to the interaction and presentation modules running on the user terminal side, including but not limited to H5 or App. "Back-end," "backstage," "face verification system," "backstage service system," etc., refer to the service cluster deployed in the cloud or at the edge. The above terms are interchangeable in this specification and claims, each referring to its respective functional set, and do not constitute a substantial difference.

[0106] (a) Real-time transmission between front-end devices and back-end servers.

[0107] The terminal (also known as the front end) application acquires video image frames from the camera in real time and transmits them to the cloud (also known as the back end) via WebRTC. After receiving the video image frames, the cloud performs face quality detection (which can be asynchronous) and returns the detection results to the terminal.

[0108] The front-end and back-end signaling communication of this invention is implemented based on WebSocket connection. Subsequently, business signaling (such as process guidance and text broadcasting instructions) is transmitted through WebSocket bidirectional connection to ensure the high efficiency and low latency of bidirectional real-time communication.

[0109] Please see Figure 8This diagram illustrates a bidirectional communication model between the front-end and back-end. The invention achieves efficient data transmission between the front-end and back-end using the WebRTC and WebSocket protocols. After a WebRTC connection is established, the front-end can transmit video frames to the back-end at a dynamically adjustable frame rate (e.g., 25 FPS). The frame rate parameter can also be dynamically set by the back-end based on the performance of the front-end device; for example, high-performance devices can use a higher frame rate to improve real-time performance.

[0110] Specifically, the front-end application connects to the back-end (also known as the face verification back-end service system) via WebRTC and transmits video image frames, i.e., transmitting the customer's face video, to the back-end. After receiving the video image frames, the back-end replies with confirmation information to the front-end.

[0111] After receiving the facial video, the backend performs facial quality checks (including facial quality detection, facial sharpness detection, facial pose monitoring, and facial occlusion checks). If the checks pass, it generates a digital human rendering inference and transmits the digital human video and audio frames to the frontend application via WebRTC. The frontend application then sends a confirmation message back to the backend via WebRTC. In this way, the computationally intensive facial recognition task is offloaded to the backend, effectively addressing the issue of insufficient computing power on H5 terminals.

[0112] Next, the front-end displays the digital human. Simultaneously, the front-end application captures the user's audio frames and sends the audio stream to the back-end service system, which responds with confirmation. Finally, the back-end service system obtains the user's intent by downloading and analyzing ASR (Automatic Speech Recognition) data.

[0113] (ii) Implementation of the digital human state machine.

[0114] Please see Figure 9 This diagram illustrates a digital human's state transition. The backend service system achieves real-time state switching through a system-defined animation state setting method. Specifically, a hierarchical state structure is adopted, defining the following states: 1. Basic state (also known as default state): The state value is 0, and the default rendering behavior is used. 2. In silent mode, the state value is 1, and pre-rendered frames are used; 3. Custom status: The backend service system can load custom broadcast resources using the loadCustom method.

[0115] The digital human performs normal business announcements in its default state. When a mute indicator is detected (i.e., mute is detected), the digital human switches from the default state to a static state. Conversely, when sound is detected in the audio channel (i.e., speech is detected), the digital human switches back from the static state to the default state. The aforementioned audio channel refers to a real-time audio transmission channel established via WebRTC, used for bidirectional transmission of audio data streams. Specifically, the backend service system transmits the digital human's voice announcements to the frontend application in real time through the audio channel, and the frontend application plays them for the user to hear.

[0116] In addition, the digital human can have other custom states. For example, in the diagram, custom state 1 and custom state 2 are used as examples. When the set conditions are met (the diagram uses "System Custom Animation State Settings" as an example), the digital human enters custom state 1. Custom state 1 has predefined images and audio. When the images and audio of custom state 1 have finished playing (that is, when the digital human has completed the action of custom state 1 and finished playing the predefined audio), it is called the completion of custom state 1. After the state is completed, the digital human switches from custom state 1 to the default state. Custom state 2 is the same and will not be repeated.

[0117] The above is just an example; the switching between different states can be flexibly combined and configured according to business needs, and can flexibly support the needs of various business scenarios. The following section details the implementation methods for transitioning between different states of the digital human.

[0118] First, the backend service system uses `currState` to record and track the current state. `currState` is a state variable used to record the current state of the digital human in real time. Its value can be 0 (basic state or default broadcast state), 1 (mute state), or other custom state identifiers. The backend service system updates the value of `currState` every time the state changes, so it can always know what rendering behavior the digital human should perform. For example, when `currState=1`, the backend service system knows it should display a static waiting animation; when `currState=0`, it knows it should generate a dynamic lip-sync animation based on the audio content.

[0119] A dedicated counter, `customIndex`, is used to maintain the custom state. Specifically, `customIndex` tracks which frame is currently being played in a custom state. Since custom states typically have predefined animation sequences (such as thank-you actions or prompts), these animations consist of a series of consecutive video frames. `customIndex` records the current playback progress. For example, if a custom animation contains 50 frames, `customIndex` will start counting from 0, incrementing by 1 for each frame played. When `customIndex` reaches 49 (or exceeds the maximum value), the background service system knows that the custom animation has finished playing and can switch back to the default state.

[0120] In addition, the system internally uses queues `customImgCycle` and `customAudioCycle` to store video and audio frame resources for specific states, allowing for the cyclical use of image and audio resources after entering a certain stage. Specifically, `customImgCycle` is a queue data structure used to store all video frame resources for a specific custom state. When the digital human needs to enter a custom state (such as a gesture animation prompting the user to answer), the backend service system loads all image frames corresponding to that state sequentially into the `customImgCycle` queue using the `loadCustom` method. This way, when playing the animation, the corresponding video frames can be retrieved from the queue according to the value of `customIndex` for rendering and pushing. The advantage of using queues is that they allow for loop playback; if an animation needs to be displayed repeatedly, `customIndex` can be reset to 0, and all frames in the queue can be traversed again without reloading the resources. Correspondingly, `customAudioCycle`, similar to `customImgCycle`, is a queue used to store audio frame resources for a specific custom state. For example, when the digital human needs to play the prompt "Do you agree to process this service? If you agree, please answer 'agree'", the corresponding audio data is pre-loaded into the `customAudioCycle` queue. The backend service system retrieves the corresponding audio frames from the queue according to the playback progress and pushes them to the frontend application synchronously with the video frames. The audio and video frames are stored separately but played synchronously, ensuring precise matching between lip movements and sound.

[0121] After each state is completed, the digital human returns to the default state and plays the animation of the silent state. The silent state uses static frames, eliminating the need for GPU streaming; it simply loops through the corresponding image frames, effectively reducing the GPU's computational burden. In other words, if an animation needs to be looped (such as a waiting animation), simply resetting `customIndex` allows playback to continuously restart from the head of the queue, while resources remain in memory, avoiding repetitive I / O operations and computational overhead. When the custom state ends and the system switches to another state, resources in the queue can be released to save memory.

[0122] The following example demonstrates the conversion between silent and default states, using the following formula:

[0123] in, This describes the state transition function in the digital human state machine. It defines which state the digital human should transition to given the current state q and the input condition σ (primarily volume E). Specifically, the function contains two main transition rules: when the digital human is in the default state and the detected volume E is less than the mute threshold... When the digital human is already in a silent state, but the detected volume E is greater than or equal to the speech threshold, the digital human will switch to a silent state. At that time, the conversion function is used to intelligently determine what type of animation the digital human should play by monitoring the audio in real time, thereby achieving a smooth switch between the silent state and the default state, avoiding frequent state jumps caused by instantaneous volume fluctuations, and ensuring the visual continuity and smooth user experience of the entire digital human interaction process.

[0124] The relevant threshold parameters in the algorithm can be set according to the specific scenario. In this solution, they have been determined after testing and verification. =0.1, =0.15. It is understood that the above parameters are merely examples and not intended to limit the scope of the invention.

[0125] Similarly, the switching between the custom state i and the default state is expressed by the following formula:

[0126] The above state transition function Used to describe the current state q after receiving a custom state signal. How does the state transition occur? Specifically, when the system is in the default state, if it receives a custom state signal i (triggered by calling the "System Custom Animation State Setting" method), the digital human switches from the default state to the corresponding custom state i. At this time, the system loads the video frame resources (customImgCycle queue) and audio frame resources (customAudioCycle queue) specific to that custom state. When the digital human is already in a certain custom state i, it continuously monitors the completion conditions of that state. Once the completion conditions are detected, the digital human will automatically return to the default state and release the resources occupied by that custom state. It could be a user-initiated action. For example, when a user's action is detected, the digital human switches from a custom state to the default state.

[0127] The aforementioned state transition mechanism ensures that the digital human can flexibly switch action behaviors at different stages of the business process. For example, it can switch to a specific gesture animation during the user interaction prompt stage and play an acknowledgment animation during the result processing stage. At the same time, it ensures that the human can smoothly return to the basic state after each custom state is completed, maintaining the continuity and controllability of the entire interaction process.

[0128] The content of the above formula can also be referenced. Figure 10 When the digital human is in the default state and the volume is less than the first threshold, the digital human switches from the default state to the mute state. When the digital human is in the mute state and the volume is greater than or equal to the second threshold, the digital human switches from the mute state back to the default state. When the digital human is in the default state and the set conditions are met, the digital human switches from the default state to custom state 1. When the digital human is in custom state 1 and the frame loop ends, the digital human switches from custom state 1 back to the default state.

[0129] The aforementioned state transition algorithm can avoid false switching caused by sudden volume interference at a certain time. That is, the present invention adopts a strategy of switching only after N consecutive occurrences. The sensitivity of switching can be adjusted by adjusting the value of N.

[0130] The table below is a reference table for the state transitions of digital humans, as shown in the table below:

[0131] The specific transition conditions are as follows: When transitioning from the default state to the mute state, the trigger condition is that the background service system detects N consecutive frames and the volume threshold is less than the set value. The action is to switch the state and record the start time simultaneously. When transitioning from the mute state to the default state, the trigger condition is that the background service system detects N consecutive frames and the volume threshold is greater than the set value. The action is to switch the state and reset the recorded start time. When transitioning from the default state to the custom state, the trigger condition is that the background service system actively calls the "System Custom Animation State Setting" method. The action is to load the corresponding custom animation resource file. When transitioning from the custom state to the default state, the trigger condition is that the background service system detects that the frame index exceeds the maximum value. The action is to release the custom animation resource. The value of N can be determined independently according to the actual situation. For example, in one example, N is between 3 and 5 frames. In this way, with a certain sensitivity, it can effectively ensure that the state will not switch erroneously, ensuring the stability and reliability of the entire system operation.

[0132] Optionally, the hierarchical state machine includes five core phases, namely: (1) First stage (also known as the stability check stage).

[0133] When the service begins, the front-end application establishes a WebRTC connection with the back-end service system, and the entire system first enters a stability check phase. During this phase, the front-end application mutes the user's microphone and does not accept any audio content; the entire audio channel is muted, and the digital human rendered by the back-end service system is also muted accordingly. Once the back-end service system passes the stability frame test, the system switches to the broadcast phase (the next phase, the second phase).

[0134] Here, the stability check phase is used to confirm, before each round of driving and rendering, whether the current frame is stable enough, whether cloud computing needs to be triggered, and whether the output can be safely used for display.

[0135] (2) The second stage (also known as the broadcasting stage).

[0136] The backend service system begins generating the digital human's business voice broadcast content, at which point sound begins to be output from the audio channel. Upon detecting the sound, the backend service system automatically switches the digital human to broadcast mode, and the digital human begins lip-syncing broadcast actions. The aforementioned real-time rendered video frames are then transmitted to the frontend application via WebRTC for display.

[0137] (3) The third stage (also known as the user interaction prompt stage).

[0138] After the broadcast is completed, the digital human enters a customized state, which prompts the user to respond. At this point, the digital human rendered by the backend service system enters an animation effect different from the broadcast state (such as a prompt gesture), for example... Figure 5 The custom state 1 shown prompts the user for a response in a user-friendly way.

[0139] At this point, the backend service system loads the predefined status and plays predefined audio content, such as asking the user whether they agree to process this service, and if they agree, asking them to answer "agree".

[0140] (4) The fourth stage (user response stage).

[0141] After the digital human finishes its presentation, it enters the user response phase. At this time, the front-end application turns on its microphone to accept the user's audio input, while the digital human rendered by the back-end service system enters a silent state. Meanwhile, the front-end displays animation frames in the silent state to await the user's response.

[0142] During this phase, the backend service system listens to the audio frames sent by the frontend application through the WebRTC audio channel, receiving the user's audio content in real time and performing real-time ASR (Actions Responsive Suggestion) every three seconds to determine if the user's intent is correct. Once the backend service system receives a correct or negative intent, this phase ends, and the intent result processing phase begins.

[0143] (5) Fifth stage (Intent result processing stage).

[0144] During the intent result processing phase, the digital human rendered by the backend service system will enter an animation different from the broadcast state, for example... Figure 5 Custom status 2 in the settings indicates that the user's response process has been completed. The backend service system then renders the thank-you animation frames and audio content indicating the end of the process, and transmits them to the frontend application for display via WebRTC.

[0145] Understandably, the five stages described above are merely examples. In practical applications, these stages can be deleted or added. For instance, a practical application might only include three stages (such as a stability check stage, a broadcast stage, and a result processing stage). Furthermore, the five stages do not have a specific order. For example, after the first stage (stability check), the application can directly proceed to the third stage for user interaction, or it can re-enter the stability check stage after user interaction, and so on. There are no restrictions.

[0146] (iii) Dynamic blending transitions in the digital human state-aware rendering pipeline.

[0147] During the digital human's broadcast, the backend system continuously renders frames, which allows the digital human to maintain smooth movements in each state and achieve a smooth transition when switching states, thus achieving a smooth, realistic, and human-like effect in the digital human's movement switching.

[0148] First, the backend detects the audio state of the digital human to identify its current state, which includes: silent state, voice broadcast state, and specific action / visual state. The specific action / visual state can be triggered by commands, such as sending a specific closing animation at the end of the desired answer response phase. When a state transition is detected, a dynamic alpha blending transition algorithm is used to comprehensively analyze the current and subsequent rendering frames, generate a blended transition frame, and push it to the frontend via WebRTC.

[0149] Please see Figure 11 This diagram illustrates a rendering process for a digital human based on a dynamic blending algorithm. First, the audio is monitored and its type determined, categorized into three types: silent, speech, and custom. When the current audio is determined to be silent, the digital human switches to a silent state, loading pre-rendered static frames to reduce computational overhead and maintain image stability. When the current audio is determined to be speech, it enters a broadcasting state, generating corresponding dynamic frames based on real-time audio characteristics (such as speech rate and rhythm) to synchronize lip movements with speech rate. When a custom state is triggered by a business instruction, specific predefined animation resources are loaded for prompts, thanks, or other business gestures.

[0150] When switching between different states, a dynamic alpha blending transition algorithm is enabled to perform pixel-by-pixel weighted fusion of the two consecutive frames, thereby maintaining visual continuity and natural transitions during the transitions between silent, broadcast, and custom states. After the transition is completed, the target frame and audio are then aligned with the timestamp by a frame synchronization mechanism, and finally output to the WebRTC channel for real-time playback on the H5 front-end, or provided to external applications such as conferencing software via a virtual camera solution.

[0151] First, the backend obtains the target frame through GPU rendering inference and determines whether the alpha blending transition algorithm is enabled. If enabled, the blending transition algorithm is used; otherwise, the current frame is directly used as the target frame for rendering, and the target frame data is sent to the frontend via WebRTC so that the frontend can display the target frame. This invention provides two blending transition algorithms, which are described in detail below.

[0152] Algorithm 1 (refer to steps 510-530 above): This algorithm is suitable for mixed transitions within a state. Its goal is to achieve a smooth transition within the same state (e.g., different lip-sync frames in the same "speaking state" or different still frames in the same "silent state"), avoiding jumps in the image between consecutive frames.

[0153] This algorithm employs a time-driven linear α-weighted approach: starting from the previous frame of the current state and ending at the currently calculated target frame, α is calculated based on the transition start time and transition duration, and then linearly fused pixel by pixel for output. This method is simple to implement, has low overhead and low latency, is suitable for high-frequency and high-stability applications, and can be used as a baseline rendering strategy to meet the continuity requirements of most business scenarios.

[0154] Alternatively, the formula for Algorithm 1 is as follows: α = min(1.0, (current system time - transition start time) / preset transition duration) For a concrete example, assuming the current time is t = 0.205s, the transition start point is t_start = 0.200s (also known as the previous frame of the transition), and the preset transition duration is T = 0.08s, then the transition coefficient α for this frame is calculated according to the above formula, resulting in α = (0.205 - 0.200) / 0.08 = 0.0625. α is always limited to a range not exceeding 1. When the digital human is about to stop speaking and enter silence, the algorithm uses this α value to generate a transition image of a smooth mouth closure.

[0155] When α=0, the output screen is equivalent to a spoken frame; when α=0.5, the output screen has half the weight between the spoken frame and the silent frame; when α=1, the output screen is completely switched to a silent frame.

[0156] Algorithm 2 (refer to steps 610-630 above): Algorithm 2 is suitable for mixed transitions between states and is a further upgrade of Algorithm 1. Its goal is to achieve smooth transitions between different states (e.g., "silent state - speaking state", "speaking state - custom action state"). Algorithm 2 introduces a non-linear α curve, which can present a more natural transition at abrupt changes such as "opening", "closing", or "entering / exiting a specific interactive action".

[0157] In one scenario, when the digital human's speaking speed changes (for example, when the system detects that the front-end is slowing down due to network lag, or when the user manually adjusts the speaking speed), the back-end dynamically calculates the α value based on the front-end's speaking speed, making the video smoother.

[0158] In one possible implementation, Algorithm 2 is used when the speech rate of the digital human exceeds a set value. The front end collects and identifies the phoneme rate of the digital human's speech and sends the collected speech rate information to the back end via WeSocket. When there are consecutive N frames showing speech rate changes, and the change threshold exceeds the set value Δe, this algorithm is used. The values ​​of N and Δe can be determined according to the situation and are not limited. For example, in a specific example, when the speech rate change exceeds 2 for 5 consecutive frames, Algorithm 2 is executed.

[0159] Optionally, unlike the fixed transition duration of Algorithm 1, the transition duration of Algorithm 2 is calculated based on the speech rate. That is, Algorithm 2 can adjust the transition duration according to the speech rate. In one example, the detected speech rate can be normalized and mapped to a fixed interval, such as mapping the speech rate to the interval [0, 1]. Then, the target value within this fixed interval is jointly calculated with the transition duration to finally obtain the transition duration related to the speech rate.

[0160] Please see Figure 12 The diagram illustrates a dynamic mixing process applied to Algorithm 2 above, specifically including the following steps: Step 1210, Audio Feature Analysis.

[0161] Here, the current audio frame is collected and analyzed, such as extracting features like energy envelope, zero-crossing rate, fundamental frequency, and phoneme rate, to estimate when a state transition will occur (such as from speaking to silence, or from speaking to a specific action).

[0162] Step 1220: Calculate the optimal transition time.

[0163] The transition duration T is calculated based on audio characteristics. For example, the transition can be slightly longer when the speech is fast to avoid abruptness, and shorter when the speech is slow to improve responsiveness.

[0164] In one example, the phoneme rate can be mapped to a ranged transition duration, such as approximately 50–150 ms, thus ensuring smoothness without excessive trailing.

[0165] Step 1230: Generation of nonlinear α curve.

[0166] This step generates α, which varies with time, based on the transition start time t0 and duration T.

[0167] Step 1240, Dynamic Mixing.

[0168] Finally, the two frames are blended to generate a composite image. In this way, the digital human can adapt to different speech speeds and, based on the perceived speed of the front-end playback, the digital human's lip movements can change with the speech speed and network environment, making the animation playback more natural.

[0169] Please see Figure 13 The diagram illustrates a flowchart of a hybrid algorithm. The front-end application collects speech rate in real time, extracting driving parameters related to speech pronunciation frame by frame, and sends the collected speech rate information to the back-end via Wesocket. The back-end performs feature extraction on the audio, and by analyzing the speech rate, it can obtain the current playback rate of the digital human. If the digital human's playback rate is too fast, the foreground image (i.e., the image from the previous frame) is used preferentially for hybridization.

[0170] The "alt" branch box in the middle of the diagram represents two processing paths that are split based on conditions. The conditions are determined by comparing the "change threshold" and the threshold constant "Δe". The overall loop progresses frame by frame. After processing each frame, the front end "displays the next frame" and continues to the next round of judgment and processing.

[0171] After receiving the speech rate information from the front end, the back end will take a branch when it detects that the speech rate of the current frame is greater than the "change threshold > Δe" relative to the previous frame. The back end first receives the speech rate data and performs N-frame calculation. That is, after receiving a new frame of speech rate data, the back end does not only look at the speech rate of this frame, but also performs joint calculations on a temporal window (with a length of N frames, such as several frames before and after) including this frame to obtain more stable, coherent parameters that can be used to drive rendering. Then, the speech rate variation algorithm is triggered to perform image frame blending calculations.

[0172] The backend then "adjusts the alpha value." The alpha value can be understood as a weight or blending coefficient in the mixed calculation, used to control the proportion of influence of speech rate on facial expressions, lip movements, facial bones, or mesh deformation, ensuring the changes are both timely and smooth. After updating the parameters, the backend performs "real-time rendering adjustments" and sends the adjusted results back to the frontend via WebSocket. Upon receiving the backend's processing results, the frontend renders the content based on the returned data.

[0173] When the backend detects that "the change threshold is less than or equal to Δe", it proceeds to the next branch: since the difference between this frame and the previous frame is small, it does not need to use a hybrid algorithm, but can directly perform real-time rendering, while the frontend directly uses the currently held parameters to display the next frame locally.

[0174] Please see Figure 14 The diagram illustrates a real-time dynamic alpha blending transition algorithm, primarily showcasing the algorithm for transitioning from a silent frame to a speaking frame, and vice versa. As can be understood, a silent frame refers to a video frame in the digital human state where the digital human is silent and not speaking, while a speaking frame refers to a video frame in the digital human state where the digital human is speaking.

[0175] First, determine whether the current frame (the frame at the start of processing) is a silent frame or a speaking frame. Based on this, there are two cases: First, if the current frame is silent, transition from silent to speaking; second, if the current frame is speaking, transition from speaking to silent. In either case, it's necessary to determine whether transition effects are enabled. If transition effects are enabled, initialize transition variables and retrieve data for the next frame. If transition effects are not enabled, skip transition initialization and directly retrieve data for the next frame. The two cases are described below.

[0176] Scenario 1: The current audio frame is a speaking frame, and the next frame is a silent frame.

[0177] In this scenario, firstly, `speaking=False` is set, indicating that the next frame's audio is considered silent, and the rendering pipeline enters the "silence processing" path, i.e., the transition from speaking to silence. Next, it checks if a custom video exists. If a custom video resource exists, a mirror index is calculated, and this index is used to locate the frame that should be displayed in the current frame within the custom resource set. That is, if a "custom video frame sequence" is configured (instead of using the default silent frame list), the nth frame is not directly retrieved in the original order. Instead, a "mirror index" is calculated based on the current time or frame number, combined with a loop rule. The "mirror index" is often used for swingback playback: when the sequence reaches the end, the index decreases in reverse, returning to the beginning, thus avoiding a smooth transition between the beginning and end. After obtaining the mirror index value, the corresponding frame is retrieved from the custom resource array as the frame to be displayed in this frame. If no custom video exists, it falls back to the default frame list, locates the target frame that should be displayed in the default silent frame sequence, and finally obtains the target_frame for this frame.

[0178] Next, it determines whether to enable transition effects. If transition effects are not enabled, no transition initialization or blending is performed, and the target_frame is directly used as the output screen.

[0179] If a transition effect is enabled, the system continues to determine whether it is currently in a transition period. If it is in a transition period, the system performs a "speaking-silence" transition calculation: lstspk_frame is selected as the exit frame (i.e., the keyframe of the previous speaking state, used for fusion with the silence target), and the alpha value is calculated as the transition time ratio, where alpha∈[0,1]. The transition is performed linearly based on the transition start time and the preset transition duration, or according to an adaptive strategy. Finally, a pixel-by-pixel linear fusion is performed, using the following fusion formula: cmb_frame= lstspk_frame ×(1 - alpha)+target_frame×alpha After merging, the current silent target frame is cached in lstspk_frame.

[0180] If a transition effect is enabled but the current state is not in a transition period, no blending is performed. The target frame is used directly as the output, and the mute target frame is written to the mute keyframe buffer lstspk_frame. This buffer is used as an anchor point when transitioning from mute to other states later. In other words, the frame currently considered mute (target_frame) is saved to a variable lstspk_frame, which acts as a "reference anchor point." When the state switches from mute to speaking, this variable and the speaking frame are used for blending calculations to ensure a smooth transition.

[0181] In terms of output, this mute path has two possible outcomes: if a transition occurs and a merged frame is calculated, the resulting transition frame cmb_frame is used as the result; if no transition occurs or transition is not enabled, the target frame target_frame is used directly as the result.

[0182] Scenario 2: The current audio frame is a silent frame, and the next audio frame is a spoken frame.

[0183] In this scenario, firstly, `speaking=true` is set, indicating that the next audio frame is determined to be a speaking frame, and the rendering pipeline enters the "speaking processing" path, i.e., the transition from silence to speaking. Next, `pstbck_frame` is called to generate the current frame. Subsequently, it is determined whether "transition effects are enabled". If transition effects are not enabled, no transition initialization or blending is performed, and `target_frame` is directly used as the output screen.

[0184] If a transition effect is enabled, the system continues to determine whether it is currently in a transition period. If it is in a transition period, the system performs a "mute-speak" transition calculation: lstsil_frame is selected as the exit frame (i.e., the frame in the previous mute state), and the alpha value is calculated as the transition time ratio, where alpha ∈ [0,1]. The transition is performed linearly based on the transition start time and the preset transition duration, or according to an adaptive strategy. Finally, pixel-by-pixel linear fusion is performed using the following formula: cmb_frame = lstsil_frame (1-alpha)+current_farme alpha After fusion is complete, the current target frame is cached in lstsil_frame.

[0185] If a transition effect is enabled but the current transition period is not in progress, no blending is performed. The target_frame is used directly as the output, and the target frame is written to the keyframe buffer lstsil_frame for use as an anchor point when it is necessary to transition from the speaking frame to other states later.

[0186] Regarding the output, this speaking path ultimately has two possibilities: if a transition occurs and a merged frame is calculated, the resulting transition frame cmb_frame is used as the result; if no transition occurs or transition is not enabled, the target frame target_frame is used directly as the result.

[0187] In both scenarios, a "target frame" is generated (either the frame obtained from the current inference is used directly, or the overmixed frame is used). The output phase then begins. In one example, the synchronization engine first aligns the target frame with its corresponding audio timestamp using a unified clock. If the current system time is earlier than the target timestamp, a short sleep compensation is performed to wait until the scheduled transmission time. Afterward, the target frame is written to the rendering buffer and sent through the WebRTC video channel, while the matching audio packet is pushed via the audio channel.

[0188] The system then checks if an exit event has been triggered, such as session termination, resource reclamation signal, or fatal error. If an exit event is triggered, resources are cleaned up sequentially, such as stopping transitional blending calculations, and the video memory and memory usage of custom frame resources and the default frame list are released. If no exit event is triggered, the system returns to "Get Next Frame Data" and continues to judge, blend, and output subsequent frames according to the same process, thereby maintaining the smoothness and visual continuity of the picture and audio.

[0189] (iv) Dynamic frame synchronization.

[0190] Please see Figure 15 This diagram illustrates an audio synchronization process. In addition to the client and the digital human, a synchronization engine exists to perform audio synchronization on the backend. First, the client sends a video frame request. The synchronization engine calculates a predicted timestamp based on the video frame request. For example, the predicted timestamp is calculated as follows: t = t0 + n × PTIME Where t0 is the session start alignment time or the reference time after the most recent resynchronization, n is the sequence number of the current frame, and PTIME is the preset frame period (e.g., approximately 40ms at 25fps). After obtaining the target time point (predicted time error), the synchronization engine measures the "clock offset" Δt between the current system time and the target time. Here, Δt is used to reduce drift caused by factors such as thread scheduling, network stack jitter, operating system timing accuracy, and the cumulative error of crystal oscillators from different devices.

[0191] In the diagram, "alt" represents branching behavior: when Δt≥0, it means that the system time has not yet reached the target transmission point, and the synchronization engine executes await sleep(Δt), that is, delaying the frame by Δt time before sending it to the front end, thus postponing the actual transmission action to the predicted time in a precise sleep manner; when Δt<0, it means that the moment has been "missed", and the engine no longer waits but immediately continues the process to avoid further backlog and image stagnation.

[0192] After waiting and correction, the synchronization engine sends a precise timestamp to the renderer, along with the timestamp used as a timing anchor. Based on this timestamp, the renderer generates or selects a rendering frame at the corresponding moment, for example, outputting the target frame showing the digitizer's current lip movements, facial expressions, or posture at that moment. Subsequently, the video frame with the precise timestamp is sent back to the client via the WebRTC media channel, and the client receives a synchronized video stream that has been time-corrected. Based on this approach, video and audio synchronization can be achieved.

[0193] Please see Figure 16 The diagram illustrates an audio synchronization process, which includes the following steps: Step 1610: Initialize the time base.

[0194] First, the time base is initialized. According to the WebRTC RFC specification document, video uses a clock frequency of 90kHz, and its time base v_base is defined as 1 / 90000 seconds / tick; audio uses a sampling rate of 16kHz, and its time base a_base is defined as 1 / 16000 seconds / sample.

[0195] Step 1620: Predict the time of the next frame.

[0196] For a video frame, the theoretical timestamp calculation formula for the nth frame is: next_time1 = start_time + n × v_ptime × v_clock_rate Where start_time is the initial time, v_ptime is the frame interval, and v_clock_rate is the clock frequency.

[0197] For audio frames, the theoretical timestamp calculation formula for the m-th frame is: next_time2 = start_time + m× v_ptime × v_clock_rate Let's illustrate with a specific example: For a video frame, assuming the initial time is 0.200 seconds, to calculate the timestamp of the 5th frame, the formula is as follows: next_time1 = 0.200 + 5 × 0.04 × 90000 = 18000 + 18000 = 36000ticks.

[0198] Here, v_ptime represents the video frame interval, currently set to 0.04 seconds (corresponding to 25fps).

[0199] For audio frames: Assuming the initial time is 0.200 seconds, to calculate the timestamp of the 100th frame, the formula is as follows: next_time2= 0.200 + 100 × 0.02 × 16000 = 3200 + 32000 = 35200.

[0200] Step 1630: Calculate the waiting time.

[0201] The formula for calculating the waiting time is: wait = start + n × v_ptime – currenttime Where currenttime is the current system time. The waiting time (wait) is used to compensate for system processing delays.

[0202] For a concrete example, assuming the current system time is 0.205 seconds, the theoretical time for the 5th video frame (or audio frame) is next_time = 0.200 + 5 × 0.040 = 0.400 seconds. Then the waiting time is wait = 0.400 - 0.205 = 0.195 seconds, and a timestamp compensation of 195 milliseconds will be performed.

[0203] Step 1640: Determine the transmission time of the video frame or audio frame based on the current time and the waiting time.

[0204] If the predicted time of the next frame is greater than the current time, the corresponding video frame or image frame will be sent to the front end after a wait time.

[0205] In a specific example, video frames are sent at approximately 40 millisecond intervals (25fps), and audio packets are sent at approximately 20 millisecond intervals (50fps). By compensating for various system delays through the necessary waiting time for executing frames, millisecond-level audio and video synchronization errors are ultimately achieved. This ensures that the audio and video received by the front end via WebRTC remain perfectly synchronized, providing users with a smooth and natural digital human interaction experience.

[0206] Steps 1610-1640 above can be referred to steps 710-730, and will not be repeated here.

[0207] Optionally, an incremental algorithm can be used to update the timestamp. Specifically, for videos, the timestamp update formula is: v_timestamp = v_timestamp + v_ptime ×v_clock_rate.

[0208] Where v_timestamp is the video timestamp.

[0209] For audio, the timestamp update formula is: a_timestamp = a_timestamp + a_ptime × sample_rate Where a_timestamp is the audio timestamp and sample_rate is the sampling frequency.

[0210] Through the above approach, after completing sleep compensation and synchronization mechanism calculations, the synchronization engine transmits precise timestamps to the digital human model, which then sends the corresponding video and audio frames to the front end. The synchronization engine works closely with the digital human renderer to ensure precise audio-visual synchronization and deliver stable and smooth output.

[0211] The above solutions can reduce the average face-scanning time for cloud-based remote identity authentication, improve the pass rate, and enable H5 and mobile terminals to have the same real-time digital human capabilities as iOS or Android after the image and audio frame data are transmitted to the backend via WebRTC in real time.

[0212] Based on the same concept, this application also provides a digital human interaction device that can execute the digital human interaction method described above.

[0213] Please see Figure 17 This application provides a schematic diagram of the structure of a digital human interaction device according to an embodiment, such as... Figure 17 As shown, the digital human interaction device includes: The monitoring module 1701 is used to monitor for any interaction stuttering during the interaction between the digital human displayed on the front-end device and the front-end user.

[0214] The acquisition module 1702 is used to acquire the first video frame displayed on the front-end device and the next second video frame to be displayed if there is an interactive lag.

[0215] The generation module 1703 is used to generate at least one mixed frame between the first video frame and the second video frame using the first video frame and the second video frame, and send it to the front-end device.

[0216] In one possible implementation, the monitoring module 1701 is specifically used to: determine whether there is a predefined state transition during the interaction process; if there is a state transition, then determine that there is an interaction lag.

[0217] In one possible implementation, the generation module 1703 is specifically configured to: determine a first transition duration between the first video frame and the second video frame, and the elapsed duration from the first video frame to the current time; determine a first mixing parameter based on the ratio of the elapsed duration to the first transition duration; and generate at least one mixed frame between the first video frame and the second video frame based on the first mixing parameter, the first video frame, and the second video frame.

[0218] In one possible implementation, the monitoring module 1701 is specifically used to: determine whether a predefined state switch exists based on the digital human's audio information; and / or, determine whether a predefined state switch exists based on the instruction information given to the digital human.

[0219] In one possible implementation, the monitoring module 1701 is specifically used to: receive the speech rate changes of the digital human reported by the front-end device; if the speech rate changes of N consecutive frames are greater than a set threshold, then it is determined that there is an interactive stuttering situation.

[0220] In one possible implementation, the generation module 1703 is specifically used to: determine a second transition duration based on the digital human's current speech rate; determine a second mixing parameter based on the second transition duration; and generate at least one mixed frame between the first video frame and the second video frame based on the second mixing parameter, the first video frame, and the second video frame.

[0221] In one possible implementation, the monitoring module 1701 is further configured to: trigger a silent video frame indicating that the digital human is in a silent state after detecting the video screen of the front-end user; trigger the digital human's broadcast audio frame when the digital human enters the broadcasting state, and trigger the digital human's broadcast video frame after detecting the audio information.

[0222] In one possible implementation, the monitoring module 1701 is further configured to: store silent video frames in a silent state through the front-end device; specifically, the monitoring module 1701 is configured to: trigger the front-end device to retrieve silent video frames from the local device and play them in a loop.

[0223] In one possible implementation, the generation module 1703 is specifically used to: calculate the timestamp of the mixed frame based on the frame interval and clock frequency for any mixed frame; if the timestamp of the mixed frame is greater than the current time, wait for a set duration; the set duration is determined based on the timestamp of the mixed frame and the current time; after the set duration, send the mixed frame to the front-end device.

[0224] In one possible implementation, a sending module 1704 is also included for transmitting audio and video to a front-end device based on WebRTC (Web Real-Time Communication).

[0225] Please see Figure 18 This shows a schematic diagram of the structure of another digital human interaction device provided in an embodiment of this application, such as... Figure 18 As shown, the digital human interaction device includes a memory 1801 and a processor 1802, with the processor 1802 coupled to the memory 1801. The memory 1801 is used to store program instructions, and the processor 1802 is used to call the program instructions stored in the memory 1801 and execute the aforementioned digital human interaction method according to the obtained program.

[0226] Optionally, the digital human interaction device may further include an interface circuit 1803, which may be a transceiver or an input / output interface. The input / output interface is used for inputting and / or outputting information; output can be understood as sending, and input as receiving. The processor 1802 can communicate with other devices in the digital human interaction device or other devices besides the digital human interaction device through the interface circuit 1803 to obtain the information required to execute the above-described digital human interaction method.

[0227] When the digital human interaction device 1800 is used to achieve Figure 4 In the method shown, the processor 1802 is used to implement the functions of the monitoring module 1701, the acquisition module 1702, the generation module 1703, and the sending module 1704.

[0228] It is understood that the processor in the embodiments of this application may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0229] The memory in the embodiments of this application may be random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disk, portable hard disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. The storage medium may also be a component of the processor.

[0230] Based on the same technical concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program or instructions, which, when executed by a processor, causes the computer to perform the aforementioned digital human interaction method.

[0231] Based on the same technical concept, embodiments of the present invention also provide a computer-readable program product, which, when executed, causes a computer to perform the aforementioned digital human interaction method.

[0232] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0233] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0234] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0235] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0236] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A digital human interaction method, characterized in that, The method is applied to a backend server and includes: During the interaction between the digital human displayed on the front-end device and the front-end user, monitor for any interaction lag. If there is a lag in the interaction, the first video frame displayed on the front-end device and the next second video frame to be displayed are obtained. At least one mixed frame between the first video frame and the second video frame is generated and sent to the front-end device using the first video frame and the second video frame.

2. The method as described in claim 1, characterized in that, The monitoring of whether there is interaction lag includes: Determine if there are predefined state transitions during the interaction process. If there are state transitions, then it is determined that there is an interaction lag.

3. The method as described in claim 2, characterized in that, The step of generating at least one mixed frame between the first video frame and the second video frame includes: Determine the first transition duration between the first video frame and the second video frame, and the elapsed time since the first video frame to the current time; The first mixing parameter is determined based on the ratio of the elapsed time to the first transition time; Based on the first mixing parameters, the first video frame, and the second video frame, at least one mixed frame is generated between the first video frame and the second video frame.

4. The method as described in claim 2, characterized in that, Determining whether a predefined state transition exists during the interaction includes: Based on the digital human's audio information, determine whether a predefined state transition exists; and / or, Based on the instructions given to the digital human, determine whether there is a predefined state transition.

5. The method as described in claim 1, characterized in that, The monitoring of whether there is interaction lag includes: Receive feedback from the front-end device regarding the digital human's speech rate changes; If the speech rate change exceeds a set threshold for N consecutive frames, then an interactive stuttering situation is identified.

6. The method as described in claim 5, characterized in that, The step of generating at least one mixed frame between the first video frame and the second video frame includes: The second transition duration is determined based on the digital human's current speech rate; The second mixing parameter is determined based on the second transition duration; Based on the second mixing parameters, the first video frame, and the second video frame, at least one mixed frame is generated between the first video frame and the second video frame.

7. The method according to any one of claims 1 to 6, characterized in that, Before monitoring for interaction lag, the process also includes: After detecting the video feed from the front-end user, a silent video frame is triggered to indicate that the digital human is in a silent state; When the digital human enters the broadcasting state, the broadcasting audio frame of the digital human is triggered, and after detecting the audio information, the broadcasting video frame of the digital human is triggered.

8. The method as described in claim 7, characterized in that, The method further includes: The front-end device stores the silent video frames in the silent state; Triggering a silent video frame that indicates the digital human is in a silent state includes: The front-end device is triggered to retrieve the silent video frame from the local device and play it in a loop.

9. The method according to any one of claims 1 to 6, characterized in that, The sending to the front-end device includes: For any hybrid frame, calculate the timestamp of the hybrid frame based on the frame interval and clock frequency; If the timestamp of the mixed frame is greater than the current time, then wait for a set duration; the set duration is determined based on the timestamp of the mixed frame and the current time. After the set duration, the mixed frame is sent to the front-end device.

10. The method according to any one of claims 1 to 6, characterized in that, Also includes: The audio and video are transmitted to the front-end device using WebRTC (Web Real-Time Communication).

Images