A digital human communication method and a digital human communication system

Abstract

This invention provides a digital human communication method and system, relating to the field of digital human interaction technology. The method includes: in response to a call initiated by a calling user, acquiring call input data; processing the call input data to generate parsed text; inputting the parsed text into a preset large model to generate contextual dialogue text; converting the contextual dialogue text into digital human-driven audio data and corresponding digital human action data; driving a preset 3D digital human action based on the digital human-driven audio data and the digital human action data and rendering and outputting it to obtain a digital human rendered audio-visual stream; converting the digital human rendered audio-visual stream into an audio-visual stream suitable for the communication system; and sending the audio-visual stream to the calling user's terminal. This invention can ensure the real-time performance and effectiveness of interaction in multimodal interaction scenarios, improving the user's interaction experience with the digital human.

Description

Technical Field

[0001] This invention relates to the field of digital human interaction technology, and more specifically, to a digital human communication method and a digital human communication system. Background Technology

[0002] A digital human is a virtual character created using technologies such as computer graphics and artificial intelligence, possessing a human-like appearance and behavioral abilities; digital human interaction refers to the communication and collaboration process between users and digital humans.

[0003] In related technologies, the interaction between users and digital humans is mainly limited to local rendering on the device side or cloud rendering of browser H5, which lacks diversity and flexibility and affects the user's interactive experience. The interaction between the call system and the digital human often runs independently, lacking effective integration and interaction methods, resulting in a lack of continuity when users switch between different systems and a poor user experience. Summary of the Invention

[0004] The present invention aims to solve at least one of the following technical problems: lack of diversity and flexibility in digital human interaction, inconsistent user switching between different systems, and poor user interaction experience.

[0005] To address the above problems, in a first aspect, the present invention provides a digital human communication method, comprising: In response to a call initiated by a calling user, the call input data is obtained, wherein the call input data includes at least one of user audio and video stream data and user keypad data. The call input data is processed to generate parsed text, including: The user audio and video stream data is subjected to multimedia transcoding and segmentation detection processing to generate a first parsed text; and / or, the user key press data is parsed according to a preset key press strategy to generate a second parsed text; the parsed text includes at least one of the first parsed text and the second parsed text. The parsed text is input into a preset large model to generate contextual dialogue text; the contextual dialogue text is converted into digital human driving audio data and corresponding digital human motion data; the preset three-dimensional digital human motion is driven according to the digital human driving audio data and the digital human motion data and rendered and output to obtain a digital human rendered audio and video stream. The digital human rendered audio and video stream is converted into an audio and video stream suitable for the call system, and the audio and video stream is sent to the calling user terminal.

[0006] Optionally, the step of performing multimedia transcoding and segmentation detection on the user audio and video stream data to generate the first parsed text includes: Obtain the first audio data from the user's audio and video stream data; According to the call description protocol, the first audio data is converted from the first encoding format to the second encoding format to obtain the second audio data; The second audio data is subjected to voice activity detection and dynamically segmented according to a preset silence time threshold to obtain at least one audio file; Speech recognition is performed on each of the audio files to obtain the first parsed text.

[0007] Optionally, the step of performing voice activity detection on the second audio data and dynamically segmenting it according to a preset silence time threshold to obtain at least one audio file includes: Semantic features are extracted from the second audio data to identify the speech and non-speech components; Based on the speech portion, the start and end boundaries of the speech portion are determined by judging the transition positions between consecutive speech frames and consecutive non-speech frames, and the speech portion is dynamically segmented according to a preset silence time threshold, so that each segmented audio segment corresponds to a complete semantic unit. Each audio segment is encapsulated into an audio file conforming to a preset format.

[0008] Optionally, the step of parsing the user key press data according to a preset key press strategy to generate a second parsed text includes: The system receives key events obtained from the user key data by the call system, wherein the key events include at least a call identifier and a key value; Based on the call identifier and the key value, query the preset function mapping table to obtain the corresponding function command; The execution function corresponding to the function instruction is invoked using the strategy pattern to generate a text interaction event containing function information, and the text interaction event is used as the second parsed text.

[0009] Optionally, the step of using the strategy pattern to call the execution function corresponding to the function instruction, generating a text interaction event containing function information, and using the text interaction event as the second parsed text includes: Obtain the function number and function name corresponding to the function instruction; Based on the function number and function name, determine the type of business logic to be executed. The type of business logic includes at least one of audio and video recording, video screenshot, Uniform Resource Locator display, or text pass-through. The key value, along with the corresponding function number, function name, and business logic type, are assembled into a text interaction event.

[0010] Optionally, inputting the parsed text into a preset large model to generate contextual dialogue text includes: A preset large model is invoked and the parsed text is used as input to generate a dialogue, obtain a dialogue record, and each time the preset large model is invoked, the dialogue record of the previous round is carried as input to obtain a preset number of dialogue records as the context dialogue text.

[0011] Optionally, the step of converting the contextual dialogue text into digital human-driven audio data and corresponding digital human motion data, driving a preset 3D digital human motion based on the digital human-driven audio data and the digital human motion data, and rendering and outputting the result to obtain a digital human rendered audio-visual stream includes: The speech synthesis service is invoked in a streaming manner to synthesize the contextual dialogue text into audio data driving the digital human. The digital human driving audio data is converted into facial expression driving data by a streaming conversion using an audio driving algorithm. The digital human motion data includes the facial expression driving data. The preset three-dimensional digital human actions are driven and rendered based on the digital human driving audio data and the digital human motion data, resulting in a digital human rendered audio and video stream. The preset three-dimensional digital human is invoked by the called user corresponding to the call input data.

[0012] Optionally, the step of driving the preset 3D digital human movements based on the digital human driving audio data and the digital human motion data, and rendering and outputting the resulting digital human rendered audio-visual stream includes: The digital human motion data and the digital human driving audio data are sent to the digital human rendering engine through a two-way communication protocol, so that the digital human rendering engine loads the digital human motion data in real time and drives the preset three-dimensional digital human motion, thereby rendering video data and synchronously playing the digital human driving audio data. The video data and the digital human driving audio data are then combined to obtain the digital human rendering audio-visual stream. The digital human rendered audio and video stream is pushed out in real time through a cloud application platform.

[0013] Optionally, converting the digital human rendered audio / video stream into an audio / video stream suitable for a call system and sending the audio / video stream to the calling user terminal includes: The digital human driving audio data in the digital human rendering audio-visual stream is converted into the first encoding format to obtain the third audio data; The video data in the digital human rendered audio and video stream is transcoded according to the call description protocol to obtain the converted video data. The converted video data and the third audio data are encapsulated according to a real-time media transmission protocol to serve as an audio and video stream suitable for the call system. The audio and video streams are sent to the calling user's terminal.

[0014] The digital human calling method provided by this invention acquires call input data in the calling user's dialing scenario, enabling the calling system to receive the user's audio, video, and / or key input, thus achieving compatibility with multiple input methods during the call. By performing multimedia transcoding and segmentation detection processing on the audio and video streams, and strategy parsing processing on the user's key data, multimodal interactive inputs such as voice, video, and key inputs are uniformly converted into standardized text that can be processed by a large model. By inputting the parsed text into a large model to generate contextual dialogue text and driving the rendering output of the 3D digital human, intelligent conversion from multimodal semantic understanding to digital human voice and action response is achieved. This allows the 3D digital human's lip movements, facial expressions, and other actions to be synchronized with the spoken content in real time, providing not only rich and flexible interaction driving methods but also improving the interaction effect. By converting the digital human's rendered audio and video streams into a format suitable for the calling system and sending them to the calling user's terminal, the audio and video streams can be played normally on the calling user's terminal device without requiring any modification to the user's terminal or the installation of special software, ensuring real-time interaction and improving the user's interaction experience with the digital human, enabling users to interact with the digital human more naturally and conveniently.

[0015] In a second aspect, the present invention also provides a digital human communication system, which applies the digital human communication method as described in any of the preceding claims, including: The call input module is used to: in response to a call initiated by a calling user, acquire call input data, wherein the call input data includes at least one of user audio and video stream data and user keypad data; The parsed text generation module is used to: process the call input data to generate parsed text, including: The user audio and video stream data is subjected to multimedia transcoding and segmentation detection processing to generate a first parsed text; and / or, the user key press data is parsed according to a preset key press strategy to generate a second parsed text; the parsed text includes at least one of the first parsed text and the second parsed text. The digital human multimodal interaction module is used to: input the parsed text into a preset large model to generate contextual dialogue text; convert the contextual dialogue text into digital human driving audio data and corresponding digital human motion data; drive a preset three-dimensional digital human motion according to the digital human driving audio data and the digital human motion data and render and output it to obtain a digital human rendered audio and video stream; The digital human rendering output module is used to: convert the digital human rendered audio and video stream into an audio and video stream suitable for a call system, and send the audio and video stream to the calling user terminal.

[0016] Thirdly, the present invention provides an electronic device, including a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the digital human communication method as described in the first aspect when executing the computer program.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the digital human communication method as described in the first aspect.

[0018] The beneficial effects of the call system, electronic device, and computer-readable storage medium provided by this invention compared to the prior art are the same as the beneficial effects of the digital human call method compared to the prior art, and will not be repeated here. Attached Figure Description

[0019] Figure 1 This invention illustrates a flowchart of a digital human communication method according to an embodiment of the present invention. Figure 1 ; Figure 2 This diagram illustrates a page for audio segmentation processing in an embodiment of the present invention. Figure 3 An interactive schematic diagram of the digital human communication method in an embodiment of the present invention is shown; Figure 4 This illustrates a flowchart of the digital human communication method in an embodiment of the present invention. Figure 2 ; Figure 5 A schematic diagram of the structure of the digital human communication system in an embodiment of the present invention is shown; Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] It should be noted that relational terms such as "first" and "second" in this invention are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] In the description of this specification, references to terms such as "embodiment," "one embodiment," and "one implementation" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or implementation is included in at least one embodiment or illustrative implementation of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or implementation. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or implementations.

[0023] This invention relates to the introduction of terminology.

[0024] 1. SDP (Session Description Protocol). A text-based protocol used to describe metadata (such as media type, encoding format, and transport address) for multimedia sessions, enabling communicating parties to negotiate session parameters. Key fields include: session-level information (session name, initiator, time, network type, etc.) and media-level information (media type, transport protocol, audio / video encoding format, port number, etc.).

[0025] 2. RTP (Real-time Transport Protocol); RTP is a streaming media protocol used for transmitting real-time audio and video data over the Internet. Based on the UDP protocol, it does not guarantee the order or reliability of data packets, but synchronization is achieved through sequence numbers and timestamps. Each media stream uses an independent RTP session, typically used in conjunction with RTCP (Real-time Transport Control Protocol) to monitor transmission quality, provide feedback on packet loss rate, latency, and other information.

[0026] 3. VAD (Voice Activity Detection); A technique for detecting the presence of speech in an audio signal. Common methods include energy-based detection, spectral feature-based detection, machine learning-based detection, zero-crossing rate-based detection, and detection methods that combine multiple features. Among these, machine learning-based methods can handle complex noisy environments and improve detection accuracy.

[0027] 4. WebRTC (Web Real-Time Communication); This invention relates to a technology that supports real-time voice, video, and data sharing across web pages and mobile applications. Based on the pion / WebRTC open-source project implemented in Go, this invention constructs a WebRTC digital human media input function that can run independently without relying on a web browser.

[0028] 5. BlendShape; A technique for achieving facial animation involves defining multiple different shapes (such as smiling, angry, surprised, etc.) in a 3D model and then blending these shapes at runtime to create dynamic facial expressions.

[0029] 6. AMR-WB (Adaptive Multi-Rate Wideband). A wideband voice coding standard developed by the 3GPP organization, primarily used in mobile communication systems to improve voice quality. This technology extends traditional voice bandwidth (50-7000Hz), supports dynamic bitrate adjustment (6.6kbps to 23.85kbps), and employs voice coding algorithms to achieve efficient audio compression.

[0030] 7. PCM (Pulse Code Modulation); Uncompressed audio sample data raw stream, which is standard digital audio data converted from analog signals through sampling, quantization, and encoding.

[0031] 8. ONNX Runtime; Developed by Microsoft, this high-performance inference engine runs ONNX (Open Neural Network Exchange) deep learning models. Its primary goal is to accelerate model inference and support cross-platform deployment, enabling deep learning models to run efficiently on CPUs, GPUs, and edge devices.

[0032] It should be noted that the above terminology definitions are only used to help understand the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0033] Reference Figure 1 As shown, this embodiment of the invention proposes a digital human communication method; The digital human communication method includes: S100: In response to the calling user initiating a call, obtain call input data, wherein the call input data includes at least one of user audio / video stream data and user keypad data.

[0034] Specifically, when a calling user dials a digital human service number linked to the called user's number via a terminal device (such as a mobile phone), the call system responds to the call request by acquiring input data during the call. This input data includes at least one of user audio and video stream data (i.e., the user's voice and / or the user's video feed) and user keystroke data (i.e., the number keys pressed by the user on the dial pad, such as "2").

[0035] S200: Process the call input data to generate parsed text, including: The user audio and video stream data is subjected to multimedia transcoding and segmentation detection processing to generate a first parsed text; and / or, the user key press data is parsed according to a preset key press strategy to generate a second parsed text; the parsed text includes at least one of the first parsed text and the second parsed text.

[0036] Specifically, different processing methods are used for different types of call input data. For user audio and video stream data, multimedia transcoding and segmentation detection are performed to convert it into formatted text, i.e., the first parsed text. For example, the user's spoken message "Please check my balance" is converted into the text "Please check my balance" after transcoding, segmentation, and speech recognition. For user key press data, it is parsed according to a pre-set key press strategy, converting the key values ​​into corresponding function commands or text information, i.e., the second parsed text. For example, when the user presses key "2", the system parses it as the command "Start recording of caller audio and video". The final parsed text can be the first parsed text, the second parsed text, or a combination of both.

[0037] S300: Input the parsed text into a preset large model to generate contextual dialogue text; convert the contextual dialogue text into digital human-driven audio data and corresponding digital human motion data; drive the preset three-dimensional digital human motion according to the digital human-driven audio data and the digital human motion data and render and output it to obtain a digital human rendered audio and video stream.

[0038] Specifically, the parsed text is fed into a pre-defined large model (e.g., a language model). Based on this input, the large model generates context-coherent dialogue text. For example, in the first round, if a user asks "What's the weather like in Beijing tomorrow?", the large model answers "Tomorrow in Beijing will be sunny, with a temperature of 22°C." In the second round, if the user only asks "What clothes should I wear?", the large model, considering the context of the previous round, understands that the user is asking "What clothes should I wear in Beijing tomorrow with a temperature of 22°C?" and generates the answer "Tomorrow in Beijing, the temperature will be 22°C; a light jacket or long-sleeved shirt is suitable." Then, this dialogue text is converted into audio data and corresponding motion data to drive the digital human. This data is used to drive a pre-defined 3D digital human model to perform corresponding actions and speech expressions, and the output is rendered in real time, forming a digital human rendered audio-visual stream containing video and audio.

[0039] S400: Convert the digital human rendered audio and video stream into an audio and video stream suitable for the call system, and send the audio and video stream to the calling user terminal.

[0040] Specifically, the digital human rendering audio and video stream generated by the S300 undergoes format conversion to conform to the audio and video formats required by traditional call systems (such as encoding formats and transmission protocols). For example, the WebRTC format audio and video stream output by the digital human rendering is transcoded into an RTP format audio and video stream that the call system can recognize. After conversion, the audio and video stream suitable for the call system is sent to the calling user's terminal, allowing the calling user to see the digital human's image and hear its voice in real time, realizing real-time dialogue between the user and the digital human; ensuring the real-time nature of the interaction.

[0041] In practical application, this embodiment acquires call input data during the calling user's dialing scenario, enabling the call system to receive the user's audio, video, and / or key input, thus achieving compatibility with multiple input methods during the call. By performing multimedia transcoding and segmentation detection on the audio and video streams, and by performing strategy parsing on the user's key data, multimodal interactive inputs such as voice, video, and key inputs are uniformly converted into standardized text that can be processed by large models. By inputting the parsed text into the large model to generate contextual dialogue text and driving the rendering output of the 3D digital human, intelligent conversion from multimodal semantic understanding to digital human voice and action response is achieved. This allows the 3D digital human's lip movements, facial expressions, and other actions to be synchronized with the spoken content in real time, providing not only rich and flexible interaction driving methods but also improving the interaction effect. By converting the digital human's rendered audio and video streams into a format suitable for the call system and sending them to the calling user's terminal, the audio and video streams can be played normally on the calling user's terminal device without requiring any modification to the user's terminal or the installation of special software, ensuring the real-time nature of the interaction and improving the user's interaction experience with the digital human, enabling users to interact with the digital human more naturally and conveniently.

[0042] This invention is primarily applied to customer service Q&A scenarios, such as bank customer service, government inquiries, and e-commerce shopping guides. For example, after dialing a customer service hotline, users can directly engage in real-time dialogue with a 3D digital human customer service representative without waiting for a human agent. Users describe their questions via voice (e.g., "What should I do if my bank card is locked?"), which is then converted into text by speech recognition technology and input into a large-scale intelligent model. The model generates a professional answer based on a business knowledge base, which is then converted into speech using speech synthesis technology, driving the digital human to speak the answer and simultaneously display facial expressions, lip movements, and other actions. This method automates and intelligentizes customer service, allowing users to access 24 / 7 online service without installing any applications, reducing waiting time, and supporting multimodal interactive input such as voice, video, and button input, improving service efficiency and user satisfaction. The digital human's facial expressions and lip movements are synchronized with speech and other actions in real time, enhancing the naturalness and immersion of the interaction.

[0043] like Figure 2 , Figure 3 and Figure 4 As shown, in an optional embodiment of the present invention, the step of performing multimedia transcoding and segmentation detection processing on the user audio and video stream data to generate the first parsed text includes: Obtain the first audio data from the user's audio and video stream data; Specifically, the text generation module interfaces with the call system to acquire and parse the user's voice during the call; it receives the user's audio and video stream transmitted by the call system via the RTP protocol, extracts the audio data portion from it, and uses it as the first audio data. This audio data uses an encoding format agreed upon by the call system, such as an AMR-WB encoded audio stream.

[0044] According to the call description protocol, the first audio data is converted from the first encoding format to the second encoding format to obtain the second audio data; Specifically, the call description protocol (e.g., Session Description Protocol, SDP) contains media parameters negotiated by both parties in the call; the SDP signaling information is parsed to determine the audio encoding format and encoding mode used by the user's audio and video streams; when it is determined that the first audio data uses AMR-WB encoding in bandwidth-efficient mode, the audio data is bitstream transcoded based on the RFC3267 protocol to convert the bandwidth-efficient mode audio stream into an Octet-aligned mode audio stream, and then transcoded into a PCM raw audio stream by a multimedia processing framework (e.g., FFmpeg) as the second audio data.

[0045] The second audio data is subjected to voice activity detection and dynamically segmented according to a preset silence time threshold to obtain at least one audio file; Specifically, the raw PCM audio stream is buffered and input in real time into a deep learning-based speech activity detection model (e.g., VAD) for voice recognition. This model uses the ONNX Runtime engine for inference acceleration to identify the human voice portion of the audio. Based on the recognition results, the audio stream is dynamically segmented according to a preset silence time threshold (e.g., 1 second), dividing continuous human voices into multiple audio segments, each corresponding to a complete semantic unit. These audio segments are then encapsulated into WAV format audio files.

[0046] Speech recognition is performed on each of the audio files to obtain the first parsed text.

[0047] Specifically, the WAV audio file obtained by segmentation and encapsulation is parsed by calling the Automatic Speech Recognition (ASR) service to convert the speech content in the audio file into text, which is then output as the first parsed text.

[0048] In practical application, this embodiment solves the technical problem that the multimedia processing framework does not support bandwidth-efficient mode by default by converting the first audio data (e.g., an audio stream in AMR-WB bandwidth-efficient mode) in the call system into the second audio data (e.g., a raw PCM audio stream). It then uses a deep learning-based speech activity detection model (e.g., VAD) to detect human voices and segment them according to a preset silence time threshold (e.g., 1 second). This segments the continuous speech into multiple audio fragments corresponding to complete semantic units and encapsulates them into audio files in a preset format (e.g., WAV format). This adapts to speech recognition services that only support file-based audio recognition interfaces, achieving near real-time speech recognition. The above method completes the entire conversion from call audio input to parsed text, providing standardized text input for subsequent large-scale model dialogues.

[0049] As an optional embodiment of the present invention, the call system is further configured with a user management center. This user management center supports the querying and configuration of digital human information. Pre-bound digital human configuration parameters can be queried through the called user's number, including: digital human identifier (used to associate with 3D digital human model files), digital human voice identifier (used for speech synthesis services), digital human application identifier (used for the 3D cloud application platform), large model intelligent agent identifier, large model intelligent agent authentication token, and intelligent agent knowledge base identifier. The above information is pre-generated or selected in the digital human platform, speech synthesis service, 3D cloud application platform, large model platform, and knowledge base platform, and then configured in the user management center along with the called user's number. When the calling user dials the digital human service number, the system queries and obtains the corresponding digital human configuration parameters in real time based on the called number, for subsequent digital human rendering and large model dialogue.

[0050] like Figure 2 As shown, in an optional embodiment of the present invention, the step of performing voice activity detection on the second audio data and dynamically segmenting it according to a preset silence time threshold to obtain at least one audio file includes: Semantic features are extracted from the second audio data to identify the speech and non-speech components; Specifically, the raw PCM audio stream obtained through transcoding is buffered and input into a deep learning-based speech activity detection model (e.g., VAD). This model first preprocesses the audio signal, including pre-emphasis, framing, and windowing, converting the audio signal into a format suitable for neural network processing. Then, a neural network model built using deep learning technology automatically learns and recognizes the features of the speech signal, outputting the identification result of whether the audio frame belongs to the speech portion or the non-speech portion.

[0051] Based on the speech portion, the start and end boundaries of the speech portion are determined by judging the transition positions between consecutive speech frames and consecutive non-speech frames, and the speech portion is dynamically segmented according to a preset silence time threshold, so that each segmented audio segment corresponds to a complete semantic unit. Specifically, based on the recognition results output by the speech activity detection model, a speech start boundary is determined when consecutive non-speech frames are converted into consecutive speech frames, and a speech end boundary is determined when consecutive speech frames are converted into consecutive non-speech frames. After detecting a speech end boundary, if the silence duration reaches a preset silence time threshold (e.g., 1 second), the speech segment is segmented. Each segmented audio fragment corresponds to a complete semantic unit, such as a complete question from a user, which can effectively achieve sentence segmentation recognition of the user's speech intent.

[0052] Each audio segment is encapsulated into an audio file conforming to a preset format.

[0053] Specifically, the segmented audio segments are processed into audio packets and packaged into audio files conforming to a preset format (e.g., WAV format) for subsequent use by the speech recognition service. Since the current speech recognition service only supports file-based audio recognition interfaces and not streaming methods, segmentation and packaging into audio files achieves adaptation to non-streaming ASR services.

[0054] In practical application, this embodiment uses a deep learning-based speech activity detection model to perform human voice recognition on audio. It dynamically segments the audio according to a preset silence time threshold (e.g., 1 second), dividing continuous human voices into multiple audio segments. Each segment corresponds to a complete semantic unit and is packaged into a WAV format audio file. This approach reduces requests for invalid audio to the speech recognition service, improving resource utilization. Furthermore, it solves the problem of speech recognition services not supporting streaming, enabling segmented audio to be recognized segment by segment, achieving a near real-time interactive experience.

[0055] As an optional embodiment of the present invention, the step of parsing the user key press data according to a preset key press strategy to generate a second parsed text includes: The system receives key events obtained from the user key data by the call system, wherein the key events include at least a call identifier and a key value; Specifically, the call system provides an HTTP key press reporting interface, through which it receives key press events obtained from user key press data. User key press data originates from the terminal keys pressed by the calling user during a call. These key press events are associated with a key press policy pre-bound to the called number (e.g., customer service hotlines 400x, 10010, etc.). Different called numbers can be configured with different key press function mappings. For example, taking the 400x service number as an example: key 1 is configured to hang up, and key 0 is configured to transfer to a live agent. The field information obtained from the key press events is shown in Table 1 below. Table 1 - Field Information Table

[0056] Based on the call identifier and the key value, query the preset function mapping table to obtain the corresponding function command; Specifically, the system uses the called user's number and keystroke values ​​as search criteria to find the corresponding function commands in a pre-defined function mapping table. The digital human information associated with the called number is pre-configured in the user management center, including digital human ID, digital human voice, digital human application, and large-scale intelligent agent ID. The function mapping table stores the correspondence between keystroke values ​​and function commands for different called numbers, allowing called users to customize keystroke logic combinations based on supported functions.

[0057] The execution function corresponding to the function instruction is invoked using the strategy pattern to generate a text interaction event containing function information, and the text interaction event is used as the second parsed text.

[0058] Specifically, a strategy pattern is adopted, where each function number implements different logic on the system according to the strategy pattern. The configured target execution function is obtained by parsing the key press strategy, and text interaction events are generated according to the instruction logic. The generated text interaction events are shown in Table 2 below.

[0059] Table 2 - Text Interaction Events

[0060] At the same time, data is recorded based on the called user's number, user keystrokes, function number, URL image, URL video, and text content. The execution function corresponding to each function number can implement different business logic.

[0061] In practical application, this embodiment achieves dynamic mapping between key values ​​and function commands by receiving key events and querying the function mapping table, allowing different called numbers to be configured with different key function combinations. The strategy pattern is used to call execution functions, ensuring that the implementation logic of different key functions is independent, facilitating function expansion and maintenance. For example, the called number 101551001 can be configured to set the logic function of key 2 as calling video recording. When the user presses key 2 once during a call, the FFmpeg tool is invoked to start recording the user's audio and video; pressing key 2 again ends the recording. This method integrates traditional call key presses with intelligent dialogue in a digital human, expanding the ways users interact with the digital human.

[0062] As an optional embodiment of the present invention, the step of using the strategy pattern to call the execution function corresponding to the function instruction, generating a text interaction event containing function information, and using the text interaction event as the second parsed text includes: Obtain the function number and function name corresponding to the function instruction; Specifically, the function number and function name are obtained from the function instructions. The system has a variety of preset functions, such as: return, start recording of calling audio and video, end recording of calling audio and video, start recording of called audio and video, end recording of called audio and video, extract and generate an image from the calling video, extract and generate an image from the called video, display an image from a URL, close an image from a URL, play a video from a URL, close a video from a URL, and pass through text. Each function corresponds to a unique function number and function name. For example, function number 2 corresponds to the function name callRecordStart (start recording of calling audio and video).

[0063] Based on the function number and function name, determine the type of business logic to be executed. The type of business logic includes at least one of audio and video recording, video screenshot, Uniform Resource Locator display, or text pass-through. Specifically, the business logic type to which a function instruction belongs is determined by its function number and function name. For example, "Start recording of calling audio / video" and "End recording of calling audio / video" belong to the audio / video recording type; "Extracting a screenshot from the calling video" belongs to the video screenshot type; "URL image display" and "URL video playback" belong to the Uniform Resource Locator (URL) display type; and "Text pass-through" belongs to the text pass-through type.

[0064] The key value, along with the corresponding function number, function name, and business logic type, are assembled into a text interaction event.

[0065] Specifically, key values, function numbers, function names, and the determined business logic type are assembled into structured text interaction events. Taking caller video recording as an example, when the user presses key "2", the generated text interaction event includes function number "2", function name "callRecordStart", key value "2", and can carry optional fields such as URL image address, URL video address, and text content.

[0066] In practical applications, this embodiment transforms button interactions into structured text interaction events by obtaining the function number and function name and determining the business logic type. This method converts traditional button input into structured data that can be recognized by the digital human multimodal interaction system, providing standardized input for the execution of subsequent business logic.

[0067] As an optional embodiment of the present invention, the step of inputting the parsed text into a preset large model to generate contextual dialogue text includes: A preset large model is invoked and the parsed text is used as input to generate a dialogue, obtain a dialogue record, and each time the preset large model is invoked, the dialogue record of the previous round is carried as input to obtain a preset number of dialogue records as the context dialogue text.

[0068] Specifically, the parsed text (i.e., the first parsed text and / or the second parsed text) is input as the question for the current round into a pre-defined large language model. On the first call, only the current parsed text is used as input; from the second call onwards, in addition to the current parsed text, the response records generated by the model in the previous round are also used as context input. This process iterates, carrying the dialogue records from the previous round with each call, accumulating until a preset number of rounds is reached (e.g., a maximum of 10 rounds). The accumulated multi-round dialogue records are then output as context dialogue text for use in subsequent steps.

[0069] In practical application, this embodiment calls a preset large (language) model and uses the parsed text as input for the current round. At the same time, it also inputs the dialogue record generated from the previous round into the model, enabling the model to understand the information or referential relationships omitted by the user in the current round based on the historical dialogue content, and generate response text that is coherent with the context. By iteratively accumulating the dialogue record of a preset number of rounds as contextual dialogue text, context management of multi-round dialogue is realized, enabling the digital human to understand the user's continuous intentions in multi-round dialogue, avoiding repeated requests for the same information, and improving the naturalness and intelligence of digital human interaction.

[0070] As an optional embodiment of the present invention, the step of converting the contextual dialogue text into digital human-driven audio data and corresponding digital human motion data, driving a preset three-dimensional digital human motion based on the digital human-driven audio data and the digital human motion data, and rendering and outputting the result to obtain a digital human rendered audio-visual stream includes: The speech synthesis service is invoked in a streaming manner to synthesize the contextual dialogue text into audio data driving the digital human. Specifically, text-to-speech (TTS) is a service that converts text data into audio. By acquiring the response text streamed from a large model, the service is invoked in batches and continuously during the text reception process to synthesize the text content into PCM format audio files, which serve as the audio data driving the digital human. Streaming refers to invoking the service while receiving the output from the large model, without waiting for the complete text to be returned before invoking.

[0071] The digital human driving audio data is converted into facial expression driving data by a streaming conversion using an audio driving algorithm. The digital human motion data includes the facial expression driving data. Specifically, PCM audio is streamed into an audio-driven algorithm, which analyzes acoustic features such as phonemes, intonation, and rhythm in the audio to generate BlendShape facial expression data in real time, serving as the facial expression driving data. Streaming conversion means processing audio data while it is being received, without waiting for the complete audio file to be ready before processing.

[0072] The preset three-dimensional digital human actions are driven and rendered based on the digital human driving audio data and the digital human motion data, resulting in a digital human rendered audio and video stream. The preset three-dimensional digital human is invoked by the called user corresponding to the call input data.

[0073] Specifically, BlendShape data and PCM audio files are sent to a 3D digital human rendering environment in real time. This environment drives the 3D digital human to perform corresponding actions based on the motion data and renders video footage, while simultaneously playing audio data, ultimately resulting in a digital human rendered audio-visual stream containing both video and audio. The 3D digital human model used corresponds to a specific called user; different called users can correspond to different digital human models.

[0074] In practical applications, this embodiment achieves real-time conversion of text to audio and then to facial expression driving data by streaming speech synthesis services (such as TTS services) and audio driving algorithms; by sending the driving data to the 3D digital human rendering environment, the digital human's movements are driven and rendered, enabling the digital human's voice and facial expressions to be generated synchronously in real time.

[0075] As an optional embodiment of the present invention, the step of driving the preset three-dimensional digital human movements based on the digital human driving audio data and the digital human motion data, and rendering and outputting them to obtain a digital human rendered audio-visual stream includes: The digital human motion data and the digital human driving audio data are sent to the digital human rendering engine through a two-way communication protocol, so that the digital human rendering engine loads the digital human motion data in real time and drives the preset three-dimensional digital human motion, thereby rendering video data and synchronously playing the digital human driving audio data. The video data and the digital human driving audio data are then combined to obtain the digital human rendering audio-visual stream. Specifically, BlendShape data and PCM audio files are sent to the Unity digitizer application in real time via bidirectional communication protocols such as WebSocket. The Unity application, acting as the digitizer rendering engine, loads the BlendShape data in real time to drive the digitizer's facial expressions and synchronously plays the PCM audio stream, enabling the digitizer to speak in sync. At the same time, the rendered video footage and audio are combined into a digitizer rendering audio-visual stream.

[0076] The digital human rendered audio and video stream is pushed out in real time through a cloud application platform.

[0077] Specifically, the digital human rendered video and the digital human speaking audio stream are pushed out in real time through the 3D cloud application platform, and are accessible via the WebRTC protocol for downstream modules to subscribe to and pull.

[0078] When establishing a connection with the 3D cloud application, a client based on a real-time communication protocol is created, a call description protocol proposal signaling is generated, and signaling exchange occurs with the 3D cloud application. After negotiating the proposal and response signaling, a connection is established, and the digital human rendering audio and video streams are obtained through a subscription method. This implementation does not depend on the browser environment and can be directly integrated into the backend service.

[0079] In practical applications, this embodiment sends facial expression driving data (e.g., BlendShape) and digital human driving audio data (e.g., PCM) to the digital human rendering engine in real time via a bidirectional communication protocol (e.g., WebSocket). After synthesizing the audio and video stream, it is pushed out through the cloud application platform using a real-time communication protocol (e.g., WebRTC), thus completing the closed loop of conversion from driving data to rendered audio and video stream.

[0080] As an optional embodiment of the present invention, the step of converting the digital human rendered audio and video stream into an audio and video stream suitable for a call system and sending the audio and video stream to the calling user terminal includes: The digital human driving audio data in the digital human rendering audio-visual stream is converted into the first encoding format to obtain the third audio data; Specifically, the audio data in the digital human's rendered audio and video stream uses a third encoding format (such as Opus encoding), while the call system requires the audio data to use a first encoding format (such as AMR-WB encoding). By converting the encoding format of the digital human's driving audio data from the third encoding format to the first encoding format, the resulting third audio data is obtained, making the audio data encoding format conform to the call system's reception requirements.

[0081] The video data in the digital human rendered audio and video stream is transcoded according to the call description protocol to obtain the converted video data. Specifically, during the call setup phase, the calling user terminal and the call system negotiate media parameters through a call description protocol (such as Session Description Protocol SDP). The negotiated parameters include video encoding format (e.g., H.264, H.265), resolution (e.g., 1280×720, 1920×1080), frame rate (e.g., 25fps, 30fps), bitrate (e.g., 1Mbps, 2Mbps), and encoding level. These negotiated parameters represent the video format requirements that the call system can correctly receive and decode.

[0082] The raw video data generated by the digital human rendering engine may have parameter settings (such as high resolution and high bitrate) that exceed the receiving capabilities of the call system or do not meet the format requirements agreed upon in the negotiation protocol. Therefore, this step transcodes the video data output by the digital human rendering engine according to the parameters agreed upon in the call description protocol. This includes, but is not limited to: adjusting the video resolution to match the screen size supported by the call system, compressing the video bitrate to adapt to the bandwidth limitations of the call network, and converting the encoding format or encoding level to meet the decoding capabilities of the call system. The transcoded video data fully meets the receiving requirements of the call system, can be transmitted normally during the call, and can be played smoothly on the user's terminal.

[0083] The converted video data and the third audio data are encapsulated according to a real-time media transmission protocol to serve as an audio and video stream suitable for the call system. Specifically, real-time media transport protocols (such as RTP) are the media transport protocols used by the call system. For example, by encapsulating transcoded video data and converted third-party audio data according to the RTP protocol format, they are packaged into data packets that conform to the RTP protocol specification, forming an audio and video stream suitable for the call system.

[0084] The audio and video streams are sent to the calling user's terminal.

[0085] Specifically, the encapsulated RTP audio and video streams are pushed to the media transmission port (including the target IP address, audio port, and video port) specified by the call system, and then forwarded by the call system to the calling user terminal, thus realizing the transmission of the digital human rendering audio and video streams to the calling user terminal.

[0086] In practical applications, this embodiment performs format conversion and transcoding on the audio and video streams rendered by the digital human, making them compatible with traditional call systems. For example, audio can be converted from encodings such as Opus to the encoding format required by the call system, and video can have its resolution, bitrate, and other parameters adjusted according to the call description protocol. After conversion, the video is encapsulated according to a real-time media transmission protocol (such as RTP) and sent to the calling user's terminal, thereby realizing a complete closed loop for digital human calls.

[0087] It should be noted that this invention relates to four core protocols: a call description protocol (such as SDP) used to negotiate media parameters (encoding format, resolution, etc.) between the two parties in a call, without transmitting data; a real-time media transport protocol (such as RTP) used to actually transmit audio and video data packets in the call system; a bidirectional communication protocol (such as WebSocket) used to send digital human driving data to the rendering engine; and a real-time communication protocol (such as WebRTC) used to pull digital human rendering audio and video streams from the cloud application platform. These four protocols together complete a closed loop from call establishment, data negotiation, media transmission to digital human driving and rendering output.

[0088] Based on the above embodiments, the following can be obtained: Figure 3 and Figure 4 The complete process involves the following steps: After a user initiates a call, the call system establishes a connection with the user's call input module via SDP signaling exchange and obtains the digital human configuration parameters (including digital human ID, digital human voice, large model agent ID, etc.) bound to the called number from the user management center. The user's call input module receives call input data transmitted based on a real-time media transmission protocol (such as RTP). This data includes at least one of the following: user key data, audio data, and video data. The audio data needs to undergo transcoding (e.g., from AMR-WB to PCM format; the transcoded PCM audio data is used for voice activity detection and segmentation, and is finally recognized as text by ASR) before subsequent use. When entering the digital human multimodal interaction, the system makes a judgment based on the input decision execution logic: if it does not conform to the digital human interaction logic (e.g., determined by function commands bound to buttons), then it calls the function function based on the event information (which can be set to the same set of business logic functions as the function command) to complete logical functions such as video recording and photo generation, and stores the results in the file storage service (e.g., storing videos for tongue diagnosis); if it conforms to the digital human interaction logic, it identifies the text content corresponding to the calling user (user button data is parsed into text, audio data is recognized into text by ASR, and video data can also be converted into text after extraction), enters the large model dialogue, generates context dialogue text, calls the speech synthesis service to synthesize digital human driving audio data, and uses the audio driving algorithm to convert the digital human driving audio data into digital human action data (e.g., BlendShape expression driving data), combines the digital human driving audio data to drive the digital human application rendering output, and after processing by the 3D cloud application system and the call system, sends the audio and video streams suitable for the call system to the user terminal.

[0089] like Figure 5 As shown, the present invention also provides a digital human communication system 200, which applies the digital human communication method described in the above embodiments, including: The call input module 210 is used to: in response to a call initiated by a calling user, acquire call input data, wherein the call input data includes at least one of user audio and video stream data and user keypad data; The parsed text generation module 220 is used to: process the call input data to generate parsed text, including: The user audio and video stream data is subjected to multimedia transcoding and segmentation detection processing to generate a first parsed text; and / or, the user key press data is parsed according to a preset key press strategy to generate a second parsed text; the parsed text includes at least one of the first parsed text and the second parsed text. The digital human multimodal interaction module 230 is used to: input the parsed text into a preset large model to generate contextual dialogue text; convert the contextual dialogue text into digital human driving audio data and corresponding digital human motion data; drive a preset three-dimensional digital human motion according to the digital human driving audio data and the digital human motion data and render and output it to obtain a digital human rendered audio and video stream. The digital human rendering output module 240 is used to: convert the digital human rendered audio and video stream into an audio and video stream suitable for a call system, and send the audio and video stream to the calling user terminal.

[0090] The specific implementation method of this embodiment can be referred to the corresponding implementation method described above, and will not be described again here.

[0091] like Figure 6 As shown, an electronic device 300 provided in this embodiment of the invention includes a memory 310 and a processor 320; the memory 310 is used to store a computer program; the processor 320 is used to implement the digital human communication method as described above when the computer program is executed.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0093] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

[0094] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for digital human communication, characterized in that, include: In response to a call initiated by a calling user, the call input data is obtained, wherein the call input data includes at least one of user audio and video stream data and user keypad data. The process of processing the call input data to generate parsed text includes: performing multimedia transcoding and segmentation detection on the user audio and video stream data to generate a first parsed text; and / or parsing the user key press data according to a preset key press strategy to generate a second parsed text; the parsed text includes at least one of the first parsed text and the second parsed text. The parsed text is input into a preset large model to generate contextual dialogue text; the contextual dialogue text is converted into digital human driving audio data and corresponding digital human motion data; the preset three-dimensional digital human motion is driven according to the digital human driving audio data and the digital human motion data and rendered and output to obtain a digital human rendered audio and video stream. The digital human rendered audio and video stream is converted into an audio and video stream suitable for the call system, and the audio and video stream is sent to the calling user terminal.

2. The digital human communication method according to claim 1, characterized in that, The step of performing multimedia transcoding and segmentation detection on the user audio and video stream data to generate the first parsed text includes: Obtain the first audio data from the user's audio and video stream data; According to the call description protocol, the first audio data is converted from the first encoding format to the second encoding format to obtain the second audio data; The second audio data is subjected to voice activity detection and dynamically segmented according to a preset silence time threshold to obtain at least one audio file; Speech recognition is performed on each of the audio files to obtain the first parsed text.

3. The digital human communication method according to claim 2, characterized in that, The step of performing voice activity detection on the second audio data and dynamically segmenting it according to a preset silence time threshold to obtain at least one audio file includes: Semantic features are extracted from the second audio data to identify the speech and non-speech components; Based on the speech portion, the start and end boundaries of the speech portion are determined by judging the transition positions between consecutive speech frames and consecutive non-speech frames, and the speech portion is dynamically segmented according to a preset silence time threshold, so that each segmented audio segment corresponds to a complete semantic unit. Each audio segment is encapsulated into an audio file conforming to a preset format.

4. The digital human communication method according to claim 1, characterized in that, The step of parsing the user key press data according to a preset key press strategy to generate a second parsed text includes: The system receives key events obtained from the user key data by the call system, wherein the key events include at least a call identifier and a key value; Based on the call identifier and the key value, query the preset function mapping table to obtain the corresponding function command; The execution function corresponding to the function instruction is invoked using the strategy pattern to generate a text interaction event containing function information, and the text interaction event is used as the second parsed text.

5. The digital human communication method according to claim 4, characterized in that, The step of using the strategy pattern to call the execution function corresponding to the function instruction, generating a text interaction event containing function information, and using the text interaction event as the second parsed text includes: Obtain the function number and function name corresponding to the function instruction; Based on the function number and function name, determine the type of business logic to be executed. The type of business logic includes at least one of audio and video recording, video screenshot, Uniform Resource Locator display, or text pass-through. The key value, along with the corresponding function number, function name, and business logic type, are assembled into a text interaction event.

6. The digital human communication method according to claim 1, characterized in that, The step of inputting the parsed text into a preset large model to generate contextual dialogue text includes: A preset large model is invoked and the parsed text is used as input to generate a dialogue, obtain a dialogue record, and each time the preset large model is invoked, the dialogue record of the previous round is carried as input to obtain a preset number of dialogue records as the context dialogue text.

7. The digital human communication method according to claim 2, characterized in that, The step of converting the contextual dialogue text into digital human-driven audio data and corresponding digital human motion data, driving preset 3D digital human motion based on the digital human-driven audio data and digital human motion data, and rendering and outputting the digital human rendered audio-visual stream includes: The speech synthesis service is invoked in a streaming manner to synthesize the contextual dialogue text into audio data driving the digital human. The digital human driving audio data is converted into facial expression driving data by a streaming conversion using an audio driving algorithm. The digital human motion data includes the facial expression driving data. The preset three-dimensional digital human actions are driven and rendered based on the digital human driving audio data and the digital human motion data, resulting in a digital human rendered audio and video stream. The preset three-dimensional digital human is invoked by the called user corresponding to the call input data.

8. The digital human communication method according to claim 7, characterized in that, The step of driving the preset 3D digital human movements based on the digital human driving audio data and the digital human motion data, and rendering and outputting the resulting digital human rendered audio-visual stream includes: The digital human motion data and the digital human driving audio data are sent to the digital human rendering engine through a two-way communication protocol, so that the digital human rendering engine loads the digital human motion data in real time and drives the preset three-dimensional digital human motion, thereby rendering video data and synchronously playing the digital human driving audio data. The video data and the digital human driving audio data are then combined to obtain the digital human rendering audio-visual stream. The digital human rendered audio and video stream is pushed out in real time through a cloud application platform.

9. The digital human communication method according to claim 8, characterized in that, The step of converting the digital human rendered audio and video stream into an audio and video stream suitable for a call system and sending the audio and video stream to the calling user terminal includes: The digital human driving audio data in the digital human rendering audio-visual stream is converted into the first encoding format to obtain the third audio data; The video data in the digital human rendered audio and video stream is transcoded according to the call description protocol to obtain the converted video data. The converted video data and the third audio data are encapsulated according to a real-time media transmission protocol to serve as an audio and video stream suitable for the call system. The audio and video streams are sent to the calling user's terminal.

10. A digital human communication system, characterized in that, The digital human communication method as described in any one of claims 1-9 includes: The call input module is used to: in response to a call initiated by a calling user, acquire call input data, wherein the call input data includes at least one of user audio and video stream data and user keypad data; The parsed text generation module is used to: process the call input data to generate parsed text, including: The user audio and video stream data is subjected to multimedia transcoding and segmentation detection processing to generate a first parsed text; and / or, the user key press data is parsed according to a preset key press strategy to generate a second parsed text; the parsed text includes at least one of the first parsed text and the second parsed text. The digital human multimodal interaction module is used to: input the parsed text into a preset large model to generate contextual dialogue text; convert the contextual dialogue text into digital human driving audio data and corresponding digital human motion data; drive a preset three-dimensional digital human motion according to the digital human driving audio data and the digital human motion data and render and output it to obtain a digital human rendered audio and video stream; The digital human rendering output module is used to: convert the digital human rendered audio and video stream into an audio and video stream suitable for a call system, and send the audio and video stream to the calling user terminal.

Images