Adaptive routing digital human interaction method and device and electronic equipment

Abstract

The application discloses a kind of self-adapting route digital human interaction method and its device, electronic equipment, it is related to artificial intelligence technical field, wherein, the method comprises: pre-processing input voice, obtain processing voice;Recognize the interactive scene to which processing voice belongs, when interactive scene is chat scene, processing voice is input to voice model, generate first voice reply audio data, and generate first synthesis video based thereon, control digital person to broadcast based on first synthesis video;When interactive scene is business scene, processing voice is processed, and acoustic feature is obtained;Acoustic feature is input to digital human driving model, and second synthesis video is output, and control digital person to broadcast based on second synthesis video.The application solves the technical problem that digital person lacks self-adapting routing mechanism between processing simple conversation and complex business in related art, which leads to the inability to simultaneously consider the low-latency response of interaction and the controllability of complex business process.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to an adaptive routing digital human interaction method, apparatus, and electronic device. Background Technology

[0002] With the development of artificial intelligence, voice interaction, multimodal large models and digital human technology, digital humans have gradually evolved from traditional one-way broadcasting and simple question and answer to intelligent interactive portals with speech recognition, semantic understanding, speech synthesis, expression-driven and real-time interactive capabilities.

[0003] Currently, digital human interaction mainly includes two types of system architectures: (1) multi-stage serial architecture, which has the advantages of clear modules, controllable process, and easy access to business systems and tool calls, but the link is long and the response latency is high; (2) end-to-end voice interaction architecture, which can directly realize voice input to voice output, with fast response speed and natural tone, but the intermediate process is uncontrollable, and in complex business, emergency handling, permission verification and multi-tool call scenarios, problems such as untraceable results, business illusion or unstable instructions are easy to occur. Therefore, the current method cannot take into account both the low latency of interaction and the controllability of complex business.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides an adaptive routing digital human interaction method, apparatus, and electronic device to at least solve the technical problem in related technologies where digital humans lack an adaptive routing mechanism between handling simple dialogues and complex business processes, resulting in the inability to simultaneously ensure low-latency response of interactions and controllability of complex business processes.

[0006] According to one aspect of the embodiments of this application, an adaptive routing digital human interaction method is provided, applied to a digital human interaction system, comprising: receiving input voice from a terminal device and preprocessing the input voice to obtain processed voice; recognizing the processed voice to determine the interaction scenario to which the processed voice belongs; if the interaction scenario is a chat scenario, inputting the processed voice into a voice model to generate first voice response audio data; generating a first synthesized video based on the first voice response audio data, and controlling the digital human to broadcast based on the first synthesized video; if the interaction scenario is a business scenario, processing the processed voice to obtain acoustic features; inputting the acoustic features into a digital human driving model to output a second synthesized video, and controlling the digital human to broadcast based on the second synthesized video.

[0007] Further, the input speech is preprocessed to obtain processed speech, including: performing speech detection on the input speech to obtain a speech detection result; if the speech detection result indicates that the input speech includes human voice, performing wake word detection on the input speech to obtain a wake word detection result; if the wake word detection result indicates that the input speech includes a wake word, performing voiceprint verification on the input speech to obtain a verification result; if the verification result indicates that the input speech passes, performing acoustic echo cancellation on the input speech to obtain processed speech.

[0008] Further, the steps of recognizing the processed speech and determining the interaction scenario to which the processed speech belongs include: using a business complexity scoring model to calculate the business complexity score of the processed speech, wherein the business complexity scoring model is trained through a set of question pairs, and each question pair in the set includes at least: multiple historical audio texts and a business complexity score description text corresponding to each historical audio text; if the business complexity score is greater than or equal to a preset scoring threshold, the interaction scenario to which the processed speech belongs is determined to be a business scenario; if the business complexity score is less than the preset scoring threshold, the interaction scenario to which the processed speech belongs is determined to be a chat scenario.

[0009] Further, the steps of processing the processed speech to obtain acoustic features include: encoding the processed speech to obtain speech encoding features, inputting the speech encoding features into a language model, generating a calling instruction, and calling the multi-source data corresponding to the processed speech based on the calling instruction; generating an event object based on the multi-source data; and generating acoustic features based on the event object.

[0010] Furthermore, the steps for generating event objects based on multi-source data include: preprocessing the multi-source data to obtain multiple structured data; associating the multiple structured data based on a preset event type to obtain associated data; and encapsulating the associated data to obtain event objects.

[0011] Furthermore, the step of generating acoustic features based on the event object includes: generating a second speech response text based on the event object and processed speech; extracting features from the second speech response text; and generating acoustic features based on the features.

[0012] Furthermore, after controlling the digital human to broadcast based on the second synthesized video, the process includes: listening to the interrupted speech on the terminal device and determining whether the interrupted speech and the input speech belong to the same user; if the interrupted speech and the input speech belong to the same user, determining whether the interrupted speech and the input speech are the same; if the interrupted speech and the input speech are different, controlling the digital human to stop broadcasting; recognizing the interrupted speech, determining the interaction scenario to which the interrupted speech belongs, and regenerating the synthesized video based on the interaction scenario.

[0013] According to another aspect of the embodiments of this application, an adaptive routing digital human interaction device is also provided, applied to a digital human interaction system, comprising: a first processing unit, configured to receive input voice from a terminal device and preprocess the input voice to obtain processed voice; a first recognition unit, configured to recognize the processed voice, determine the interaction scenario to which the processed voice belongs, and, in the case of a chat scenario, input the processed voice to a voice model to generate first voice response audio data; a first generation unit, configured to generate a first synthesized video based on the first voice response audio data, and control the digital human to broadcast based on the first synthesized video; a second processing unit, configured to process the processed voice to obtain acoustic features when the interaction scenario is a business scenario; and a third processing unit, configured to input the acoustic features to a digital human driving model, output a second synthesized video, and control the digital human to broadcast based on the second synthesized video.

[0014] Further, the first processing unit includes: a first detection module for performing speech detection on the input speech to obtain a speech detection result; a second detection module for performing wake-up word detection on the input speech when the speech detection result indicates that the input speech includes a human voice, to obtain a wake-up word detection result; a first verification module for performing voiceprint verification on the input speech when the wake-up word detection result indicates that the input speech includes a wake-up word, to obtain a verification result; and a first cancellation module for performing acoustic echo cancellation on the input speech when the verification result indicates that the input speech passes, to obtain processed speech.

[0015] Furthermore, the first recognition unit includes: a first calculation module, used to calculate a business complexity score for the processed speech using a business complexity scoring model, wherein the business complexity scoring model is trained through a set of question pairs, and each question pair in the set of question pairs includes at least: multiple historical audio texts and a business complexity score description text corresponding to each historical audio text; a first determination module, used to determine that the interaction scenario to which the processed speech belongs is a business scenario when the business complexity score is greater than or equal to a preset scoring threshold; and a second determination module, used to determine that the interaction scenario to which the processed speech belongs is a chat scenario when the business complexity score is less than the preset scoring threshold.

[0016] Furthermore, the second processing unit includes: a first encoding module, used to encode the processed speech to obtain speech encoding features, input the speech encoding features into a language model, generate a calling instruction, and call the multi-source data corresponding to the processed speech based on the calling instruction; a first generation module, used to generate an event object based on the multi-source data; and a second generation module, used to generate acoustic features based on the event object.

[0017] Furthermore, the first generation module includes: a first preprocessing submodule, used to preprocess multi-source data to obtain multiple structured data; a first association submodule, used to associate multiple structured data based on preset event types to obtain associated data; and a first encapsulation submodule, used to encapsulate the associated data to obtain event objects.

[0018] Furthermore, the second generation module includes: a first generation submodule, used to generate a second voice response text based on the event object and processed speech; and a second generation submodule, used to extract features from the second voice response text and generate acoustic features based on the features.

[0019] Furthermore, the adaptive routing digital human interaction device also includes: a first monitoring module, used to monitor the interrupted speech of the terminal device after controlling the digital human to broadcast based on the second synthesized video, and determine whether the interrupted speech and the input speech belong to the same user; a first judgment module, used to determine whether the interrupted speech and the input speech are the same if they belong to the same user; a first control module, used to control the digital human to stop broadcasting if the interrupted speech and the input speech are different; and a first recognition module, used to recognize the interrupted speech, determine the interaction scenario to which the interrupted speech belongs, and regenerate the synthesized video based on the interaction scenario.

[0020] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described adaptive routing digital human interaction methods.

[0021] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described adaptive routing digital human interaction methods.

[0022] In this invention, input voice from a terminal device is received and preprocessed to obtain processed voice. The processed voice is then recognized to determine the interaction scenario to which it belongs. If the interaction scenario is a chat scenario, the processed voice is input into a voice model to generate first voice response audio data. Based on the first voice response audio data, a first synthesized video is generated, and the digital human is controlled to broadcast based on the first synthesized video. If the interaction scenario is a business scenario, the processed voice is processed to obtain acoustic features. The acoustic features are input into a digital human driving model to output a second synthesized video, and the digital human is controlled to broadcast based on the second synthesized video. This invention solves the technical problem in related technologies where digital humans lack an adaptive routing mechanism between handling simple dialogues and complex business processes, resulting in an inability to simultaneously achieve low-latency response in interactions and controllability in complex business processes.

[0023] In this invention, input voice from a terminal device is received and preprocessed to obtain processed voice. The processed voice is then identified to determine its corresponding interaction scenario. When the interaction scenario is identified as a chat scenario, the processed voice is input into a voice model to generate first voice response audio data, and a first synthesized video is generated based on this audio data, controlling a digital human to broadcast. When the interaction scenario is identified as a business scenario, the processed voice is processed to obtain acoustic features, which are then input into a digital human driving model to output a second synthesized video, controlling the digital human to broadcast. By establishing a dual-path adaptive routing mechanism based on the interaction scenario, simple dialogues use a low-latency end-to-end channel, while complex business processes use a controllable multi-stage business channel. A feature-level interconnection mechanism is also established within the multi-stage business channel to reduce redundant conversions between audio, text, and lip-sync features, improving real-time performance and audio-visual synchronization stability. This achieves low-latency response for simple dialogues and controllability for complex business processes. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an adaptive routing digital human interaction method is shown.

[0026] Figure 2 This is a flowchart of an adaptive routing digital human interaction method according to Embodiment 1 of this application;

[0027] Figure 3 This is a schematic diagram of a full-duplex digital human interaction system with dual-path adaptive routing for an integrated air-space-ground digital intelligence tower, according to an embodiment of this application.

[0028] Figure 4 This is a flowchart of a dual-channel adaptive full-duplex digital human interaction according to an embodiment of this application;

[0029] Figure 5 This is a flowchart of a dual-path adaptive routing determination according to an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of a feature-level through-link of a multi-stage service channel according to an embodiment of this application;

[0031] Figure 7 This is a flowchart of a multi-source data fusion and structured event object generation method based on an embodiment of this application;

[0032] Figure 8 This is a flowchart of a full-duplex interruption and state reset according to an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of an adaptive routing digital human interaction device according to an embodiment of this application;

[0034] Figure 10 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected and involved in this invention are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data comply with the relevant laws, regulations, and standards of the relevant regions, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface. After receiving consent from the aforementioned user or organization, the relevant information is obtained. If the user chooses to refuse, the process enters the expert decision-making process.

[0038] In this invention, by continuously monitoring the user's natural speech during the digital human's broadcast, and combining preprocessing and interruptible monitoring mechanisms, the system makes real-time judgments on user interruptions. When a valid new instruction is detected, the system automatically stops the current broadcast, clears the audio queue, video frame queue, and lip-sync drive cache. Therefore, it reduces problems such as continued broadcasting of old content, crosstalk between old and new responses, and residual old audio or lip movements, improving the consistency of the interactive state. Furthermore, it unifies multi-source information such as high-altitude tower video, smoke and fire identification results, satellite remote sensing hotspots, UAV data, meteorological data, and forest resource ledgers into a structured fire event object, ensuring that digital human broadcasting, response plan generation, instruction issuance, and report generation are all based on the same event context. Therefore, it reduces manual comparison and repetitive descriptions caused by data dispersion from multiple systems, making fire events traceable, updatable, and reusable in multiple rounds of interaction. Simultaneously, multiple modules are set up in the multi-stage business channel, enabling complex business instructions to be progressively parsed, invoked, confirmed, and recorded. Compared to fully end-to-end voice interaction, this approach preserves auditable intermediate processes and business statuses in scenarios such as fire situation inquiries, drone verification, plan generation, command issuance, and report generation, improving the reliability of complex business processing. Furthermore, by constructing a feature-level interconnection mechanism, speech coding features, semantic features, acoustic features, and digital human-driven features are prioritized for transmission between speech recognition, large-scale model inference, speech synthesis, and the digital human-driven model. This reduces redundant conversions between audio, text, and lip-sync features, lowers data copying and serialization overhead between the CPU and GPU, and improves the digital human's response speed, lip-sync stability, and regeneration efficiency after interruption. In addition, echo cancellation and voiceprint lock verification reduce the risk of misidentification caused by the digital human's own broadcasting voice, background noise, and the voices of others. Through the simultaneous output of voice, lip-sync, facial expressions, subtitles, and visual information, commanders can simultaneously hear feedback and confirm the fire location, risk level, response plan, and task status, thereby improving the real-time performance, reliability, and confirmability of emergency command interaction.

[0039] The present invention will now be described in detail with reference to various embodiments.

[0040] Example 1

[0041] According to an embodiment of this application, an embodiment of an adaptive routing digital human interaction method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an adaptive routing digital human interaction method is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions may also be included. In addition, it may include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera, wherein the network interface can be connected to wired and / or wireless networks. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0043] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the adaptive routing digital human interaction method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned adaptive routing digital human interaction method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0046] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Under the aforementioned operating environment, this application provides the following: Figure 2 The adaptive routing method for digital human interaction is shown. Figure 2 This is a flowchart of the adaptive routing digital human interaction method according to Embodiment 1 of this application, as follows: Figure 2 As shown, the method includes the following steps:

[0048] Step S201: Receive input voice from the terminal device and preprocess the input voice to obtain processed voice.

[0049] In this embodiment of the invention, users can input voice into the digital human interaction system through a microphone, mobile terminal, command center terminal, patrol terminal or vehicle-mounted equipment. The voice input can be daily question and answer voice, or business command voice such as data query (such as fire query, etc.), equipment scheduling, plan confirmation, instruction issuance, report generation, etc.

[0050] Optionally, preprocessing may include at least: speech activity detection, wake word detection, AEC (Acoustic Echo Cancellation), and voiceprint verification of the input speech.

[0051] This digital human interaction method is applied to digital human interaction systems. Figure 3 This is a schematic diagram of a full-duplex digital human interaction system with dual-path adaptive routing for an integrated air-space-ground digital intelligence tower, according to an embodiment of this application. Figure 3 As shown, the voice input module is connected to the full-duplex access and preprocessing module to receive natural voice input from users or command personnel. Users can input voice into the system through microphones, mobile terminals, command center terminals, patrol terminals, or vehicle-mounted equipment. Voice input can be everyday question-and-answer voice, or business command voice such as fire situation inquiries, equipment scheduling, plan confirmation, instruction issuance, and report generation. The full-duplex access and preprocessing module is connected to the lightweight intent recognition and business complexity assessment module. The full-duplex access and preprocessing module is used to perform voice detection (i.e., VAD detection, wake word detection), wake word detection, AEC echo cancellation, voiceprint lock verification, and interruptible listening on the input voice. Among them, voice activity detection is used to determine whether there is valid voice in the input audio, wake word detection is used to determine whether an interactive state has been entered, AEC echo cancellation is used to suppress misidentification caused by the digital human's own broadcast voice being re-collected by the microphone, voiceprint lock verification is used to determine whether the current voice comes from an authorized speaker, and interruptible listening is used to continuously receive natural interruptions from users during the digital human's broadcast. The lightweight intent recognition and business complexity assessment module is connected to the dual-path adaptive routing module. This module extracts intent features, business entities, risk levels, tool invocation requirements, and contextual states from the current voice input to determine whether the current interaction is a simple daily chat or a complex business scenario. For example, when the input voice consists only of low-risk phrases such as "Hello," "Who are you?" or "Explain your function," the system classifies it as a simple dialogue. When the input voice contains business entities or action words such as "Tower 3," "Fire situation," "Drone review," "Issue instructions," or "Generate report," the system classifies it as a complex business scenario. The dual-path adaptive routing module is connected to both the end-to-end voice channel module and the multi-stage business channel module. Based on the business complexity assessment results, the dual-path adaptive routing module selects different processing paths. When the interaction is determined to be simple daily chat, low-risk Q&A, or an interaction that does not require external tool invocation, the system selects the end-to-end voice channel module. When the interaction is determined to be complex business such as fire inquiry, fire assessment, drone dispatch, emergency response, tool invocation, permission verification, or report generation, the system selects the multi-stage business channel module.

[0052] like Figure 3As shown, the end-to-end voice channel module is connected to the digital human audio-visual synchronous output module. The end-to-end voice channel module can use a speech-to-speech model or an end-to-end large voice model to directly complete the conversion from voice input to voice output. This channel module reduces the intermediate links between the current ASR (Automatic Speech Recognition), text reasoning and TTS (Text-to-Speech), and is suitable for simple dialogue scenarios to reduce response latency. The multi-stage business channel module connects with the air-space-ground multi-source data module and the digital human audio-visual synchronous output module. The multi-stage business channel module includes an ASR sub-module, an Agent / LLM (i.e., an intelligent agent combining LLM (Large Language Model)) sub-module, a tool invocation / business system sub-module, a structured event object generation sub-module, and a TTS sub-module. The ASR sub-module converts user speech into text or semantic features; the Agent / LLM sub-module performs semantic understanding, intent recognition, business reasoning, and decision generation; the tool invocation / business system sub-module invokes external business data such as the tower video platform, smoke and fire recognition results (which can be detected by a smoke and fire recognition model trained from collected historical fire data), drone data, meteorological data, and resource ledgers; the structured event object generation sub-module encapsulates multi-source data into a unified event object; and the TTS sub-module converts the response results of the Agent / LLM sub-module into speech or acoustic features. The multi-source air-space-ground data module is bidirectionally connected to the tool invocation / business system submodule within the multi-stage business channel module. The multi-source air-space-ground data module provides business data such as high-altitude video from iron towers, smoke and fire identification results, satellite remote sensing hotspots, UAV data, meteorological data, and forest resource ledgers. The tool invocation / business system submodule retrieves relevant data from the multi-source air-space-ground data module based on invocation instructions generated by the Agent / LLM submodule and returns it to the structured event object generation submodule or the Agent / LLM submodule. The digital human audio-visual synchronization output module is connected to the business execution and report generation module. This module synchronously outputs the voice, acoustic features, text, subtitles, event information, and visual information from the end-to-end voice channel or multi-stage business channel into a digital human broadcast screen. The output content includes voice broadcasting, lip-syncing, facial expressions, subtitle display, fire location display, handling process display, and task status display. The business execution and report generation module is connected to the multi-round interaction and status management module. The business execution and report generation module is used to perform fire broadcasting, plan confirmation, instruction issuance, and report generation based on the digital human interaction results. For example, after confirmation by the command personnel, the system can issue a review task to the drone system, push patrol instructions to the forest ranger terminal, synchronize the fire event to the emergency platform, or generate a standardized handling report.The multi-turn interaction and state management module forms a feedback connection with the full-duplex access and preprocessing module. The multi-turn interaction and state management module is used to manage the session state, message ID (Identifier), current channel state, structured event object state, digital human broadcast state, and user interruption state. When the user continues to ask questions or naturally interrupts during the digital human broadcast, this module will send the new voice input back to the full-duplex access and preprocessing module, thereby realizing multi-turn interaction and dynamic channel switching.

[0053] Optionally, the end-to-end voice channel can be implemented using open-source speech-to-speech large models, streaming speech large models, multimodal speech models, or lightweight speech models; the multi-stage business channel can be implemented using ASR+LLM+TTS, ASR+Agent+TTS, ASR+RAG (Retrieval-Augmented Generation) + tool calls + TTS, or speech understanding models + business workflows + TTS, etc. The two channels can be deployed on the same server, or separately on the cloud, edge nodes, or local terminals.

[0054] Step S202: Recognize the processed speech, determine the interaction scenario to which the processed speech belongs, and if the interaction scenario is a chat scenario, input the processed speech into the speech model to generate the first speech response audio data.

[0055] In this embodiment of the invention, a business complexity scoring model can be used to perform a preliminary understanding of the input speech, extracting information such as intent features, business entities, risk levels, tool invocation requirements, and contextual states. Business entities may include tower numbers, monitoring points, fire events, drone numbers, forest area names, response personnel, and material locations. Tool invocation requirements may include querying videos, calling small models (such as smoke and fire recognition models, which can be trained using collected historical fire data), dispatching drones, retrieving resource ledgers, and generating reports.

[0056] Optionally, by constructing a set of question pairs, a business complexity scoring model can be trained based on the constructed set of question pairs. Each question pair in the set of question pairs includes at least: multiple historical audio texts and business complexity scoring description text corresponding to each historical audio text.

[0057] For example, when the input voice is low-risk speech such as "Hello," "Who are you?" or "Tell me about your function," the model can classify it as a simple dialogue (i.e., a chat scenario). When the input voice is a business entity or action word, such as "Tower No. 3," "Fire situation," "Drone review," "Issuing instructions," or "Generating reports," the model can classify it as a complex business scenario.

[0058] In this embodiment of the invention, a dual-path adaptive routing mechanism is constructed, that is, the processed speech is identified through a business complexity scoring model to determine the interaction scenario to which the processed speech belongs. When the interaction scenario is a chat scenario, the processed speech can be directly input into the speech model (e.g., an existing large speech model, which is not limited here), and the speech model generates the first speech response audio data based on the input processed speech.

[0059] Step S203: Based on the first voice response audio data, generate a first synthesized video, and control the digital human to broadcast based on the first synthesized video.

[0060] Optionally, after generating the first voice response audio data, the digital human can be synchronously output as a broadcast screen through the digital human driving model (which can call an open-source model) based on the first voice response audio data, that is, the first synthesized video can be generated, and the digital human can be controlled to broadcast based on the first synthesized video.

[0061] Optionally, the digital human driving model can be any algorithmic model capable of converting acoustic features or audio signals into digital human facial driving parameters (including lip movements, facial expressions, and head posture).

[0062] Step S204: In the case that the interaction scenario is a business scenario, the processed speech is processed to obtain acoustic features.

[0063] In this embodiment of the invention, the business scenario involves complex task processing interactions, such as querying fire information, dispatching drones, and generating reports, which requires calling external systems and generating structured event objects. The processed speech is converted into text or semantic features and input into the Agent / LLM for semantic understanding and business reasoning. The Agent / LLM outputs response semantic features or text features, and then TTS can be used to generate acoustic features based on the response semantic features.

[0064] Step S205: Input the acoustic features into the digital human driving model, output the second synthesized video, and control the digital human to broadcast based on the second synthesized video.

[0065] In this embodiment of the invention, the generated acoustic features can be input into the digital human driving model to generate a second synthetic video, including lip-sync parameters, facial expression parameters, and motion parameters, thereby controlling the digital human to broadcast based on the second synthetic video.

[0066] Figure 4 This is a flowchart of a dual-channel adaptive full-duplex digital human interaction according to an embodiment of this application, such as... Figure 4As shown, the process includes: First, the user / commander inputs natural speech through the terminal microphone, such as, "Tower digital human, check if there is a fire near tower 3," "Dispatch a drone to verify the smoke point just now," "Generate this response report," etc. Through full-duplex access and preprocessing, the system can perform VAD voice detection, wake-word detection, AEC echo cancellation, and voiceprint lock verification on the input speech, while maintaining an interruptible listening state. If no valid speech is detected, the system continues to listen; if valid speech is detected and voiceprint verification is passed, the subsequent recognition process begins. AEC echo cancellation uses the digital human's broadcast audio as a remote reference signal to suppress echoes in the microphone-collected audio, preventing the digital human from speaking to itself. Then, through lightweight intent recognition and business complexity assessment, the system can perform a preliminary understanding of the input speech, extracting intent features, business entities, risk levels, tool call requirements, and contextual states. Business entities may include tower number, monitoring point, fire event, drone number, forest area name, response personnel, and material location, etc. Tool invocation requests can include querying videos, calling small model results, dispatching drones, retrieving resource ledgers, and generating reports. The system determines whether the input is simple daily conversation. If the input is a simple greeting, routine question and answer, or low-risk interaction, it enters the end-to-end voice channel. If the input involves fire events, business system calls, emergency response, permission verification, or report generation, it enters the multi-stage business channel. When the system selects the end-to-end voice channel, the user's voice input directly enters the end-to-end voice model or voice-to-speech model (the current open-source model can be called as needed, without restriction). The model directly generates a voice response, which is sent to the digital human audio-visual synchronization output module to drive digital lip movements, facial expressions, and subtitles. When the system selects the multi-stage business channel, the ASR submodule converts the user's voice into text or intermediate semantic features; the Agent / LLM submodule parses the user's intent and generates task instructions; the tool invocation / business system submodule calls multi-source data from air, space, and ground according to the task instructions, such as high-altitude tower video, smoke and fire recognition results, satellite remote sensing hotspots, drone data, meteorological data, and forest resource ledgers, depending on business needs. For example, when a user queries the fire situation near a certain iron tower, the system can simultaneously retrieve the tower's video, smoke and fire identification results, satellite hotspots, and meteorological data for cross-validation. The structured event object generation submodule generates structured event objects / response plans based on the returned results, while the TTS submodule converts the text into acoustic features or speech. The output results of the end-to-end voice channel or multi-stage business channel are uniformly fed into the digital human audio-visual synchronization output module. This module generates voice broadcasts, lip movements, facial expressions, and subtitles based on acoustic features or speech and a digital human-driven model, and simultaneously displays visual information such as fire location coordinates, risk level, response suggestions, and task status. The system executes voice broadcasts, plan confirmations, command issuance, and report generation based on the interaction results.For example, the digital human can announce the location and risk level of a fire and ask whether to issue a drone verification task. After the user confirms, the system issues the task to the drone platform and writes the execution result to an event object. Through multi-round interaction and dynamic switching, the user can continue the conversation or interrupt the digital human's announcement in real time. The system continuously listens to the user's voice and re-performs intent recognition and business complexity assessment after each round of input, thereby achieving dynamic switching between end-to-end voice channels and multi-stage business channels.

[0067] In summary, by receiving and preprocessing input voice from terminal devices and identifying interactive scenarios, in chat scenarios, the processed voice is input into a voice model to generate first voice response audio data, and based on this data, a first synthesized video is generated to control the digital human's broadcast. In business scenarios, the processed voice is processed to obtain acoustic features, which are then input into the digital human driving model to output a second synthesized video, and the digital human is controlled to broadcast based on the second synthesized video. Through the aforementioned dual-path adaptive routing mechanism, dynamic switching is achieved between the end-to-end voice channel and the multi-stage business channel, realizing low-latency natural interaction for simple dialogues and precise and controllable processing of complex emergency business. This solves the technical problem in related technologies where digital humans lack an adaptive routing mechanism for handling simple dialogues and complex business processes, resulting in an inability to simultaneously achieve low-latency response in interactions and controllability in complex business processes.

[0068] To accurately obtain the processed speech, in the adaptive routing digital human interaction method provided in Embodiment 1 of this application, speech detection is performed on the input speech to obtain a speech detection result; if the speech detection result indicates that the input speech includes a human voice, wake word detection is performed on the input speech to obtain a wake word detection result; if the wake word detection result indicates that the input speech includes a wake word, voiceprint verification is performed on the input speech to obtain a verification result; if the verification result indicates that the input speech passes, acoustic echo cancellation is performed on the input speech to obtain the processed speech.

[0069] In this embodiment of the invention, when the digital human is in standby mode, the VAD algorithm is used to perform speech detection on the input speech to determine whether there is a valid human voice signal in the audio stream, thereby filtering out background noise (such as wind noise, electrical noise, etc.) and saving subsequent processing resources. The wake-up word is a specific phrase used by the user to initiate interaction (e.g., "Digital Human #1"). Its purpose is to distinguish between unintentional speech and intentional interaction requests, ensuring that only specific wake-up commands are responded to, and avoiding accidental activation due to others speaking or the user talking to themselves in a non-interactive state.

[0070] Voiceprint verification extracts the speaker's voiceprint features and compares them with pre-stored voiceprint templates of authorized users. This prevents unauthorized personnel from misoperating or maliciously interfering, ensuring that only authorized personnel can initiate business operations (such as dispatching drones or issuing commands). Acoustic echo cancellation subtracts the echo of the digital human's own voice from the mixed signal captured by the microphone. For example, if the digital human is broadcasting: "Suspected smoke point detected," and the microphone simultaneously captures the commander saying "Generate report," acoustic echo cancellation can prevent the digital human from hearing its own words and being misidentified as a new user command. Through a four-layer filtering mechanism of VAD detection, wake-word detection, voiceprint verification, and AEC echo cancellation, the problems of false triggering, unidentified individuals, and echo interference in the complex environment of the integrated air-space-ground digital intelligence tower are solved.

[0071] Optionally, voiceprint verification can be replaced by account login, password verification, face recognition, employee ID card recognition, device binding, role and permission table, or multi-factor authentication mechanism; AEC echo cancellation can be implemented using DSP (Digital Signal Processing) algorithms, deep learning echo cancellation models, beamforming, noise suppression, or sound source separation methods; the wake-up method can also be replaced by a fixed wake-up word or button wake-up, gesture wake-up, automatic wake-up from event alarms, or authorized wake-up from the command platform.

[0072] To improve resource utilization efficiency, it is necessary to determine the interaction scenario to which the processed voice belongs. In the adaptive routing digital human interaction method provided in Embodiment 1 of this application, a business complexity scoring model is used to calculate the business complexity score of the processed voice. The business complexity scoring model is obtained by training a set of question pairs. Each question pair in the question pair set includes at least: multiple historical audio texts and a business complexity score description text corresponding to each historical audio text. If the business complexity score is greater than or equal to a preset scoring threshold, the interaction scenario to which the processed voice belongs is determined to be a business scenario. If the business complexity score is less than the preset scoring threshold, the interaction scenario to which the processed voice belongs is determined to be a chat scenario.

[0073] In this embodiment of the invention, the business complexity scoring model is obtained by training a language model with a small number of parameters (which can be selected according to the actual situation) based on a set of question pairs. The question pair consists of multiple historical audio texts (i.e., text data after transcription of the user's historical speech) and business complexity scoring description text corresponding to each historical audio text (i.e., labels or descriptions generated by experts or the system that reflect the complexity of the text). The purpose is to enable the model to learn the mapping relationship from text features (such as the number of entities and verb types) to complexity scores.

[0074] The preset scoring threshold is a pre-defined numerical limit used to delineate the boundary between simple dialogues and complex business processes. If the business complexity score is greater than or equal to the preset scoring threshold, the interaction scenario to which the voice processing belongs is determined to be a business scenario. If the business complexity score is less than the preset scoring threshold, the interaction scenario to which the voice processing belongs is determined to be a chat scenario. This chat scenario does not involve external system calls and does not require the generation of structured data.

[0075] Figure 5 This is a flowchart of a dual-path adaptive routing determination according to an embodiment of this application, such as... Figure 5As shown, the judgment process includes: Step S501, receiving the current round of voice input. The system obtains the valid voice input of the current round from the full-duplex access and preprocessing module. Step S502, extracting interaction features. The system extracts intent features, business entities, risk levels, tool call requirements, and context states from the current voice or its transcription results. Intent features are used to determine whether the user is chatting, querying, confirming, scheduling, or generating a report; business entities are used to identify tower numbers, fire locations, drone numbers, emergency resources, etc.; risk levels are used to determine whether high-risk emergency operations are involved; tool call requirements are used to determine whether access to external systems is required; context states are used to determine whether the current situation is in the process of handling an event. Step S503, determining whether structured event objects are involved. If the user input involves fire events, monitoring points, drone missions, handling resources, or report generation, the system determines that it involves structured event objects. If it is only a regular greeting or simple chat, it can be determined that it does not involve structured event objects. Step S504, calculating the business complexity score. The system can calculate a business complexity score based on factors such as the number of business entities, whether an external system is required, whether it is in an event handling context, whether it is a high-risk operation, and whether structured results need to be generated (this score can also be obtained by training a business complexity scoring model). The higher the score, the more likely it is to select a multi-stage business channel. Step S505: Determine if it is a simple daily chat. If the score is below a preset threshold and does not involve structured event objects, does not require calling external systems, and is not a high-risk operation, the system determines it to be a simple daily chat; otherwise, it is determined to be a complex business. Steps S506 to S507: When determined to be a simple daily chat, the system selects an end-to-end voice channel, outputs the voice result, and drives the digital human. Steps S508 to S510: When determined to be a complex business, the system selects a multi-stage business channel, executes ASR, Agent / LLM, tool calls, and TTS, outputs the business result, and drives the digital human. Step S511: Update the session context and event state. Regardless of which channel is selected, the system updates the session context after output; if structured event objects are involved, the event state can also be updated. Step S512: Determine whether a channel switch is needed in the next round. After the next round of input arrives, the system re-determines whether a channel switch is needed. If a switch is needed, the interaction features are re-extracted; if no switch is needed, step S513 is executed to maintain the current channel and continue interaction.

[0076] Optionally, dual-path adaptive routing can also be implemented through rule engines, keyword matching, intent classification models, lightweight large models, risk scoring models, or a combination of these methods. Business complexity scoring can also be calculated by weighting factors such as the number of business entities, whether external system calls are involved, whether the system is in an event handling state, whether permission verification is required, and whether structured reports need to be generated. Alternatively, a trained classification model can directly output the selection results for end-to-end channels or multi-stage business channels.

[0077] In order to accurately generate acoustic features, in the adaptive routing digital human interaction method provided in Embodiment 1 of this application, the processed speech is encoded to obtain speech encoding features, and the speech encoding features are input into a language model to generate a calling instruction. Based on the calling instruction, the multi-source data corresponding to the processed speech is called; based on the multi-source data, an event object is generated; and based on the event object, acoustic features are generated.

[0078] In this embodiment of the invention, the speech coding feature is a high-dimensional vector representation of the processed speech, which is converted into semantic, phonemic, and prosodic information by an encoder, rather than the original PCM (Pulse Code Modulation Audio) audio or intermediate text. This preserves the fine-grained features of the speech and allows the speech coding feature to be input into a language model (e.g., an agent or LLM with semantic understanding and reasoning capabilities) to generate a calling instruction, avoiding the redundant process of first converting audio to text and then into model input. For example, if the processed speech is to retrieve the video of Tower 3, the encoder converts it into a speech coding feature containing Tower 3, the video, and the semantic meaning of retrieval. After this feature is input into the language model, the model understands the intent and generates a calling instruction: Retrieve the Tower video, with the parameter Tower ID=3.

[0079] Based on the invocation command, the system calls and processes multi-source data corresponding to the voice (including but not limited to: high-altitude video of the iron tower, results of the small model for firework recognition, satellite remote sensing hotspots, drone data, meteorological data, and forest resource ledgers, etc.). According to the command generated by the language model, specific business data related to the user's intent can be obtained in real time. Then, based on the multi-source data, event objects are generated. The event objects are structured data encapsulated by cleaning, spatiotemporally aligning, and cross-validating the scattered multi-source data (such as video frames, confidence values, and meteorological parameters). For example, {Event ID: "E001", Iron Tower ID: "3", Coordinates: "(x,y)", Confidence: 0.95, Wind Direction: "Southeast", Status: "Pending Review"}.

[0080] Acoustic features are intermediate acoustic representations (such as Mel spectrum, phoneme duration, prosodic parameters, etc.) generated from the business response content contained in the event object, rather than the final PCM audio file. This enables feature-level integration and directly provides high-precision driving data for digital human driving models, avoiding the delay and accuracy loss of re-extracting features from PCM audio, while preserving complete prosodic and emotional information.

[0081] Figure 6 This is a schematic diagram of a feature-level through-link of a multi-stage service channel according to an embodiment of this application, such as... Figure 6 As shown, current multi-stage links are typically (e.g.) Figure 6 The upper part of the block diagram shows: audio input enters ASR, outputting text; text enters LLM / Agent, outputting text; text enters TTS, outputting PCM audio; then audio features are extracted from the PCM audio to generate lip movements, driving the data human to broadcast. This link suffers from problems such as repetitive conversions, frequent copying between the CPU and GPU, loss of prosodic information, and high latency. Based on this, the embodiments of the present invention construct a feature-level interconnected link (such as... Figure 6 As shown in the lower half of the block diagram, this link, while maintaining modularity and controllability, prioritizes the transmission of unified intermediate feature representations between adjacent modules, rather than frequently transmitting discrete text, complete PCM audio, or file-based audio. Specifically, user audio input is first processed by the speech encoding module to generate speech encoding features. These speech encoding features can include one or more of the following: semantic features, phoneme features, prosodic features, speaker features, emotion features, and timestamp features. Mapping the speech encoding features to embedding vectors that can be accepted by a large language model can be achieved through a projection layer (e.g., a linear transformation), which is then used by the Agent / LLM for semantic understanding and business reasoning, outputting response semantic features or text features. This output can be used to simultaneously generate text identifiers for caption display, log recording, and business auditing, or it can serve as input conditional features for the TTS submodule. Based on the response semantic features, the TTS submodule generates acoustic features, which can include one or more of the following: Mel spectrum features, audio codec tokens (discrete symbols generated based on the semantic encoder-decoder model), audio latent space features, phoneme duration features, and prosodic control features. Subsequently, the digital human-driven model directly reads the acoustic features generated by the TTS submodule to generate lip-sync parameters, facial expression parameters, and motion parameters, without needing to extract lip-sync features again from the synthesized PCM audio. This feature-level integration mechanism reduces redundant processing of audio-to-text, text-to-audio, and audio-to-lip-sync features, preserving prosody, emotion, and temporal information in speech, and improving the digital human's response speed, lip-sync stability, and regeneration efficiency after full-duplex interruptions.

[0082] In one implementation, feature tensors can be transferred between ASR, Agent / LLM, TTS, and digital human driven models via a shared feature cache in GPU memory to reduce data copying and serialization overhead between the CPU and GPU.

[0083] Optionally, feature-level transmission in the multi-stage business channel module can be implemented using different intermediate representations such as speech coding features, text embedding, LLM hidden states, audio codec tokens, EL spectrum, and digital human driving parameters; the sharing method can be GPU memory caching, shared memory, message queues, streaming tensor interfaces, or inter-service RPC interfaces. The digital human output is not limited to 2D digital humans; it can be replaced with 3D digital humans, cartoon digital humans, image digital humans, video digital humans, virtual anchors, robot screen avatars, or voice assistants with only subtitles and visual panels.

[0084] To improve the accuracy of event objects, in the adaptive routing digital human interaction method provided in Embodiment 1 of this application, multi-source data is preprocessed to obtain multiple structured data; based on preset event types, the multiple structured data are associated to obtain associated data; and the associated data is encapsulated to obtain event objects.

[0085] In this embodiment of the invention, preprocessing includes operations such as data cleaning, format standardization, and timestamp alignment. For example, the time format, coordinate format, device number, alarm type, and confidence level format of output data from different systems are unified into a recognizable data structure. Furthermore, spatiotemporal alignment and event association can be performed based on preset event types. For instance, data from different sources can be associated with the same suspected fire event (i.e., a preset event type) based on timestamps, geographical coordinates, tower numbers, monitoring points, drone inspection areas, and satellite hotspot locations.

[0086] Optionally, multi-source cross-validation can also be performed on the associated data. For example, when the smoke and fire identification model outputs a suspected smoke point, the system can combine satellite remote sensing hotspots, meteorological conditions, drone verification results, and nearby tower videos for verification to reduce the error rate of multi-source data.

[0087] Next, the associated data is encapsulated in a structured manner to generate a structured event object. This object may include fields such as: event number, tower number, fire point coordinates, smoke and fire confidence level, satellite hotspot matching results, drone verification status, wind speed and direction, surrounding roads, water sources, personnel, materials, and current handling status.

[0088] Optionally, the generated structured event object can be invoked in multi-turn dialogues. For example, if the user later asks, "How did that smoke point go?", the system can link the current event object to the previous fire event without relying on the user's complete description again.

[0089] Optionally, multi-source data from air, space, and ground sources is not limited to tower video, smoke and fire identification, satellite hotspots, drones, meteorological data, and resource ledgers. It can also be replaced or expanded to data sources such as radar, infrared thermal imaging, IoT sensors, patrol personnel reports, historical event databases, road traffic data, and emergency material systems. The fields of structured event objects can also be replaced with disaster event objects, inspection event objects, hidden danger event objects, or equipment alarm event objects, depending on the scenario, for use in scenarios such as forest fire prevention, straw burning ban, low-altitude inspection, and urban governance.

[0090] Figure 7 This is a flowchart illustrating the fusion of air, space, and ground multi-source data and the generation of structured event objects according to an embodiment of this application, such as... Figure 7 As shown, the multi-source data input includes high-altitude video of iron towers, smoke and fire identification results, satellite remote sensing hotspots, drone data for filling gaps, meteorological data, and forest resource ledgers. First, the above data can be cleaned and standardized. For example, the time format, coordinate format, device number, alarm type, and confidence level format output from different data source systems can be unified into a data structure that the system can recognize. Then, spatiotemporal alignment and event association are performed. The system associates data from different sources with the same suspected fire event based on timestamps, geographic coordinates, iron tower numbers, monitoring points, drone inspection areas, and satellite hotspot locations. Next, multi-source cross-validation is performed. For example, when the smoke and fire identification mini-model outputs a suspected smoke point, the system can combine satellite remote sensing hotspots, meteorological conditions, drone verification results, and nearby iron tower video for verification to reduce false alarms and missed alarms. Finally, structured encapsulation is performed to generate a structured fire event object, which includes at least the following fields: event number, iron tower number, fire point coordinates, smoke and fire confidence level, satellite hotspot matching result, drone verification status, wind speed and direction, surrounding roads, water sources, personnel, materials, and current handling status. After the structured fire event object is generated, it can be broadcast by a digital human, or input into the response plan generation module to generate response plan suggestions through a pre-trained model. It can also be used by the instruction issuance and report generation module.

[0091] To improve the accuracy of generated acoustic features, the adaptive routing digital human interaction method provided in Embodiment 1 of this application generates a second voice response text based on the event object and processed speech; features of the second voice response text are extracted, and acoustic features are generated based on these features. These acoustic features can be directly passed to the digital human driving model to drive the digital human's lip movements, facial expressions, and head movements, generating natural and smooth broadcast videos without first synthesizing a complete audio file, thus reducing the redundant overhead of audio encoding and decoding.

[0092] In this embodiment of the invention, the user's processed speech is first encoded to obtain speech encoding features. These features may include one or more of semantic features, phoneme features, prosodic features, speaker features, emotion features, and timestamp features. Then, the speech encoding features and event objects are input to an Agent / LLM for semantic understanding and business reasoning, outputting response semantic features or text features. This output can be used to simultaneously generate text identifiers for subtitle display, log recording, and business auditing, or it can serve as input condition features for TTS. Based on the response semantic features, TTS generates acoustic features, which may include one or more of the following: Mel-spectral features, audio codec tokens, audio latent space features, phoneme duration features, and prosodic control features.

[0093] To accurately regenerate the synthesized video, in the adaptive routing digital human interaction method provided in Embodiment 1 of this application, the interrupted speech of the terminal device is monitored, and it is determined whether the interrupted speech and the input speech belong to the same user; if the interrupted speech and the input speech belong to the same user, it is determined whether the interrupted speech and the input speech are the same; if the interrupted speech and the input speech are different, the digital human is controlled to stop broadcasting; the interrupted speech is identified to determine the interaction scene to which the interrupted speech belongs, and the synthesized video is regenerated based on the interaction scene.

[0094] In this embodiment of the invention, when the digital human is in a broadcasting state, that is, when the digital human is outputting speech, lip movements, subtitles, and visual information, it can continuously monitor the user's natural speech (i.e., interrupted speech) on the terminal device and determine whether the interrupted speech and the original input speech belong to the same user. If the interrupted speech and the input speech belong to the same user, it determines whether the interrupted speech and the input speech are the same. If the interrupted speech and the input speech are different, it controls the digital human to stop broadcasting, that is, the speech at this time is new speech, controls the digital human to stop broadcasting, stops the current TTS output, clears the audio queue, video frame queue, and lip movement driver buffer to prevent old speech segments, old video frames, and old lip movement states from continuing to be output, re-identifies the interrupted speech, determines the interaction scenario to which the interrupted speech belongs, and regenerates the synthesized video based on the interaction scenario, thus realizing full-duplex interaction of broadcasting, monitoring, understanding, and automatic interruption.

[0095] Figure 8 This is a flowchart of a full-duplex interruption and state reset according to an embodiment of this application, such as... Figure 8As shown, the process includes: Step S801, the digital human is currently in broadcasting mode. The digital human is outputting speech, lip movements, subtitles, and visual information. Step S802, continuously listening to the user's natural speech. The system does not turn off the microphone during the digital human's broadcasting, but continuously receives the user's speech and reduces backflow of the digital human's broadcasting sound through AEC echo cancellation. Step S803, determining whether a valid interruption has been detected. The system determines whether a valid user interruption has been detected through VAD and voiceprint verification. If no valid interruption is detected, the broadcasting continues; if a valid interruption is detected, proceed to Step S804, determining whether the interruption is a valid new instruction (i.e., determining whether the interrupted speech and the input speech belong to the same user), and obtaining the determination result. The system performs preliminary identification of the interrupted speech, determining whether it is related to business instruction changes, follow-up questions, pause requests, confirmation requests, or event status updates. Step S805, based on the determination result, determining whether to take over the current round. If the judgment result indicates that the interruption is an invalid duplicate instruction, the system ignores the invalid interruption and continues listening; if the judgment result indicates that the interruption is a valid new instruction, the system takes over the current round and proceeds to step S806. Steps S806 to S810 can execute broadcast stop and queue clearing: the system stops the current digital human broadcast, stops the current TTS output, clears the audio queue, clears the video frame queue, and clears the lip-sync driver buffer to prevent old voice segments, old video frames, and old lip-sync states from continuing to be output. Step S811, update message ID / session state. The system generates a new message ID for the new user input and updates the current session state to avoid crosstalk between old and new responses. Step S812, send the new input to the dual-path adaptive routing module. The system sends the user interruption content as new input to the dual-path adaptive routing judgment process. Step S813, regenerate the response based on the latest input. The system reselects the end-to-end voice channel or multi-stage service channel based on the new input and the current event context, and generates a new response. Step S814, the digital human resumes broadcasting. The digital human re-outputs speech, lip movements, subtitles, and visuals based on the new response content. In other words, the above steps reset the content, including old speech segments, old video frames, old lip movement states, and old message states. This process differs from solutions that only interrupt broadcasting via buttons or fixed wake words. It enables automatic takeover of natural interruptions during the digital human's broadcasting process and simultaneously clears old audio and video states, ensuring consistency throughout the full-duplex interaction.

[0096] The adaptive routing digital human interaction method provided in this application can obtain reliable processed speech by performing speech detection, wake-word detection, voiceprint verification, and acoustic echo cancellation on the input speech. A business complexity scoring model is used to calculate a score, distinguishing between chat scenarios and business scenarios. In the business scenario, the processed speech is encoded into speech coding features and input into a language model to generate calling instructions. Multi-source data is called and preprocessed into structured data. Related data is obtained based on preset event types and encapsulated to generate event objects. A second voice reply text is generated based on the event object and the processed speech, its features are extracted, and acoustic features are generated. During digital human broadcasting, interrupting voices are monitored to determine if they belong to the same user and have different instructions. If so, broadcasting stops, the interrupting voice is re-identified, the interaction scenario is determined, and a new synthesized video is generated. This method achieves dynamic switching between end-to-end and multi-stage channels based on intent, balancing low latency and controllability of interaction. Voiceprint locking, AEC, and interruption reset mechanisms improve interaction security and state consistency in complex environments. Feature-level continuous transmission reduces repeated encoding and decoding, improving the real-time performance and reliability of digital human interaction.

[0097] The following describes in detail another optional implementation method.

[0098] In this embodiment, we assume the following application scenario: a forest fire emergency command scenario, where the commander says, "Tower Digital Human, check if there is a fire near Tower 3." After receiving the voice through the voice input module, the system enters the full-duplex access and preprocessing module. The system performs VAD detection, wake-up word detection, AEC echo cancellation, and voiceprint lock verification to confirm that the voice input is from an authorized commander.

[0099] Subsequently, the lightweight intent recognition and business complexity assessment module identified business entities such as "Tower No. 3" and "fire", and determined that the input involved structured event objects and external business system calls. Therefore, the dual-path adaptive routing module selected the multi-stage business channel.

[0100] In the multi-stage business process, the ASR submodule converts speech into text or semantic features; the Agent / LLM submodule interprets the user intent as "querying about fire conditions near Tower 3"; the tool invocation / business system submodule invokes high-altitude video of the tower, a small model for smoke and fire identification, satellite remote sensing hotspots, and meteorological data; the structured event object generation submodule cleans, aligns, cross-validates, and encapsulates the returned data in a structured manner to generate a fire event object. This object includes fields such as tower number, coordinates of suspected fire points, smoke and fire confidence level, satellite hotspot matching results, wind speed and direction, nearby roads, and available drones.

[0101] Subsequently, the system generates broadcast voice or acoustic features through the TTS submodule, and the digital human audio-visual synchronization output module generates lip movements, facial expressions, subtitles, and visual information about the fire location based on these voice or acoustic features. Digital human broadcast: Suspected smoke point detected southeast of Tower 3. The confidence level for smoke and fire identification is high; it is recommended to use a drone for verification.

[0102] During the digital human's broadcast, the commanding officer naturally interjected: "Don't issue any orders yet; send a drone to verify first." The system continuously monitored the user's voice and confirmed through voiceprint verification that the voice belonged to an authorized commanding officer. The system further determined that the interjection was a valid new instruction, thus triggering a full-duplex interruption and content reset process: stopping the current broadcast, stopping the current TTS output, clearing the old audio queue, old video frame queue, and lip-sync driver cache, updating the message ID and session state, and sending the new input back to the dual-path adaptive routing module.

[0103] The system reselects the multi-stage business channel. The Agent / LLM submodule parses the new instruction to generate a drone verification task. The tool call / business system submodule calls the drone system to generate the verification task and update the structured fire event object. The digital human then rebroadcasts: "Drone verification task has been generated. The target is a suspected smoke point southeast of Tower 3. Please confirm whether to execute."

[0104] If the subsequent command personnel say, "Okay, that's a quick response," the system reassesses the business complexity, determines that the input is a simple daily chat, and does not involve external system calls or structured event updates. Therefore, it switches to an end-to-end voice channel, quickly generates a natural response, and drives the digital human to complete the output of speech, lip movements, and subtitles.

[0105] In this embodiment, low-latency natural interaction can be provided in simple chat scenarios, and feature-level data transmission can be provided in complex emergency business scenarios. This reduces the overhead of data copying and serialization between the CPU and GPU, improves the response speed and lip-sync stability of the digital human, realizes low-latency response in simple dialogues and controllable processing of complex business, and supports natural interruption, automatic interruption, state reset and event context update during the digital human's broadcast, realizing full-duplex interaction of broadcasting, listening, understanding and automatic interruption at the same time.

[0106] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0107] Example 2

[0108] This application also provides an adaptive routing digital human interaction device. It should be noted that the adaptive routing digital human interaction device of this application can be used to execute the adaptive routing digital human interaction method provided in this application. The adaptive routing digital human interaction device provided in this application will be described below.

[0109] According to an embodiment of this application, an apparatus for implementing the above-described adaptive routing digital human interaction method is also provided. Figure 9 This is a schematic diagram of an adaptive routing digital human interaction device according to an embodiment of this application, such as... Figure 9 As shown, the adaptive routing digital human interaction device may include: a first processing unit 90, a first recognition unit 91, a first generation unit 92, a second processing unit 93, and a third processing unit 94.

[0110] The first processing unit 90 is used to receive the input voice from the terminal device and preprocess the input voice to obtain processed voice.

[0111] The first recognition unit 91 is used to recognize the processed speech, determine the interaction scenario to which the processed speech belongs, and input the processed speech into the speech model to generate the first speech response audio data when the interaction scenario is a chat scenario.

[0112] The first generation unit 92 is used to generate a first synthesized video based on the first voice response audio data, and to control the digital human to broadcast based on the first synthesized video;

[0113] The second processing unit 93 is used to process the processed speech to obtain acoustic features when the interaction scenario is a business scenario.

[0114] The third processing unit 94 is used to input acoustic features into the digital human driving model, output the second synthesized video, and control the digital human to broadcast based on the second synthesized video.

[0115] The adaptive routing digital human interaction device provided in this application embodiment can receive input voice from a terminal device through a first processing unit 90, preprocess the input voice to obtain processed voice, recognize the processed voice through a first recognition unit 91 to determine the interaction scenario to which the processed voice belongs, and input the processed voice into a voice model to generate first voice response audio data when the interaction scenario is a chat scenario, and generate a first synthesized video based on the first voice response audio data through a first generation unit 92, and control the digital human to broadcast based on the first synthesized video, and process the processed voice through a second processing unit 93 when the interaction scenario is a business scenario to obtain acoustic features, and input the acoustic features into a digital human driving model through a third processing unit 94 to output a second synthesized video, and control the digital human to broadcast based on the second synthesized video.

[0116] Optionally, the first processing unit 90 includes: a first detection module, used to perform speech detection on the input speech and obtain a speech detection result; a second detection module, used to perform wake-up word detection on the input speech when the speech detection result indicates that the input speech includes a human voice, and obtain a wake-up word detection result; a first verification module, used to perform voiceprint verification on the input speech when the wake-up word detection result indicates that the input speech includes a wake-up word, and obtain a verification result; and a first cancellation module, used to perform acoustic echo cancellation on the input speech when the verification result indicates that the input speech passes, and obtain processed speech.

[0117] Optionally, the first recognition unit 91 includes: a first calculation module, used to calculate a business complexity score for processing speech using a business complexity scoring model, wherein the business complexity scoring model is trained through a set of question pairs, and each question pair in the set of question pairs includes at least: multiple historical audio texts and a business complexity score description text corresponding to each historical audio text; a first determination module, used to determine that the interaction scenario to which the processed speech belongs is a business scenario when the business complexity score is greater than or equal to a preset scoring threshold; and a second determination module, used to determine that the interaction scenario to which the processed speech belongs is a chat scenario when the business complexity score is less than the preset scoring threshold.

[0118] Optionally, the second processing unit 93 includes: a first encoding module, used to encode the processed speech to obtain speech encoding features, input the speech encoding features into a language model, generate a calling instruction, and call the multi-source data corresponding to the processed speech based on the calling instruction; a first generation module, used to generate an event object based on the multi-source data; and a second generation module, used to generate acoustic features based on the event object.

[0119] Optionally, the first generation module includes: a first preprocessing submodule for preprocessing multi-source data to obtain multiple structured data; a first association submodule for associating multiple structured data based on preset event types to obtain associated data; and a first encapsulation submodule for encapsulating the associated data to obtain event objects.

[0120] Optionally, the second generation module includes: a first generation submodule for generating a second voice response text based on the event object and processed speech; and a second generation submodule for extracting features from the second voice response text and generating acoustic features based on the features.

[0121] Optionally, the adaptive routing digital human interaction device further includes: a first monitoring module, used to monitor the interrupted speech of the terminal device after controlling the digital human to broadcast based on the second synthesized video, and determine whether the interrupted speech and the input speech belong to the same user; a first judgment module, used to determine whether the interrupted speech and the input speech are the same if they belong to the same user; a first control module, used to control the digital human to stop broadcasting if the interrupted speech and the input speech are different; and a first recognition module, used to recognize the interrupted speech, determine the interaction scenario to which the interrupted speech belongs, and regenerate the synthesized video based on the interaction scenario.

[0122] The aforementioned adaptive routing digital human interaction device may also include a processor and a memory. The first processing unit 90, the first recognition unit 91, the first generation unit 92, the second processing unit 93, the third processing unit 94, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0123] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, acoustic features are input into the digital human driving model, a second synthesized video is output, and the digital human is controlled to broadcast based on the second synthesized video.

[0124] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0125] It should be noted that the first processing unit 90, the first identification unit 91, the first generation unit 92, the second processing unit 93, and the third processing unit 94 mentioned above correspond to steps S201 to S205 in Embodiment 1. The instances and application scenarios implemented by the above units and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0126] Example 3

[0127] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced with a mobile terminal or an electronic device, etc.

[0128] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0129] In this embodiment, the computer terminal can execute the following steps of the adaptive routing digital human interaction method: receiving input voice from the terminal device and preprocessing the input voice to obtain processed voice; recognizing the processed voice to determine the interaction scenario to which the processed voice belongs; if the interaction scenario is a chat scenario, inputting the processed voice into a voice model to generate first voice response audio data; generating a first synthesized video based on the first voice response audio data, and controlling the digital human to broadcast based on the first synthesized video; if the interaction scenario is a business scenario, processing the processed voice to obtain acoustic features; inputting the acoustic features into a digital human driving model to output a second synthesized video, and controlling the digital human to broadcast based on the second synthesized video.

[0130] Optionally, the computer terminal described above can execute the program code for the following steps in the adaptive routing digital human interaction method: performing speech detection on the input speech to obtain a speech detection result; if the speech detection result indicates that the input speech includes a human voice, performing wake word detection on the input speech to obtain a wake word detection result; if the wake word detection result indicates that the input speech includes a wake word, performing voiceprint verification on the input speech to obtain a verification result; if the verification result indicates that the input speech passes, performing acoustic echo cancellation on the input speech to obtain processed speech.

[0131] Optionally, the computer terminal described above can execute the following steps in the adaptive routing digital human interaction method: using a business complexity scoring model to calculate the business complexity score of the processed speech, wherein the business complexity scoring model is trained through a set of question pairs, and each question pair in the set includes at least: multiple historical audio texts and a business complexity score description text corresponding to each historical audio text; if the business complexity score is greater than or equal to a preset scoring threshold, determining that the interaction scenario to which the processed speech belongs is a business scenario; if the business complexity score is less than the preset scoring threshold, determining that the interaction scenario to which the processed speech belongs is a chat scenario.

[0132] Optionally, the computer terminal described above can execute the following steps of the adaptive routing digital human interaction method: encoding the processed speech to obtain speech encoding features, inputting the speech encoding features into a language model, generating a calling instruction, and calling the multi-source data corresponding to the processed speech based on the calling instruction; generating an event object based on the multi-source data; and generating acoustic features based on the event object.

[0133] Optionally, the computer terminal described above can execute the program code for the following steps in the adaptive routing digital human interaction method: preprocessing multi-source data to obtain multiple structured data; associating the multiple structured data based on preset event types to obtain associated data; and encapsulating the associated data to obtain event objects.

[0134] Optionally, the computer terminal described above can execute the program code for the following steps in the adaptive routing digital human interaction method: generating a second voice response text based on the event object and processed speech; extracting features from the second voice response text, and generating acoustic features based on the features.

[0135] Optionally, the computer terminal can execute the following steps of the adaptive routing digital human interaction method: listen to the interrupted voice of the terminal device and determine whether the interrupted voice and the input voice belong to the same user; if the interrupted voice and the input voice belong to the same user, determine whether the interrupted voice and the input voice are the same; if the interrupted voice and the input voice are different, control the digital human to stop broadcasting; recognize the interrupted voice, determine the interaction scene to which the interrupted voice belongs, and regenerate the synthesized video based on the interaction scene.

[0136] Optionally, Figure 10 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 10 As shown, the electronic device may include: one or more ( Figure 10(Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0137] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the adaptive routing digital human interaction method and device in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned adaptive routing digital human interaction method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0138] The processor can access the information and application programs stored in the memory via the transmission device to execute the steps described above in the adaptive routing digital human interaction method.

[0139] This application provides an adaptive routing solution for digital human interaction. By constructing a dual-path interaction architecture that combines an end-to-end voice channel with a multi-stage business channel, and dynamically selecting the processing path based on user intent, business complexity, risk level, and tool call requirements, it can automatically switch to the low-latency end-to-end channel in simple daily conversation scenarios, and automatically switch to the controllable multi-stage business channel in complex business scenarios such as fire inquiries and command issuance. This achieves dual-modal adaptive interaction that balances interaction response speed and business processing controllability, thereby solving the technical problem in related technologies where digital humans lack an adaptive routing mechanism for handling simple conversations and complex business processes, resulting in the inability to simultaneously achieve low-latency response and controllability of complex business processes.

[0140] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only. Electronic devices can also be terminal devices such as smartphones, tablets, PDAs, and mobile internet devices (MIDs). Figure 10 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 10 The different configurations shown.

[0141] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0142] Example 4

[0143] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the adaptive routing digital human interaction method provided in Embodiment 1.

[0144] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0145] This application also provides a computer program product, which, when executed on a data processing device, is a program adapted to perform steps of an adaptive routing digital human interaction method.

[0146] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0152] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A digital human interaction method with adaptive routing, characterized in that, Applications in digital human interaction systems include: Receive input voice from terminal device and preprocess the input voice to obtain processed voice; The processed speech is recognized to determine the interaction scenario to which it belongs. If the interaction scenario is a chat scenario, the processed speech is input into the speech model to generate first speech response audio data. Based on the first voice response audio data, a first synthesized video is generated, and the digital human is controlled to broadcast based on the first synthesized video; When the interaction scenario is a business scenario, the processed speech is processed to obtain acoustic features; The acoustic features are input into the digital human driving model, a second synthesized video is output, and the digital human is controlled to broadcast based on the second synthesized video.

2. The adaptive routing digital human interaction method according to claim 1, characterized in that, The steps of preprocessing the input speech to obtain processed speech include: The input speech is subjected to speech detection to obtain the speech detection result; If the speech detection result indicates that the input speech includes a human voice, a wake word detection is performed on the input speech to obtain a wake word detection result; If the wake-up word detection result indicates that the input speech includes a wake-up word, the input speech is subjected to voiceprint verification to obtain the verification result; If the verification result indicates that the input speech passes, acoustic echo cancellation is performed on the input speech to obtain the processed speech.

3. The adaptive routing digital human interaction method according to claim 1, characterized in that, The step of recognizing the processed speech and determining the interaction scenario to which the processed speech belongs includes: A business complexity scoring model is used to calculate the business complexity score of the processed speech. The business complexity scoring model is trained through a set of question pairs. Each question pair in the set of question pairs includes at least: multiple historical audio texts and a business complexity score description text corresponding to each historical audio text. If the business complexity score is greater than or equal to a preset score threshold, the interaction scenario to which the processed voice belongs is determined to be the business scenario. If the business complexity score is less than the preset score threshold, the interaction scenario to which the processed voice belongs is determined to be the chat scenario.

4. The adaptive routing digital human interaction method according to claim 1, characterized in that, The step of processing the spoken language to obtain acoustic features includes: The processed speech is encoded to obtain speech encoding features, and the speech encoding features are input into a language model to generate a calling instruction. Based on the calling instruction, the multi-source data corresponding to the processed speech is called. Based on the multi-source data, an event object is generated; The acoustic features are generated based on the event object.

5. The adaptive routing digital human interaction method according to claim 4, characterized in that, The steps for generating an event object based on the multi-source data include: The multi-source data is preprocessed to obtain multiple structured data sets; Based on preset event types, the multiple structured data are associated to obtain associated data; The associated data is encapsulated to obtain the event object.

6. The adaptive routing digital human interaction method according to claim 4, characterized in that, The step of generating the acoustic features based on the event object includes: Based on the event object and the processed voice, a second voice response text is generated; Extract features from the second voice response text, and generate the acoustic features based on the features.

7. The adaptive routing digital human interaction method according to claim 1, characterized in that, After controlling the digital human to broadcast based on the second synthesized video, the process includes: Listen to the interrupted voice on the terminal device and determine whether the interrupted voice and the input voice belong to the same user; If the interrupted voice and the input voice belong to the same user, determine whether the interrupted voice and the input voice are the same; If the interrupted speech is different from the input speech, control the digital human to stop broadcasting; The interrupted speech is identified to determine the interaction scenario to which it belongs, and a synthesized video is regenerated based on the interaction scenario.

8. A digital human interaction device with adaptive routing, characterized in that, Applications in digital human interaction systems include: The first processing unit is used to receive the input voice from the terminal device and preprocess the input voice to obtain processed voice. The first recognition unit is used to recognize the processed speech, determine the interaction scenario to which the processed speech belongs, and, if the interaction scenario is a chat scenario, input the processed speech into the speech model to generate first speech response audio data. The first generation unit is used to generate a first synthesized video based on the first voice response audio data, and to control the digital human to broadcast based on the first synthesized video; The second processing unit is used to process the processed speech to obtain acoustic features when the interaction scenario is a business scenario. The third processing unit is used to input the acoustic features into the digital human driving model, output the second synthesized video, and control the digital human to broadcast based on the second synthesized video.

9. A computer program product, characterized in that, The method includes a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the adaptive routing digital human interaction method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the adaptive routing digital human interaction method according to any one of claims 1 to 7.

Images