Digital human AGI dialogue system based on cloud side-end collaborative architecture

Abstract

The invention provides a digital human AGI dialogue system based on a cloud side-end collaborative architecture, and relates to the technical field of digital humans, the system is characterized in that a sensing module, a processing module, a decision module, a rendering module and an output module are deployed at a side end, and a decision module, a driving module and a rendering module are deployed at a cloud end; the sensing module collects and preprocesses an input signal of a user; the processing module is connected with the sensing module and is used for extracting features of the input signals and generating context vectors; the decision-making module is connected with the processing module, and generates a decision-making result containing an answer text and an emotion label according to the context vector; the driving module is connected with the decision module, generates an audio stream and a phoneme sequence according to the answer text, and calculates skeleton driving parameters and mouth shape driving parameters of the digital human; the rendering module is connected with the driving module to generate a rendered picture; and the interaction module is connected with the driving module and the rendering module, and aligns the rendered picture and the audio stream to obtain an output result. And low time delay and high performance are realized by adopting cloud edge collaboration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital human, in particular to a digital human AGI dialogue system based on a cloud-edge-end collaborative architecture. BACKGROUND

[0002] In recent years, "digital human" technology is rapidly moving from concept verification to industrial application. From virtual anchors, digital employees to intelligent cockpit assistants, digital humans have become a new generation of human-computer interaction carriers that connect the physical world and digital space and provide immersive intelligent services. An ideal digital human not only needs to have human-like general intelligence (AGI) and be able to conduct deep and coherent multi-round dialogue, but also needs to have lively and natural emotional expression to establish user trust and emotional connection.

[0003] However, current digital human products are trapped in a dilemma in terms of technical implementation: either pursuing extreme picture rendering quality and complex AI reasoning, resulting in high system latency and slow response, which destroys the real-time nature and immersion of the interaction; or greatly simplifying the model and rendering effect in order to ensure smoothness, resulting in stiff digital human expressions and mechanical movements, lacking the necessary expressiveness. The fundamental reason is that highly realistic expression and ultra-low latency response are a pair of contradictions that are difficult to reconcile at the technical bottom layer. SUMMARY

[0004] The purpose of the present application is to provide a digital human AGI dialogue system based on a cloud-edge-end collaborative architecture, and the technical problem to be solved is how to realize the unification of low latency and high expressiveness in the digital human dialogue system.

[0005] The present application is realized by the following technical solutions:

[0006] A digital human AGI dialogue system based on a cloud-edge-end collaborative architecture, comprising:

[0007] A perception module, the perception module is deployed at the edge end; the perception module is used to collect input signals of a user and pre-process the input signals;

[0008] A processing module, the processing module is deployed at the edge end, and the processing module is connected to the perception module; the processing module is used to analyze the pre-processed input signals, extract features from the input signals, and generate a context vector;

[0009] A decision module, the decision module is deployed at the edge end and the cloud end, and the decision module is connected to the processing module; the decision module is used to generate a decision result containing an answer text and an emotional label based on a large language model and a local knowledge base according to the context vector;

[0010] A driving module, the driving module is deployed in the cloud, and the driving module is connected with the decision module; the driving module is used for receiving the decision result, generating an audio stream and a phoneme sequence according to the answer text, and calculating a digital human skeleton driving parameter and a digital human lip driving parameter through a sentiment-action mapping model based on the sentiment label and the phoneme sequence;

[0011] A rendering module, the rendering module is deployed in the edge and the cloud, and the rendering module is connected with the driving module; the rendering module is used for receiving the skeleton driving parameter and the lip driving parameter, and generating a digital human rendering picture;

[0012] An interaction module, the interaction module is deployed in the edge, and the interaction module is connected with the driving module and the rendering module; the interaction module is used for synchronously aligning the rendering picture and the audio stream to obtain an output result.

[0013] The collection and preprocessing of the input signal are placed in the edge, which solves the network transmission delay caused by directly uploading the original data to the cloud, quickly processes the data at the data source, and reduces the transmission delay. The basic task with high real-time requirement (such as simple question and answer and intent recognition) is processed by the edge, the embedded neural network processor (NPU) is used for acceleration, and the network delay is reduced; the complex task with high computing power requirement (such as multi-step logical reasoning and complex data calculation) is processed by the cloud, the powerful computing power of the cloud is used to generate the personification answer, and the expressiveness of the content layer is ensured; both low latency and high expressiveness are achieved. The driving module generates driving parameters through the sentiment-action mapping model, so that the expression and action of the digital human are no longer mechanical and disconnected with the content, but have emotional changes, thereby realizing high expressiveness in the visual layer; the rendering module is deployed in the edge and the cloud, the edge rendering and the cloud rendering are switched according to whether the computing power of the edge is sufficient, and the problem of local lag and reduced real-time performance is solved. The interaction module aligns the rendering picture and the audio stream, improves the consistency of the audio-visual experience, and thus realizes high expressiveness.

[0014] Further, the perception module comprises:

[0015] A voice collection unit, the voice collection unit is used for collecting a voice input signal and preprocessing the voice input signal;

[0016] A touch collection unit, the touch collection unit is used for collecting a coordinate signal of a touch point, and converting the coordinate signal into an interaction instruction through the EventSystem of the Unity engine.

[0017] The environmental noise and invalid audio are filtered at the data source, which reduces the subsequent processing difficulty and computational complexity. The user can interact with the digital human through touch, which enhances the liveliness of the digital human and the user's immersion, and improves the expressiveness.

[0018] Further, the processing module is used to analyze the preprocessed input signal, and extract features from the input signal. The specific steps include:

[0019] The fundamental frequency and energy value are extracted from the voice input signal according to the preset frame frequency to obtain voice information.

[0020] The voice information is converted into text information by the Xunfei ASR offline engine, the text information is segmented and entity recognized, and the historical dialogue stored in the edge is associated to generate a context vector representing the current dialogue context.

[0021] The fundamental frequency is related to the pitch, the energy value is related to the volume, and the speech rate is related to the speech emotion. The extracted feature data is used as the direct basis for generating digital human facial expressions and micro-movements. The feature data is updated in real time at a preset frame frequency, so that the generated digital human facial expressions and micro-movements can be synchronized with the user's voice, avoiding lag or stiffness. The Xunfei ASR offline engine is used for voice-to-text conversion, rather than cloud-based ASR, avoiding network transmission delay and cloud service queuing delay. The historical dialogue stored in the edge is associated to generate a context vector containing the cause and effect of the dialogue context, improving the expressiveness of the content.

[0022] Further, the decision module deployed in the edge is used to process any one or more of the basic tasks in the basic task set. After receiving the context vector, the Spark lightweight large model and the local knowledge base are used to generate a decision result containing answer text and emotion labels.

[0023] The decision module deployed in the cloud is used to process any one or more of the complex tasks in the complex task set. After receiving the context vector, the Spark cloud large model is used to generate a decision result containing answer text and emotion labels.

[0024] Small-computing and delay-sensitive tasks are fixed in the edge for processing, reducing network transmission delay. The Spark lightweight large model and the local knowledge base are used to enable them to run on the edge 3588 motherboard NPU, achieving single-round inference less than or equal to 500ms, ensuring that users receive timely feedback and maintain smooth dialogue flow. Tasks that require strong knowledge and reasoning capabilities, such as multi-step logical reasoning, complex calculations, deep semantic analysis, and open-domain creative generation, are scheduled to the cloud. The cloud's computing resources can be fully utilized to improve the digital human's high expressiveness in content.

[0025] Further, the driving module includes:

[0026] The speech synthesis unit is connected to the decision module. The speech synthesis unit is used to synthesize the answer text into an audio stream and a phoneme sequence with emotion parameters by the Xunfei TTS offline engine.

[0027] The driving computing unit is connected to the speech synthesis unit. The driving computing unit is used to map the intensity value of the emotion label to the corresponding expression blendshape weight and skeletal driving parameters through linear interpolation using the emotion-action mapping model. The phoneme sequence is then converted into lip-shape driving parameters using the expression blendshape weight and skeletal driving parameters.

[0028] The audio stream synthesized using iFlytek's offline TTS engine contains emotional parameters. The generated speech is not a mechanical, monotonous electronic voice, but rather speech with variations in intonation, rhythm, and stress, conveying emotions such as joy, seriousness, and empathy, thus enhancing auditory expressiveness. The generated phoneme sequence is the core data source driving the digital human's mouth movements. First, based on the intensity value of the emotional tag, linear interpolation is used to calculate the overall expression blendshape weight (such as the amplitude of a smile) and skeletal driving parameters (such as the amplitude of nodding and the degree of body swaying), ensuring that the digital human's facial and body movements express emotions consistent with the dialogue content, rather than just mouth movements. Building upon the emotional tone, the phoneme sequence is then used to drive the mouth, generating lip-sync parameters. This ensures that the digital human's facial expressions are vivid while the lip movements correspond to the audio, thereby improving the synchronization rate of expression, movement, and audio.

[0029] Furthermore, when there is sufficient computing power at the edge, the above-mentioned skeletal driving parameters and lip-sync driving parameters are used to generate a digital human rendering image through the URP pipeline, ETC2 texture compression and dynamic resolution of the Unity engine at the edge.

[0030] When insufficient computing power is detected at the edge, rendering is switched to a server cluster deployed in the cloud. The server cluster uses Docker to containerize Unity instances and Kubernetes for resource scheduling.

[0031] When edge computing power is sufficient, rendering is performed at the edge, reducing network encoding, transmission, and decoding latency required for cloud rendering. When edge GPUs cannot handle the load, the system switches to a cloud server cluster. The cloud's computing power can render higher-precision, more complex lighting (such as ray tracing), and more realistic images, ensuring visual performance. The use of Docker containerization and Kubernetes scheduling ensures system stability under heavy loads, thereby indirectly guaranteeing service continuity and real-time performance.

[0032] Furthermore, the aforementioned cloud and edge devices adopt the NICE DCV protocol; the rendered images generated in the cloud are encoded and transmitted to the edge device via the NICE DCV protocol, and then decoded and displayed at the edge device.

[0033] Further, the above-mentioned interaction module is used for aligning the audio stream and the rendering picture frame by timestamp, and when it is detected that the audio-video synchronization error is greater than 50 ms, the rendering frame rate or the playing buffer time of the audio stream is adjusted.

[0034] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0035] The collection and preprocessing of the input signal are placed at the edge end, solving the network transmission delay caused by directly uploading the original data to the cloud, quickly processing the data at the data source, and reducing the transmission delay. The basic task with high real-time requirement (such as simple question and answer and intent recognition) is processed by the edge end, the embedded neural network processor (NPU) is used for acceleration, and the network delay is reduced; the complex task with high computing power requirement (such as multi-step logical reasoning and complex data calculation) is processed by the cloud, the powerful computing power of the cloud is used to generate the personification answer, and the expressiveness in the content level is ensured; both low latency and high expressiveness are achieved. The driving module generates driving parameters through the emotion-action mapping model, so that the expression and action of the digital person are no longer mechanical and disconnected with the content, but have emotional changes, thereby realizing high expressiveness in the visual level; the rendering module is deployed on both the edge end and the cloud, and according to whether the computing power of the edge end is sufficient, the edge end rendering and the cloud rendering are switched, thereby solving the problem of local lag and reducing real-time performance. The interaction module aligns the rendering picture and the audio stream, improves the consistency of the audio-visual experience, and thus realizes high expressiveness. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the example embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor. In the drawings:

[0037] Figure 1 The connection diagram of the digital person AGI dialogue system.

[0038] Markings and corresponding component names in the drawings:

[0039] 1, perception module; 11, voice acquisition unit; 12, touch acquisition unit; 2, processing module; 3, decision module; 4, driving module; 41, voice synthesis unit; 42, driving calculation unit; 5, rendering module; 6, interaction module. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description will be given below in combination with embodiments and drawings. The schematic embodiments of the present application and their descriptions are only used to explain the present application, and do not limit the present application.

[0041] In order to realize a digital human dialogue system with deep semantic understanding, emotional expression and autonomous knowledge evolution capability, the present application is based on a "cloud-edge-end collaborative" architecture, combines general artificial intelligence (AGI), real-time 3D driving and multi-modal interaction technology, and is used to support more than 10 rounds of dialogue context memory, voice-expression synchronization rate greater than 98%, end-to-end delay less than 300 ms, local knowledge base retrieval response less than 100 ms, and rendering capability with adaptive hardware performance. Specific examples are as follows.

[0042] First embodiment:

[0043] In combination Figure 1 A digital human AGI dialogue system based on a cloud-edge-end collaborative architecture, comprising:

[0044] A perception module 1 deployed at the edge end, which is used to collect input signals of a user and pre-process the input signals;

[0045] A processing module 2 deployed at the edge end, which is connected to the perception module 1; the processing module 2 is used to analyze the pre-processed input signals, extract features from the input signals, and generate a context vector;

[0046] A decision module 3 deployed at the edge end and the cloud end, which is connected to the processing module 2; the decision module 3 is used to generate a decision result containing an answer text and an emotional label based on a large language model and a local knowledge base according to the context vector;

[0047] A driving module 4 deployed at the cloud end, which is connected to the decision module 3; the driving module 4 is used to receive the decision result, generate an audio stream and a phoneme sequence according to the answer text, and calculate the skeletal driving parameters and the lip driving parameters of the digital human through an emotion-action mapping model based on the emotional label and the phoneme sequence;

[0048] A rendering module 5 deployed at the edge end and the cloud end, which is connected to the driving module 4; the rendering module 5 is used to receive the skeletal driving parameters and the lip driving parameters, and generate a rendering picture of the digital human;

[0049] An interaction module 6 deployed at the edge end, which is connected to the driving module 4 and the rendering module 5; the interaction module 6 is used to synchronize and align the rendering picture and the audio stream to obtain an output result.

[0050] It adopts a six-layer architecture of "perception-processing-decision-driving-rendering-interaction", integrates the advantages of Unity's real-time driving and cloud-edge collaboration, and strengthens AGI's autonomous decision-making and knowledge evolution capabilities. Through the cloud-edge collaboration architecture, it integrates real-time rendering and AGI intelligence, and data logic runs through the entire "input-processing-decision-output" link, ensuring that the digital human has both a smooth interactive experience and human-like understanding and expression capabilities.

[0051] Second embodiment:

[0052] Based on the first embodiment, the sensing module 1 includes:

[0053] The voice acquisition unit 11 may be an array microphone. The array microphone acquires the voice input signal and preprocesses the voice input signal. The preprocessing is based on an AI noise reduction model to filter environmental noise.

[0054] The touch acquisition unit 12 can be a 65-inch touchscreen with a resolution of 1920x1080. It acquires the coordinate signals of touch points on the touchscreen and converts these signals into interactive commands using Unity's EventSystem to predict touch intentions with a response latency of less than 10ms. For example, clicking on a digital human triggers a focus animation. The touch acquisition unit 12 connects to the interaction module 6 for touch feedback animation.

[0055] By filtering out environmental noise and invalid audio at the data source, the difficulty and computational load of subsequent processing are reduced. Users can interact with the digital human through touch, enhancing the digital human's vividness and user immersion, and improving its expressiveness.

[0056] Third embodiment:

[0057] Based on any of the above embodiments, the processing module 2 is used to parse the preprocessed input signal and extract features from the input signal. The specific steps include:

[0058] The fundamental frequency and energy value are extracted from the above speech input signal at a preset frame rate (10ms / frame) to obtain speech information;

[0059] The iFlytek ASR offline engine converts the aforementioned voice information into text information, performs word segmentation and entity recognition on the text information, and associates it with the historical dialogue stored at the edge to generate a context vector representing the current dialogue context.

[0060] The above fundamental frequency is related to tone, energy value is related to volume, and speech speed is related to voice emotion. The extracted feature data is directly used as the basis for generating digital human facial expressions and micro-movements. The feature data is updated in real time at a preset frame frequency, so that the generated digital human facial expressions and micro-movements can be synchronized with the user's voice, avoiding motion lag or stiffness. The voice-to-text conversion is performed using the Xunfei ASR offline engine instead of the cloud ASR, avoiding network transmission delay and cloud service queuing delay. The historical dialogue stored on the edge is associated to generate a context vector containing the context of the dialogue, improving the expressiveness of the content.

[0061] The use scenario can be referred to as follows: a 4-array microphone is used, which has a 3-meter pickup radius. Background noise is filtered through beamforming. Audio data is transmitted to the NPU of the 3588 mainboard at a sampling rate of 44.1 kHz. The Xunfei ASR offline engine is used to convert 16 kHz voice information into text information, and the conversion accuracy is greater than 95%. The fundamental frequency, energy value, and speech speed are extracted from the voice input signal. The energy value is used for expression driving, and the speech speed is used for lip synchronization speed adjustment. The feature data is temporarily stored in the memory at a frequency of 10 ms / frame. The text information output by the Xunfei ASR offline engine is segmented (jieba library), entity recognition is performed (based on the spark model), and the dialogue history is associated (stored in the edge solid state disk (SSD), retaining the last 50 rounds of dialogue). A context vector is generated (used for large model understanding of context).

[0062] Fourth embodiment:

[0063] On the basis of any of the above embodiments, the decision module 3 deployed on the edge is used to process any one or more of the basic tasks in the basic task set. After receiving the context vector, the spark lightweight large model and the local knowledge base are used to generate a decision result containing answer text and emotion label.

[0064] The decision module 3 deployed on the cloud is used to process any one or more of the complex tasks in the complex task set. After receiving the context vector, the spark cloud large model is used to generate a decision result containing answer text and emotion label.

[0065] Small-computing and delay-sensitive tasks are fixed on the edge for processing, reducing network transmission delay. The spark lightweight large model and the local knowledge base are used to enable them to run on the NPU of the edge 3588 mainboard, with single-round inference less than or equal to 500 ms, ensuring that the user receives timely feedback and maintains the smoothness of the dialogue. Tasks that require strong knowledge and reasoning capabilities, such as multi-step logical reasoning, complex calculations, deep semantic analysis, and open-domain creative generation, are scheduled to the cloud. The computing resources of the cloud can be fully utilized to improve the high expressiveness of digital human content.

[0066] Reference use case, hybrid deployment of "Xinghuo V3.0 offline version and enhanced cloud version": local model for basic question answering and intent recognition, single round reasoning less than 500ms, 3588 NPU acceleration; cloud model for complex reasoning such as mathematical calculation and multi-step logic, LAN call delay less than 200ms. Before using the knowledge base retrieval, the Xinghuo document slicing tool is used to divide the PDF / Word into 500-2000 word segments, extract the BERT vector and store it in Milvus, support 100,000 level document retrieval, time less than 100ms; through cloud synchronization of industry knowledge base at midnight every day, user feedback error knowledge triggers immediate correction. In addition, the dialogue state tracking model based on Transformer replaces the traditional finite state machine, dynamically tracks the change of user intent, such as switching from "query product" to "complaint problem".

[0067] Fourth embodiment:

[0068] On the basis of any of the above embodiments, the driving module 4 comprises:

[0069] The speech synthesis unit 41 is connected to the decision module 3; the speech synthesis unit 41 is used to synthesize the answer text into an audio stream with emotional parameters and a phoneme sequence through the Xunfei TTS offline engine;

[0070] The driving calculation unit 42 is connected to the speech synthesis unit 41; the driving calculation unit 42 is used to map the intensity value of the emotional label to the corresponding expression Blendshape weight and skeletal driving parameter through linear interpolation method through the emotional-action mapping model, and convert the phoneme sequence to mouth shape driving parameter through the expression Blendshape weight and skeletal driving parameter.

[0071] The synthesized audio stream with emotional parameters by the Xunfei TTS offline engine is not a mechanical, flat and straight electronic voice, but a voice containing changes in tone, rhythm, stress, etc., with emotional colors such as joy, seriousness, sympathy, etc., improving the expressiveness at the auditory level. The generated phoneme sequence is the core data source for driving the digital human mouth movement; first, according to the intensity value of the emotional label, the overall expression Blendshape weight (such as the amplitude of a smile) and the skeletal driving parameter (such as the amplitude of a nod and the degree of body shaking) are calculated through linear interpolation method, so that the digital human expresses the emotions consistent with the content of the conversation through the face and body movements when speaking, rather than only the mouth movement; on the basis of the emotional tone, the mouth shape driving parameter is generated by driving the mouth with the phoneme sequence, ensuring that the digital human's facial expression is lively, and the mouth shape also corresponds to the audio, thereby improving the synchronization rate of expression, action and audio.

[0072] For reference, the use cases combine iFlytek TTS phoneme sequences (such as "a" and "i") with a dual-drive mechanism (Blendshape controls lip shape + bone displacement adjusts jaw), achieving a synchronization error of less than 50ms. The animation state machine adopts a layered design, including a base layer, an expression layer, and an action layer. The base layer switches between standby and interactive states, such as switching from "standby" to "listening" when a user asks a question. The expression layer uses Blend Tree to achieve "emotion-animation" expression mixing, with the weight determined by the emotion tag. The action layer triggers specific actions based on the dialogue content, such as "nodding" to indicate affirmation and "waving" to indicate negation.

[0073] Fifth embodiment:

[0074] Based on any of the above embodiments, the above rendering module 5 is used to generate a digital human rendering image by means of the above bone driving parameters and lip driving parameters through the URP pipeline, ETC2 texture compression and dynamic resolution of the edge Unity engine when the edge computing power is sufficient.

[0075] When insufficient computing power is detected at the edge, rendering is switched to a server cluster deployed in the cloud. The server cluster uses Docker to containerize Unity instances and Kubernetes for resource scheduling.

[0076] When edge computing power is sufficient, rendering is performed at the edge, reducing network encoding, transmission, and decoding latency required for cloud rendering. When edge GPUs cannot handle the load, the system switches to a cloud server cluster. The cloud's computing power can render higher-precision, more complex lighting (such as ray tracing), and more realistic images, ensuring visual performance. The use of Docker containerization and Kubernetes scheduling ensures system stability under heavy loads, thereby indirectly guaranteeing service continuity and real-time performance.

[0077] In a specific embodiment, the cloud and edge terminals adopt the NICE DCV protocol; the rendered image generated in the cloud is encoded and transmitted to the edge terminal via the NICE DCV protocol, and then decoded and displayed at the edge terminal.

[0078] In a specific embodiment, the above-mentioned interaction module 6 is used to align the audio stream and the rendered screen frame using timestamps. When an audio-visual synchronization error greater than 50ms is detected, the rendering frame rate or the playback buffer time of the audio stream is adjusted.

[0079] The reference use scenario adopts cloud edge collaborative rendering, the hardware of local rendering is realized by edge all-in-one machine (3588 mainboard), URP pipeline, ETC2 texture compression, dynamic resolution (720p-1080p adaptive) are enabled, and 60FPS is maintained; the expression driving logic is put into a sub-thread through the Unity Jobs System, so that the CPU occupation of the main thread is reduced by 20%. The cloud rendering is realized by an NVIDIA A10 server (single card supporting 4-way 1080P 30fps rendering), a Docker containerized deployment Unity instance, Kubernetes scheduling resources, data interaction is realized by using the NICE DCV protocol, the local area network delay is less than 50ms, and the encoding is H.265 (code rate 4-8Mbps); the edge end is displayed after decoding. The rendering picture is output to a 65-inch touch screen through streaming / local output, and real-time subtitles can also be superimposed; the voice is synthesized through the Xunfei TTS offline engine, the emotional tone is supported, for example, the pitch is raised when "joy", the voice is played through 2*10W loudspeakers, the audio and picture time stamps are aligned, and touch feedback animation is realized.

[0080] The above specific embodiments further specifically explain the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A digital human AGI dialogue system based on a cloud-edge-device collaborative architecture, characterized in that, include: A sensing module (1) is deployed at the edge; the sensing module (1) is used to collect the user's input signal and preprocess the input signal; The processing module (2) is deployed at the edge and is connected to the perception module (1). The processing module (2) is used to parse the preprocessed input signal, extract features from the input signal, and generate a context vector. The decision module (3) is deployed at the edge and in the cloud. The decision module (3) is connected to the processing module (2). The decision module (3) is used to generate decision results containing answer text and sentiment tags based on the context vector, according to the large language model and local knowledge base. The driving module (4) is deployed in the cloud and is connected to the decision module (3). The driving module (4) is used to receive the decision result, generate an audio stream and phoneme sequence according to the answer text, and calculate the skeletal driving parameters and lip-sync driving parameters of the digital human based on the emotion tag and phoneme sequence through the emotion-action mapping model. The rendering module (5) is deployed at the edge and in the cloud. The rendering module (5) is connected to the driving module (4). The rendering module (5) is used to receive skeletal driving parameters and lip-sync driving parameters to generate the rendering image of the digital human. An interaction module (6) is deployed at the edge. The interaction module (6) connects the driver module (4) and the rendering module (5). The interaction module (6) is used to synchronize and align the rendered screen and audio stream to obtain the output result.

2. The digital human AGI dialogue system according to claim 1, characterized in that, The sensing module (1) includes: A voice acquisition unit (11) is used to acquire voice input signals and preprocess the voice input signals; Touch acquisition unit (12) is used to acquire the coordinate signals of touch points and convert the coordinate signals into interactive instructions through Unity engine's EventSystem.

3. The digital human AGI dialogue system according to claim 1, characterized in that, The processing module (2) is used to parse the preprocessed input signal and extract features from the input signal. The specific steps include: The fundamental frequency and energy value are extracted from the voice input signal according to the frequency of a preset frame to obtain voice information; The speech information is converted into text information using the iFlytek ASR offline engine. The text information is then segmented and entity recognized, and associated with historical dialogues stored at the edge to generate a context vector representing the current dialogue context.

4. The digital human AGI dialogue system according to claim 1, characterized in that, The decision module (3) deployed at the edge is used to process any one or more basic tasks in the basic task set. After receiving the context vector, it generates a decision result containing the answer text and sentiment label through the Spark Lightweight Large Model and the local knowledge base. The decision module (3) deployed in the cloud is used to process any one or more complex tasks in the complex task set. After receiving the context vector, it generates a decision result containing the answer text and sentiment label through the Spark Cloud Big Model.

5. The digital human AGI dialogue system according to claim 1, characterized in that, The driving module (4) includes: The speech synthesis unit (41) is connected to the decision module (3); the speech synthesis unit (41) is used to synthesize an audio stream and phoneme sequence with emotional parameters from the answer text through the iFlytek TTS offline engine; The driving calculation unit (42) is connected to the speech synthesis unit (41). The driving calculation unit (42) is used to map the intensity value of the emotion label to the corresponding expression blendshape weight and skeletal driving parameters through the emotion-action mapping model and the expression blendshape weight and skeletal driving parameters. The phoneme sequence is converted into lip shape driving parameters through the expression blendshape weight and skeletal driving parameters.

6. The digital human AGI dialogue system according to claim 1, characterized in that, The rendering module (5) is used to generate a digital human rendering image by means of the bone driving parameters and lip driving parameters through the URP pipeline, ETC2 texture compression and dynamic resolution of the Unity engine at the edge when the computing power at the edge is sufficient. When insufficient computing power is detected at the edge, rendering is switched to a server cluster deployed in the cloud. The server cluster uses Docker containerization to deploy Unity instances and Kubernetes for resource scheduling.

7. The digital human AGI dialogue system according to claim 6, characterized in that, The cloud and edge terminals use the NICE DCV protocol; the rendered images generated in the cloud are encoded and transmitted to the edge terminal via the NICE DCV protocol, and then decoded and displayed at the edge terminal.

8. The digital human AGI dialogue system according to claim 1, characterized in that, The interaction module (6) is used to align the audio stream and the rendered screen frame by timestamp. When the audio-visual synchronization error is detected to be greater than 50ms, the rendering frame rate or the playback buffer time of the audio stream is adjusted.

Images