A digital human question answering system

Abstract

The present application provides a kind of digital human question and answer system, the system includes: speech receiving processing module, for receiving user speech input in real time, and the input speech is split, the acoustic characteristics of each frame speech are extracted in real time, while the acoustic characteristics of continuous frame are cached, form streaming data queue;Voice feature inference module is used to input pre-training question and answer model with streaming data queue, and output question and answer text result and speech synthesis instruction in real time;Digital human synchronous driving module is used to generate synchronous voice signal based on question and answer text result and speech synthesis instruction, synchronous voice signal is mapped to lip mapping library and action template library, generate the lip sequence and body movement sequence matched with voice time sequence, complete digital human broadcast.The present application returns the question and answer result after receiving user text or speech input, and drives digital human to complete voice broadcast.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a digital human question-and-answer system. Background Technology

[0002] As an important application in the field of human-computer interaction, digital human question-answering systems are gradually moving from demonstration scenarios to service scenarios. Traditional digital human question-answering systems are mainly built on their own computing chips. They receive text or voice input from users, perform model inference, return the question-answering results, and drive the digital human to complete the voice broadcast.

[0003] The existing technology has the following drawbacks and shortcomings:

[0004] Due to poor environmental adaptability, traditional digital human question-answering systems cannot be directly deployed on computing hardware that is not self-provided. They rely too much on self-provided software and hardware, which will lead to data security risks when using other software and hardware to run traditional digital human question-answering systems.

[0005] Streaming processing suffers from high latency. Traditional digital human question-and-answer systems often employ batch data processing, resulting in significant delays in parsing and reasoning for continuous user voice input, which fails to meet the demands of real-time interaction.

[0006] The lip-sync is insufficient. The lip-sync generation and body movement matching of traditional digital human question-and-answer systems have not been optimized in combination with the characteristics of other computing power systems. If other computing power systems are used directly to drive the system, the timing of the digital human's voice broadcast will be out of sync with the lip-sync and movement, resulting in a stiff interactive experience.

[0007] The inference efficiency is not well adapted, the proprietary AI framework has poor compatibility with other computing chips, the computing power utilization is low during model inference, there is a waste of resources, and the inference strategy cannot be dynamically adjusted according to the load of other hardware. Summary of the Invention

[0008] The purpose of this invention is to provide a digital human question-and-answer system that receives and processes user voice input in real time, and streams question-and-answer results and digital human announcements to achieve low-latency interaction and improve user engagement and satisfaction.

[0009] To achieve the above objectives, the present invention provides a digital human question-and-answer system, the system comprising:

[0010] The voice receiving and processing module is used to receive user voice input in real time, split the input voice, extract the acoustic features of each frame of voice in real time, and cache the acoustic features of consecutive frames to form a streaming data queue.

[0011] The speech feature inference module is used to input the streaming data queue into the pre-trained question answering model and output the question answering text results and speech synthesis instructions in real time;

[0012] The digital human synchronization driver module is used to generate synchronized speech signals based on question-and-answer text results and speech synthesis instructions. The synchronized speech signals are then mapped to a lip-sync mapping library and an action template library to generate lip-sync sequences and body action sequences that match the speech timing, thus completing the digital human's broadcast.

[0013] Furthermore, the acoustic features of each frame of speech are extracted in real time, specifically including:

[0014] S11. After removing the DC offset in the input speech using a first-order differential filtering algorithm, a Hanning window is applied to obtain the windowed speech signal.

[0015] S12. Convert the windowed speech signal into a frequency domain signal to obtain a complex spectrum. Take the square of the modulus of the complex spectrum to obtain the power spectrum.

[0016] S13. Set up several triangular Mel filters according to preset rules, and perform convolution operation between the power spectrum and the Mel filter bank through the vector multiplication operator to obtain the Mel spectrum;

[0017] S14. The Mel spectrum is transformed by the natural logarithm to obtain the Mel logarithmic spectrum. The Mel logarithmic spectrum is then transformed by the discrete cosine transform and feature selection DCT to output the Mel frequency cepstral coefficients.

[0018] S15. Perform intra-frame normalization on the Mel frequency cepstral coefficients and output the MFCC feature vector for each frame of speech.

[0019] Furthermore, a streaming data queue is formed, specifically including:

[0020] S21. Create a circular buffer in the system shared memory area, wherein the storage unit of the circular buffer is a tuple of frame features and timestamps;

[0021] S22. Generate timestamps for the MFCC feature vectors of each frame, forming tuples. Insert the tuples into the circular buffer in real time in ascending order of timestamps through atomic operations. When the buffer is full, overwrite the earliest storage unit according to the first-in-first-out rule.

[0022] S23. Monitor the effective data length of the circular buffer. When the preset trigger condition is met, read all consecutive effective frame feature sequences in the buffer to form a streaming data queue.

[0023] S24. After reading, the read storage unit is marked and cleared, and a cyclic redundancy check is performed before enqueuing the streaming data into the queue. If the check fails, the data is discarded.

[0024] Furthermore, the construction of the pre-trained question-answering model specifically includes:

[0025] S31. Obtain text corpus and question-answer pair data in the target domain to form a training dataset for the target domain.

[0026] S32. Based on the neural network architecture, construct a basic language model, input the target domain training dataset into the basic language model for domain training, and obtain the target domain question answering model;

[0027] S33. Perform streaming input / output adaptation and lightweight processing of preset rules on the target domain question answering model to obtain the question answering model.

[0028] Furthermore, the generation of question-and-answer text results specifically includes:

[0029] S41. The question-answering model generates an initial candidate text set based on the features of the starting frame of the received streaming data queue.

[0030] S42. When a new preset frame is added, the existing encoding state is reused to dynamically update the initial candidate text set, including adding candidate texts and eliminating candidate texts with low confidence.

[0031] S43. Repeat step S42 until the preset text output termination condition is met, and determine the output text from the final candidate text set according to the preset filtering rules.

[0032] Furthermore, the generation of speech synthesis instructions specifically includes:

[0033] S51. Receive the text output by the question-answering model, parse it, and output the prosody control parameters corresponding to the text;

[0034] S52. Generate speech synthesis instructions based on the output text and prosody control parameters.

[0035] Furthermore, the digital human synchronization driver module is specifically used to execute:

[0036] S61. Receive the question-and-answer text result and the speech synthesis instruction, and generate a synchronous speech signal carrying timing information based on the speech synthesis instruction;

[0037] S62. By analyzing the acoustic features of the synchronized speech signal in real time, mapping and interpolating them to the lip shape mapping library, a lip shape sequence is obtained.

[0038] S63. By integrating question-and-answer text results, synchronized voice signals, and preset emotional states, the results are mapped and arranged into the action template library to obtain a sequence of body movements.

[0039] S64. Align and fuse the lip-sync sequence and body movement sequence according to the timing of the synchronized speech signal to drive the digital human model to complete the broadcast.

[0040] Furthermore, step S62 specifically includes:

[0041] S71. Based on the timing of the synchronized speech signal, determine the phoneme sequence corresponding to the currently processed speech segment and the start and end times of each phoneme.

[0042] S72. Based on the phoneme sequence and the start and end times of each phoneme, and combined with the acoustic characteristics of the synchronous speech signal, query the preset lip-shape mapping library to determine the target lip-shape data corresponding to each phoneme.

[0043] S73. Based on the start and end times of each phoneme, perform real-time interpolation calculations between the target lip shape data of adjacent phonemes to generate a lip shape sequence that is time-aligned with the synchronous speech signal.

[0044] Furthermore, step S63 specifically includes:

[0045] S81. Perform semantic analysis on the question-and-answer text results and extract action-related semantic features;

[0046] S82. Perform prosodic analysis on the synchronized speech signal and extract prosodic features;

[0047] S83. Based on action-related semantic features, the prosodic features, and preset emotional states, retrieve and match corresponding candidate action templates from a preset action template library;

[0048] S84. Based on the temporal sequence of the synchronized speech signal, the candidate action templates are spatiotemporally arranged and fused to generate a limb action sequence corresponding to the synchronized speech signal.

[0049] Furthermore, upon system startup, the adaptation layer automatically detects the current hardware and software environment, specifically including:

[0050] S91. Based on the detected hardware and software environment, dynamically load the corresponding driver interfaces, system adaptation components, and framework compatible components from the resource library.

[0051] S92. Utilize the loaded driver interface, system adaptation components, and framework compatible components to complete the adaptation and initialization of the system operating environment.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] This invention provides a digital human question-and-answer system that, through streaming reception, segmentation, and inference, eliminates the need to wait for complete sentences to finish before processing, thus providing a human-like, real-time dialogue response and enhancing the naturalness of interaction and user experience. Streaming data processing combined with a pre-trained question-and-answer model enables smooth scheduling of computing resources, avoiding peak pressure from processing large amounts of data at once. This not only improves processing efficiency but also enhances the system's stability and reliability when facing long-duration audio or high concurrency. The system drives the generation of synchronous speech signals that perfectly match semantics and rhythm through the coordination of speech synthesis instructions and text results. Then, through temporal mapping technology, the speech signals are converted into coordinated lip movements, facial expressions, and body movements, making the digital human's broadcast vivid and natural. This invention receives and processes user voice input in real time and streams question-and-answer results and digital human broadcasts, achieving low-latency interaction and improving user engagement and satisfaction. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the structure of a digital human question-answering system provided in an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of the real-time extraction of acoustic features of each frame of speech provided in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the process for forming a streaming data queue provided in an embodiment of the present invention;

[0058] Figure 4 A schematic diagram illustrating the construction process of the pre-trained question-answering model provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram illustrating the process of generating question-and-answer text results provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the speech synthesis instruction generation process provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram illustrating the specific execution flow of the digital human synchronization driving module provided in an embodiment of the present invention;

[0062] Figure 8 This is a schematic diagram of the process for generating lip shape sequences provided in an embodiment of the present invention;

[0063] Figure 9 This is a schematic diagram of the process for generating limb movement sequences provided in an embodiment of the present invention;

[0064] Figure 10 This is a schematic diagram illustrating the process of automatically detecting the current hardware and software environment through an adaptation layer, as provided in an embodiment of the present invention. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0066] Reference Figure 1 This embodiment provides a digital human question-answering system, the system comprising:

[0067] The voice receiving and processing module is used to receive user voice input in real time, split the input voice, extract the acoustic features of each frame of voice in real time, and cache the acoustic features of consecutive frames to form a streaming data queue.

[0068] The speech feature inference module is used to input the streaming data queue into the pre-trained question-answering model and output the question-answering text results and speech synthesis instructions in real time.

[0069] The digital human synchronization driver module is used to generate synchronized speech signals based on question-and-answer text results and speech synthesis instructions. The synchronized speech signals are then mapped to a lip-sync mapping library and an action template library to generate lip-sync sequences and body action sequences that match the speech timing, thus completing the digital human's broadcast.

[0070] In this embodiment, after the system starts, the adaptation layer automatically detects the current hardware, operating system type, and AI framework version, loads the corresponding hardware driver interface, system adaptation plugin, and framework compatibility patch, and completes the environment adaptation preparation. Subsequently, the voice receiving and processing module captures the user's voice stream in real time and divides the voice stream into continuous micro-frame units in chronological order. Each frame of voice is processed in real time to extract acoustic features that represent corresponding pitch, spectrum, and other information; at the same time, the system continuously caches the acoustic features of continuous frames, forming a dynamically updated streaming data queue. The voice feature inference module continuously reads the streaming data queue and inputs it into the pre-trained question-answering model. By analyzing the input acoustic feature stream in real time, it dynamically recognizes the voice content, understands the user's intent, and synchronously generates the text results of the question and answer and instructions such as prosody and rhythm to guide speech synthesis.

[0071] The digital human synchronization driver module first generates a high-quality synchronized speech signal based on the text results and speech synthesis instructions. The system uses this synchronized speech signal as a timeline reference, mapping it to a lip-sync library and an action template library: precisely driving the lips, chin, and other parts to generate lip-sync sequences corresponding to each phoneme; simultaneously, based on semantic content and speech rhythm, it triggers and choreographs predefined body movement sequences. Finally, the synthesized speech, lip-sync animation, and body animation are precisely time-aligned and rendered to complete the digital human's broadcast.

[0072] As a preferred embodiment, the acoustic features of each frame of speech are extracted in real time, specifically including:

[0073] S11. After removing the DC offset in the input speech using a first-order differential filtering algorithm, a Hanning window is applied to obtain a windowed speech signal.

[0074] S12. Convert the windowed speech signal into a frequency domain signal to obtain a complex spectrum. Take the square of the modulus of the complex spectrum to obtain the power spectrum.

[0075] S13. Set up several triangular Mel filters according to preset rules, and perform convolution operation between the power spectrum and the Mel filter bank through vector multiplication operator to obtain the Mel spectrum.

[0076] S14. The Mel spectrum is transformed by natural logarithm to obtain the Mel logarithmic spectrum. The Mel logarithmic spectrum is then transformed by discrete cosine transform and feature selection DCT to output the Mel frequency cepstral coefficients.

[0077] S15. Perform intra-frame normalization on the Mel frequency cepstral coefficients and output the MFCC feature vector for each frame of speech.

[0078] In this embodiment, the original speech signal often suffers from DC offset (zero-frequency component) interference in spectrum analysis due to the recording equipment. First-order differential filtering effectively suppresses DC offset. Subsequently, a Hanning window is applied to reduce spectral leakage caused by finite-length truncation (framing) of an infinitely long signal, allowing frame edges to smoothly decay to zero, thus improving the accuracy of subsequent spectrum analysis. The perceptual characteristics of speech (such as timbre) are primarily determined by its spectral energy distribution. Fast Fourier Transform converts the time-domain signal to the frequency domain, revealing frequency components. The square of the modulus of the complex spectrum yields the power spectrum, reflecting the energy distribution of the signal at each frequency component, which is the basis for subsequent processing. Human ear perception of frequency is not linear, but resolution is high in the low-frequency band and low in the high-frequency band. The Mel scale is a psychoacoustic scale based on auditory experiments. By convolving the power spectrum with a set of overlapping triangular filter banks, a linear frequency scale is mapped to the Mel scale, achieving compression and dimensionality reduction of high-frequency information.

[0079] The human ear perceives sound intensity approximately logarithmically. Taking the natural logarithm of the Mel spectrum aligns with auditory characteristics while compressing the dynamic range. The subsequent Discrete Cosine Transform (DCT) converts the highly correlated Mel logarithmic spectral coefficients into a set of low-order, approximately uncorrelated cepstral coefficients. These low-order coefficients represent the envelope shape of the spectrum (related to the speech content), while higher-order coefficients represent the fine structure of the spectrum (more related to the excitation source / individual characteristics). Feature filtering (usually retaining low-order coefficients) effectively reduces dimensionality and focuses content information. Different speakers, recording channels, or environments can cause overall shifts or scale differences in feature vectors. Normalization of the MFCC vector for each frame eliminates the influence of volume differences between speakers.

[0080] Specifically, the MFCC feature extraction algorithm of this invention is optimized based on the traditional MFCC algorithm, specifically for the NPU parallel computing characteristics and low latency requirements of current chips. The acoustic feature extraction process for each 15ms frame of speech is as follows, and it is fully compatible with operator interfaces of frameworks such as MindSpore / PaddlePaddle:

[0081] After receiving the audio signal from the microphone, the sampling clock is automatically calibrated to avoid frame feature distortion caused by hardware sampling deviations. A first-order differential filtering algorithm is used to remove DC offset from the audio signal, reducing environmental noise interference. A Hanning window is applied to the audio segments to suppress spectral leakage at frame edges. A parallel FFT operator is invoked to convert the windowed time-domain audio signal into a frequency-domain signal. The FFT points are set to 1024, outputting a 513-dimensional complex spectrum. The power spectrum is obtained by taking the modulus and squaring the complex spectrum, reducing the amount of imaginary number operations and improving NPU computational efficiency.

[0082] Twenty-four custom triangular Mel filters are used, with their center frequencies uniformly distributed along the Mel scale. The filter coefficients are pre-loaded into the NPU's shared memory to avoid redundant computation. The power spectrum is convolved with the Mel filter bank using the NPU's vector multiplication operator, yielding a 24-dimensional Mel spectrum (each dimension corresponding to the output energy of one filter). A natural logarithmic transform is applied to the 24-dimensional Mel spectrum to compress the feature dynamic range and adapt it to the input characteristics of the target model. A 13th-order Discrete Cosine Transform (DCT) and feature-selective DCT transform are employed to convert the 24-dimensional Mel logarithmic spectrum into 13-dimensional MFCC coefficients (containing one DC component and twelve AC components). The DCT matrix is ​​pre-compiled into constant operators of the target framework, reducing memory usage.

[0083] Intra-frame normalization (mean 0, variance 1) is performed on the 13-dimensional MFCC coefficients using the target BatchNorm lightweight operator to eliminate the influence of speaker volume differences. To improve recognition robustness, first-order and second-order difference features can be superimposed to expand the feature vector to 39 dimensions (13+13+13), but in streaming scenarios, the 13-dimensional core features are used by default to control latency. Each frame of speech ultimately outputs a 13-dimensional (or 39-dimensional) standardized MFCC feature vector in float32 format to meet the input requirements of the target inference engine.

[0084] As a preferred embodiment, forming a streaming data queue specifically includes:

[0085] S21. Create a circular buffer in the system shared memory area. The storage unit of the circular buffer is a tuple of frame features and timestamps.

[0086] S22. Generate timestamps for the MFCC feature vectors of each frame, forming tuples. Insert the tuples into the circular buffer in real time in ascending order of timestamps through atomic operations. When the buffer is full, overwrite the earliest storage unit according to the first-in-first-out rule.

[0087] S23. Monitor the effective data length of the circular buffer. When the preset trigger condition is met, read all consecutive effective frame feature sequences in the buffer to form a streaming data queue.

[0088] S24. After reading, the read storage unit is marked and cleared, and a cyclic redundancy check is performed before enqueuing the streaming data into the queue. If the check fails, the data is discarded.

[0089] In this embodiment, the circular buffer is a fixed-size first-in-first-out (FIFO) data structure. By eliminating the overhead of moving data after an element is dequeued in a regular queue, insertion and deletion achieve O(1) time complexity. Creating a circular buffer in the shared memory area allows modules such as voice reception and feature inference, which may be located in different processes or threads, to access the same data area with zero copy, reducing the data copying overhead and latency of inter-process communication. In concurrent production (feature extraction) and consumption (feature inference) scenarios, there is a race condition for reading and writing the buffer index. Atomic operations achieve lock-free or minimal lock contention-safe access, avoiding thread blocking and context switching overhead caused by using mutexes, and ensuring low latency. Inserting in ascending order by timestamp ensures strict order preservation of data sequence, providing a foundation for subsequent model processing of correct context dependencies.

[0090] When the buffer is full, the oldest data is overwritten to ensure that the system does not block the production end due to a full buffer when the consumer end cannot keep up with the production end speed, thus preventing input voice loss or system lag. Preset trigger conditions (such as "valid data length reaches N frames" or "buffer data time span exceeds T milliseconds") achieve a balance between batch processing and stream processing. By accumulating a small batch of continuous frames before outputting, it can meet the model's need for context windows while avoiding the excessive overhead of frame-by-frame inference. Mark-and-sweep ensures the cyclic reuse of buffer space. Cyclic redundancy checks are performed before enqueuing to detect data corruption in shared memory or during atomic operations. If the check fails, the frame is discarded to prevent erroneous features from contaminating the inference model and causing unpredictable output, thereby improving the overall robustness of the system.

[0091] Specifically, to ensure the real-time performance and data continuity of streaming inference, a circular buffer is used to construct a streaming data queue. The core design includes: building a circular buffer based on the target operating system's shared memory to avoid time-consuming data copying between processes; the queue storage unit is a tuple of frame features and timestamps, with each storage unit occupying 52 bytes (13-dimensional float32 features = 52 bytes) + 8 bytes (millisecond-level timestamp) = 60 bytes; the maximum queue length is set to 10 frames, and the minimum effective length is set to 3 frames, balancing latency and inference accuracy—less than 3 frames can easily lead to incomplete semantics, while more than 10 frames increase latency. After each frame's MFCC feature extraction is completed, it automatically carries the current system timestamp and is enqueued in ascending order of timestamp to avoid frame out-of-order issues caused by network jitter or hardware interruptions; a real-time mechanism of "enqueuing upon extraction completion" is adopted, with the enqueue operation implemented through atomic operations of the target system to avoid multi-threaded concurrency conflicts.

[0092] When the queue length reaches the 10-frame limit, the earliest frame is automatically discarded to ensure the queue always stores the latest speech features and avoids memory overflow. The inference engine uses an on-demand reading mechanism. When the queue length is ≥3 frames, it automatically reads all cached consecutive frame features (up to 10 frames) in the queue to form a batch input model of the feature sequence. If the user interrupts speech input (detecting consecutive 20ms silence frames), even if the queue length is <3 frames, the remaining frame features are immediately read and inference is triggered to avoid interactive stuttering. After reading the features, the inference engine releases the corresponding storage unit through a mark-and-sweep mechanism, overwriting the old unit only when a new frame is enqueued, reducing memory operation time. The queue storage unit is allocated addresses according to the NPU memory alignment requirements of the target chip, improving the efficiency of NPU reading feature data and reducing memory access latency. Before enqueuing, CRC32 verification is performed on the MFCC features. If corrupted feature data is detected, the frame is marked as invalid and automatically skipped during enqueuing to avoid affecting subsequent inference.

[0093] As a preferred embodiment, the construction of the pre-trained question-answering model specifically includes:

[0094] S31. Obtain text corpus and question-answer pair data in the target domain to form a training dataset for the target domain.

[0095] S32. Based on the neural network architecture, construct a basic language model, input the target domain training dataset into the basic language model for domain training, and obtain the target domain question answering model.

[0096] S33. Perform streaming input / output adaptation and lightweight processing of preset rules on the target domain question answering model to obtain the question answering model.

[0097] In this embodiment, the general-purpose large-scale pre-trained language model possesses powerful language understanding and generation capabilities, but lacks specialized terminology, knowledge structure, and expression style specific to the target domain. By collecting domain-specific text corpora and structured question-and-answer pairs, a training set is constructed, including domain knowledge density, enhancing the model's grasp of facts, concepts, and relationships within that vertical domain. Domain-specific question-and-answer patterns enable the model to learn to answer in a format and style consistent with domain conventions. This involves fine-tuning and aligning the parameter space of the general-purpose language model to the target domain. Employing a transfer learning strategy, the basic language model, pre-trained on massive amounts of general-purpose text, learns the language's syntax, semantics, and some common sense. Starting from this, supervised fine-tuning using the target domain training dataset allows for the rapid and efficient adaptation of the model's general knowledge to the target domain at a relatively low cost, improving performance on domain-specific tasks. The neural network architecture (typically a Transformer) provides powerful sequence modeling and feature interaction capabilities.

[0098] Traditional batch processing models are generally unsuitable for the streaming, low-latency, and resource-constrained environments required by digital human systems. Digital human systems process continuous streams of speech features. Through streaming input-output adaptation, question-answering models can handle incomplete incremental input and perform incremental generation or streaming output to support interactive modes that allow users to listen, think, and answer simultaneously. On edge devices or high-concurrency servers, model inference speed, memory usage, and power consumption are critical. Lightweight processing reduces computational and storage overhead while maintaining model performance as much as possible.

[0099] Specifically, the question-answering model is built on the MindSpore 1.10+ / PaddlePaddle 2.4+ framework, using a lightweight Transformer variant (MobileBERT-Lite) as its basic skeleton. Redundant attention heads and fully connected layers from the traditional Transformer are removed, retaining a 6-layer encoder (1 / 4 the number of layers in traditional BERT) and 8 attention heads (1 / 2 the number in traditional BERT), keeping the basic parameter count below 8M. It natively supports incremental feature input. For MFCC streaming features (13 dimensions / frame) at 15ms / frame, a dynamic sequence input interface is designed to adapt to streaming processing logic. The output layer adopts a "multi-candidate generation and confidence ranking" structure, with two parallel branches: a candidate text branch (generating 3-5 semantically similar question-answer texts) and a confidence branch (calculating the semantic matching score for each candidate text), balancing real-time performance and accuracy.

[0100] Throughout training and inference, native operators of the target framework are used. For example, on the MindSpore side, `mindspore.nn.Dense` (optimized memory access) and `mindspore.nn.MultiHeadAttention` (supports NPU parallel computing) are prioritized to replace third-party operators. On the PaddlePaddle side, `paddle.nn.Linear` (integrated with NPU tensor computation optimization) and `paddle.nn.MultiHeadAttention` (supports dynamic sequence length) are used to avoid operator compatibility issues. For the target framework's memory allocation mechanism, a "pre-allocation and dynamic reuse" strategy is adopted to avoid latency caused by frequent memory allocation / release. Distributed training is performed on the target system's computing server, for example, using MindSpore's `DataParallel` or PaddlePaddle's `DistributedDataParallel`, splitting the training data into 8-way slices, improving training efficiency by 6 times while ensuring full compatibility between model parameters and the inference environment.

[0101] The lightweight three-dimensional optimization includes: structured pruning, employing a dual strategy of attention head pruning and fully connected layer neuron pruning. Using MindSpore / PaddlePaddle's model compression tool, five core attention heads out of eight are retained, and the number of neurons in each fully connected layer is compressed from 2048 to 1434. Core components are selected based on gradient contribution to ensure that the semantic understanding accuracy of the model decreases by no more than 3% after pruning. Operator fusion, fusing convolutional layers + BatchNorm layers + activation layers, and attention layers + residual connection layers into composite operators. For example, on the MindSpore side, the fusion operator is encapsulated using `mindspore.ops.Custom` to reduce the number of operators. The number of data copies between nodes is reduced; the PaddlePaddle side uses the paddle.static.nn.fusion_conv_bn_act interface to merge three independent operators into one computation step, improving inference speed; low-precision quantization is used, with INT8 quantization applied to model weights to adapt to the low-precision computing units of the target chip's NPU, and quantization calibration is performed using the target domain question-answering dataset to reduce quantization errors; inference adaptation uses MindSpore's QuantAwareTraining or PaddlePaddle's paddle.quantization tool to generate quantized models, improving NPU computing power utilization.

[0102] The question-answering encoder retains the intermediate encoding state of each frame's features. When inputting features for the next frame, it directly reuses the previous state without re-encoding all features, reducing redundant calculations. By maintaining a "historical semantic cache pool" with a capacity of 10, it stores the core semantics of the last 5 rounds of interaction. When inferring features for the new frame, it combines the historical cache to improve the coherence of long-sentence question-answering.

[0103] As a preferred embodiment, the generation of question-and-answer text results specifically includes:

[0104] S41. The question-answering model generates an initial candidate text set based on the features of the starting frame of the received streaming data queue.

[0105] S42. When a new preset frame is added, the existing encoding state is reused to dynamically update the initial candidate text set, including adding candidate texts and eliminating candidate texts with low confidence.

[0106] S43. Repeat step S42 until the preset text output termination condition is met, and determine the output text from the final candidate text set according to the preset filtering rules.

[0107] In this embodiment, after receiving the initial frame features from the streaming data queue, the model must perform preliminary reasoning based on incomplete and potentially ambiguous early information to generate an initial candidate text set. This covers multiple potentially reasonable semantic directions, avoiding premature determination of a single output and subsequent correction difficulties, essentially establishing a "search space" for the subsequent optimization process. When adding a preset number of speech features, completely re-encoding and re-decoding all received frames is extremely inefficient. Reusing existing encoding states enables incremental encoding, calculating only new frames and fusing them with historical states, reducing computational complexity. Based on this, the candidate set is dynamically updated, simulating the human process of listening, thinking, and revising understanding. New information may support some existing candidates or generate new candidates, while eliminating candidates weakened by new evidence or with low confidence. Streaming generation requires a clear stop signal; a preset text output termination condition determines when generation ends. After termination, a unique output text is determined from the final candidate set according to preset filtering rules, ensuring the determinism and optimality of the result.

[0108] Specifically, the question-answering model receives the first 3 frames of MFCC features and generates 1-2 preliminary candidate texts. For every 2 new frames of features, the model reuses the previous encoding state and updates the candidate text set (adding 2-3 candidates and eliminating older candidates with confidence < 0.6). The final text is output when any of the following signals are detected: speech terminates when a silence frame is detected for 20ms consecutively (judged by the MFCC feature energy threshold; energy < 0.01 is considered silence). In streaming scenarios, semantic integrity judgment prioritizes 'silence frame detection + keyword matching'. Short texts (< 3 characters) directly output candidate results, while long texts are progressively improved through incremental decoding to balance real-time performance and accuracy (judged by a lightweight parser). When the streaming queue reaches 10 frames, the candidate text with the highest current confidence is forcibly output. The text with the highest confidence (≥ 0.75) is selected as the output from the candidate text set. If the text with the highest confidence is < 0.75, a standardized response of "Please say it again" is output to ensure interaction integrity.

[0109] As a preferred embodiment, the generation of speech synthesis instructions specifically includes:

[0110] S51. Receive the text output by the question-answering model, parse it, and output the prosodic control parameters corresponding to the text.

[0111] S52. Generate speech synthesis instructions based on the output text and prosody control parameters.

[0112] In this embodiment, since plain text lacks the prosodic information necessary for speech expression, computational linguistics analysis and prediction models are used to infer the corresponding suprasegmental features from the text, thereby obtaining the rhythm, intonation, stress, and emotional coloring of the speech, which is key to the naturalness of the speech. The sequence of phonetic symbols in the text is time-aligned and fused with continuous prosodic control parameters, packaged into a unified instruction set that the model can directly interpret and execute. This provides the speech synthesis engine with a detailed "score," which not only specifies the "notes" (phonemes) but also strictly defines the "beat, intensity, and vibrato" (duration, energy, and fundamental frequency) of each note.

[0113] Specifically, the speech synthesis instructions are in a standardized JSON format, including three core fields: text content, prosodic parameters, and adaptation parameters, to ensure compatibility with the target system's engine interface. The prosodic parameter generation logic includes: speech rate, defaulting to 1.0x speed; if the question-and-answer text is a long sentence (>15 characters), it automatically decreases to 0.9x speed; if it is a short sentence (<5 characters), it increases to 1.1x speed; intonation, determined by text sentiment analysis (a lightweight NaiveBayes classifier)—interrogative sentences (“What business do you need to handle?”) have their intonation increased by 1.1x, declarative sentences remain at 1.0x, and exclamatory sentences are increased by 1.05x; pause positions are automatically generated based on text punctuation (commas 150ms, periods 300ms, semicolons 200ms) and semantic breakpoints (segmented by keywords, such as adding a 100ms pause after “government affairs processing procedures”). Instruction output is generated synchronously with the question-and-answer text results and pushed in real-time to the audio-visual synchronization module via the target system's inter-process communication interface to ensure consistency with the timing of lip movements and actions.

[0114] In a preferred embodiment, the digital human synchronization driving module is specifically used to perform:

[0115] S61. Receive the question and answer text results and the speech synthesis instruction, and generate a synchronous speech signal carrying timing information based on the speech synthesis instruction.

[0116] S62. By analyzing the acoustic features of the synchronized speech signal in real time, mapping and interpolating them to the lip shape mapping library, a lip shape sequence is obtained.

[0117] S63. By integrating the question-and-answer text results, synchronized voice signals, and preset emotional states, the results are mapped and arranged into the action template library to obtain a sequence of body movements.

[0118] S64. Align and fuse the lip-sync sequence and body movement sequence according to the timing of the synchronized speech signal to drive the digital human model to complete the broadcast.

[0119] In this embodiment, the digital human synchronization driving module is the multimodal output execution engine of the digital human question-answering system. It transforms the discretized and symbolic content (text) and expressive instructions (speech parameters) generated upstream into continuous, high-fidelity audiovisual streaming media output, ensuring strict temporal and semantic consistency among audio, visual lip movements, and body movements. The speech synthesis instructions already contain precise phoneme duration and fundamental frequency information, allowing the generated synchronized speech signal to embed precise timestamps and prosodic markers, providing a reference timeline for all subsequent visual-driven synchronization and avoiding errors and delays caused by extracting temporal information from synthesized audio. Lip movements and the acoustic characteristics of speech, especially the spectrum and energy, are strongly correlated; for example, the lip movements for the vowels / a / and / i / differ significantly. By analyzing the acoustic characteristics of the speech signal in real time and mapping them to a predefined lip movement mapping library, a lip movement sequence highly matching the current pronunciation is generated. Mapping and interpolation ensure smooth and continuous lip movement animation across a limited set of visual bases.

[0120] The driving logic of body movements (such as gestures, nodding, and torso rotation) is more complex than that of lip movements, requiring the integration of three types of information: semantics, prosody, and emotional state. Semantics (the question-and-answer text result) determines the type and meaning of the action (e.g., "listing" corresponds to a counting gesture, "emphasis" corresponds to a gesture of intensification), prosody (the rhythm and stress of the synchronized speech signal) determines the timing and pace of the action, and the pre-defined emotional state determines the amplitude and style of the action (e.g., large amplitude and fast movement when excited). By mapping and arranging semantics, prosody, and pre-defined emotional states into a predefined action template library, non-verbal expressions that are consistent with the speech content can be achieved. The separately generated lip-sync sequences (driving the facial grid) and body movement sequences (driving the body skeleton) are then finally aligned temporally and spatially with the synchronized speech signal to ensure audio-visual synchronization. All sequences must be played synchronously based on the timestamp of the speech signal and presented in a conflict-free manner on the digital human model.

[0121] Specifically, the lip-sync library is designed to cover core Mandarin pronunciation, adapt to real-time digital human operation, and be compatible with target computing power systems, as detailed below:

[0122] The core lip shape selection is based on 30 basic lip shapes supporting combined pronunciation mapping. It covers composite phonemes (e.g., 'jiang': lip shape ID12 (j) + ID03 (i) + ID25 (ang) + 2 frames of transitional lip shapes) through 'phoneme sequence splicing + transitional lip shape interpolation' to ensure no pronunciation omissions. A 4K camera and facial motion capture module were used to collect pronunciation samples from several real people of different genders and ages. For basic vowels such as "a / o / e / i / u / ü" and initials such as "b / p / m / f", lip shape videos of single-phoneme pronunciation, disyllabic words, and commonly used short phrases were recorded, with pronunciation timestamps recorded simultaneously. The original videos were converted to YUV420 format by a target codec library (e.g., Huawei Cloud Codec), and the single-frame image resolution was compressed to 512×512 (adapting to digital human model input) and stored in the target database.

[0123] The phoneme labels for each frame of video were manually annotated using a labeling tool (e.g., 'a' corresponds to lip shape ID01, 'sh' corresponds to lip shape ID15), with an annotation accuracy of ≤1 frame (approximately 16.7ms at 60fps). Based on the digital human facial skeleton binding structure, facial key points were extracted from the annotated videos, and 3D lip shape mesh data was generated using the MindSpore lightweight pose estimation model. The 3D lip shape mesh was converted into a parameter set that the digital human could drive (e.g., lip opening degree 0-100, lip eversion degree 0-50, tongue height 0-30), with each basic lip shape corresponding to a fixed parameter combination, stored in JSON format. The target system's algorithm (LZ4 compression algorithm) was used to compress the 3D mesh data of 30 basic lip shapes to ≤10KB per lip shape, reducing NPU memory usage. A three-dimensional index table of "phoneme-lip shape ID-parameter set" was established to support quick lookup of corresponding lip shapes by phoneme. For digital human models of different precision (low-poly 10k polygons, high-poly 50k polygons), a lip shape parameter scaling interface was provided to adapt to the rendering capabilities of the target system's display terminal.

[0124] The motion template library is designed for natural interaction and low latency. It covers 8 types of head movements and 12 types of basic gestures, as shown in Table 1. Specifically, it includes:

[0125] Table 1 Action Type and Scene Binding

[0126]

[0127] Based on core application scenarios such as government affairs and finance, we sorted out high-frequency interactive dialogues and matched them with corresponding action types to ensure a strong correlation between actions and semantics. We used motion capture equipment to collect real-person action data, or manually adjusted the keyframes of the digital human skeleton through target plugins to generate motion animation clips. We decomposed the motion animation clips into digital human skeleton driving parameters (such as neck joint rotation angle, shoulder joint displacement, and finger flexion), and each action template corresponds to a fixed parameter time sequence (such as: guiding gestures, including raising, pointing, holding, and retracting, four stages). We established a "question and answer text semantics - action type" mapping table. For example: question text ("What business do you need to handle?") → head slightly tilted (side tilt + 5°) + palm upward gesture; instruction text ("Please provide an image") → head looking straight ahead + pointing to the screen gesture. We used target model compression tools (such as PaddleSlim) to sample and optimize the action keyframes, compressing the number of keyframes for each action template from 30 frames to 15 frames, maintaining the smoothness of the action and reducing storage volume. We used a target distributed file system to store action templates and cached high-frequency actions through a target caching system.

[0128] Using the speech signal output by the target system's TTS engine as the core, the system generates a speech signal through four steps: "phoneme segmentation - lip-syncing - transition - timestamp calibration," ensuring precise synchronization with the speech. When the target system's TTS engine (such as Huawei Cloud TTS) generates the speech signal, it simultaneously outputs "speech waveform + phoneme timing sequence" (format: phoneme ID - start timestamp - end timestamp), for example: "Hello" → phonemes "n" (0ms-80ms), "i" (80ms-150ms), "h" (150ms-220ms), "ao" (220ms-350ms). The phoneme segmentation accuracy is ≤10ms, and the accuracy is verified through the target speech processing operator (MindSporeAudio module) to ensure no phoneme omissions or misalignments. Based on the "phoneme-mouth shape ID" index table, each phoneme is mapped to a basic mouth shape in the mouth shape mapping library. For example, phoneme "a" → mouth shape ID01 (open mouth wide), phoneme "sh" → mouth shape ID15 (lips and teeth close together). Mouth shape parameter variants that match the pronunciation duration of the phoneme are selected first (e.g., long vowel "i" corresponds to a longer mouth shape duration, short vowel "b" corresponds to a rapid mouth shape switch) to avoid mouth shape and pronunciation rhythm being out of sync.

[0129] For basic lip shape differences between adjacent phonemes (e.g., "a" → "o"), 1-2 frames of transitional lip shapes are automatically inserted (based on interpolation calculations of basic lip shape parameters, such as a linear decrease in lip opening from 100 to 60) to avoid abrupt lip shape transitions. Transitional lip shape generation is achieved through a target interpolation operator (PaddlePaddleops.interpolate), reducing computation time and ensuring real-time streaming processing. Each lip shape frame (basic lip shape + transitional lip shape) is assigned a timestamp consistent with the speech frame, and timing deviations are corrected using the DTW algorithm in the audio-video synchronization calibration module. The final generated lip shape sequence frame rate is aligned with the speech sampling rate (48kHz corresponds to 30fps lip shape frames), ensuring complete synchronization between lip shape changes and speech pronunciation rhythm. Combining question-and-answer text semantics and speech prosodic features, action templates are dynamically selected and their timing adjusted to ensure coordination between actions, speech, and lip shapes.

[0130] A lightweight parser (trained on MindSpore) extracts the semantic type (question, statement, instruction, reassurance), sentence length (short sentences < 5 characters, long sentences > 15 characters), and keywords (such as "click", "query", "apply") of the question-and-answer text. Based on the semantic parsing results, an appropriate action is selected from the action template library. For example: long statement ("The government service process you consulted requires 3 working days") → head looking straight ahead + hands spreading out in explanation gesture (the duration of the action is consistent with the duration of the sentence pronunciation); instruction keywords ("click", "view") → pointing gesture to the screen (the start time of the action is synchronized with the pronunciation time of the keyword). Selection logic: if there is no explicit appropriate action in the text, the default selection is "slight head turning left and right" (once every 2 seconds, with an amplitude of ±10°) to simulate the state of a real person listening and avoid the digital human being being static and stiff.

[0131] By using the time scaling operator of the target system, which is implemented based on PaddlePaddle.ops.interpolat or MindSpore.nn.Resize, parallel computing optimization is performed on the NPU of the target system to ensure that the keyframe rhythm remains unchanged during duration stretching / compression. The total duration of the action template is adjusted to be consistent with the duration of the speech signal. For example: if the speech duration is 1.2 seconds, the action template is extended from 800ms to 1.2 seconds (the keyframe time is stretched proportionally). The prosodic features of the speech signal (speech rate, pauses, intonation) are extracted to adjust the rhythm of the action. For example: fast speech rate (>180 words / minute) → reduced action amplitude and faster switching; speech pause (≥150ms) → the action maintains the current posture and avoids redundancy of actions during the pause; increased intonation (at the end of interrogative sentences) → the head is slightly raised at the end of the action to enhance the interactive appeal.

[0132] Action start time calibration ensures that keyframes of actions (such as raising a gesture or turning the head) are synchronized with the position of speech accents. For example, the speech accent is at "apply" (250ms-300ms) → the keyframe for raising a gesture corresponds to 270ms. The timestamps of action data are aligned with the timestamps of speech and lip movements, and deviations are corrected through dynamic time warping (e.g., if the action is delayed by 10ms, the entire action is moved forward by 10ms), ultimately reducing the synchronization error between action and speech. If the user inputs continuously (multiple rounds of question and answer), the ending frame of the previous round of action is automatically spliced ​​with the starting frame of the next round of action, and transition frames (1-2 frames) are used to avoid action breaks and ensure smooth interaction. The generated sequence of action parameters (bone rotation angle, displacement) is compressed into INT16 format (originally FLOAT32), and further compressed using the target system's compression algorithm (LZ4) to reduce the transmission and storage pressure on the NPU. Action generation and lip movement sequence generation are processed in parallel by the target system's inference engine through multi-threading (CPU / NPU collaboration), reducing the total time consumption and not affecting the overall question and answer response latency within 200ms.

[0133] In a preferred embodiment, step S62 specifically includes:

[0134] S71. Based on the timing of the synchronized speech signal, determine the phoneme sequence corresponding to the currently processed speech segment and the start and end times of each phoneme.

[0135] S72. Based on the phoneme sequence and the start and end times of each phoneme, and combined with the acoustic characteristics of the synchronous speech signal, query the preset lip-sync mapping library to determine the target lip-sync data corresponding to each phoneme.

[0136] S73. Based on the start and end times of each phoneme, perform real-time interpolation calculations between the target lip shape data of adjacent phonemes to generate a lip shape sequence that is time-aligned with the synchronous speech signal.

[0137] In this embodiment, a phoneme is the smallest unit of articulation that constitutes speech, and each phoneme corresponds to a target vocal tract configuration, i.e., a reference lip shape. Therefore, using phonemes as the basic unit for lip shape driving is the most direct and controllable way to ensure lip shape accuracy. By obtaining phoneme-level forced alignment information, i.e., the precise boundaries (start and end times) of each phoneme on the time axis, the known phoneme duration prediction results generated upstream when generating the target segment of speech are directly reused, ensuring the authority and zero latency of the temporal information. The essence of the lip shape mapping library is a function from discrete symbols (phonemes) and continuous features (acoustic context) to a continuous lip shape parameter space. Simply relying on phoneme symbols for static mapping (e.g., phoneme / a / -> fixed lip shape A) cannot reflect the coarticulation phenomenon, i.e., the lip shape changes caused by preceding and following phonemes.

[0138] Therefore, by querying the current phoneme and its real-time acoustic features (such as MFCC, which reflects the instantaneous vocal tract shape), context-aware dynamic lip-sync can be achieved, making the generated lip shape closer to the actual pronunciation of the current speech. Lip-sync mapping models typically output target lip shape data at the phoneme center or a fixed frequency (e.g., every 10ms), but this does not match the higher, more uniform frame rate (e.g., 60 frames per second) required for graphics rendering, and the lip shape data will jump when phonemes switch. The purpose of real-time interpolation is: first, upsampling, adapting low-frequency lip shape control signals to high-frequency rendering frame rates; second, smoothing, generating natural transition animations between target lip shapes of adjacent phonemes, simulating the continuity and inertia of oral muscle movements, and avoiding visually abrupt changes.

[0139] In a preferred embodiment, step S63 specifically includes:

[0140] S81. Perform semantic analysis on the question-and-answer text results and extract action-related semantic features.

[0141] S82. Perform prosodic analysis on the synchronized speech signal and extract prosodic features.

[0142] S83. Based on the action-related semantic features, the prosodic features, and the preset emotional state, retrieve and match the corresponding candidate action templates from the preset action template library.

[0143] S84. Based on the temporal sequence of the synchronized speech signal, the candidate action templates are spatiotemporally arranged and fused to generate a limb action sequence corresponding to the synchronized speech signal.

[0144] In this embodiment, body movements do not occur randomly, but are influenced by the synergistic effects of language content (what is said), expression mode (how it is said), and the speaker's state (what emotion is used to express it). Therefore, control features need to be extracted from three independent but related data sources: semantic features (i.e., what action is performed), which are semantic units that can trigger or associate with the target action, such as interrogative words, enumeration items, emphasis words, and spatial descriptive words, from the question-and-answer text results, determining the basic type and intention of the action; prosodic features (i.e., when to perform the action), which are fundamental frequency contours, energy envelopes, and pause information extracted from synchronous speech signals, with stress points, intonation transitions, and phrase boundaries being the preferred times for the action to occur, and prosodic features determining the rhythm, starting point, and intensity changes of the action; and emotional state (i.e., how the action is expressed), where preset emotional labels (such as neutral, pleasant, positive, and authoritative) provide stylized modulation parameters for the action, affecting the kinematic characteristics of the action, such as amplitude, speed, and fluency.

[0145] Generating physically accurate body movements from scratch requires extremely high computational costs and complex kinematic models. This is addressed by using a pre-defined movement template library. Each template in the library is a pre-made movement fragment with rich metadata tags, such as semantic tags, applicable prosodic patterns, and emotional styles. The most suitable candidate movement templates are retrieved from the library using composite query conditions based on multi-source features. The retrieved templates are independent fragments, which are arranged, trimmed, and blended in real-time animation according to a global timeline (synchronous speech signal timing), and transition issues between templates are resolved to form a continuous, abrupt final movement sequence.

[0146] As a preferred embodiment, upon system startup, the current hardware and software environment is automatically detected through the adaptation layer, specifically including:

[0147] S91. Based on the detected hardware and software environment, dynamically load the corresponding driver interfaces, system adaptation components, and framework compatible components from the resource library.

[0148] S92. Utilize the loaded driver interface, system adaptation components, and framework compatible components to complete the adaptation and initialization of the system operating environment.

[0149] In this embodiment, the system must accurately identify the hardware architecture, accelerator type, operating system kernel, and core system library version of the current operating environment during the startup phase. This is achieved through layered detection and feature extraction, including identifying the CPU instruction set, accelerator type, and driver version at the hardware level; and identifying the operating system distribution, kernel version, GLIBC version, and critical dependency libraries at the software level.

[0150] By constructing a versioned and hierarchical resource repository, pre-compiled or pre-configured components for different environment combinations are stored. These include: driver interfaces, which encapsulate low-level operations for different computing hardware, such as a unified adaptation layer for Hygon's ROCm / HIP interface and Phytium's math library interface; system adaptation components, which handle operating system differences, such as the differences between Kylin and UnionTech UOS in system calls, security modules, file system paths, and process scheduling strategies; and framework compatibility components, which solve compatibility issues between different deep learning frameworks (such as Baidu PaddlePaddle and PyTorch) in model format, operator definition, and memory management. After loading the components, the system needs to complete runtime binding and state initialization, establish a mapping from the unified API to the specific implementation, and configure the hardware and software environment to the optimal state.

[0151] Specifically, the system obtains precise hardware identifiers by calling standardized hardware query interfaces to read system files and execute vendor-specific query commands; it then parses operating system distribution files, checks kernel versions, dynamic link library dependencies, and the deep learning framework runtime environment to complete software environment identification; and encodes the detection results into structured environment descriptors to generate environment fingerprints. The structured environment descriptors are used as index keys. An example of the resource library directory structure is: / lib / adaptors / aarch64 / kylin / ascend / mindspore / ; each directory contains the dynamic link libraries, configuration files, kernel binaries, etc., required by the target environment. By constructing an intelligent loader, the resource library is retrieved based on the environment fingerprint, an optimal matching algorithm is used to locate the most suitable component set, and the target components are loaded through dynamic linking or plugin mechanisms.

[0152] All loaded components must conform to predefined abstract interfaces. For example, the computation driver interface uniformly provides standard function pointers such as `mem_alloc`, `kernel_launch`, and `stream_sync`. The loaded driver interface initialization function is called to establish a communication context with the target accelerator and load the corresponding computation graph compiler. Based on the loaded system adaptation components, memory page configuration is adjusted, CPU affinity is set, and security module callback functions of the target operating system are mounted. Through framework-compatible components, the model computation graph is converted into an intermediate representation of the target framework, and custom operators are registered. All component version compatibility is checked, and lightweight functional testing is performed to ensure the adapted system is in a stable and usable state.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A digital human question-and-answer system, characterized in that, The system includes: The voice receiving and processing module is used to receive user voice input in real time, split the input voice, extract the acoustic features of each frame of voice in real time, and cache the acoustic features of consecutive frames to form a streaming data queue. Forming a streaming data queue specifically includes: S21. Classify and normalize the Mel frequency cepstral coefficients of the speech signal, output the MFCC feature vector for each frame of speech, and create a circular buffer in the system shared memory area. The storage unit of the circular buffer is a tuple of frame features and timestamp. S22. Generate timestamps for the MFCC feature vectors of each frame, forming tuples. Insert the tuples into the circular buffer in real time in ascending order of timestamps through atomic operations. When the buffer is full, overwrite the earliest storage unit according to the first-in-first-out rule. S23. Monitor the effective data length of the circular buffer. When the preset trigger condition is met, read all consecutive effective frame feature sequences in the buffer to form a streaming data queue. S24. After reading, the read storage unit is marked and cleared, and a cyclic redundancy check is performed before enqueuing the streaming data into the queue. If the check fails, the data is discarded. The speech feature inference module is used to input the streaming data queue into the pre-trained question answering model and output the question answering text results and speech synthesis instructions in real time; The digital human synchronization driver module is used to generate synchronized speech signals based on question-and-answer text results and speech synthesis instructions, map the synchronized speech signals to the lip-shape mapping library and the action template library, generate lip-shape sequences and body action sequences that match the speech timing, and complete the digital human broadcast; The digital human synchronization driver module includes: S61. Receive the question-and-answer text result and the speech synthesis instruction, and generate a synchronous speech signal carrying timing information based on the speech synthesis instruction; S62. By analyzing the acoustic features of the synchronized speech signal in real time, mapping and interpolating them to the lip shape mapping library, a lip shape sequence is obtained. S63. By integrating question-and-answer text results, synchronized voice signals, and preset emotional states, the results are mapped and arranged into the action template library to obtain a sequence of body movements. S64. Align and fuse the lip-sync sequence and body movement sequence according to the timing of the synchronized speech signal to drive the digital human model to complete the broadcast.

2. The digital human question-and-answer system according to claim 1, characterized in that, Real-time extraction of acoustic features for each frame of speech, specifically including: S11. After removing the DC offset in the input speech using a first-order differential filtering algorithm, a Hanning window is applied to obtain the windowed speech signal. S12. Convert the windowed speech signal into a frequency domain signal to obtain a complex spectrum. Take the square of the modulus of the complex spectrum to obtain the power spectrum. S13. Set up several triangular Mel filters according to preset rules, and perform convolution operation between the power spectrum and the Mel filter bank through the vector multiplication operator to obtain the Mel spectrum; S14. The Mel spectrum is transformed by natural logarithm to obtain the Mel logarithmic spectrum. The Mel logarithmic spectrum is then transformed by discrete cosine transform and feature selection DCT to output the Mel frequency cepstral coefficients. S15. Perform intra-frame normalization on the Mel frequency cepstral coefficients and output the MFCC feature vector for each frame of speech.

3. The digital human question-and-answer system according to claim 1, characterized in that, The construction of a pre-trained question-answering model specifically includes: S31. Obtain text corpus and question-answer pair data in the target domain to form a domain training dataset; S32. Based on the neural network architecture, construct a basic language model, input the domain training dataset into the basic language model for domain training, and obtain a domain question answering model; S33. Perform streaming input / output adaptation and lightweight processing of preset rules on the domain question answering model to obtain the question answering model.

4. The digital human question-and-answer system according to claim 1, characterized in that, The generation of question-and-answer text results specifically includes: S41. The question-answering model generates an initial candidate text set based on the features of the starting frame of the received streaming data queue. S42. When a new preset frame is added, the existing encoding state is reused to dynamically update the initial candidate text set, including adding candidate texts and eliminating candidate texts with low confidence. S43. Repeat step S42 until the preset text output termination condition is met, and determine the output text from the final candidate text set according to the preset filtering rules.

5. The digital human question-and-answer system according to claim 4, characterized in that, The generation of speech synthesis instructions specifically includes: S51. Receive the text output by the question-answering model, parse it, and output the prosody control parameters corresponding to the text; S52. Generate speech synthesis instructions based on the output text and prosody control parameters.

6. The digital human question-and-answer system according to claim 1, characterized in that, Step S62 specifically includes: S71. Based on the timing of the synchronized speech signal, determine the phoneme sequence corresponding to the currently processed speech segment and the start and end times of each phoneme. S72. Based on the phoneme sequence and the start and end times of each phoneme, and combined with the acoustic characteristics of the synchronous speech signal, query the preset lip-shape mapping library to determine the target lip-shape data corresponding to each phoneme. S73. Based on the start and end times of each phoneme, perform real-time interpolation calculations between the target lip shape data of adjacent phonemes to generate a lip shape sequence that is time-aligned with the synchronous speech signal.

7. The digital human question-and-answer system according to claim 1, characterized in that, Step S63 specifically includes: S81. Perform semantic analysis on the question-and-answer text results and extract action-related semantic features; S82. Perform prosodic analysis on the synchronized speech signal and extract prosodic features; S83. Based on action-related semantic features, the prosodic features, and preset emotional states, retrieve and match corresponding candidate action templates from a preset action template library; S84. Based on the temporal sequence of the synchronized speech signal, the candidate action templates are spatiotemporally arranged and fused to generate a limb action sequence corresponding to the synchronized speech signal.

8. The digital human question-and-answer system according to claim 1, characterized in that, When the system starts up, the adaptation layer automatically detects the current hardware and software environment, specifically including: S91. Based on the detected hardware and software environment, dynamically load the corresponding driver interfaces, system adaptation components, and framework compatible components from the resource library. S92. Utilize the loaded driver interface, system adaptation components, and framework compatible components to complete the adaptation and initialization of the system operating environment.

Images