Customer service interaction methods and systems for digital human lip-syncing.

Abstract

This invention discloses a customer service interaction method and system for lip-sync driving of digital humans, relating to the field of data processing. The method includes: real-time data acquisition based on customer service interaction scenarios, generating real-time speech stream data, and extracting multimodal features to obtain a speech feature tensor; constructing a lip-sync key point sequence based on the tensor; performing deformation driving on the digital human facial mesh according to the key point sequence to generate a dynamic lip flow that drives the digital human to perform lip movements, obtaining real-time interactive feedback data; and feeding the feedback data back to the lip-sync key point sequence for online parameter fine-tuning, generating lip movement compensation parameters to perform interactive compensation for the digital human's lip movements. This application solves the problems of poor lip movement coherence, lack of consistency with speech rhythm and emotion, and large lip-sync deviation in existing digital human lip-sync driving methods, achieving the effects of improving lip movement coherence, ensuring lip shape matches speech rhythm and emotion, and reducing lip-sync deviation.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a customer service interaction method and system for digital human lip-syncing. Background Technology

[0002] The lip-sync effect of digital humans in customer service interactions is a key factor in determining the realism of intelligent customer service interactions and improving user experience. Current mainstream technologies for driving lip-sync in digital humans mostly extract basic speech features and match them with a limited set of preset lip-shape templates, or generate lip movements based on simple phoneme-lip shape mapping relationships. Existing methods generally suffer from insufficient mining of multi-dimensional speech features and incomplete consideration of lip shape constraints, resulting in a lack of coherence in the generated lip movements, a mismatch with the rhythm and emotional expression of the speech, and significant deviations in lip-sync.

[0003] Currently, the lip-sync driving technology for digital human customer service suffers from technical problems such as poor continuity of lip movements, lack of consistency with speech rhythm and emotion, and large deviation in lip-sync. Summary of the Invention

[0004] This application provides a customer service interaction method and system for lip-sync driving of digital humans. It employs real-time acquisition of speech streams in customer service interaction scenarios, performing multimodal feature extraction and tensor transformation to obtain speech feature tensors. These tensors are then input into a lip-sync driving model to generate a sequence of lip-shape key points matching the speech. Based on this key point sequence, deformation driving is applied to the digital human's facial mesh to generate a dynamic lip flow, driving the digital human to complete synchronized lip movements. Simultaneously, real-time interactive feedback data is collected, and the lip-shape key point sequence is fine-tuned online using this feedback data. Compensation parameters are generated for real-time interactive optimization of lip movements. These techniques solve the technical problems of poor lip movement coherence, lack of synchronization with speech rhythm and emotion, and large lip-sync deviation in existing digital human customer service lip-sync driving methods. The system achieves the technical effects of improving lip movement coherence, ensuring high synchronization between lip movements and speech rhythm and emotion, and reducing lip-sync deviation.

[0005] This application provides a customer service interaction method for lip-sync driving of digital humans, comprising: real-time data acquisition based on customer service interaction scenarios, generating real-time voice stream data for multimodal feature extraction, and obtaining a voice feature tensor; lip-sync driving based on the voice feature tensor to construct a lip-shape key point sequence; deformation driving of the digital human facial mesh according to the lip-shape key point sequence to generate a lip dynamic stream driving the digital human to perform lip movements matching customer service responses, and obtaining real-time interaction feedback data; and online parameter fine-tuning based on the real-time interaction feedback data fed back to the lip-shape key point sequence to generate lip movement compensation parameters for interaction compensation of the digital human's lip movements.

[0006] In a possible implementation, real-time data is collected based on customer service interaction scenarios to generate real-time voice stream data for multimodal feature extraction, obtaining a voice feature tensor. The following processing is then performed: voice activity detection is performed based on the real-time voice stream data, voice detection results are generated, background noise is suppressed, and multiple clean voice segments are read; phoneme decoding is performed on the multiple clean voice segments to generate a phoneme sequence; acoustic feature analysis is performed on the multiple clean voice segments to extract acoustic prosodic features; emotion classification is performed based on the multiple clean voice segments, and emotion category probability distribution data is calculated; according to the phoneme sequence, the acoustic prosodic features and the emotion category probability distribution data are concatenated to obtain concatenated feature pairs; tensor transformation is performed on the concatenated feature pairs to construct the voice feature tensor.

[0007] In a possible implementation, the multiple clean speech segments are traversed for phoneme decoding to generate a phoneme sequence, and the following processes are performed: the multiple clean speech segments are traversed for equal-length segmentation to generate multiple short time frames; cepstral calculation is performed based on the multiple short time frames to obtain cepstral coefficient features; Viterbi decoding is performed according to the cepstral coefficient features to construct an initial phoneme sequence; and time boundary labeling is performed on the initial phoneme sequence to generate the phoneme sequence.

[0008] In a possible implementation, lip-syncing is driven based on the speech feature tensor to construct a lip-sync keypoint sequence, and the following processes are performed: a lip-syncing driving model is constructed; the speech feature tensor is temporally modeled using the lip-syncing driving model to generate a temporal sub-model; the speech feature tensor is spatially topologically modeled using the lip-syncing driving model to generate a spatial topological sub-model; mapping analysis is performed based on the temporal sub-model and the spatial topological sub-model to construct a speech-lip-syncing mapping relationship; decoding is performed according to the speech-lip-syncing mapping relationship, and multiple facial feature points are analyzed for temporal trajectory changes according to three-dimensional spatial coordinates to generate the lip-sync keypoint sequence.

[0009] In a possible implementation, the construction process of the lip-sync driving model involves the following steps: audio generation analysis is performed based on the speech feature tensor to construct an audio feature encoder; deep audio extraction is performed using the audio feature encoder to determine audio embedding data; temporal dependency analysis is performed based on the audio embedding data to capture speech signal context information; temporal modeling is performed based on the speech signal context information to generate a temporal modeling network; geometric co-modeling is performed using lip-sync keypoints to construct a spatial topology modeling network; the temporal modeling network and the spatial topology modeling network are mapped according to the lip-sync keypoints to construct a motion decoder; and the temporal modeling network, the spatial topology modeling network, and the motion decoder are integrated to construct the lip-sync driving model.

[0010] In a possible implementation, temporal modeling is performed based on the context information of the speech signal to generate a temporal modeling network, and the following processes are performed: multi-scale analysis is performed based on the context information of the speech signal to extract multi-scale temporal features; parallel convolution is performed based on the multi-scale temporal features to capture multiple speech features, including short-time phoneme features and long-time prosodic features; temporal analysis is performed based on the short-time phoneme features to construct a first temporal feature map; temporal analysis is performed based on the long-time prosodic features to construct a second temporal feature map; and modeling fusion is performed based on the first temporal feature map and the second temporal feature map to construct the temporal modeling network.

[0011] In a possible implementation, lip shape key points are introduced for geometric co-modeling to construct a spatial topology modeling network. The following processes are performed: a 3D facial scan is performed based on a digital human to construct a facial topology map of the digital human; feature propagation is performed by traversing the facial topology map to determine multiple facial key points; the multiple facial key points are aggregated to capture the spatial dependencies of the key points; multi-layer convolutional iteration is performed based on the multiple facial key points to determine lip shape key points; the lip shape key points are combined and co-modeled according to the spatial dependencies of the key points to construct the spatial topology modeling network.

[0012] In a possible implementation, the digital human's facial mesh is deformed according to the lip shape key point sequence to generate a lip dynamic flow that drives the digital human to perform lip movements matching the customer service response, thereby obtaining real-time interactive feedback data. The following processing is then performed: an expression basis set is introduced into the digital human; the facial topology map of the digital human is fused with the expression basis set to construct an expression solver; the lip shape key point sequence is synchronized to the expression solver for analysis to generate expression basis weight coefficients; the expression basis set is fused and calculated according to the expression basis weight coefficients to construct a lip deformation network for temporal smoothing, generating a lip dynamic flow; the customer service response voice stream is retrieved, and the lip dynamic flow and the customer service response voice stream are rendered synchronously to construct a driving signal; the driving signal drives the digital human to perform lip movements for real-time acquisition, generating the real-time interactive feedback data.

[0013] In a possible implementation, the customer service response voice stream is retrieved, and the lip dynamics stream is synchronously rendered with the customer service response voice stream to construct a driving signal. This driving signal then drives the digital human to perform lip movements for real-time acquisition, generating the real-time interactive feedback data. The following processing is then performed: The digital human's response is analyzed based on the speech feature tensor to generate a customer service response voice stream; the customer service response voice stream and the lip dynamics stream are time-axis aligned to construct a lip-phonetic synchronization mapping relationship; bidirectional driving analysis is performed according to the lip-phonetic synchronization mapping relationship to generate a composite driving signal, which includes an audio driving component and a lip-phonetic driving component; the audio driving component is sent to an audio rendering module to render the digital human, generating customer service response voice data; the lip-phonetic driving component is sent to a graphics rendering module to render the digital human, generating digital human lip position data; and the digital human is driven to perform lip movements for real-time acquisition based on the customer service response voice data and the digital human lip position data, generating the real-time interactive feedback data.

[0014] This application also provides a customer service interaction system for lip-sync driving of digital humans, including: a voice data feature extraction module, used to collect real-time data based on customer service interaction scenarios, generate real-time voice stream data for multimodal feature extraction, and obtain a voice feature tensor; a lip-sync key point sequence construction module, used to drive lip-sync based on the voice feature tensor and construct a lip-sync key point sequence; a real-time interaction feedback module, used to drive the deformation of the digital human facial mesh according to the lip-sync key point sequence, generate a dynamic lip flow to drive the digital human to perform lip movements that match the customer service response, and obtain real-time interaction feedback data; and a lip-sync interaction compensation module, used to fine-tune parameters online based on the real-time interaction feedback data fed back to the lip-sync key point sequence, and generate lip movement compensation parameters to perform interaction compensation for the digital human's lip shape.

[0015] The proposed customer service interaction method and system for lip-sync driving of digital humans, as described in this application, firstly involves real-time data acquisition based on customer service interaction scenarios. This generates real-time speech stream data for multimodal feature extraction, yielding a speech feature tensor. Next, lip-sync driving is performed based on this speech feature tensor to construct a lip-sync keypoint sequence. Then, deformation driving is applied to the digital human's facial mesh according to this lip-sync keypoint sequence, generating a dynamic lip flow that drives the digital human to perform lip movements matching the customer service response, obtaining real-time interactive feedback data. Finally, based on this real-time interactive feedback data, online parameter fine-tuning is performed on the lip-sync keypoint sequence to generate lip movement compensation parameters for interactive compensation of the digital human's lip movements. Through this process, the proposed method and system achieve the technical effects of improving the coherence of lip movements, ensuring a high degree of consistency between lip movements and speech rhythm and emotion, and reducing lip-sync deviation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1 This is a flowchart illustrating a customer service interaction method for digital human lip-syncing driving, provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the structure of a customer service interaction system for digital human lip-syncing driving provided in an embodiment of this application.

[0019] Figure labeling: 10 for speech data feature extraction module, 20 for lip shape key point sequence construction module, 30 for real-time interactive feedback module, and 40 for lip shape interaction compensation module. Detailed Implementation

[0020] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0021] This application provides a customer service interaction method for digital human lip-syncing, such as... Figure 1 As shown, the method includes:

[0022] Step S100: Real-time data collection is performed based on customer service interaction scenarios to generate real-time voice stream data for multimodal feature extraction, thereby obtaining voice feature tensors.

[0023] Specifically, a high-definition microphone array is used as the acquisition device to collect voice data in real time in a customer service scenario, generating single-channel real-time voice stream data containing all audio information of the conversation between customer service and user. A multimodal feature extraction process is initiated, using Python combined with the Librosa audio processing library for feature extraction. First, the real-time voice stream data undergoes pre-emphasis, framing, and windowing processing. Then, phoneme-related features, acoustic prosodic features, and emotional features are extracted sequentially. Finally, all extracted features are concatenated in a fixed order, and through tensor transformation, a voice feature tensor is obtained, which serves as the input to the lip-sync driving model.

[0024] In one possible implementation, real-time data acquisition is performed based on customer service interaction scenarios to generate real-time voice stream data for multimodal feature extraction, obtaining a voice feature tensor. Step S100 further includes step S110, performing voice activity detection based on the real-time voice stream data, generating voice detection results, performing background noise suppression, and reading multiple clean voice segments. Specifically, the WebRTC VAD voice activity detection algorithm is used for voice activity detection, setting a detection threshold, for example, 0.5. Values ​​greater than 0.5 are considered valid voice, and values ​​less than or equal to 0.5 are considered silence or noise. Frame-by-frame detection is performed on the real-time voice stream data to generate voice detection results, marking each frame as valid or invalid voice. The Wiener filtering algorithm is used for background noise suppression. Through invalid voice frames in the voice detection results, the spectral features of background noise are extracted to establish a noise model, and then the valid voice frames are filtered to remove noise components. Based on the speech detection results, consecutive valid speech frames are spliced ​​into multiple clean speech segments. The duration of each clean speech segment is controlled between 0.5 and 2 seconds. Segments shorter than 0.5 seconds are merged into adjacent segments, and segments longer than 2 seconds are split into multiple segments.

[0025] Step S120: The multiple clean speech segments are traversed for phoneme decoding to generate a phoneme sequence. Specifically, the Kaldi speech recognition toolkit is used for phoneme decoding. A pre-trained phoneme decoding model based on a Hidden Markov Model (HMM) is loaded. Each clean speech segment is traversed, and pre-emphasis, framing, and windowing are performed on each segment. Cepstral coefficient features are extracted and input into the phoneme decoding model. The model uses the Viterbi decoding algorithm to select the phoneme sequence with the highest probability from all possible phoneme combinations. The sequence length corresponds to the number of frames in the clean speech segment. The generated phoneme sequence is deduplicated to remove consecutively repeated identical phonemes, resulting in a standard phoneme sequence, which serves as the basis for lip-sync matching.

[0026] Step S130: Based on the multiple clean speech segments, perform acoustic feature analysis to extract acoustic prosodic features. Specifically, use the Librosa audio processing library for acoustic feature analysis, traversing each clean speech segment and sequentially extracting four acoustic prosodic features: fundamental frequency, speech rate, volume amplitude, and spectral bandwidth. The fundamental frequency is extracted using the YIN algorithm, with an extraction range of 50-500Hz. One fundamental frequency value is calculated for each frame, and the average fundamental frequency within the frame is taken as the fundamental frequency feature for that frame. Speech rate is calculated by statistically analyzing the ratio of the number of phonemes to the duration of each clean speech segment. Volume amplitude is obtained by calculating the root mean square of the speech signal in each frame, with one amplitude data point per frame. Spectral bandwidth is obtained by calculating the standard deviation of the spectrum in each frame, reflecting the degree of spectral dispersion. The four features are concatenated frame by frame to form the acoustic prosodic feature vector for each frame. The feature vectors of all frames are arranged in chronological order to form the acoustic prosodic features of the clean speech segment, influencing the amplitude and rhythm of lip movements.

[0027] Step S140: Perform emotion classification based on the multiple clean speech segments and calculate the probability distribution data of emotion categories. Specifically, a CNN-LSTM hybrid model is used for emotion classification. The model is pre-trained using a publicly available customer service voice emotion dataset to identify the emotional tendencies in clean speech segments and determine the emotion category conveyed by the speech, such as calmness, patience, or urgency. Each clean speech segment is traversed, and Mel-frequency cepstral coefficient features are extracted and input into the trained emotion classification model. The model extracts local features through convolutional layers, captures temporal features through LSTM layers, and finally outputs probability values ​​for different emotion categories through a Softmax activation function. Each emotion category corresponds to a probability, and the sum of all probabilities is 1, forming the emotion category probability distribution data for that clean speech segment, which is used to adjust the emotional adaptability of lip movements.

[0028] Step S150: According to the phoneme sequence, the acoustic prosodic features and the emotion category probability distribution data are concatenated to obtain concatenated feature pairs. Specifically, the phoneme sequence is encoded using one-hot encoding, mapping each phoneme to a one-hot vector, forming a phoneme feature matrix. The acoustic prosodic features are standardized using Z-score standardization, mapping feature values ​​to between 0 and 1, resulting in a standardized acoustic prosodic feature matrix. The emotion category probability distribution data is copied according to the time step to form an emotion feature matrix. The three matrices are column-concatenated in the order of phoneme feature matrix - acoustic prosodic feature matrix - emotion feature matrix. The feature vector at each time step is a concatenated feature, and all concatenated features are arranged in chronological order to form concatenated feature pairs.

[0029] Step S160: Perform tensor transformation on the cascaded feature pairs to construct the speech feature tensor. Specifically, the PyTorch framework is used for tensor transformation, converting the cascaded feature pairs into tensor data types in PyTorch. The tensor dimensions are adjusted using the view function, adding batch dimensions to convert the feature matrix into a three-dimensional tensor. The tensor is then normalized, mapping the values ​​of all elements to the range of 0-1. A min-max normalization algorithm is used, where each element is subtracted from its minimum value and then divided by the difference between its maximum and minimum values, resulting in the final speech feature tensor that meets the input requirements of the lip-sync driving model.

[0030] In one possible implementation, the multiple clean speech segments are traversed for phoneme decoding to generate a phoneme sequence. Step S120 further includes step S121, traversing the multiple clean speech segments for equal-length segmentation to generate multiple short-time frames. Specifically, the Python Librosa library is used for equal-length segmentation. Each clean speech segment is traversed, and the frame length and frame shift are set. The clean speech segments are segmented at fixed time intervals to obtain multiple short-time speech segments of the same length. Before segmentation, the clean speech segments are padded with zeros. If the end of a segment is less than one frame length, it is padded with zeros to one frame length to ensure that all short-time frames have the same length after segmentation. The number of sampling points in each short-time frame is the sampling rate × frame length.

[0031] Step S122: Perform cepstral calculation based on the multiple short-time frames to obtain cepstral coefficient features. Specifically, the cepstral calculation is performed using the Python SciPy library. The following steps are executed sequentially for each short-time frame: 1. Pre-emphasize the short-time frame using the following formula: y(n) = x(n) - pre-emphasis coefficient × x(n-1), where x(n) is the nth sampling point of the short-time frame, x(n-1) is the (n-1)th sampling point of the short-time frame, and y(n) is the value after pre-emphasis; 2. Apply a Hanning window to the pre-emphasized short-time frame, with the window length consistent with the frame length, to reduce spectral leakage; 3. Perform a fast Fourier transform on the windowed short-time frame to obtain spectral data; 4. Take the absolute value of the spectral data and then calculate the logarithm to obtain the logarithmic spectrum; 5. Perform an inverse fast Fourier transform on the logarithmic spectrum to obtain cepstral data; 6. Extract the first 13 coefficients of the cepstral data as the cepstral coefficient features of the short-time frame. The cepstral coefficient features of all short frames are arranged in chronological order to form a cepstral coefficient feature matrix, which is used for phoneme decoding.

[0032] Step S123: Perform Viterbi decoding according to the cepstral coefficient features to construct the initial phoneme sequence. Specifically, use the Viterbi decoding algorithm from the Kaldi toolkit, load the pre-trained HMM-GMM phoneme model, and take the cepstral coefficient feature matrix as input. The model first performs GMM probability calculation on the cepstral coefficient features of each short-time frame to obtain the probability that the frame belongs to the HMM state of each phoneme. Through the Viterbi algorithm, find the state path with the highest probability. The phoneme sequence corresponding to this path is the initial phoneme sequence. The length of the initial phoneme sequence is consistent with the number of short-time frames, and each short-time frame corresponds to one phoneme.

[0033] Step S124: Traverse the initial phoneme sequence and perform time boundary labeling to generate the phoneme sequence. Specifically, Python code is used to traverse the initial phoneme sequence. First, deduplication is performed, merging consecutively repeated identical phonemes into one phoneme, and recording the start and end short-time frame indices corresponding to each merged phoneme. Based on the frame shift time of the short-time frames, the start and end times of each phoneme are calculated: start time = start short-time frame index × frame shift time, end time = end short-time frame index × frame shift time + frame length. Each phoneme is bound to its corresponding start and end times and arranged in chronological order to generate a standard phoneme sequence.

[0034] Step S200: Based on the speech feature tensor, perform lip-sync driving to construct a lip-sync key point sequence.

[0035] Specifically, a pre-trained lip-sync driving model is loaded, and the speech feature tensor is used as the model input. The model calculates lip movement data synchronized with speech pronunciation to drive the lip movements of the digital human. The model first extracts audio embedding data through an audio feature encoder, then captures the temporal dependencies of speech through a temporal modeling network, captures facial spatial relationships through a spatial topology modeling network, and finally maps the temporal and spatial features to lip keypoint coordinates through a motion decoder, outputting a lip keypoint sequence. The sequence length is consistent with the time step of the speech feature tensor, and each time step contains the three-dimensional spatial coordinates of the lip keypoints of the digital human, corresponding to the lip shape at that time step.

[0036] In one possible implementation, lip-sync synchronization is driven based on the speech feature tensor to construct a lip-sync keypoint sequence. Step S200 further includes step S210, constructing a lip-sync synchronization driving model, and performing temporal modeling on the speech feature tensor through the lip-sync synchronization driving model to generate a temporal sub-model. Specifically, the lip-sync synchronization driving model is built based on the PyTorch framework to realize the mapping from the speech feature tensor to the lip-sync keypoint sequence. The overall model adopts an encoder-decoder structure, including four core modules: an audio feature encoder, a temporal modeling network, a spatial topology modeling network, and a motion decoder. The speech feature tensor is input to the audio feature encoder to extract audio embedding data. Then, the audio embedding data is input to the temporal modeling network to capture the contextual dependencies of the speech signal. The network output is a temporal feature, and a temporal sub-model is constructed based on this temporal feature. The input of the temporal sub-model is the audio embedding data, and the output speech signal is the temporal feature, which is used to capture the temporal correlation of lip movements.

[0037] Step S220: Spatial topological modeling is performed on the speech feature tensor using the lip-sync driving model to generate a spatial topological sub-model. Specifically, spatial topological modeling is based on a digital facial topology map. First, a pre-constructed digital facial topology map is loaded, and the facial topology map is converted into an adjacency matrix. The adjacency matrix is ​​concatenated with the audio embedding data of the speech feature tensor and input into the spatial topology modeling network. The network captures the spatial dependencies of facial key points through graph convolution operations and outputs the spatial features of lip-sync key points. Based on these spatial features, a spatial topological sub-model is constructed. The input of the spatial topological sub-model is the adjacency matrix and the audio embedding data, and the output is the spatial features of lip-sync key points, used to ensure the rationality of lip movements.

[0038] Step S230: Based on the temporal sub-model and the spatial topology sub-model, a mapping analysis is performed to construct a speech-lip-shape mapping relationship. Specifically, an attention mechanism is used for mapping analysis. First, the temporal features output by the temporal sub-model and the spatial features output by the spatial topology sub-model are extracted. The temporal features are used as the query vector Q, and the spatial features are used as the key vector K and value vector V. The attention weights of the temporal and spatial features are calculated using the scaling dot product attention formula. The association mapping between the temporal and spatial features is realized through the attention weights, that is, the temporal feature (speech-related) at each time step corresponds to the spatial feature (lip-shape-related) at one time step. This correspondence is organized into a mapping table, which contains the spatial coordinate range of the lip-shape keypoints corresponding to each speech feature, forming a speech-lip-shape mapping relationship.

[0039] Step S240: Decode the speech-lip-shape mapping relationship according to the given information, construct multiple facial feature points, and perform temporal trajectory change analysis based on three-dimensional spatial coordinates to generate the lip-shape key point sequence. Specifically, input the speech-lip-shape mapping relationship into a motion decoder. The decoder decodes the mapping relationship and outputs the three-dimensional coordinates of the lip-shape key points at each time step. Perform temporal trajectory change analysis on the output coordinate data. Use the Python Matplotlib library to plot the trajectory of the x, y, and z coordinates of each lip-shape key point changing over time. Analyze the smoothness of the trajectory, correct any unsmooth trajectories, and ensure the smoothness of the lip-shape movements. Arrange the corrected lip-shape key point coordinates of each time step in chronological order to generate the lip-shape key point sequence.

[0040] In one possible implementation, the construction process of the lip-sync driving model, step S210, further includes step S211, which involves performing audio generation analysis based on the speech feature tensor to construct an audio feature encoder. Specifically, using the speech feature tensor as the core input, the tensor is first analyzed for dimensionality to clarify core parameters such as time step and feature dimension. Then, the encoder structure is designed based on the audio characteristics of the customer service voice scenario. The encoder adopts a combination architecture of a 3-layer one-dimensional convolutional neural network and a 2-layer fully connected layer. The convolutional kernel sizes are set to 3, 5, and 7, respectively, with a stride of 1 for each layer. Local temporal features and global semantic features in the speech feature tensor are extracted through layer-by-layer convolution. The fully connected layer is responsible for feature dimensionality reduction and nonlinear mapping, compressing the high-dimensional features output by the convolution. At the same time, batch normalization and ReLU activation functions are introduced to alleviate the gradient vanishing problem. Finally, an audio feature encoder adapted to customer service voice is constructed to convert the original speech feature tensor into high-dimensional, highly recognizable audio embedding features.

[0041] Step S212 involves deep audio extraction using the audio feature encoder to determine the audio embedding data. Specifically, the standardized speech feature tensor is input into the constructed audio feature encoder, which sequentially passes through convolutional layers, activation layers, pooling layers, and fully connected layers. The encoder extracts deep information such as phoneme association features, prosodic features, and emotional features of the speech layer by layer. The output feature vectors are arranged in a matrix according to time steps. This matrix is ​​the audio embedding data, containing the core temporal and semantic information of the speech, and serves as the basic input for temporal and spatial modeling.

[0042] Step S213: Perform temporal dependency analysis based on the audio embedding data to capture the contextual information of the speech signal. Specifically, a bidirectional long short-term memory (Bi-LSTM) network is used to perform temporal dependency analysis on the audio embedding data. The audio embedding data is input into the Bi-LSTM network frame by frame according to time steps. The forward LSTM captures the positive contextual information of the speech from the start to the end of the time sequence, and the backward LSTM captures the negative contextual information from the end to the start of the time sequence. By concatenating the outputs of the forward and backward LSTMs, a contextual feature vector for each time step is obtained. This vector integrates the speech information of the current frame and the frames before and after it, thus fully capturing the short-term and long-term temporal dependencies of the speech signal.

[0043] Step S214: Perform temporal modeling based on the contextual information of the speech signal to generate a temporal modeling network. Specifically, using the contextual feature vector output by Bi-LSTM as input, a multi-scale temporal modeling network is constructed to separate and enhance the short-term phoneme features and long-term prosodic features of the speech. The final generated temporal modeling network can accurately capture the temporal features of speech at different time scales, and the output temporal feature matrix directly reflects the temporal rhythm and change pattern of lip movements.

[0044] Step S215 involves introducing lip shape key points for geometric co-modeling to construct a spatial topology modeling network. Specifically, by combining the physical topology of the digital face, the audio embedding data and the spatial geometric features of the lip shape key points are co-modeled to establish the correlation between speech features and the spatial position of the lip shape. The spatial feature matrix output by the final generated spatial topology modeling network can reflect the spatial dependency and positional constraints of each lip shape key point.

[0045] Step S216: The temporal modeling network and the spatial topology modeling network are mapped to lip shape keypoints to construct a motion decoder. Specifically, the motion decoder adopts a decoding architecture of temporal features + spatial features to coordinate mapping. First, the temporal feature matrix output by the temporal modeling network and the spatial feature matrix output by the spatial topology modeling network are aligned in dimension. The temporal features are then mapped to a dimension matching the number of keypoints through a fully connected layer, and then concatenated with the spatial features point by point to form a fused feature. Subsequently, through 3 deconvolutional layers and 1 linear output layer, the fused feature is mapped to the three-dimensional coordinates of each lip shape keypoint at each time step, and a coordinate matrix is ​​output, completing the mapping from features to the spatial position of the lip shape.

[0046] Step S217: Integrate the temporal modeling network, the spatial topology modeling network, and the motion decoder to construct the lip-sync synchronization driving model. Specifically, based on the PyTorch framework, the audio feature encoder, temporal modeling network, spatial topology modeling network, and motion decoder are connected in a sequence of input → encoding → temporal modeling → spatial modeling → decoding → output. Define the forward propagation function of the model: the speech feature tensor is first input into the audio feature encoder to obtain audio embedding data; the audio embedding data is split into two paths, one path is input into the temporal modeling network to obtain temporal features, and the other path is combined with the facial topology map and input into the spatial topology modeling network to obtain spatial features; the temporal features and spatial features are input into the motion decoder, outputting a three-dimensional coordinate sequence of lip-sync key points. At the same time, a loss function and optimizer are added to the model to complete the overall integration of the model and form an end-to-end lip-sync synchronization driving model.

[0047] In one possible implementation, temporal modeling is performed based on the context information of the speech signal to generate a temporal modeling network. Step S214 further includes step S2141, which involves performing multi-scale analysis based on the context information of the speech signal to extract multi-scale temporal features. Specifically, multi-scale sliding window analysis is performed on the context feature vector output by Bi-LSTM, setting two windows of different scales: a small window corresponds to short-term features, capturing rapid changes at the phoneme level; a large window corresponds to long-term features, capturing changes at the prosodic level. Mean pooling and max pooling are performed on the feature vector within each window, and the pooling results are concatenated to obtain the multi-scale feature vector for each time step, achieving comprehensive extraction of short-term and long-term temporal features of the speech.

[0048] Step S2142: Perform parallel convolution based on the multi-scale temporal features to capture multiple speech features, including short-term phoneme features and long-term prosodic features. Specifically, construct two parallel one-dimensional convolutional neural network branches: the first branch targets short-term phoneme features, setting the convolution kernel size to 3, the stride to 1, and the activation function to LeakyReLU, focusing on capturing rapidly changing phoneme features; the second branch targets long-term prosodic features, setting the convolution kernel size to 15, the stride to 3, and the activation function to ReLU, focusing on capturing prosodic variation features. The multi-scale temporal features are simultaneously input into both branches; the first branch outputs a short-term phoneme feature matrix, and the second branch outputs a long-term prosodic feature matrix.

[0049] Step S2143: Perform temporal analysis based on the short-time phoneme features to construct a first temporal feature map. Specifically, input the short-time phoneme feature matrix into a single-layer unidirectional LSTM. The LSTM processes the features frame by frame according to time steps, capturing the continuous change patterns of phonemes within a short period of time, and outputs a temporal feature vector for each time step. Arrange this vector according to the time step and feature dimension to form the first temporal feature map. This feature map directly corresponds to the rapid and subtle movements of the lips, such as the opening and closing of the lips and teeth, and the position of the tongue tip.

[0050] Step S2144: Perform temporal analysis based on the long-term prosodic features to construct a second temporal feature map. Specifically, the long-term prosodic feature matrix is ​​input into a single-layer gated recurrent unit (GRU). The GRU focuses on capturing long-distance temporal dependencies and outputs a prosodic temporal feature vector for each time step. This vector is arranged according to the time step and feature dimension to form the second temporal feature map. This feature map corresponds to the overall amplitude and rhythmic changes of lip movements, such as a large lip opening and closing amplitude when the volume is high, and a high lip movement frequency when the speech speed is fast.

[0051] Step S2145: Based on the first temporal feature map and the second temporal feature map, modeling and fusion are performed to construct the temporal modeling network. Specifically, an attention fusion mechanism is used to fuse the two temporal feature maps: the first temporal feature map is used as the query vector Q, and the second temporal feature map is used as the key vector K and value vector V. The attention weight for each time step is calculated using the scaling dot product attention formula. The weight reflects the degree of correlation between short-term phoneme features and long-term prosodic features. The attention-weighted features are added element-wise to the original two feature maps, and then dimensionality is reduced through a fully connected layer to finally form the output of the temporal modeling network, completing the fusion modeling of short-term and long-term temporal features.

[0052] In one possible implementation, lip-shaped key points are introduced for geometric co-modeling to construct a spatial topology modeling network. Step S215 further includes step S2151, which involves performing a 3D facial scan of the digital human to construct a facial topology map. Specifically, a structured light 3D scanner is used to perform a high-precision scan of the digital human's face to obtain 3D point cloud data of the facial surface. The point cloud data is converted into a triangular mesh model using a Poisson reconstruction algorithm, and then the mesh is partitioned based on facial anatomical features to mark the core facial key points. Using key points as nodes and the physical connections between key points as edges, an adjacency matrix is ​​constructed, ultimately forming a digital human facial topology map that includes spatial location and topological connection relationships.

[0053] Step S2152 involves traversing the facial topology map to perform feature propagation, identifying multiple facial key points, and aggregating these key points to capture their spatial dependencies. Specifically, based on the adjacency matrix of the facial topology map, graph convolution is used for feature propagation: the audio embedding data is mapped to a dimension matching the number of key points, serving as the initial features for the key points; features are propagated layer by layer through two layers of graph convolution. During feature propagation, the features of each key point are fused with the features of its neighboring key points, ultimately outputting a facial key point feature matrix. This matrix captures the spatial dependencies between key points, such as the position of the corner of the mouth key point being constrained by the upper and lower lip key points.

[0054] Step S2153: Based on the multiple facial key points, perform multi-layer convolutional iterations to determine the lip shape key points. Specifically, construct three stacked graph convolutional layers for multi-layer convolutional iterations: the first graph convolution focuses on global facial features, extracting global spatial features of all key points; the second graph convolution uses a masking operation to retain only the key point features of the perilip region, narrowing the scope of attention; the third graph convolution further focuses on the lip shape key points, concentrating weights on the lip shape key points through an attention mask, outputting a lip shape key point feature matrix, and extracting the spatial features of the lip shape key points.

[0055] Step S2154 involves combining and collaboratively modeling the lip-shaped key points according to their spatial dependencies to construct the spatial topology modeling network. Specifically, the lip-shaped key point feature matrix and the adjacency matrix of the facial topology graph retaining only the lip-shaped key points are input again into two layers of graph convolution. Each layer of graph convolution focuses on modeling the spatial collaborative relationships between the lip-shaped key points. The output lip-shaped key point spatial feature matrix is ​​integrated with the graph convolutional network structure, adjacency matrix, and feature propagation logic to form a complete spatial topology modeling network. This network ensures that the output lip-shaped key point coordinates conform to the physical topological constraints of the digital human face, avoiding lip movements that do not conform to physiological structures.

[0056] Step S300: Deformation driving of the digital human facial mesh is performed according to the lip shape key point sequence to generate lip dynamic flow to drive the digital human to perform lip shape actions that match the customer service response, and obtain real-time interactive feedback data.

[0057] Specifically, using the lip shape keypoint sequence as the core driving force, combined with the digital human facial topology map and expression base set, the system drives real-time deformation of the lip region of the digital human facial mesh through processes such as expression calculation, weight fusion, temporal smoothing, and lip-phonetic synchronization rendering. This ensures that lip movements are completely synchronized with the pronunciation, rhythm, and emotion of the customer service response speech. Simultaneously, real-time execution data of the digital human's lip movements, such as keypoint coordinate errors, deformation frame rates, and lip-phonetic synchronization deviations, are collected to form real-time interactive feedback data for fine-tuning of model parameters.

[0058] In one possible implementation, the digital human's facial mesh is deformed according to the lip shape keypoint sequence to generate a dynamic lip flow that drives the digital human to perform lip movements matching customer service responses, obtaining real-time interactive feedback data. Step S300 further includes step S310, introducing the digital human's expression base set, and fusing it with the facial topology map of the digital human to construct an expression solver. Specifically, the expression base set contains multiple preset core lip expression bases of the digital human, such as closed lips, open lips, pouting lips, wide lips, round lips, etc., and each expression base corresponds to the deformation weight of the facial mesh. Based on the facial topology map, the spatial position of each expression base in the expression base set is associated with the lip shape keypoints to construct a mapping table between expression bases and keypoint coordinates. The expression solver uses a linear hybrid skinning algorithm as its core architecture, integrating the mapping table, facial topology map, and expression base set to convert the target coordinates of the lip shape keypoints into corresponding expression base weight coefficients, realizing the calculation from keypoint coordinates to facial mesh deformation.

[0059] Step S320: The lip shape keypoint sequence is synchronized to the expression solver for analysis to generate expression base weight coefficients. Specifically, the lip shape keypoint sequence is input into the expression solver frame by frame according to time steps. The solver first compares the target coordinates of the lip shape keypoints at each time step with the reference coordinates corresponding to the expression base, and calculates the contribution weight of each expression base using the least squares method, so that the linear combination of multiple expression bases can approximate the target coordinates. The output expression base weight coefficients are a matrix, where each value represents the fusion ratio of the corresponding expression base at that time step, and the sum of the weight coefficients is 1 to ensure the rationality of expression fusion.

[0060] Step S330: The expression base set is fused according to the expression base weight coefficients to construct a lip deformation network for temporal smoothing, generating a lip dynamic flow. Specifically, the expression base set is linearly fused step by step, with the lip mesh deformation at each time step = Σ(expression base i × weight coefficient i), resulting in an initial lip deformation sequence. A lip deformation network is constructed to temporally smooth the initial deformation sequence: a one-dimensional convolutional neural network layer filters high-frequency noise in the deformation sequence, and a moving average layer eliminates abrupt changes between adjacent frames, making the lip movement transition smoother. The final output lip dynamic flow is a matrix containing the three-dimensional coordinates of all mesh vertices of the digital human's lips at each time step, reflecting the continuous and smooth deformation process of the lips.

[0061] Step S340: Retrieve the customer service response voice stream, synchronously render the lip movement dynamics stream and the customer service response voice stream to construct a driving signal, and use the driving signal to drive the digital human to perform lip movements for real-time acquisition, generating the real-time interactive feedback data. Specifically, the lip movement dynamics stream and the customer service response voice stream are time-axis aligned and synchronously rendered to generate a composite signal that drives the digital human to perform lip movements, and execution data is collected to obtain real-time interactive feedback data.

[0062] In one possible implementation, the customer service response voice stream is retrieved, and the lip movement dynamics stream is rendered synchronously with the customer service response voice stream to construct a driving signal. This driving signal drives the digital human to perform lip movements for real-time acquisition, generating the real-time interactive feedback data. Step S340 further includes step S341, which analyzes the digital human's response based on the speech feature tensor to generate a customer service response voice stream. Specifically, the speech feature tensor is input into a pre-trained TTS (Text-to-Speech) model. The TTS model first reconstructs the text content, prosody, and emotional features of the customer service response from the speech feature tensor, and then generates a customer service response voice stream that matches the original speech features based on the digital human's speech timbre library. The duration, speed, and emotion of the voice stream are consistent with the original customer service interaction voice.

[0063] Step S342 involves aligning the customer service response speech stream with the lip movement dynamics stream along the time axis to construct a lip-phoneme synchronization mapping relationship. Specifically, using the frame timestamps of the speech stream as a reference, the time steps of the lip movement dynamics stream are mapped one-to-one with the speech frames. First, the phoneme time boundaries of the speech stream are extracted. Then, each time step of the lip movement dynamics stream is mapped to the start / end timestamp of the phoneme. The time step interval of the lip movement dynamics stream is adjusted by linear interpolation to ensure that the start / end time of the lip movements is completely synchronized with the speech pronunciation. The final constructed lip-phoneme synchronization mapping table includes the lip movement dynamics stream time step index and synchronization error threshold corresponding to each speech frame.

[0064] Step S343: Perform bidirectional drive analysis according to the stated lip-sync mapping relationship to generate a composite drive signal, which includes an audio drive component and a lip-sync drive component. Specifically, the bidirectional drive analysis verifies synchronicity simultaneously from both audio and lip-sync dimensions: in the audio dimension, it verifies whether the playback time of the speech frame matches the lip movement time; in the lip-sync dimension, it verifies whether the amplitude of the lip movement matches the speech volume / prosody. Based on the verification results, a composite drive signal is generated: the audio drive component includes the speech stream playback command, volume adjustment parameters, and frame rate parameters; the lip-sync drive component includes the coordinate data of the lip dynamic flow, deformation smoothing parameters, and synchronization compensation parameters. The two components are bound by timestamps and executed synchronously.

[0065] Step S344: The audio driving component is sent to the audio rendering module to render the digital human and generate customer service response voice data. Specifically, after receiving the audio driving component, the audio rendering module first decodes the voice stream, then adjusts the voice volume according to the volume adjustment parameters, controls the voice playback speed according to the frame rate parameters, and finally generates customer service response voice data that conforms to the timbre characteristics of the digital human, and outputs the real-time status of voice playback.

[0066] Step S345: The lip-shape driving component is sent to the graphics rendering module to render the digital human, generating lip position data. Specifically, the graphics rendering module, based on the digital human facial mesh model, receives coordinate data and deformation parameters from the lip-shape driving component. Using a GPU-accelerated mesh deformation algorithm, it updates the 3D coordinates of the lip mesh vertices in real time, rendering the real-time shape of the digital human's lips. Simultaneously, it collects data such as the coordinates of key lip points, the number of mesh triangles, and the rendering frame rate for each rendering frame to generate lip position data, reflecting the actual execution effect of the lip-shape movement.

[0067] Step S346: Real-time acquisition of lip movements is performed by driving the digital human to execute lip-shape movements based on the customer service response voice data and the digital human lip position data, generating the real-time interactive feedback data. Specifically, the voice data output by the audio rendering module and the lip position data output by the graphics rendering module are synchronously sent to the digital human driving engine. The engine synchronously plays the voice and drives lip-shape movements according to the timestamp. Simultaneously, the real-time acquisition module is activated to collect data such as lip-tone synchronization deviation (the time difference between lip movements and speech pronunciation in each frame), lip-shape keypoint coordinate error (the mean square error between predicted coordinates and actual rendered coordinates), system performance data (rendering frame rate, CPU / GPU utilization), and emotion matching degree (the score of the fit between lip-shape movement amplitude and voice emotion). This data is organized by time step to form structured real-time interactive feedback data, including fields such as timestamp, deviation value, error value, and performance indicators.

[0068] Step S400: Based on the real-time interactive feedback data fed back to the lip shape key point sequence, online parameter fine-tuning is performed to generate lip shape motion compensation parameters for interactive compensation of the digital human's lip shape.

[0069] Specifically, anomaly analysis is performed on real-time interactive feedback data, setting thresholds such as lip-sync deviation > 10ms and keypoint coordinate error > 0.5mm to filter out abnormal time steps exceeding these thresholds. For abnormal time steps, compensation parameters are calculated. The time compensation parameter adjusts the timestamp of the lip-shape keypoint sequence based on the lip-sync deviation, shifting it forward / backward by the corresponding number of milliseconds. The position compensation parameter calculates the keypoint offset based on the coordinate error and corrects the coordinate values. The amplitude compensation parameter adjusts the amplitude coefficient of the lip-shape movement based on the emotional matching degree. These compensation parameters are integrated into the lip-shape keypoint sequence according to the time step, generating a corrected keypoint sequence, which is fed back to the digital human driving engine in real time to adjust the generation logic of the lip dynamic flow. Simultaneously, the compensation parameters are used as fine-tuning signals to update some weights of the lip-shape synchronization driving model. Online fine-tuning is performed using mini-batch gradient descent to ensure that the subsequently generated lip-shape keypoint sequences are more accurate, gradually reducing outliers in the interactive feedback data, achieving real-time interactive compensation for digital human lip-shape movements, and improving the accuracy and naturalness of lip-sync.

[0070] This application employs a method of real-time acquisition of speech streams in customer service interaction scenarios, followed by multimodal feature extraction and tensor transformation to obtain speech feature tensors. These tensors are then input into a lip-sync driving model to generate a sequence of lip-shape key points that match the speech. Based on this key point sequence, the digital human's facial mesh is deformed to generate a dynamic lip flow and drive the digital human to complete synchronized lip movements. Simultaneously, real-time interactive feedback data is collected, and the lip-shape key point sequence is fine-tuned online using this feedback data. Compensation parameters are then generated to optimize the real-time interaction of lip movements. This approach solves the technical problems of poor lip movement continuity, lack of harmony with speech rhythm and emotion, and large lip-sync deviation in existing digital human customer service lip-sync driving models. It achieves the technical effects of improving lip movement continuity, ensuring high harmony between lip movements and speech rhythm and emotion, and reducing lip-sync deviation.

[0071] In the above text, refer to Figure 1 A customer service interaction method for digital human lip-sync synchronization driving according to an embodiment of the present invention is described in detail. Next, reference will be made to... Figure 2 A customer service interaction system for lip-syncing driving of a digital human is described according to an embodiment of the present invention.

[0072] The customer service interaction system for lip-sync driving of digital humans according to embodiments of the present invention addresses the technical problems of poor lip movement continuity, lack of coordination with speech rhythm and emotion, and large lip-sync deviation in existing digital human customer service lip-sync driving systems. It achieves the technical effects of improving lip movement continuity, ensuring high coordination between lip movements and speech rhythm and emotion, and reducing lip-sync deviation. The customer service interaction system for lip-sync driving of digital humans includes: a speech data feature extraction module 10, a lip-sync key point sequence construction module 20, a real-time interactive feedback module 30, and a lip-sync interaction compensation module 40.

[0073] The speech data feature extraction module 10 is used to collect real-time data based on customer service interaction scenarios, generate real-time speech stream data for multimodal feature extraction, and obtain a speech feature tensor; the lip shape key point sequence construction module 20 is used to drive lip shape synchronization based on the speech feature tensor and construct a lip shape key point sequence; the real-time interaction feedback module 30 is used to drive the deformation of the digital human face mesh according to the lip shape key point sequence, generate a lip dynamic flow to drive the digital human to perform lip shape actions that match the customer service response, and obtain real-time interaction feedback data; the lip shape interaction compensation module 40 is used to perform online parameter fine-tuning based on the real-time interaction feedback data fed back to the lip shape key point sequence, generate lip shape action compensation parameters to perform interaction compensation for the digital human's lip shape.

[0074] The detailed description of the specific configuration of the voice data feature extraction module 10 is as follows: As mentioned above, based on real-time data collection in a customer service interaction scenario, real-time voice stream data is generated for multimodal feature extraction to obtain a voice feature tensor. The voice data feature extraction module 10 may further include: a voice activity detection unit for performing voice activity detection based on the real-time voice stream data, generating voice detection results for background noise suppression, and reading multiple clean voice segments; a phoneme decoding unit for traversing the multiple clean voice segments for phoneme decoding to generate a phoneme sequence; an acoustic feature analysis unit for performing acoustic feature analysis based on the multiple clean voice segments to extract acoustic prosodic features; an emotion classification unit for performing emotion classification based on the multiple clean voice segments and calculating emotion category probability distribution data; a feature concatenation unit for concatenating the acoustic prosodic features and the emotion category probability distribution data according to the phoneme sequence to obtain concatenated feature pairs; and a tensor transformation unit for performing tensor transformation on the concatenated feature pairs to construct the voice feature tensor.

[0075] The process involves traversing multiple clean speech segments to perform phoneme decoding and generate a phoneme sequence. The phoneme decoding unit may further include: an equal-length segmentation subunit for traversing the multiple clean speech segments to perform equal-length segmentation and generate multiple short-time frames; a cepstral calculation subunit for performing cepstral calculation based on the multiple short-time frames to obtain cepstral coefficient features; a Viterbi decoding subunit for performing Viterbi decoding according to the cepstral coefficient features to construct an initial phoneme sequence; and a time boundary labeling subunit for traversing the initial phoneme sequence to perform time boundary labeling and generate the phoneme sequence.

[0076] The detailed configuration of the lip-shape key point sequence construction module 20 is explained as follows: As mentioned above, based on the speech feature tensor, lip-shape synchronization is driven to construct a lip-shape key point sequence. The lip-shape key point sequence construction module 20 may further include: a temporal modeling unit for constructing a lip-shape synchronization driving model, performing temporal modeling on the speech feature tensor through the lip-shape synchronization driving model to generate a temporal sub-model; a spatial topology modeling unit for performing spatial topology modeling on the speech feature tensor through the lip-shape synchronization driving model to generate a spatial topology sub-model; a mapping analysis unit for performing mapping analysis based on the temporal sub-model and the spatial topology sub-model to construct a speech-lip-shape mapping relationship; and a time trajectory change analysis unit for decoding according to the speech-lip-shape mapping relationship, constructing multiple facial feature points for time trajectory change analysis according to three-dimensional spatial coordinates, and generating the lip-shape key point sequence.

[0077] The construction process of the lip-sync driving model may further include the following: an audio generation and analysis subunit for performing audio generation and analysis based on the speech feature tensor to construct an audio feature encoder; a deep audio extraction subunit for performing deep audio extraction through the audio feature encoder to determine the audio embedding data; a temporal dependency analysis subunit for performing temporal dependency analysis based on the audio embedding data to capture speech signal context information; a temporal modeling subunit for performing temporal modeling based on the speech signal context information to generate a temporal modeling network; a geometric co-modeling subunit for introducing lip-sync keypoints for geometric co-modeling to construct a spatial topology modeling network; a displacement mapping subunit for performing displacement mapping on the temporal modeling network and the spatial topology modeling network according to the lip-sync keypoints to construct a motion decoder; and an integration subunit for integrating the temporal modeling network, the spatial topology modeling network, and the motion decoder to construct the lip-sync driving model.

[0078] The process involves performing temporal modeling based on the context information of the speech signal to generate a temporal modeling network. The temporal modeling subunit may further include: a multi-scale analysis component for performing multi-scale analysis based on the speech signal context information to extract multi-scale temporal features; a parallel convolution component for performing parallel convolution based on the multi-scale temporal features to capture multiple speech features, including short-term phoneme features and long-term prosodic features; a temporal analysis component for performing temporal analysis based on the short-term phoneme features to construct a first temporal feature map; and a temporal analysis component for performing temporal analysis based on the long-term prosodic features to construct a second temporal feature map; and a modeling fusion component for performing modeling fusion based on the first and second temporal feature maps to construct the temporal modeling network.

[0079] Specifically, the process involves introducing lip shape key points for geometric co-modeling to construct a spatial topology modeling network. The geometric co-modeling sub-unit can further include: a facial 3D scanning component for performing facial 3D scanning based on a digital human to construct a facial topology map; a feature propagation component for traversing the facial topology map to perform feature propagation, determining multiple facial key points, aggregating these key points, and capturing their spatial dependencies; a multi-layer convolution iteration component for performing multi-layer convolution iteration based on the multiple facial key points to determine lip shape key points; and a combined co-modeling component for combining the lip shape key points according to their spatial dependencies to construct the spatial topology modeling network.

[0080] The detailed description of the specific configuration of the real-time interactive feedback module 30 is as follows: As mentioned above, the digital human facial mesh is deformed according to the lip shape key point sequence to generate a lip dynamic flow that drives the digital human to perform lip movements matching the customer service response, thereby obtaining real-time interactive feedback data. The real-time interactive feedback module 30 may further include: an expression solver construction unit for introducing the digital human's expression base set, fusing the facial topology map of the digital human with the expression base set to construct an expression solver; an expression base weight coefficient generation unit for synchronizing the lip shape key point sequence to the expression solver for analysis to generate expression base weight coefficients; a fusion calculation unit for performing fusion calculation on the expression base set according to the expression base weight coefficients, constructing a lip deformation network for temporal smoothing processing, and generating a lip dynamic flow; and a synchronous rendering unit for retrieving the customer service response voice stream, synchronously rendering the lip dynamic flow and the customer service response voice stream to construct a driving signal, and driving the digital human to perform lip movements for real-time acquisition through the driving signal to generate the real-time interactive feedback data.

[0081] The process includes retrieving the customer service response voice stream, synchronously rendering the lip dynamics stream and the customer service response voice stream to construct a driving signal, and using the driving signal to drive the digital human to perform lip movements for real-time acquisition, generating the real-time interactive feedback data. The synchronous rendering unit may further include: a response analysis subunit for performing response analysis on the digital human based on the speech feature tensor to generate a customer service response voice stream; a time axis alignment subunit for aligning the customer service response voice stream and the lip dynamics stream on the time axis to construct a lip-phone synchronization mapping relationship; a bidirectional driving analysis subunit for performing bidirectional driving analysis according to the lip-phone synchronization mapping relationship to generate a composite driving signal, which includes an audio driving component and a lip-phone driving component; a rendering subunit for sending the audio driving component to an audio rendering module to render the digital human and generate customer service response voice data; sending the lip-phone driving component to a graphics rendering module to render the digital human and generate digital human lip position data; and a real-time lip movement acquisition subunit for driving the digital human to perform lip movements for real-time acquisition according to the customer service response voice data and the digital human lip position data to generate the real-time interactive feedback data.

[0082] The customer service interaction system for digital human lip-syncing provided in the embodiments of the present invention can execute the customer service interaction method for digital human lip-syncing provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0083] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A customer service interaction method for digital human lip-syncing, characterized in that, The method includes: Real-time data collection is performed based on customer service interaction scenarios to generate real-time voice stream data for multimodal feature extraction, thereby obtaining voice feature tensors. Based on the speech feature tensor, lip-syncing is driven to construct a sequence of lip-syncing key points. The digital human facial mesh is deformed according to the lip key point sequence to generate a dynamic lip flow that drives the digital human to perform lip movements that match the customer service response, thereby obtaining real-time interactive feedback data. Based on the real-time interactive feedback data, online parameter fine-tuning is performed on the lip shape key point sequence to generate lip shape motion compensation parameters for interactive compensation of the digital human lip shape. A lip-sync driving model is constructed, and the speech feature tensor is temporally modeled using the lip-sync driving model to generate a temporal sub-model. The lip-sync driving model is used to perform spatial topology modeling on the speech feature tensor to generate a spatial topology sub-model. Based on the temporal sub-model and the spatial topological sub-model, a mapping analysis is performed to construct a speech-lip shape mapping relationship; Decode the speech-lip shape mapping relationship, construct multiple facial feature points, perform temporal trajectory change analysis according to three-dimensional spatial coordinates, and generate the lip shape key point sequence. A digital human facial expression base set is introduced, and an expression solver is constructed by fusing the digital human's facial topology map with the facial expression base set. The lip shape key point sequence is synchronized to the expression solver for analysis to generate expression base weight coefficients; The expression basis set is fused and calculated according to the expression basis weight coefficients, and a lip deformation network is constructed for temporal smoothing to generate lip dynamic flow; The customer service response voice stream is retrieved, and the lip dynamics stream is rendered synchronously with the customer service response voice stream to construct a driving signal. The driving signal is used to drive the digital human to perform lip movements for real-time acquisition, generating the real-time interactive feedback data.

2. The customer service interaction method for digital human lip-syncing as described in claim 1, characterized in that, Based on customer service interaction scenarios, real-time data collection is performed to generate real-time voice stream data for multimodal feature extraction, obtaining voice feature tensors. The methods include: Based on the real-time speech stream data, speech activity detection is performed, speech detection results are generated, background noise is suppressed, and multiple clean speech segments are read. The multiple clean speech segments are traversed to perform phoneme decoding and generate a phoneme sequence; Acoustic feature analysis is performed on the multiple clean speech segments to extract acoustic prosodic features; Based on the multiple clean speech segments, emotion classification is performed, and the probability distribution data of emotion categories is calculated. According to the phoneme sequence, the acoustic prosodic features and the emotion category probability distribution data are concatenated to obtain concatenated feature pairs; The cascaded feature pairs are subjected to tensor transformation to construct the speech feature tensor.

3. The customer service interaction method for digital human lip-syncing as described in claim 2, characterized in that, The method involves iterating through multiple clean speech segments to perform phoneme decoding and generate a phoneme sequence, including: The multiple clean speech segments are traversed and segmented into equal-length segments to generate multiple short time frames; Based on the multiple short-time frames, cepstral calculations are performed to obtain cepstral coefficient features; Viterbi decoding is performed based on the cepstral coefficient features to construct an initial phoneme sequence; The initial phoneme sequence is traversed and time boundary markings are performed to generate the phoneme sequence.

4. The customer service interaction method for digital human lip-syncing as described in claim 1, characterized in that, The construction process and methods of the lip-sync driving model include: Based on the speech feature tensor, audio generation analysis is performed to construct an audio feature encoder; Deep audio extraction is performed using the audio feature encoder to determine the audio embedding data. Perform time-dependent analysis based on the audio embedding data to capture the contextual information of the speech signal; Temporal modeling is performed based on the contextual information of the speech signal to generate a temporal modeling network; We introduce lip-shaped key points for geometric co-modeling and construct a spatial topology modeling network. The temporal modeling network and the spatial topology modeling network are displacement-mapped according to lip key points to construct a motion decoder; The temporal modeling network, the spatial topology modeling network, and the motion decoder are integrated to construct the lip-sync driving model.

5. The customer service interaction method for digital human lip-syncing as described in claim 4, characterized in that, The method for generating a temporal modeling network based on the contextual information of the speech signal includes: Multi-scale analysis is performed based on the contextual information of the speech signal to extract multi-scale temporal features; Parallel convolution is performed based on the multi-scale temporal features to capture multiple speech features, including short-time phoneme features and long-time prosodic features. Based on the short-time phoneme features, a temporal analysis is performed to construct a first temporal feature map; Based on the long-term prosodic features, a temporal analysis is performed to construct a second temporal feature map; The time-series modeling network is constructed by modeling and fusing the first and second time-series feature maps.

6. The customer service interaction method for digital human lip-syncing as described in claim 4, characterized in that, Introducing lip-shaped key points for geometric co-modeling and constructing a spatial topology modeling network, the methods include: Based on digital human facial 3D scanning, construct a facial topology map of the digital human; The facial topology is traversed to perform feature propagation, multiple facial key points are identified, and the multiple facial key points are aggregated to capture the spatial dependencies of the key points. Multi-layer convolution iteration is performed based on the multiple facial key points to determine the lip shape key points; The lip-shaped key points are combined and collaboratively modeled according to their spatial dependencies to construct the spatial topology modeling network.

7. The customer service interaction method for digital human lip-syncing as described in claim 1, characterized in that, The method includes: retrieving the customer service response voice stream, synchronously rendering the lip movement dynamics stream with the customer service response voice stream, constructing a driving signal, and using the driving signal to drive the digital human to perform lip movements for real-time acquisition, generating the real-time interactive feedback data. Based on the aforementioned speech feature tensor, the digital human's response is analyzed to generate a customer service response speech stream; The customer service response voice stream and the lip movement dynamic stream are aligned on the time axis to construct a lip-sound synchronization mapping relationship; Bidirectional drive analysis is performed according to the lip-sync mapping relationship to generate a composite drive signal, which includes an audio drive component and a lip-sync drive component. The audio driving component is sent to the audio rendering module to render the digital human and generate customer service response voice data; The lip shape driving component is sent to the graphics rendering module to render the digital human and generate lip position data of the digital human. The digital human is driven to perform lip movements in real time based on the customer service response voice data and the digital human lip position data, and the real-time interactive feedback data is generated.

8. A customer service interaction system for digital human lip-syncing, characterized in that, The system is used to implement the customer service interaction method for digital human lip-syncing drive as described in any one of claims 1-7, the system comprising: The voice data feature extraction module is used to collect real-time data based on customer service interaction scenarios, generate real-time voice stream data for multimodal feature extraction, and obtain voice feature tensors. The lip shape key point sequence construction module is used to construct a lip shape key point sequence based on the speech feature tensor for lip shape synchronization driving. The real-time interactive feedback module is used to drive the deformation of the digital human facial mesh according to the lip shape key point sequence, generate a dynamic flow of lips to drive the digital human to perform lip shape actions that match the customer service response, and obtain real-time interactive feedback data. The lip shape interaction compensation module is used to perform online parameter fine-tuning based on the real-time interaction feedback data fed back to the lip shape key point sequence, and generate lip shape action compensation parameters to perform interactive compensation on the digital human's lip shape.

Images