Interactive digital human interaction method and system based on mobile terminal computing power

Abstract

This invention discloses an interactive digital human interaction method and system based on mobile terminal computing power, relating to the fields of mobile terminal-side multimodal interaction and real-time digital human rendering technology. It addresses time-varying mismatch issues in terminal-side digital human interaction, such as unstable audio-visual alignment, lip-sync asynchrony, and frame drops and stuttering caused by computing power fluctuations. The mobile terminal simultaneously acquires microphone audio and camera images, aligning them with a unified timestamp, and extracts acoustic or prosodic, facial expression, and head posture interaction features to form a feature package. Further, on the terminal side, speech recognition, intent classification, and slot extraction are performed, and emotional states are fused to generate interactive driving commands. Based on the commands, the terminal performs speech synthesis, lip-sync, facial expression, or head posture driving, and rendering output on the terminal side, while simultaneously monitoring and adaptively correcting computing power and time-varying mismatch to ensure stable and consistent interaction.

Description

Technical Field

[0001] This invention relates to the field of mobile terminal-side multimodal interaction and real-time rendering technology for digital humans, and more specifically, to an interactive digital human interaction method and system based on the computing power of mobile terminals. Background Technology

[0002] With the development of mobile internet and AIGC, interactive digital humans are widely used in customer service, companionship, tour guiding, and content creation. Existing solutions mostly rely on cloud inference and rendering, which, while easy to concentrate computing power, is prone to increased end-to-end latency, audio-visual asynchrony, and lip-sync lag when mobile networks fluctuate. At the same time, uploading sensitive data such as voice or face data to the cloud also brings privacy and compliance pressures. On the other hand, pure edge-side solutions are limited by the computing power, power consumption, and real-time scheduling of mobile terminals. The interaction link often exhibits a three-domain coupling time-varying mismatch: "blocky voice generation, uneven video frame intervals, and different update rhythms of driving parameters." The generation rhythm, driving rhythm, and display rhythm are split in stages, and in the highly significant mouth area, this is nonlinearly amplified into obvious audio-visual inconsistency, which is difficult to suppress stably by traditional fixed delay compensation or single threshold strategies.

[0003] To address the above problems, this invention proposes a solution. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide an interactive digital human interaction method and system based on the computing power of a mobile terminal to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In a preferred embodiment, it includes: During the interactive session, the mobile terminal simultaneously collects microphone audio sampling sequences and camera image frame sequences, and generates a terminal-side interactive feature packet bound to a second timestamp; The mobile terminal performs end-side intent parsing on the end-side interaction feature packet, generates an interaction-driven instruction with a reference timestamp field, and writes it into the timing buffer of the interaction session; The mobile terminal reads the interactive driving instructions, generates a voice output waveform sequence and a video output image frame sequence on the terminal side, and aligns and outputs them according to the reference timestamp field; The mobile terminal performs end-side monitoring, end-side state update, and end-side correction output generation for time-varying mismatch in three domains: voice output waveform sequence, video output image frame sequence, lip shape driving coefficient sequence, smooth expression coefficient, and smooth head posture. It analyzes video frame interval, audio block interval, cross-modal time offset residual, calculates phase drift and local saliency, outputs reproducible correction results, and writes them into the timing buffer of the interactive session.

[0007] In a preferred embodiment, the mobile terminal synchronously acquires microphone audio sampling sequences and camera image frame sequences during an interactive session. The audio sampling segments are marked with a first timestamp, and the image frames are marked with a second timestamp. Both the first and second timestamps are monotonically increasing timing values ​​from the same clock domain. The mobile terminal reads the sampling rate and audio window length parameters based on the second timestamp, determines the audio sampling set corresponding to the frame index, and captures the audio window. The terminal performs on-side noise reduction on the audio window to obtain the noise-reduced audio.

[0008] In a preferred embodiment, the mobile terminal performs key point detection on the face region image to obtain key points of the upper lip center, lower lip center, left eye corner, and right eye corner, and calculates the lip opening degree sequence. Simultaneously, it calculates the audio energy sequence on the noise-reduced frequency according to the audio sampling set, calculates the cross-correlation between the audio energy sequence and the lip opening degree sequence, and takes the relative delay corresponding to the maximum cross-correlation as the time offset correction amount, and corrects the subsequent audio window truncation benchmark accordingly. Under the alignment result, the mobile terminal binds and encapsulates the speech acoustic features, prosodic features, facial expression features, and head posture features with the second timestamp into a terminal interaction feature package. The terminal interaction feature package includes the second timestamp, the aligned audio window index range, and the face bounding box.

[0009] In a preferred embodiment, the mobile terminal calculates the semantic and interactive control-related structured results of the terminal-side interactive feature package locally, and forms an interactive driving instruction data structure with a reference timestamp field: starting from the speech acoustic features and prosodic features, the terminal-side speech recognition parameter resources are loaded to obtain the frame-by-frame symbol posterior probability distribution and decoded using beam search to output the recognized text. Then, the recognized text is normalized according to the terminal-side fixed text rule table to obtain normalized text. In the conversation context, the normalized text is used to retrieve the person memory index to obtain candidate memory fragments and select the first few fragments as the evidence set. The evidence set, person settings and dialogue security rules are input into the dialogue generation model to generate the response text and output the evidence citation identifier set and credibility score. At the same time, a person identifier field, evidence citation field, credibility field and security policy result field are added to the instruction and written into the response text.

[0010] Furthermore, the normalized text is classified by the execution end-side intent to obtain intent labels and their probabilities, and the execution end-side slots are extracted to obtain a semantic slot set. Based on prosodic features and facial expression features, statistics are calculated within a time window defined by the aggregation window length parameter to form an emotion input vector, and the emotion state parameters are output by the end-side emotion estimation model. Finally, the baseline timestamp, normalized text, intent labels and intent confidence, slot set, emotion state parameters, and expression coefficients and head posture parameters obtained by first-order exponential smoothing of facial expression features and head posture features are encapsulated into an interaction-driven instruction, and written into the temporal buffer of the interaction session incrementally according to the baseline timestamp, while retaining the corresponding feature package index.

[0011] In a preferred embodiment, during an interactive session, the mobile terminal reads the reference timestamp, text, emotion, and driving parameter fields from the interactive driving instructions. It uses the smoothed expression coefficients and smoothed head pose from the driving parameter fields as the facial and head driving inputs for the interactive digital human. In speech output generation, the mobile terminal loads the on-device speech synthesis parameter resources based on the text and emotion fields. It encodes the normalized text into a text embedding sequence and inputs it along with the emotion state parameters into an acoustic feature generation model to obtain an acoustic feature sequence. This sequence is then input into a vocoder model to generate a speech output waveform sequence, and a reference timestamp field consistent with the interactive driving instructions is appended to this waveform sequence. In video output generation, the mobile terminal loads digital human resources to obtain a neutral geometric representation and a situational variable basis. It obtains the current expression mesh based on a linear combination of the smoothed expression coefficients and applies the smoothed head pose to the head-related vertices or bones defined by the binding parameters to update the pose.

[0012] In a preferred embodiment, the Mel spectrum of the speech output waveform sequence is calculated and input into the audio to the lip-sync mapping model to obtain a lip-sync driving coefficient sequence aligned with the video frame index. The lip-sync driving coefficients and mouth deformation bases are then matched one-to-one according to the end-side fixed index table and superimposed onto the expression mesh to form an updated facial mesh. Subsequently, this mesh, head rigid body transformation and binding parameters are input into the end-side rendering process. Vertex projection is performed according to the rendering camera parameters and view transformation parameters, and a video output image frame sequence is generated according to the lighting and shading parameters. Finally, the mobile terminal maps the audio sampling index and video frame index to the same timeline based on the reference timestamp field, and schedules playback and display according to the end-side timing values ​​of the audio playback clock and screen refresh clock to output speech and video.

[0013] In a preferred embodiment, the mobile terminal performs end-side monitoring, end-side state updating, and end-side correction output generation for time-varying coupling mismatch between the speech output waveform sequence, video output image frame sequence, lip-shape driving coefficient sequence, smoothed expression coefficient, and smoothed head posture during the interactive session: a unified time index is established on the end side, the reference timestamp in the interactive driving instruction is read as the video time reference, the speech sampling index is mapped to the same time axis according to the sampling rate, and an audio sampling set aligned with the video frame index is defined; in the end-side monitoring, time variables with time indexes are calculated and recorded, including the video frame interval, the audio block interval calculated according to the fixed block length and the generated completion timing value, the cross-modal time offset residual obtained by cross-correlation of the speech energy sequence and the lip opening degree sequence in a sliding window, the calculated phase drift determined by the lip shape coefficient change amplitude and the effective update threshold, and the local salience composed of the frame difference energy and gray-level variance ratio of the mouth region and the non-mouth region.

[0014] In a preferred embodiment, during end-side state updates, speech continuity risk, video continuity risk, cross-modal mismatch risk, and salience amplification risk are defined as components of the state vector, and each component is updated according to a deterministic rule based on a trigger function, threshold parameters, and attenuation coefficients. During end-side correction output generation, when speech continuity risk is triggered, the splicing length is determined according to the state, and cross-fade-in / fade-out weights are generated. The two waveform segments at the splicing boundary are weighted and fused to obtain a corrected waveform. When video continuity risk is triggered, a correction pixel region is selected based on the salience amplification risk. The inter-frame motion field is estimated in this region, and the current frame and the remapped previous frame are fused according to the fusion coefficient to obtain the corrected waveform. When the risk of cross-modal mismatch is triggered in a positive frame, the phase correction amount is obtained by using the sign consistency operator based on the cross-modal offset residual and phase drift, and the lip-shape driving coefficient is resampled in time. Then, the corrected speech boundary jump metric, the frame difference energy of the corrected region, and the cross-modal offset residual are recalculated and compared with the target threshold. Under the conditions of meeting the upper limit of the number of repetitions, the minimum fusion coefficient, or the maximum correction amount limit, the splicing length, fusion coefficient, or lip-shape change amplitude constraint are adjusted according to deterministic rules and the correction is repeated. Finally, the time variable set, state variable, trigger judgment result, correction parameters, and the corrected index value are bound to the reference timestamp field and written into the timing buffer of the interactive session.

[0015] In a preferred embodiment, it includes: an edge-side acquisition alignment feature generation module, an edge-side intent parsing instruction generation module, an edge-side digital human-driven audiovisual generation module, a three-domain coupling mismatch monitoring and correction module, and signal connections between the modules. The edge-side acquisition alignment feature module is used by the mobile terminal to simultaneously acquire microphone audio sampling sequences and camera image frame sequences during an interactive session, and generate an edge-side interaction feature package bound to a second timestamp; The terminal-side intent parsing instruction generation module is used by the mobile terminal to perform terminal-side intent parsing on the terminal-side interaction feature packet, generate interaction-driven instructions with a reference timestamp field, and write them into the timing buffer of the interaction session. The edge-side digital human-driven audiovisual generation module is used by the mobile terminal to read interactive driving instructions, generate voice output waveform sequences and video output image frame sequences on the edge, and align and output them according to the reference timestamp field. The three-domain coupling mismatch monitoring and correction module is used by mobile terminals to perform end-side monitoring, end-side state updates, and end-side correction output generation for time-varying mismatch in three-domain coupling of voice output waveform sequence, video output image frame sequence, lip shape driving coefficient sequence, smooth expression coefficient, and smooth head posture. It analyzes video frame interval, audio block interval, cross-modal time offset residual, calculates phase drift and local saliency, outputs reproducible correction results, and writes them into the timing buffer of the interactive session.

[0016] The technical effects and advantages of this invention, which is based on the computing power of mobile terminals, of the interactive digital human interaction method and system are as follows: Compared to conventional edge-side digital human solutions, this invention uses a unified timestamp throughout the acquisition, parsing, synthesis, and rendering process. This ensures that audio window truncation, feature binding, and output scheduling have a traceable and consistent time reference, reducing inherent synchronization errors. Furthermore, addressing the mismatch between blocky speech, non-uniform video, and asynchronous lip-syncing in edge-side interactions, this invention constructs a time-varying variable monitoring and risk state update mechanism. This mechanism provides differentiated corrections for speech splicing boundaries, video temporal consistency, and lip-syncing phase, and can select local correction areas based on lip salience, balancing stability and detail. Finally, through quality constraints and iterative parameter tuning, a closed-loop control system is formed, thereby improving continuity, reducing sudden misalignments, and enhancing the naturalness of subjective interaction under scenarios with fluctuating computing power. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the interactive digital human interaction method and system interaction driving instructions and timing buffer based on the computing power of mobile terminals according to the present invention.

[0018] Figure 2 This is a flowchart of the interactive digital human interaction method and system interaction method based on mobile terminal computing power of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example: This invention discloses an interactive digital human interaction method based on the computing power of a mobile terminal, such as... Figure 1 As shown, it includes: Step 1: End-side data acquisition, alignment, and characterization; First, the mobile terminal obtains and parses the QR code on the tombstone carrier to obtain the PersonID, a unique identifier corresponding to the target person, and the session entry information. The PersonID serves as a unique index binding the person's memory data and digital human resources. The session entry information includes at least a ResourceURI and an Access Token. Based on the PersonID, the mobile terminal locates and loads the digital human resource package and the person's memory index bound to that PersonID from on-device or remote storage. The digital human resource package includes at least the person's geometric representation parameters, material and texture parameters, binding and deformation parameters, speech synthesis parameters, and rendering configuration parameters, and is derived from offline production or offline training and stored as a loadable file.

[0021] After the mobile terminal completes loading, it creates an interactive session context (PersonID) and initializes the security policy field and the revocable flag field within the context to limit the permission boundaries and access control of the generated dialogue.

[0022] During an interactive session, the mobile terminal synchronously acquires raw interaction samples on the device side and generates a device-side interaction feature package bound to a timestamp. The raw interaction samples consist of an audio sampling sequence output from the microphone and an image frame sequence output from the camera, with the sample segments in the audio sampling sequence using a first timestamp. The image frames in the image frame sequence are marked with a second timestamp. The first and second timestamps are both given by monotonically increasing timing values ​​generated by the same clock domain of the mobile terminal, and are used to establish a reproducible time correspondence between audio segments and image frames in this step.

[0023] The mobile terminal performs on-device preprocessing on the original interactive samples to obtain preprocessed audio and preprocessed images. The preprocessed audio is used for audio-image alignment and audio feature calculation in this step, while the preprocessed image is used for face region localization, keypoint extraction, facial expression feature calculation, and head pose feature calculation in this step. The mobile terminal then performs brightness and color normalization on the image frames to obtain normalized image frames. Where t is the frame index, The data source is the image frames output by the camera. The statistics used for normalization are calculated from the pixel values ​​of the current frame or several recent frames and do not introduce external data; the mobile terminal in The upper side performs face detection to obtain the face bounding box. Where (x,y) and (w,h) are pixel-domain coordinates and dimensions, directly derived from the output of the edge-side face detection model. The mobile terminal uses... Cropping and scaling to obtain the face region image Scaling the target size The parameters are configured for the edge side and are derived from the input size requirements of the edge-side facial key points or mesh model, and are then stored in a fixed manner. This step is used to limit the input size of the cropped and scaled image to ensure that the scale of the key points is consistent with that of the mesh inference output.

[0024] Mobile terminals use image frame timestamps Based on this, an audio window corresponding to the image frame is extracted from the audio sampling sequence to obtain the time-domain audio signal. Where n is the index of the sampling point and The data source is microphone output audio samples. To ensure the reproducibility of the audio window truncating rules, the mobile terminal reads the microphone sampling rate. and read the audio window length parameter. And define the audio sample set corresponding to frame index t as: ;

[0025] Thus, the audio window is defined as And ensure that the audio window corresponding to each frame of the image is... , and Uniquely determined. Mobile terminal to Perform end-side noise suppression to generate noise-reduced frequencies Noise suppression employs spectral subtraction and is defined by the following deterministic calculation: the mobile terminal uses short-time analysis frame length parameters. With frame shift parameters The audio is divided into short-time frames, and the sample set of the q-th short-time frame is defined as follows: ;

[0026] The short-time Fourier transform X(f,q) and its amplitude spectrum |X(f,q)| are calculated for each short-time frame, where f is the frequency index. The mobile terminal calculates the short-time energy for each short-time frame: ;

[0027] And set the energy threshold parameter. ,when The short frame is then classified as a non-speech frame and included in the noise statistics set. The mobile terminal takes the median or mean of the time value on the corresponding |X(f,q)| in the noise statistics set according to the frequency dimension as the noise amplitude estimate |N(f,q)|. The selection of the median or mean is fixed and specified by the terminal configuration and is used to determine the aggregation rule for noise estimation in this step. The mobile terminal calculates the noise suppression amplitude spectrum: ;

[0028] Where α is the over-subtraction coefficient, which is derived from offline evaluation or factory configuration and permanently stored, and is used in this step to control the noise subtraction intensity to balance residual noise and speech distortion; the mobile terminal will With the original phase After combining, perform an inverse short-time Fourier transform to obtain the noise-reduced frequency. And it is stipulated that all audio features in this step shall be provided by The calculation is performed to maintain input consistency.

[0029] The mobile terminal performs audio-visual alignment to determine the amount of time offset correction between the audio and image in this step. The upper side facial landmark detection was performed to obtain the lip and eye landmarks: upper lip center landmark. Key point of the lower lip center Key points of the left corner of the eye Key points of the right eye corner The above key points are all in the end-side key point model. The inference output is based on data sourced from camera images. The mobile terminal defines the lip opening / closing sequence as follows: ;

[0030] The numerator is the distance between the centers of the upper and lower lips, and the denominator is the distance between the left and right corners of the eyes, used for scale normalization. It is the Euclidean norm; It is a positive constant, originating from the edge configuration and permanently stored, and is used in this step to prevent numerical divergence when the denominator approaches zero due to critical point anomalies. The mobile terminal uses noise reduction frequency... The audio energy sequence a(t) synchronized with frame index t is calculated, and its calculation is defined as follows: ;

[0031] in For the aforementioned , and A uniquely determined set of audio samples. Mobile terminal calculates cross-correlation: ;

[0032] And take: ;

[0033] Where τ is a candidate value for relative time delay. This is the relative delay estimate when the cross-correlation is at its maximum; the mobile terminal uses... The culling reference for the audio window is corrected so that the audio window and image frame involved in audio feature calculation in this step are determined based on the same alignment result, and the input for this correction is... The calculation result and output are the offset of the audio window index, without introducing external data.

[0034] After alignment, the mobile terminal extracts edge interaction features from the aligned noise-reduced frequency and face region image. The edge interaction features include at least speech acoustic features, prosodic features, facial expression features and head posture features. The calculation method, parameter source and parameter usage of each type of feature are as follows.

[0035] When a mobile terminal extracts acoustic features of speech, it calculates the short-time Fourier transform of the aligned, noise-reduced frequency to obtain the amplitude spectrum |X(f,q)|, and then uses the Mel filter bank. Calculate the Mel channel energy: ;

[0036] Calculate the Mel spectrum again: ;

[0037] Where m is the Mel channel index. The filter parameters are determined by a preset parameter table on the end side, which includes at least the lower cutoff frequency, center frequency, and upper cutoff frequency for each filter. A triangular weighting function is used on the frequency axis; the parameter table is generated and stored offline, and in this step, it is used to map the linear frequency amplitude spectrum to Mel band energy to form stable spectral characteristics. Constants It is a positive number and originates from the end-side configuration and is permanently stored, and is used in this step to avoid The values ​​are singular. The mobile terminal outputs M(m,q) as the speech acoustic feature, and the feature data is obtained from the microphone audio after end-to-end noise reduction. .

[0038] When extracting prosodic features on a mobile terminal, short-time energy is calculated within the same short-time frame segmentation: ;

[0039] And calculate the fundamental frequency The fundamental frequency is calculated using the autocorrelation method and defined according to the following deterministic rule: The autocorrelation is calculated for each short-time frame signal: ;

[0040] And in the lag range Internal extraction Then, based on the sampling rate calculate ;in The parameters configured for the device side are derived from the offline-defined fundamental frequency range of human voice and are stored permanently. In this step, these parameters are used to limit the search range for autocorrelation peaks to avoid falling into meaningless hysteresis values. The mobile terminal defines the prosodic features as follows: ;

[0041] in It is a positive constant, originating from the terminal configuration and permanently stored, and is used in this step to ensure the stability of logarithmic operations. (Mobile terminal output) As a prosodic feature, and its data source is noise-reduced frequency. .

[0042] When a mobile terminal extracts facial expression features, The upper-side 3D face mesh regression is used to obtain the set of observed mesh vertices. ,in The coordinates of the i-th vertex are derived from the inference output of the edge-side mesh regression model. The mobile terminal uses pre-set face parameter resources to provide a neutral mesh. With table case variable base ,in and The parameters are generated and stored as a loadable parameter file by offline modeling or offline training, and are used in this step to represent facial expressions using finite-dimensional coefficients; the mobile terminal defines the facial expression coefficient vector as follows: Define the expression grid as follows: ;

[0043] To ensure the reproducibility of the correspondence between observations and the model, the observation grid... With base model mesh The same topology and the same vertex index are used, and the topology and index are fixed by the face parameter resources preset on the edge side. The edge-side mesh regression model directly outputs the vertex sequence under this topology. and The indexes correspond one-to-one. The mobile terminal solves the problem using least-squares fitting. Its objective function is defined as ;

[0044] Here, λ is the regularization coefficient, derived from offline evaluation or factory configuration and permanently stored, and is used in this step to suppress fitting instability caused by excessively large amplitude of the expression coefficient. The mobile terminal uses Gaussian, Newton's, or gradient descent methods to iteratively solve the above optimization problem on the device side, with initial values... The solution result or zero vector from the previous frame is used to define the initial conditions. The iteration stopping condition is defined as the objective function's decrease falling below a threshold or reaching the maximum number of iterations. The threshold is used to determine whether the error change meets the convergence requirement, thus triggering the iteration to stop. The maximum number of iterations is used to force a stop if the threshold is not met to avoid infinite loops. Both conditions are derived from the terminal configuration and are permanently stored, and are used in this step to limit the iteration solution overhead. Mobile terminal output. This is a facial expression feature, calculated from the camera image via edge-side grid regression output and edge-side preset face parameter resources.

[0045] When extracting head pose features from a mobile terminal, the head pose is defined as a rotation matrix. With translation vector The mobile terminal selects 3D reference points from a pre-installed 3D face reference model on the device. ,in The three-dimensional coordinates are derived from the parameters of the face reference model fixed on the device side; the mobile terminal selects the coordinates from the two-dimensional key point set. Two-dimensional observation points corresponding to semantics The semantic correspondence is defined by the keypoint semantic index table fixed on the terminal side and stored locally, thus determining the pairing rules between 3D points and 2D points. The mobile terminal reads the camera intrinsic parameter matrix. ,in The parameters are derived from factory calibration parameters or end-side self-calibration results and stored locally. In this step, they are used to project 3D points onto the pixel plane to calculate the reprojection error. The mobile terminal solves for the pose by minimizing the reprojection error. ;

[0046] in The projection function from homogeneous coordinates to pixel coordinates is determined by the camera imaging model and implemented as a fixed function on the device side. The mobile terminal uses PnP or iterative least squares to solve the minimization problem, with initial values ​​taken from the previous frame's pose result or unit rotation and zero translation to define the initial conditions. The stopping condition is defined as the reprojection error decrease falling below a threshold or reaching the maximum number of iterations. The threshold is used to determine convergence and trigger stopping, while the maximum number of iterations is used to force stopping if convergence fails to avoid infinite loops. Both are configured and stored on the device side and used to limit the solution overhead in this step. Mobile terminal output. This is a head pose feature, which is obtained by inferring the camera image from the edge keypoints, calculating the edge pre-set 3D reference model and the edge camera intrinsic parameters.

[0047] Finally, the mobile terminal combines the speech acoustic features M(m,q) and prosodic features obtained under the same frame index t. Facial expression features Head posture features Compared with the base timestamp Bind and encapsulate into a client-side interaction feature package; wherein the bound content includes at least the following: Aligned audio window index range and face bounding boxes involved in image calculations This allows each feature in the edge interaction feature package to be traced back to the specific sensor output and specific time segment in the original interaction sampling;

[0048] Step 2: Endpoint intent parsing and driver instruction generation;

[0049] During the interactive session, the mobile terminal performs terminal-side intent parsing on the terminal-side interactive feature package generated in step one, and converts the intent parsing result into interactive driving instructions that can be directly used for expression and driving within the interactive session. Here, terminal-side intent parsing is limited to calculating the terminal-side interactive feature package on the mobile terminal itself to obtain semantic and interactive control-related structured results. Interactive driving instructions are limited to instruction data structures with timestamps and clearly defined fields formed in this step, used to express executable quantities such as the text content, interactive intent category, emotional state parameters, and facial and head driving parameters corresponding to the user input in this interactive session.

[0050] The mobile terminal reads the speech acoustic features M(m,q) and prosodic features from the terminal-side interaction feature packet. The mobile terminal performs speech recognition on the device side to obtain the recognized text. The recognized text is defined as a character sequence or word sequence output by the speech recognition system, and its data source is the inference output of the device-side speech recognition model on M(m,q). To ensure the speech recognition process is fully transparent, the mobile terminal loads speech recognition parameter resources on the device side. These resources include acoustic model parameters and decoding parameters, and are derived from offline training and stored as a loadable file. The mobile terminal defines the acoustic model output as the symbol posterior probability distribution P(s|q) generated for each time frame q, where s is an element in the symbol set, and the symbol set is defined by the device-side fixed word table. The mobile terminal uses beam search decoding to obtain the recognized text. The input to beam search decoding is P(s|q) and language constraint parameters, which are derived from the device-side fixed language model parameters or the device-side fixed word graph structure. The beam width parameter is denoted as... ,in The parameters are configured and stored on the device side, and used in this step to limit the number of decoding candidates to control the computational overhead of the device-side decoding. The mobile terminal takes the text sequence with the highest score from the beam search output as the recognized text output, and compares this recognized text with the reference timestamp of the device-side interaction feature packet. Bind to preserve the time correspondence within this step.

[0051] The mobile terminal performs on-device text normalization on the recognized text to obtain normalized text. Normalized text is defined as a text sequence obtained by unifying capitalization, standardizing number pronunciation, expanding common colloquial abbreviations, and completing punctuation in the recognized text. The mapping rules used for text normalization originate from a pre-defined text rule table stored locally on the device and are used in this step to ensure that synonyms are represented uniformly in subsequent calculations. The mobile terminal uses the normalized text as input text for subsequent on-device intent parsing and only uses two text concepts: recognized text and normalized text.

[0052] The mobile terminal obtains the normalized text Q of the user's input statement within the Context(PersonID), and performs a similarity search on the person's memory index using Q as the search query to obtain a candidate memory fragment set {Mi}. Each memory fragment Mi contains at least the fragment content, fragment source, timestamp or time range TimeSpan, and fragment identifier MID. The mobile terminal selects the Top-K fragments from {Mi} to form an evidence set E(PersonID,Q).

[0053] The mobile terminal inputs the evidence set E(PersonID,Q), the person profile (PersonID), and the dialogue security rules RuleSet into the dialogue generation model to generate the response text A. The Profile (PersonID) includes at least the person's tone style, address preferences, and the range of answers that can be given. The RuleSet includes at least the constraints that the user will not refuse to answer if there is no evidence to support the fabrication of sensitive information or will not output content that exceeds the user's permissions. The dialogue generation model can be deployed on the device or the cloud. In addition to the response text A, its output also includes a set of reference identifiers {MIDref} and a credibility score Conf, which are used to identify the evidence fragments on which A is based.

[0054] Next, the mobile terminal calculates the response emotion control parameters and prosodic target parameters based on the response text A, and uses them together with the facial expression features and head posture features in the terminal interaction feature package to calculate the subsequent driving parameter fields, so that the response voice and facial expression or head posture present a consistent emotional style.

[0055] The mobile terminal adds a PersonID field, an evidence citation field {MIDref}, a credibility field Conf, and a security policy result field SafeFlag to the interaction-driven command, and writes the reply text A into the text field or reply field of the interaction-driven command so that it can be directly read when the terminal side generates voice and video output in step three.

[0056] Furthermore, the mobile terminal performs on-device intent classification based on normalized text to obtain interaction intent tags; wherein, the interaction intent tag is defined as one or more category identifiers selected from the on-device fixed intent set, the intent set including at least one or more of question-and-answer intent, command intent, casual conversation intent, and control intent, and this intent set is defined by the on-device fixed configuration and stored locally. The mobile terminal loads intent classification model parameter resources on the device side and encodes the normalized text, the intent classification model parameter resources being derived from offline training and fixed storage; the mobile terminal encodes the text as... ,in This is a vector and the output of the intent classification model for normalized text inference. The mobile terminal defines the posterior probability of the intent classification category as P(c|). ), where c is the category identifier in the intent set; the mobile terminal selects As an interaction intent label, and will and corresponding probability Record them together;

[0057] The mobile terminal performs edge-side slot extraction based on normalized text to obtain a semantic slot set. The semantic slot set is defined as a collection of key-value pairs, where the key is the slot name and the value is a fragment from the normalized text or a structured value parsed from the fragment. The slot name set originates from a slot definition table fixed on the device and is stored locally to avoid unclear slot name origins. The mobile terminal loads sequence labeling model parameter resources on the device and outputs a labeled sequence to the normalized text. These sequence labeling model parameter resources are derived from offline training and are stored locally. The mobile terminal segments the normalized text according to the labeled sequence to obtain text fragments corresponding to each slot name. These text fragments are then converted into structured values ​​according to the parsing rules in the slot definition table, thus forming the semantic slot set. The parsing rules may include date and time parsing rules, quantity unit parsing rules, and entity name normalization rules. All of these rules originate from a fixed rule table on the device and are stored locally, ensuring that the value of each item in the semantic slot set is rationally determined.

[0058] Mobile terminals based on prosodic features With facial expression features Calculate the emotional state parameters; whereby the emotional state parameters are defined as a continuous value vector or discrete category and its intensity value used to characterize the user's emotional state in this step, and each component of the emotional state parameters is obtained by a reproducible calculation method on the mobile terminal. To ensure that the calculation method is fully disclosed, the mobile terminal first performs prosodic features... Statistics are calculated within a time aggregation window, wherein the time aggregation window is determined by the aggregation window length parameter. definition, The data originates from the device-side configuration and is permanently stored, and is used in this step to define the time coverage range for calculating sentiment-related statistics; the mobile terminal calculates the mean of the fundamental frequency component within this time aggregation window. ,variance and the mean of the logarithmic energy component ,variance The above statistics are all derived from the window. The components are calculated using standard statistical formulas. The mobile terminal simultaneously analyzes facial expression features. Calculate the mean vector of expression coefficients within the same time aggregation window. With the vector of change ,in Defined as the difference between the maximum and minimum values ​​within a window, both of which are within the window's range. The sequence is calculated. The mobile terminal defines the input feature vector of the emotional state parameters as: ;

[0059] Each component has its source and calculation method defined. The mobile terminal loads the emotion estimation model parameter resources on the device side and outputs the emotion state parameters. The parameters of the emotion estimation model are obtained from offline training and are stored permanently. The output format is defined by the terminal-side fixed configuration as either a posterior probability vector of the emotion category or a continuous value vector of the emotion dimension; the mobile terminal uses this configuration as a fixed constraint for this step, making... The meaning of the fields is explicit in the implementation; for example, when using a continuous value vector of the emotion dimension. It can include, for example, a pleasure component and an arousal component;

[0060] The mobile terminal generates interaction-driven instructions based on recognized text, normalized text, interaction intent tags, semantic slot sets, and emotion state parameters. To ensure the structure of the interaction-driven instructions is fully disclosed, in this step, the interaction-driven instructions are defined to include the following fields, with fixed meanings: a base timestamp field. This field is used to identify the time correspondence between instructions and edge-side interaction feature packets; wherein, the reference timestamp field is consistent with the second timestamp, both originating from monotonically increasing timing values ​​in the same clock domain, and both are used to identify the reproducible time correspondence with image frames in the image frame sequence. The text field is used to carry standardized text; the intent field is used to carry interaction intent tags. The intent confidence field is used to carry... The slot field is used to carry a set of semantic slots; the emotion field is used to carry emotion state parameters. The driving parameter field is used to carry the facial and head driving parameters calculated in this step. To obtain the driving parameter field, the mobile terminal analyzes facial expression features. Head posture features Temporal smoothing is performed to reduce inter-frame jitter. Temporal smoothing uses first-order exponential smoothing and is defined as follows: for any sequence to be smoothed Define smooth output Where β∈(0,1) is the smoothing coefficient, which is derived from the terminal configuration and stored permanently, and is used to control the smoothing intensity in this step; the mobile terminal will Take them as expression coefficient vectors respectively The parameterized representation of head pose is used, and the smoothed result is denoted as... and The parameterized representation of attitude can be achieved using rotation vectors or quaternions, with the method of adoption specified by the terminal-side fixed configuration, thus avoiding the introduction of undefined new terms. The mobile terminal defines the driver parameter field as containing... and The structured data is then encapsulated together with other fields into interactive driving instructions;

[0061] The mobile terminal writes the generated interactive driving instructions into the timing buffer of the interactive session; the timing buffer is defined as a sequential storage structure in the local memory of the mobile terminal, the storage order is arranged in ascending order according to the base timestamp field, and the corresponding terminal-side interactive feature packet index is retained when each interactive driving instruction is written.

[0062] Step 3: Driving the interactive digital human on the device and generating the audiovisual output on the device;

[0063] During the interactive session, the mobile terminal reads the interactive driving instructions generated in step two and generates the voice and video outputs of the interactive digital human on the terminal side based on the interactive driving instructions. The interactive digital human is defined as a renderable virtual character described by digital human resources pre-set on the mobile terminal side. These digital human resources include at least geometric representation parameters, material and texture parameters, binding and deformation parameters, voice synthesis parameters, and rendering configuration parameters. All of these parameter resources are derived from offline production or offline training and are stored as loadable files. The voice output is defined as an audio waveform sequence calculated on the terminal side and playable by a speaker. The video output is defined as an image frame sequence rendered on the terminal side and displayable on the screen. All outputs in this step are calculated from the interactive driving instructions and the terminal-side pre-set resources.

[0064] The mobile terminal reads the base timestamp field, text field, intent field, slot field, emotion field, and driving parameter field from the interaction driving instructions, and then uses the smoothing expression coefficient from the driving parameter field. With smooth head posture As the facial and head driving inputs for interactive digital humans; among which and The data source is the exponential smoothing calculation result in step two, and the meaning of its fields has been fixed in the interactive driving instructions;

[0065] The mobile terminal also reads the PersonID field and the evidence citation field {MIDref} from the interactive driving instructions, which are used to provide a traceable source identifier on the display or for family members to review log records.

[0066] When the mobile terminal generates voice output on the device side, it performs on-device speech synthesis based on the text field and emotion field of the interaction-driven command to obtain the voice output waveform sequence. To ensure the speech synthesis process is fully transparent, the mobile terminal loads speech synthesis parameter resources on the device side. These resources include at least text-to-acoustic feature mapping parameters and acoustic feature-to-waveform mapping parameters, and are derived from offline training and stored permanently. The mobile terminal encodes the normalized text corresponding to the text fields into text embedding sequences. ,in The data is a vector sequence output by an edge-side text encoding model. The parameters of this text encoding model are part of the speech synthesis parameter resource and are stored permanently. The mobile terminal records the emotion state parameters corresponding to the emotion field as follows: and use it as a control condition input, with The common input is fed into the end-side acoustic feature generation model to generate an acoustic feature sequence. ,in The definition is a sequence of temporal feature vectors used to represent speech in this step, and its data source is the inference output of the edge acoustic feature generation model. The mobile terminal will use the acoustic feature sequence... The input is fed into the end-side vocoder model to generate a speech output waveform sequence s[n], where n is the audio sampling index. The data source for the speech output waveform sequence s[n] is the inference output of the end-side vocoder model, and the speech output waveform sequence can be played by the mobile terminal's speaker. The mobile terminal adds a reference timestamp field consistent with the interactive driving command to the speech output waveform sequence to establish the audio-video time correspondence in this step.

[0067] When the mobile terminal generates video output on the device side, it drives the facial deformation and head posture of the interactive digital human based on the driving parameter field, and performs rendering on the device side to obtain the video output image frame sequence. To ensure the driving and rendering processes are fully exposed, the mobile terminal loads the digital human resources on the device side and reads the neutral geometric representation from the digital human resources. Table of Variable Base Binding parameters and rendering configuration parameters; including neutral geometry representation. Defined as a set of three-dimensional mesh vertices of an interactive digital human under neutral facial expressions, representing a situational variable basis. Defined as the set of 3D displacement basis vectors for mesh vertices, the binding parameters are defined as the set of weights that associate mesh vertices with bones or control points, and the rendering configuration parameters are defined as the camera parameters, lighting parameters, and shading parameters required for edge rendering. All of these parameters are fixed resources on the edge. The mobile terminal defines the current expression mesh of the interactive digital human as: ;

[0068] in To smooth the expression coefficient The kth component, The data source is end-side cured. , With the interactive driving instructions provided The linear combination calculation results. The mobile terminal defines the rigid body transformation of the interactive digital human's head as... And apply it to the pose update of head-related vertices or head-related bones of the interactive digital human, wherein the range of head-related vertices or head-related bones is explicitly given by the binding parameters in the digital human resource;

[0069] When generating lip-syncing, the mobile terminal calculates the lip-syncing coefficients based on the speech output waveform sequence s[n] and applies the lip-syncing coefficients and smoothing expression coefficients together to the expression mesh. The lip-shape driving coefficient is defined as a coefficient vector that controls the relevant deformation of the mouth, and its calculation input is the speech output waveform sequence s[n], and its output is a coefficient sequence aligned with the video frame index t. To ensure the source and calculation method of the lip-sync driving coefficients are fully disclosed, the mobile terminal calculates the short-time Fourier transform of the speech output waveform sequence s[n] to obtain the amplitude spectrum |S(f,q)|, and uses a pre-set Mel filter bank on the terminal side. Calculate the Mel spectrum: ;

[0070] in The Mel spectrum features of the speech synthesis waveform. and The source and meaning are consistent with step one. The mobile terminal loads the audio-to-lip-sync model parameter resources on the device side and... Input the model to output the lip-shape driving coefficients. The audio-to-lip-sync model parameter resources are derived from offline training and are stored permanently; the mobile terminal will... Each component corresponds one-to-one with a pre-set mouth deformation base in the digital human resource. This correspondence is provided by a mouth deformation index table fixed on the end side and stored locally, thus giving each component of the mouth shape driving coefficient a specific target. The mobile terminal defines the update of the facial expression mesh after mouth shape driving as follows: ;

[0071] in Let R be the r-th mouth deformation basis, and R be the number of mouth deformation basis elements. Lip shape driving coefficient The r-th component, The data source is the fixed grid resources on the edge and the driving quantity calculated from the interactive driving commands and voice output.

[0072] Mobile terminals will With head rigid body transformation And the binding parameters are used as inputs to the on-side rendering process to generate video output image frames. To ensure the rendering process is fully transparent, the mobile terminal determines the rendering camera parameters on the device side. With view transformation parameters ,in and The source is either the rendering configuration parameters fixed in the digital human resources or the parameters calculated from the mobile terminal screen display configuration, and it is used in this step to project the 3D geometry onto the screen pixel coordinate system. The mobile terminal assigns each vertex... Calculate the screen projection point: ;

[0073] in The projection function is given and is fixed at the end. These are pixel coordinates. The mobile terminal performs lighting and shading calculations based on rendering configuration parameters to obtain pixel color values ​​and outputs video image frames. The lighting and shading parameters are derived from the fixed configuration of digital human resources;

[0074] The mobile terminal at the device side combines the voice output waveform sequence s[n] with the video output image frames. Alignment is performed based on the reference timestamp field. To ensure clear alignment rules, the mobile terminal maps the audio sampling index n and the video frame index t to the same time axis based on the reference timestamp field, and schedules playback and display according to the terminal-side timing values ​​of the mobile terminal's audio playback clock and screen refresh clock. The terminal-side timing values ​​are derived from monotonically increasing timing values ​​in the same clock domain of the mobile terminal, and are used to ensure that audio playback and video display use the same time reference in this step. This ensures that the output audio and video have a clear time correspondence, and each output is calculated by interactive driving instructions and terminal-side fixed resources.

[0075] Step 4: Adaptive scheduling of edge computing resources and constraints on edge output quality;

[0076] It should be noted that in the interactive digital human process based on the computing power of mobile terminals, the abnormalities in the terminal interaction are often not single-point problems such as audio discontinuity and video frame drops, but rather a type of time-varying mismatch problem involving three domains: within the same interactive session, the generation of the voice output waveform sequence s[n] exhibits blocky and non-stationary progression, and the video output image frame sequence The generation exhibits non-uniform frame intervals and local abrupt progression, while the lip-sync driving coefficient sequence With smooth expression coefficient Smooth head posture Furthermore, each relies on different end-side computation chains and different time bases for advancement, causing the generated beats to drive the beat display to exhibit phased splits on the time axis. These splits have nonlinear amplification characteristics: when the mouth region is in a highly significant state, even a tiny cross-modal phase shift can be perceived and amplified into a significant audio-visual inconsistency, while when the mouth region is in a low significant state, the same shift is not significant. This results in the anomaly presenting as a non-stationary form that is intermittent and sudden, making it difficult to characterize using conventional single delay compensation or fixed threshold strategies.

[0077] Therefore, in this embodiment, during the interactive session, the mobile terminal outputs the waveform sequence s[n] of the voice output and the image frame sequence of the video output from the terminal side. lip shape driving coefficient sequence Smoothing coefficient With smooth head posture The time-varying coupling mismatch between the two is addressed by performing end-side monitoring, end-side state updating, and end-side correction output generation. End-side monitoring is defined as calculating several time variables from the generated data sequence on the mobile terminal and recording their time indices. End-side state updating is defined as updating a set of end-side state variables according to the time variables using deterministic rules. End-side correction output generation is defined as performing reproducible correction calculations on the voice output and video output based on the end-side state variables and outputting the correction results.

[0078] To avoid unexplained biases caused by using different time bases for different sequences, mobile terminals first establish a unified time index on the device side. The mobile terminal then reads the base timestamp field from the interactive driving instructions. As a video time reference, the sampling index n of the audio output waveform sequence is mapped to time values ​​on the same time axis. ;in The calculation is based on the audio sampling rate. ,and The audio sampling configuration is derived from the mobile terminal and can be read on the device side. ,in The timing value corresponding to the first sampling point of the voice output waveform sequence is recorded by monotonically increasing timing values ​​in the same clock domain, thereby enabling... and Comparable. The mobile terminal denotes the video frame index on the same timeline as t, and defines the audio sample set aligned with the video frame index t as... ,in The output alignment window length parameter is derived from the end-side configuration and is stored permanently. It is used to specify the time coverage range of sampling on the speech output waveform sequence during alignment.

[0079] The mobile terminal monitors coupling mismatch at the device side and generates a reproducible set of time variables. The mobile terminal calculates the device-side video frame interval. ,in The data source is a baseline timestamp field, and it is used to characterize the sequence of video output image frames. The generation of beat variations. The mobile terminal detects audio block boundaries and calculates the end-side audio block intervals on the speech output waveform sequence. The audio block boundary is defined as the boundary between two adjacent consecutive output waveform segments, where the output submission gap is determined by the output submission timing value recorded at the terminal. To ensure the reproducibility of this determination, the mobile terminal processes the voice output waveform sequence according to a fixed block length parameter. Divide the data into blocks and record the completion time value for each block. ,in Derived from end-side configuration and fixed storage, and used to specify the recording granularity. Timing values ​​originating from the same clock domain Used to characterize the block-like output tempo variation of speech output. Mobile terminal calculates cross-modal time offset residuals. ,in Defined as the offset of the peak value of the cross-correlation between the speech output energy sequence and the lip opening sequence within the sliding window; the speech output energy sequence is calculated from the speech output waveform sequence using the following formula: ;

[0080] in The aforementioned aligned sampling set; the lip opening / closing sequence is calculated from the mouth key points in the video output image frame using the following formula: ;

[0081] in , , , These are the video output image frames. The key points, including the center of the upper lip, the center of the lower lip, and the left and right corners of the eyes, were obtained from edge-side key point detection. The key point detection model parameters were derived from edge-side fixed resources and stored locally, and the key point input data came from [source missing]. ; This is a positive constant, derived from the terminal configuration and permanently stored, used to prevent the denominator from approaching zero and causing numerical divergence. The mobile terminal uses a sliding window... Internal cross-correlation calculation: ;

[0082] and take ,in The cross-correlation window length parameter is derived from the terminal-side configuration and is permanently stored, specifying the time coverage range for cross-correlation calculations. The mobile terminal calculates the phase drift on the terminal side. ,in Defined as a sequence of mouth shape driving coefficients The difference between the most recent valid update time and the time corresponding to the video frame index t; to ensure that valid updates are reproducible, the mobile terminal defines the lip movement variation amplitude. And set an effective lip-sync update threshold. ,in Derived from end-side configuration and stored, and used to determine whether a significant change in mouth shape has occurred; when Record this moment as a valid update moment and update the index of the most recent valid update. Thus defining Local saliency in mobile terminal computing Local saliency is defined as a joint index of contrast and motion energy of the mouth region relative to non-mouth regions in a video output image frame; the set of pixels in the mouth region. Depend on and And the bounding box determined by the key points at the corners of the mouth is expanded, with the expansion ratio parameter. Derived from end-side configuration and fixed storage, used to define the mouth region boundary; the set of pixels in the non-mouth region is defined as... ,in This represents the entire set of pixels in a single frame. The mobile terminal calculates the differential energy between adjacent frames in the mouth region and the non-mouth region separately: ; ;

[0083] And define the contrast item This is the ratio of the grayscale variance of the mouth region to the grayscale variance of the non-mouth region, where the grayscale is obtained by a linear transformation of the color vector and the transformation coefficients are derived from end-side solidification. The mobile terminal defines local saliency as: ;

[0084] The logarithmic function is used to compress the dynamic range, and The input comes entirely from and .

[0085] The mobile terminal maintains a set of time-varying state variables on the device side and updates these state variables according to deterministic rules based on the aforementioned set of time-varying variables. The mobile terminal defines the time-varying state variables as state vectors. ,in Indicates a risk status for voice continuity. Indicates the risk status of video continuity. Indicates the risk status of cross-modal mismatch. This indicates a significantly amplified risk state. All of these risk states are internal scalars on the device side and are calculated and updated in this step. To avoid the uninterpretability caused by simple weighted summation, the mobile terminal uses an update operator with memory for each risk state and explicitly distinguishes between triggering and decay. The mobile terminal defines a triggering function. ,in For the corresponding threshold parameter, It is a positive constant and is configured and stored permanently on the end side to avoid the denominator being zero; The output is a normalized trigger strength, used to map time variables of different dimensions to a unified interval within this step. The mobile terminal sets the threshold parameter. , , , , Each threshold parameter is derived from offline evaluation or end-side configuration and is stored in a fixed manner, and is used to trigger and determine speech block interval fluctuation, video frame interval fluctuation, cross-modal shift, phase drift and local saliency, respectively.

[0086] In an optional implementation, to ensure the operability of setting the threshold parameters θaud, θvid, θav, θphase, and θlip and to be consistent with the data distribution on the device side, the mobile terminal determines the threshold based on the quantiles of the variables at each time point under the "normal interaction" condition, according to the collected data of the target device during the offline evaluation phase: θaud is taken as the high quantile of ΔTaud(kt), θvid is taken as the high quantile of ΔTvid(t), θav is taken as the high quantile of |δav(t)|, θphase is taken as the high quantile of |δphase(t)|, and θlip is taken as the high quantile of Slip(t); the high quantile can be selected as a fixed level from the 90th percentile to the 99th percentile and configured and stored on the device side to ensure that the thresholds are reproducible and transferable between different devices.

[0087] In an optional implementation, the attenuation coefficient , , , All end-side configuration parameters within (0,1) are taken and stored in a fixed manner, and can be set according to the expected duration of the abnormality: when it is desired that the risk state gradually decays to 1 / e of the original value within consecutive Q frames, λ can be set to exp(−1 / Q) and this λ can be used as the end-side configuration and stored in a fixed manner, so that the memory length of the risk state can be reproduced.

[0088] In an optional implementation, the trigger threshold parameters ηaud, ηvid, ηav, and ηlip are all derived from the end-side configuration and stored permanently. They can be set to a fixed threshold consistent with the output range of the trigger function ϕ(⋅), such as a fixed value between 0.3 and 0.8. Among them, ηaud is used to control the sensitivity of z1(t) to trigger speech splicing boundary smoothing correction, ηvid is used to control the sensitivity of z2(t) to trigger video temporal consistency correction, ηav is used to control the sensitivity of z3(t) to trigger end-side phase correction, and ηlip is used to control the sensitivity of z4(t) to the risk of significant amplification.

[0089] Mobile terminal update voice continuity risk status: ;

[0090] in The attenuation coefficient is derived from the end-side configuration and is stored permanently. It is used to control the degree of retention of historical risks in this step. The latest audio block index, aligned with the current video frame index t at time t, is... and The comparison does not introduce new data. Mobile terminal video continuity risk status update: ;

[0091] in This is the attenuation coefficient. Mobile terminal updates cross-modal mismatch risk status: ;

[0092] in The attenuation coefficient is the absolute value used to characterize only the intensity of the offset, ignoring the direction of the offset. Mobile terminal update amplification risk status: ;

[0093] in The decay coefficient is used. All the above updates adopt a trigger-first, decay-following approach, so that the risk state rises rapidly when an anomaly occurs and decays by the coefficient when the anomaly subsides, thus forming a reproducible time-varying state evolution.

[0094] Furthermore, the mobile terminal performs edge-side correction output generation for both voice and video outputs based on time-varying state variables. The correction action employs different, traceable correction operators and adjustment methods for different risk states. This applies to situations involving voice continuity risks. When the trigger condition is met, splicing boundary smoothing correction is performed on the speech output waveform sequence; the trigger condition is defined as follows: ,in The trigger threshold parameter, derived from the device-side configuration and stored, is used to determine whether speech smoothing correction needs to be performed. The mobile terminal determines the smoothing splicing length. And it changes dynamically with the risk status, defined as follows: ;

[0095] in and These are the minimum and maximum splicing length parameters, respectively, derived from end-side configuration and stored permanently, and used to define the upper and lower bounds for splicing corrections. (Symbols omitted) This indicates the number of sampling points to be rounded up to the nearest integer. The mobile terminal generates a cross-fade-in / fade-out weight sequence at the stitching boundary. ,in And generate using a fixed end-side function. This generation method does not rely on external data and guarantees monotonically increasing weights. The mobile terminal records the last sequence of the waveform segment before the splicing boundary as... The initial sequence of the second waveform is denoted as Both sequences are directly taken from the speech output waveform sequence s[n], and the corrected concatenated sequence is calculated: ;

[0096] This yields the corrected speech output waveform sequence, where the correction intensity is determined by... Follow The changes enable dynamic adjustment without the need for additional weighted summation.

[0097] Mobile terminals in video continuity risk status When the trigger condition is met, time consistency correction is performed on the video output image frames; the trigger condition is defined as follows: ,in To trigger the threshold parameter, the mobile terminal first determines the set of pixel regions that need correction in order to avoid unnecessary loss of detail caused by uniform processing of the entire frame. And make its risk status amplified with significance. Dynamic changes: when Meet the triggering conditions season Otherwise ,in The trigger threshold parameter, configured and stored on the device side, is used to determine whether to focus on the mouth area. The mobile terminal... Estimating inter-frame motion fields The input is and The output is a pixel displacement vector; the motion estimation method is configured and fixed on the end side as either the block matching method or the optical flow method. The block size and search range of the block matching method or the number of iterations of the optical flow method are configured and fixed on the end side, thereby ensuring that the motion field estimation is reproducible.

[0098] In an optional implementation, when the terminal configuration is fixed as block matching, the mobile terminal sets the block size to Bblk and the search range to Rsrch, both of which are terminal configuration parameters and are fixedly stored. Bblk specifies the pixel side length of the local matching window, and Rsrch specifies the maximum displacement radius for each block search in the previous frame F[t−1], thus obtaining a reproducible pixel displacement vector u(x,t) on Ωfix(t). To avoid unstable jitter introduced by motion field estimation, the mobile terminal can further fix the block matching cost function as a sum of squares based on pixel intensity differences, and configure and fix the cost aggregation method on the terminal to ensure consistency.

[0099] In an optional implementation, when the edge configuration is fixed as optical flow method, the mobile terminal sets the number of iterations of optical flow method to Niter and the number of pyramid layers to Npy. Both are edge configuration parameters and are fixed and stored. Niter is used to limit the number of iterations for each layer to control edge overhead, and Npy is used to estimate u(x,t) at different scales to improve stability under large displacement conditions, thereby ensuring that the u(x,t) estimated on Ωfix(t) has a reproducible computation path.

[0100] Furthermore, to avoid holes caused by remapping at the boundary of Ωfix(t), the mobile terminal samples Fwarp[t−1] within Ωfix(t) using the same interpolation method as the terminal-side fixed implementation. The interpolation method can be fixed as either bilinear interpolation or nearest neighbor interpolation. When bilinear interpolation is used, the mobile terminal calculates Fwarp[t−1](x) with a fixed neighborhood pixel weight rule to ensure that the remapping result is reproducible.

[0101] The mobile terminal obtains the previous frame by remapping the motion field. And calculate the corrected frame: ; ; in The fusion coefficient, which changes dynamically with the risk status within this step, is defined as follows: ; in As the baseline fusion coefficient, and The adjustment coefficients are all derived from the terminal configuration and are permanently stored. This represents the clipping function, used to ensure that the fusion coefficients fall within the effective range. The above definition allows γ(t) to decrease when the risk of video continuity increases to enhance the temporal consistency correction, while increasing γ(t) when the risk of salience amplification increases to avoid excessive ghosting in the mouth area. This results in differentiated correction strength under different risk combinations and a clear purpose for each coefficient. Mobile terminals in... External retention To reduce invalid processing, and Output as corrected video output image frames.

[0102] Mobile terminals in cross-modal mismatch risk state When the triggering condition is met, end-side phase correction is performed on the temporal relationship between the lip-shape driving coefficient sequence and the driving input to suppress the instantaneous mismatch between the speech output and the lip shape appearance; the triggering condition is defined as follows: ,in The trigger threshold parameter is used to calculate the phase correction amount on the mobile terminal. And make it follow and The combined state changes are defined as follows: ; in The maximum correction amount parameter is derived from the end-side configuration and is stored permanently to limit the correction amplitude; For a fixed-side sign consistency operator, its output takes values ​​in [−1, 1] and is defined according to the following rules: when and When both signs are the same and their absolute values ​​exceed their respective thresholds, the output is: When the two signs are opposite, the output is 0; when only one exceeds the threshold, the output is the sign of that sign. For the sign function, the threshold is taken as... and This operator is used in this step to distinguish between cases where the offset direction is consistent and reliable, and cases where the offset direction conflicts and is unreliable, thereby avoiding erroneous corrections caused by simple weighted summation under conflicting evidence. The mobile terminal relies on... The lip-shape driving coefficient sequence is resampled over time, and the corrected lip-shape driving coefficients are defined as follows: ,in Obtained through linear interpolation: If If it falls between two frame indices, then it is determined by the indices of the two adjacent frames. The interpolation weights are calculated based on the interpolation values. The ratio relative to the frame interval is calculated without introducing external data. When generating subsequent video output image frames, the mobile terminal uses... Alternative Participates in mouth deformation driving; In an optional implementation, the maximum correction parameter Discrete positions are configured and permanently stored on the end side using "video frame interval" as the scale to ensure... It will not cause lip-sync abrupt changes by spanning too many frames; for example, mobile terminals can... The interval is set to a fixed level of 0.25 frame interval, 0.5 frame interval, or 1 frame interval, and this level is fixed by the end-side configuration table corresponding to the target device's computing power level or display refresh rate, thereby enabling... The upper limit is consistent with the video frame index progression and remains reproducible.

[0103] In an optional implementation, the threshold and All are derived from edge-side configuration and hard-coded storage, and can be used with... , Using the same offline evaluation quantile determination method, thus enabling the sign-consistent operator The determination of trusted offsets remains consistent across different devices and scenarios.

[0104] Next, to ensure that the correction process does not introduce new unexplainable anomalies, the mobile terminal performs consistency checks on the correction output and limits the side effects of correction using deterministic rules. After applying speech splicing smoothing, the mobile terminal recalculates the mean squared error jump metric of the splicing boundary. After applying video temporal consistency correction, the mobile terminal recalculates the differential energy between adjacent frames on Ωfix(t) to obtain Jvidfix(t), and compares it with the target threshold parameter θvidfix. θvidfix is ​​configured and stored on the device side to determine if the correction has reached the target. When Jvidfix(t) is still greater than θvidfix and γ(t) is still greater than the minimum fusion coefficient parameter γmin, the mobile terminal decrements γ(t) by a step size Δγ and repeats the fusion correction. Both γmin and Δγ are configured and stored on the device side to limit the fusion strength and adjust the step size. After applying lip-sync driven phase correction, the mobile terminal recalculates the cross-modal temporal offset residual. The absolute value and obtain and compare it with the target threshold parameter Comparison; when the goal is not achieved and Achieved At that time, the mobile terminal will The variation amplitude is constrained to not exceed the upper bound of the variation amplitude of the previous frame. ,in Derived from end-side configuration and stored in memory, it is used to limit lip-sync jumps caused by correction.

[0105] In an optional implementation, to avoid excessive ghosting caused by fusion correction and to control end-side overhead, the mobile terminal sets γmin to a lower limit value configured and stored on the end-side and sets Δγ to a fixed step size configured and stored on the end-side. Here, γmin ensures that a sufficient proportion of the current frame F[t](x) is retained in Ffix[t](x) to avoid historical ghosting dominating, and Δγ ensures that the intensity change of each repeated correction has a reproducible discrete step pattern. The mobile terminal can further set an upper limit parameter for the number of video repetition corrections. And store it permanently; when the number of repeated corrections reaches [a certain number], [the system will continue to work]. Forcefully stop repeated corrections to avoid infinite loops and ensure real-time performance on the device side.

[0106] In an optional implementation, to avoid lip-sync phase correction in | |reach If it is still impossible to make |δavfix(t)| reach the target threshold and introduce a sudden lip jump, the mobile terminal sets the upper bound θΔm of the lip change amplitude as a constraint parameter configured and stored on the terminal side. Moreover, θΔm can be determined based on the high quantile of dm(t)=‖m(t)−m(t−1)‖2 under normal interaction conditions, so that the constraint of the change amplitude of mcorr(t) is consistent with the statistical distribution of the terminal side lip driving coefficient sequence and remains reproducible.

[0107] In one optional implementation, to avoid excessive stretching of the speech splicing boundary smoothing correction, which could lead to audio quality degradation or increased latency, the mobile terminal will use Nmax and... All are set as upper limit parameters configured and stored on the edge side, where Nmax is used to limit the maximum level of Nb(t). Used to limit the maximum number of repeated corrections when θaudfix is ​​not reached; when it is reached... If θaudfix is ​​not reached afterward, the mobile terminal retains the current correction result and binds the triggered parameters and indicator values ​​with the baseline timestamp field to the timing buffer of the interactive session to ensure subsequent analysis and reproduction.

[0108] Finally, the mobile terminal will use the time-varying variable set, time-varying state variable, trigger condition determination result, and correction parameters calculated in this step, including , , γ(t), And the corrected indicator value is bound to the baseline timestamp field and written to the timing buffer of the interactive session.

[0109] This invention also proposes an interactive digital human interaction system based on the computing power of mobile terminals, such as... Figure 2As shown, it includes: a terminal acquisition alignment feature generation module, a terminal intent parsing instruction generation module, a terminal digital human driving audiovisual generation module, a three-domain coupling mismatch monitoring and correction module, and signal connections between the modules;

[0110] The edge-side acquisition alignment feature module is used by the mobile terminal to simultaneously acquire microphone audio sampling sequences and camera image frame sequences during an interactive session, and generate an edge-side interaction feature package bound to a second timestamp;

[0111] The terminal-side intent parsing instruction generation module is used by the mobile terminal to perform terminal-side intent parsing on the terminal-side interaction feature packet, generate interaction-driven instructions with a reference timestamp field, and write them into the timing buffer of the interaction session.

[0112] The edge-side digital human-driven audiovisual generation module is used by the mobile terminal to read interactive driving instructions, generate voice output waveform sequences and video output image frame sequences on the edge, and align and output them according to the reference timestamp field.

[0113] The three-domain coupling mismatch monitoring and correction module is used by mobile terminals to perform end-side monitoring, end-side state updates, and end-side correction output generation for time-varying mismatch in three-domain coupling of voice output waveform sequence, video output image frame sequence, lip shape driving coefficient sequence, smooth expression coefficient, and smooth head posture. It analyzes video frame interval, audio block interval, cross-modal time offset residual, calculates phase drift and local saliency, outputs reproducible correction results, and writes them into the timing buffer of the interactive session.

[0114] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0115] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0116] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0119] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An interactive digital human interaction method based on mobile terminal computing power, characterized in that, include: During the interactive session, the mobile terminal simultaneously collects microphone audio sampling sequences and camera image frame sequences, and generates a terminal-side interactive feature packet bound to a second timestamp; The mobile terminal performs end-side intent parsing on the end-side interaction feature packet, generates an interaction-driven instruction with a reference timestamp field, and writes it into the timing buffer of the interaction session; The mobile terminal reads the interactive driving instructions, generates a voice output waveform sequence and a video output image frame sequence on the terminal side, and aligns and outputs them according to the reference timestamp field; The mobile terminal performs end-side monitoring, end-side state update, and end-side correction output generation for time-varying mismatch in three domains: voice output waveform sequence, video output image frame sequence, lip shape driving coefficient sequence, smooth expression coefficient, and smooth head posture. It analyzes video frame interval, audio block interval, cross-modal time offset residual, calculates phase drift and local saliency, outputs reproducible correction results, and writes them into the timing buffer of the interactive session.

2. The interactive digital human interaction method based on mobile terminal computing power according to claim 1, characterized in that: During the interactive session, the mobile terminal synchronously acquires microphone audio sampling sequences and camera image frame sequences. Audio sampling segments are marked with a first timestamp, and image frames are marked with a second timestamp. Both the first and second timestamps are monotonically increasing timing values ​​from the same clock domain. The mobile terminal reads the sampling rate and audio window length parameters based on the second timestamp, determines the audio sampling set corresponding to the frame index, and extracts the audio window. The terminal performs on-side noise reduction on the audio window to obtain the noise-reduced audio.

3. The interactive digital human interaction method based on mobile terminal computing power according to claim 2, characterized in that: The mobile terminal performs key point detection on the face region image to obtain key points of the upper lip center, lower lip center, left eye corner, and right eye corner, and calculates the lip opening degree sequence. At the same time, it calculates the audio energy sequence on the noise-reduced frequency according to the audio sampling set, calculates the cross-correlation between the audio energy sequence and the lip opening degree sequence, and takes the relative delay corresponding to the maximum cross-correlation as the time offset correction amount, and corrects the subsequent audio window truncation benchmark accordingly. Under the alignment result, the mobile terminal binds and encapsulates the speech acoustic features, prosodic features, facial expression features, and head posture features with the second timestamp into a terminal interaction feature package. The terminal interaction feature package includes the second timestamp, the aligned audio window index range, and the face bounding box.

4. The interactive digital human interaction method based on mobile terminal computing power according to claim 3, characterized in that: The mobile terminal computes the semantic and interactive control-related structured results of the terminal-side interaction feature package on the local machine, and forms an interactive driving instruction data structure with a reference timestamp field: starting from the speech acoustic features and prosodic features, the terminal-side speech recognition parameter resources are loaded to obtain the frame-by-frame symbol posterior probability distribution and decoded using beam search to output the recognized text. Then, the recognized text is normalized according to the terminal-side fixed text rule table to obtain normalized text. In the conversation context, the normalized text is used to retrieve the person memory index to obtain candidate memory fragments and select the first few fragments as the evidence set. The evidence set, person settings and dialogue security rules are input into the dialogue generation model to generate the response text and output the evidence citation identifier set and credibility score. At the same time, the person identifier field, evidence citation field, credibility field and security policy result field are added to the instruction and written into the response text. Furthermore, the normalized text is classified by the execution end-side intent to obtain intent labels and their probabilities, and the execution end-side slots are extracted to obtain a semantic slot set. Based on prosodic features and facial expression features, statistics are calculated within a time window defined by the aggregation window length parameter to form an emotion input vector, and the emotion state parameters are output by the end-side emotion estimation model. Finally, the baseline timestamp, normalized text, intent labels and intent confidence, slot set, emotion state parameters, and expression coefficients and head posture parameters obtained by first-order exponential smoothing of facial expression features and head posture features are encapsulated into an interaction-driven instruction, and written into the temporal buffer of the interaction session incrementally according to the baseline timestamp, while retaining the corresponding feature package index.

5. The interactive digital human interaction method based on mobile terminal computing power according to claim 4, characterized in that: During the interactive session, the mobile terminal reads the reference timestamp, text, emotion, and driving parameter fields from the interactive driving instructions. It uses the smoothed expression coefficient and smoothed head pose from the driving parameter fields as the facial and head driving inputs for the interactive digital human. In speech output generation, the mobile terminal loads the edge speech synthesis parameter resources based on the text and emotion fields, encodes the normalized text into a text embedding sequence, and inputs it together with the emotion state parameters into the acoustic feature generation model to obtain an acoustic feature sequence. Then, it inputs the sequence into the vocoder model to generate a speech output waveform sequence and adds a reference timestamp field consistent with the interactive driving instructions to the waveform sequence. In video output generation, the mobile terminal loads the digital human resources to obtain a neutral geometric representation and a situational variation basis. It obtains the current expression mesh based on the linear combination of the smoothed expression coefficients and applies the smoothed head pose to the head-related vertices or bones defined by the binding parameters to update the pose.

6. The interactive digital human interaction method based on mobile terminal computing power according to claim 5, characterized in that: The Mel spectrum of the speech output waveform sequence is calculated and input into the audio to the lip mapping model to obtain the lip driving coefficient sequence aligned with the video frame index. The lip driving coefficients and the mouth deformation basis are matched one by one according to the end-side solidified index table and superimposed on the expression mesh to form an updated facial mesh. Then, the mesh, head rigid body transformation and binding parameters are input into the end-side rendering process. The vertex is projected according to the rendering camera parameters and view transformation parameters and the video output image frame sequence is generated according to the lighting and shading parameters. Finally, the mobile terminal maps the audio sampling index and video frame index to the same timeline based on the reference timestamp field, and schedules playback and display according to the terminal timing values ​​of the audio playback clock and the screen refresh clock to output voice and video.

7. The interactive digital human interaction method based on mobile terminal computing power according to claim 6, characterized in that: During interactive sessions, the mobile terminal performs end-side monitoring, end-side state updates, and end-side correction output generation to address time-varying coupling mismatches between the speech output waveform sequence, video output image frame sequence, lip-shape driving coefficient sequence, smoothed facial expression coefficient, and smoothed head posture. A unified time index is established on the end-side, and the reference timestamp in the interactive driving command is read as the video time reference. The speech sampling index is mapped to the same time axis according to the sampling rate, and an audio sampling set aligned with the video frame index is defined. In end-side monitoring, time-indexed time variables are calculated and recorded, including the video frame interval, the audio block interval calculated based on the fixed block length and the generated completion timing value, the cross-modal time offset residual obtained by cross-correlation of the speech energy sequence and the lip opening degree sequence in a sliding window, the calculated phase drift determined by the lip shape coefficient change amplitude and the effective update threshold, and the local salience composed of the frame difference energy and gray-level variance ratio between the mouth region and the non-mouth region.

8. The interactive digital human interaction method based on mobile terminal computing power according to claim 7, characterized in that: In the end-side state update, speech continuity risk, video continuity risk, cross-modal mismatch risk, and saliency amplification risk are defined as components of the state vector, and each component is updated according to deterministic rules based on the trigger function, threshold parameters, and attenuation coefficients. In the end-side correction output generation, when speech continuity risk is triggered, the splicing length is determined according to the state and cross-fade-in / fade-out weights are generated. The two waveform segments at the splicing boundary are weighted and fused to obtain the corrected waveform. When video continuity risk is triggered, the correction pixel region is selected based on the saliency amplification risk. The inter-frame motion field is estimated in this region, and the current frame and the remapped previous frame are fused according to the fusion coefficient to obtain the corrected frame. Cross-modal mismatch risk is addressed by... When the mismatch risk is triggered, the phase correction amount is obtained by using the sign consistency operator based on the cross-modal offset residual and phase drift, and the lip-shape driving coefficient is resampled in time. Then, the corrected speech boundary jump metric, the frame difference energy of the corrected region and the cross-modal offset residual are recalculated and compared with the target threshold. Under the conditions of meeting the upper limit of the number of repetitions, the minimum fusion coefficient or the maximum correction amount, the splicing length, fusion coefficient or lip shape change amplitude constraint are adjusted according to deterministic rules and the correction is repeated. Finally, the time variable set, state variable, trigger judgment result, correction parameter and the corrected index value are bound to the reference timestamp field and written into the timing buffer of the interactive session.

9. An interactive digital human interaction system based on mobile terminal computing power, used to implement the interactive digital human interaction method based on mobile terminal computing power as described in any one of claims 1-8, characterized in that, include: The module includes: edge-side acquisition alignment and feature generation module, edge-side intent parsing instruction generation module, edge-side digital human-driven audiovisual generation module, three-domain coupling mismatch monitoring and correction module, and signal connections between modules. The edge-side acquisition alignment feature module is used by the mobile terminal to simultaneously acquire microphone audio sampling sequences and camera image frame sequences during an interactive session, and generate an edge-side interaction feature package bound to a second timestamp; The terminal-side intent parsing instruction generation module is used by the mobile terminal to perform terminal-side intent parsing on the terminal-side interaction feature packet, generate interaction-driven instructions with a reference timestamp field, and write them into the timing buffer of the interaction session. The edge-side digital human-driven audiovisual generation module is used by the mobile terminal to read interactive driving instructions, generate voice output waveform sequences and video output image frame sequences on the edge, and align and output them according to the reference timestamp field. The three-domain coupling mismatch monitoring and correction module is used by mobile terminals to perform end-side monitoring, end-side state updates, and end-side correction output generation for time-varying mismatch in three-domain coupling of voice output waveform sequence, video output image frame sequence, lip shape driving coefficient sequence, smooth expression coefficient, and smooth head posture. It analyzes video frame interval, audio block interval, cross-modal time offset residual, calculates phase drift and local saliency, outputs reproducible correction results, and writes them into the timing buffer of the interactive session.

Images