Control system and control method for listening and recording content identification optimization

Abstract

The invention relates to the technical field of man-machine interaction, in particular to a control system and a control method for listening and recording content recognition optimization, which are used for aligning audios, texts, expressions and limb modalities at a frame level and driving a 3D digital human to render in real time. Enabling all media events to have global timestamps based on the time reference; the method comprises the following steps: collecting audios at a collection end according to a fixed frame rate, and preprocessing the collected audios; executing time sequence identification on the received audio with the timestamp at the server side to obtain word-by-word / phoneme time mapping; based on the phoneme time mapping, generating an initial mouth shape viseme timeline as a baseline drive; and a compensation controller is arranged at the rendering end, and the compensation controller is based on the identification delay and the network arrival delay monitored in real time. According to the invention, the synchronization precision, stability and interaction naturalness of digital human real-time driving are improved.

Description

Technical Field

[0001] This invention relates to a control system and method for optimizing audio recording content recognition, and particularly to a control system and method for optimizing audio recording content recognition, belonging to the field of human-computer interaction technology. Background Technology

[0002] Existing digital human-driven and audio content recognition technologies typically rely on speech recognition, lip-syncing, and basic facial expression-driven algorithms to convert audio into animation. However, these systems generally lack a unified time reference and a sophisticated multimodal synchronization mechanism. When audio, text, lip movements, facial expressions, and body movements are generated separately in different modules and network states, the lack of collaborative control methods for frame-level or millisecond-level alignment makes audio-visual synchronization highly dependent on network quality and recognition speed. At the same time, existing systems mostly use fixed buffers or simple delay strategies, which cannot adapt to network jitter, ASR fluctuations, or changes in confidence of multi-source data. This leads to the accumulation of lip movement deviation and the drift of facial expression rhythm after long-term use, resulting in an unnatural performance. The existing technology mainly suffers from the following core problems: First, the temporal dispersion of multimodal data (audio, text, vision, facial expressions, and body language) makes accurate alignment difficult due to the lack of a unified timestamp system; second, ASR delay, network jitter, or false recognition introduces temporal gaps, resulting in lip-sync discrepancies, inconsistencies between tone and expression, and frame skipping or stuttering in animations; third, the lack of dynamic compensation mechanisms such as predictive compensation, post-correction, interpolation rearrangement, and confidence fusion makes it impossible to cover up short-term errors and converge long-term errors, thus significantly reducing the realism, immersion, and usability of digital humans. Therefore, there is an urgent need to improve a control system and control method for optimizing audio content recognition to solve the above-mentioned problems. Summary of the Invention

[0003] The purpose of this invention is to provide a control system and method for optimizing audio content recognition, in order to solve the following core problems in the prior art: First, the temporal dispersion of multimodal data (audio, text, vision, facial expressions, body language) makes accurate alignment difficult due to the lack of a unified timestamp system; Second, ASR delay, network jitter, or misidentification introduces temporal gaps, resulting in lip-sync discrepancies, inconsistencies between tone and expression, and frame skipping or stuttering in animation; Third, the lack of dynamic compensation mechanisms such as prediction compensation, post-correction, interpolation rearrangement, and confidence fusion leads to the inability to cover up short-term errors and the inability to converge long-term errors, thereby significantly reducing the realism, immersion, and usability of digital humans.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A control system and method for optimizing audio content recognition are disclosed, used to align audio, text, facial expressions, and body modalities at the frame level and drive real-time rendering of a 3D digital human. The system includes the following steps: establishing a unified time reference between the acquisition end and the server end, ensuring all media events have a global timestamp based on this time reference; acquiring audio at a fixed frame rate at the acquisition end and preprocessing the acquired audio; performing temporal recognition on the received timestamped audio at the server end to obtain a word / phoneme time mapping; generating an initial lip-sync timeline as a baseline driver based on the phoneme time mapping; and setting a compensation controller at the rendering end, which dynamically selects and combines compensation strategies based on real-time monitored recognition latency and network arrival latency to maintain multimodal frame-level alignment. The compensation strategies include: adaptive jitter buffering, short-term predictive frame interpolation, timeline interpolation or rearrangement, fine correction based on late arrival information, and confidence-driven multimodal fusion and error masking.

[0005] Furthermore, it also includes the following steps: The aligned timeline, processed by the compensation controller, is sent to the 3D digital human rendering engine for rendering, and a structured log is recorded for backtracking and optimization.

[0006] Furthermore, the preprocessing includes: automatic gain control, echo cancellation, speech activity detection, and speech enhancement.

[0007] Furthermore, the adaptive jitter buffering mechanism implemented by the compensation controller specifically includes: Based on the statistical characteristics of media arrival delays over a recent period, the buffer duration of the rendering end is dynamically adjusted to balance network jitter and end-to-end latency, and the buffer duration is used as the time budget for the prediction and correction modules.

[0008] Furthermore, the short-term predictive frame interpolation strategy employed by the compensation controller is specifically as follows: A lightweight prediction model is adopted, which predicts the vision sequence in the short term based on the most recent audio features on the edge and the rendered lip-sync history. This prediction is used to temporarily drive rendering when the ASR results or alignment information are delayed. When the actual delayed alignment information arrives, the actual information is used first for backoff correction.

[0009] Furthermore, the timeline interpolation or rearrangement strategy executed by the compensation controller is specifically as follows: When correcting the pre-arranged animation timeline or audio timing, time interpolation is used to rearrange keyframes, and high-quality resampling of audio segments is performed when necessary.

[0010] Furthermore, the fine correction strategy based on the later arrival information executed by the compensation controller is specifically as follows: Upon receiving high-confidence alignment information that arrives late, the optimal time offset is calculated within a local time window based on short-time audio and phoneme / alignment information, and time correction is applied in a progressively smooth manner over several subsequent frames.

[0011] Furthermore, the confidence-driven multimodal fusion and error masking strategy executed by the compensation controller is specifically as follows: Based on the confidence scores from ASR, phoneme alignment, prediction / lip reading models, and network / noise estimation, the results of each modality are adaptively weighted and fused. When the weighted overall confidence level or network conditions are lower than a preset threshold, error masking or semi-synchronous rollback behavior is triggered. The behavior includes: maintaining the most recent valid lip shape and enabling basic facial loop animation to mask short-term asynchrony, while recording relevant events for subsequent optimization.

[0012] Furthermore, it includes: an acquisition module for acquiring audio at a fixed frame rate, attaching a global timestamp based on a unified time reference to each frame, and outputting timestamped audio frames after preprocessing the acquired audio; the preprocessing includes automatic gain control, echo cancellation, speech activity detection, and speech enhancement; The time synchronization module is used to establish and maintain the unified time reference between the acquisition end and the server end, and realize the mapping and timestamp alignment between the media clock of the acquisition end and the clock of the server end; the time sequence recognition module is used to receive the timestamped audio frames and perform speech recognition and pronunciation alignment, and output the time mapping and corresponding confidence scores for each word / phoneme. The baseline mapping module is used to generate an initial lip-sync timeline based on the phoneme time mapping and the predefined or trained phoneme-to-lip-sync vision mapping relationship. A compensation controller, located at the rendering end, communicates with the timing identification module and the baseline mapping module. Based on real-time monitored identification delay, media arrival delay statistics, and delay correction information, the compensation controller dynamically selects and combines several compensation sub-modules to maintain modal alignment. The compensation sub-modules include: The adaptive jitter buffer submodule is used to dynamically adjust the rendering buffer duration based on the statistical characteristics of media arrival delay and to provide a time budget for subsequent compensation. The prediction-driven short-term interpolation submodule is used to temporarily predict short-term lip movements based on end-side audio features and rendered lip movement history, and drive rendering before the real alignment information arrives. The interpolation / reordering submodule is used to perform time interpolation or reordering on the scheduled animation keyframes, and to perform high-quality resampling on audio segments when necessary. The subsequent fine correction submodule is used to calculate the correction amount based on the local time window when high-confidence alignment information with a delay is received, and to apply the correction to subsequent frames in a gradual smoothing strategy. The confidence-driven fusion and error masking submodule is used to fuse confidence from ASR, phoneme alignment, prediction / lip reading models and network / noise estimation, and form driving decisions accordingly; when the confidence or network conditions are insufficient, error masking or semi-synchronous backoff behavior is triggered and recovery prompts are generated. The rendering module is used to receive the alignment timeline and motion instructions output by the compensation controller, and drive the real-time rendering of the 3D digital human's lip movements, facial expressions and body movements. The log and learning module is used to record event timestamps, compensation decisions, correction parameters, modal confidence and network statistics in a structured manner, and uses data segments with low confidence or high alignment error as active learning samples for subsequent offline annotation and model fine-tuning; The compensation controller can adaptively adjust the use and weight of each compensation sub-module according to the scenario during operation.

[0013] Furthermore, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a control method for optimizing the recognition of recorded content as described in any one of claims 1 to 8.

[0014] This invention has at least the following beneficial effects: In this invention, a unified time base is used to ensure that audio, ASR, phoneme, vision, and action commands work under the same timebase; secondly, forced alignment and end-side preprocessing are combined to improve recognition stability; at the same time, dynamic compensation strategies such as adaptive jitter buffering, predictive frame interpolation, interpolation resampling, post-to-precision correction, and confidence fusion are used to achieve real-time correction of various delays and jitter, maintaining multimodal synchronization at the frame level; in abnormal situations, error masking is used to maintain rendering naturalness; and active learning closed-loop continuously optimizes model performance to improve the synchronization accuracy, stability, and interaction naturalness of real-time digital human driving. Attached Figure Description

[0015] Figure 1 This is a flowchart of a control method for optimizing the recognition of recorded content according to the present invention. Detailed Implementation

[0016] The following will describe in detail the implementation of this application with reference to the accompanying drawings and embodiments, so that the implementation process of how this application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0017] like Figure 1 As shown, this embodiment provides a control system and method for optimizing audio content recognition, used to align audio, text, facial expressions, and body modalities at the frame / millisecond level and drive real-time rendering of a 3D digital human. The method includes the following steps: establishing a clock mapping relationship between the acquisition end and the server, ensuring all media events have a global timestamp based on the same timebase; the acquisition end divides the audio into frames at a fixed frame rate and attaches a global timestamp to each frame, performing low-latency preprocessing on the acquisition side (automatic gain control, echo cancellation, speech activity detection, and speech enhancement); the server performs ASR on the received time-stamped audio and outputs a token-by-token timestamp and confidence level, obtaining a mapping between phoneme and corresponding time through pronunciation alignment; and generating an initial visual timeline based on the phoneme and a predefined or trained phoneme→viseme mapping table, which serves as the driving force for 3D digital human rendering. The baseline timeline for digital lip reading is established. A compensation controller is set up at the rendering end. Based on real-time monitoring of ASR latency, network arrival latency statistics, and late arrival correction information, the compensation controller dynamically selects and combines compensation strategies as needed to maintain frame-level alignment of each modality. Compensation strategies include adaptive jitter buffering, short-term prediction / frame interpolation, interpolation or rearrangement of the scheduled timeline, fine correction based on late arrival information, and error masking / backoff. The aligned timeline and motion commands processed by the compensation controller are sent to the 3D digital human rendering engine to perform real-time rendering of lip movements, facial expressions, and body movements. The timestamps, compensation decisions, correction parameters, confidence levels of each modality, and the final rendering timeline are structured and recorded to support auditing and subsequent optimization. This is achieved through establishing a unified time base, time-stamped acquisition at the edge, ASR temporal recognition, and phoneme→viseme. A baseline lip-sync timeline is generated through mapping, and combined with a compensation control loop, multimodal frame-level / millisecond-level alignment is achieved. This ensures high synchronization between audio, text, lip movements, facial expressions, and body language, fundamentally avoiding issues such as lip-sync misalignment, semantic lag, and facial expression disconnect. Combined with adaptive jitter buffering and predictive-driven short-term frame interpolation, stable and continuous temporal output can be maintained even under network jitter, ASR latency, or short-term frame loss. This enables low-latency, high-fidelity real-time digital human expression capabilities, significantly enhancing the immersion, naturalness, and credibility of the interaction.

[0018] The adaptive jitter buffering mechanism implemented by the compensation controller is as follows: Based on the statistical characteristics of media arrival delay in recent times, the buffer duration of the rendering end is dynamically adjusted to balance network jitter and end-to-end latency, and the buffer duration is used as the time budget for the prediction and correction modules. Specifically, the compensation controller implements an adaptive jitter buffering mechanism, and the buffering duration B is dynamically determined according to the following formula: In the formula, This is the set of media arrival delay samples within the most recent sliding window. The parameter represents the sample standard deviation and has the following range of values: , , , The sliding window length is 300-1000ms; the buffer duration B is used as the rendering end queuing / scheduling window and as the time budget input for the prediction and correction modules, and each By updating B once, interpolation and resampling are performed on the animation keyframes and audio, enabling the system to flexibly correct minor temporal shifts caused by delay, jitter or model error without compromising speech quality and motion continuity. This effectively ensures continuous matching of lip movements, facial expressions and speech rhythm, thereby significantly improving the naturalness of digital human movements and the accuracy of audio-visual synchronization. The compensation controller employs a short-term predictive frame interpolation strategy, specifically: A lightweight prediction model is adopted, which predicts the vision sequence in the short term based on the most recent audio features on the edge and the rendered lip movement history. This is used to temporarily drive rendering when the ASR results or alignment information are delayed. When the real delayed alignment information arrives, the real information is used first for backoff correction. Specifically, the compensation controller implements a prediction-driven short-term interpolation module, which employs a lightweight prediction model. Output the future short-term Visem series: Input features Including recent End-side audio features of the frame, the most recently rendered viseme sequence, and optional speaker embeddings; total prediction window duration. The model structure is preferably one of the following and supports quantized deployment at the edge: a) Two-layer GRU, 128-512 hidden cells, dropout 0-0.3; b) Transformer-lite; Prediction results include confidence level Upon receiving the actual ASR / phoneme alignment information, the actual alignment is used first, and the prediction result is used as a reference for backoff or compensation; when When model resources are insufficient, the system prioritizes Viseme estimation based on edge-side lip-reading CNN as a backup. By performing local optimal offset calculation and progressive smoothing correction on delayed high-confidence recognition results, the system can correct slight temporal deviations generated in early rendering without interrupting the rendering process, achieving "uninterrupted correction." This mechanism ensures that the digital human output maintains stable and consistent audio-visual alignment even during long-term operation. The timeline interpolation or rearrangement strategy executed by the compensation controller is as follows: When correcting the pre-arranged animation timeline or audio timing, the keyframes are rearranged using time interpolation, and the audio segments are resampled with high quality when necessary. Specifically, this involves analyzing the time sequence of keyframes in the animation. Applying interpolation operators Get rearrangement time When audio temporal correction is required to maintain audio-visual consistency, WSOLA or high-precision resampling algorithms are used to resynthesize audio segments to minimize audio quality loss and ensure that the subjective MOS drop is less than 0.2. The inputs for interpolation / resampling include pre-arranged animation keyframes, the target timebase timestamp provided by ASR / phoneme, and audio segments in the buffer. By weighted fusion of multiple source confidences such as ASR, phoneme alignment, prediction models, and network noise, the system can automatically select the most reliable data source to drive decisions in complex noisy environments, weak network conditions, or unstable recognition conditions. Combining error masking and semi-synchronous backoff, natural performance can be maintained in invalid data segments, significantly improving system robustness and interaction stability. The compensation controller executes a fine correction strategy based on back-arrival information, specifically as follows: Upon receiving high-confidence alignment information that arrives late, the optimal time offset is calculated within a local time window based on short-time audio and phoneme / alignment information, and time correction is applied in a progressively smooth manner over several subsequent frames. Specifically, when the compensation controller receives a later high-confidence ASR / phoneme timestamp or a later audio segment, it calculates the optimal time offset Δt within a local time window W and applies a progressively smooth correction. The calculation method is selected from: a) Cross-correlation method: using short-time audio feature sequences With phoneme / time feature sequence Calculate the cross-correlation function: and take ,in =200-400ms; b) DTW method: Calculate the DTW cumulative cost matrix and find the minimum cost path. ,Depend on Extracting the average translation ;get Then, smoothing correction is applied frame by frame over the next N frames, with a correction amount of... ,in Parameter range: , , By performing dynamic scheduling and frame interpolation of vision, facial expressions and body movements based on the aligned unified timeline, the rendering engine can stably achieve high frame rate, low latency and continuous motion presentation, enabling digital humans to maintain natural, coherent and believable performance in speech, live broadcast and interactive scenarios. The confidence-driven multimodal fusion and error masking strategy executed by the compensation controller is as follows: Based on the confidence scores from ASR, phoneme alignment, prediction / lip reading models, and network / noise estimation, the results of each modality are adaptively weighted and fused. When the weighted overall confidence or network conditions are lower than a preset threshold, error masking or semi-synchronous rollback behavior is triggered. The behavior includes: maintaining the most recent valid lip shape and enabling basic facial loop animation to mask short-term asynchrony, while recording relevant events for subsequent optimization. Specifically, it is obtained from the output of ASR. Aligned with phoneme From prediction / lip reading Estimated from network / noise ; Let the initial weight vector be... Calculate weights and normalization , where the function A smooth mapping is preferred: parameter It can be obtained through offline / online training; Calculate the weighted confidence score ,like , ,or ,or If an error occurs, it triggers an error masking or semi-synchronous rollback behavior: maintain the most recent lip shape, enable basic facial looping, and display "Restoring" on the UI / subtitles; all decisions and statistical information are recorded in structured logs and labeled and returned as active learning samples. By recording full timestamps, recognition results, compensation information, network parameters, and error indicators, and using abnormal samples for active learning, the compensation strategy, prediction model, and phoneme-viseme mapping relationship can be continuously optimized, so that the system performance can be continuously improved through long-term operation and adapted to more scenarios and speaking styles. The system includes: an acquisition module for acquiring audio at a fixed frame rate and appending a global timestamp based on a unified time base to each frame; and a preprocessing module for outputting timestamped audio frames after preprocessing the acquired audio. Preprocessing includes automatic gain control, echo cancellation, speech activity detection, and speech enhancement. A time synchronization module establishes and maintains a unified time base between the acquisition and server ends, mapping and aligning the media clock at the acquisition end with the clock at the server end. A timing recognition module receives timestamped audio frames and performs speech recognition and pronunciation alignment, outputting word / phoneme time mappings and corresponding confidence scores. A baseline mapping module maps phoneme time maps and uses predefined or trained phoneme-to-lip shape visions. The mapping relationship generates an initial lip-sync timeline; a compensation controller, located at the rendering end, communicates with the timing recognition module and the baseline mapping module; the compensation controller dynamically selects and combines several compensation sub-modules to maintain modal alignment based on real-time monitored recognition delay, media arrival delay statistics, and delay correction information; the compensation sub-modules include: an adaptive jitter buffer sub-module, used to dynamically adjust the rendering buffer duration based on the statistical characteristics of media arrival delay and provide a time budget for subsequent compensation; a prediction-driven short-term frame interpolation sub-module, used to temporarily predict short-term lip-sync based on end-side audio features and rendered lip-sync history, and drive rendering before the arrival of true alignment information; and an interpolation / rearrangement sub-module, using... The system performs time interpolation or rearrangement on pre-arranged animation keyframes and performs high-quality resampling of audio segments when necessary. The subsequent fine-correction submodule calculates the correction amount based on a local time window upon receiving delayed high-confidence alignment information and applies the correction to subsequent frames using a gradual smoothing strategy. The confidence-driven fusion and error-masking submodule fuses confidence scores from ASR, phoneme alignment, prediction / lip-reading models, and network / noise estimation to form driving decisions. When confidence or network conditions are insufficient, it triggers error-masking or semi-synchronous backoff behavior and generates recovery prompts. The rendering module receives the alignment timeline and motion commands output by the compensation controller and drives the lip movements and facial expressions of the 3D digital human. Real-time rendering of facial expressions and body movements; a log and learning module for structured recording of event timestamps, compensation decisions, correction parameters, modal confidence, and network statistics, using low-confidence or high-alignment-error data segments as active learning samples for subsequent offline annotation and model fine-tuning; the compensation controller, during runtime, can adaptively adjust the use and weight of each compensation sub-module according to the scenario. Through the unified collaborative work of the acquisition, synchronization, recognition, compensation, rendering, and learning modules, end-to-end temporal optimization, dynamic correction, and real-time driven rendering of the recorded content can be achieved, thereby constructing a high-precision, high-fault-tolerant, and highly immersive human-computer interaction system, laying a stable system foundation for applications in education, exhibitions, live streaming, training, and other scenarios.

[0019] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a control method for optimizing the recognition of audio recordings as described in any one of claims 1 to 8. By integrating multimodal alignment, jitter control, timing compensation, predictive backfilling, and active learning capabilities at the system level, a complete "acquisition-recognition-compensation-rendering-feedback" closed-loop architecture is constructed. This enables the digital human's expression to remain natural, continuous, and accurate even in complex network environments and under varying speech rates, thereby improving the system's engineering usability, scalability, and industry value.

[0020] If certain terms are used in the specification and claims to refer to specific components, those skilled in the art will understand that hardware manufacturers may use different names to refer to the same component. This specification and claims do not distinguish components based on differences in name, but rather on differences in function. The term "comprising" as used throughout the specification and claims is an open-ended term and should be interpreted as "comprising but not limited to." "Approximately" means that within an acceptable margin of error, those skilled in the art can solve the technical problem and substantially achieve the technical effect within a certain margin of error.

[0021] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes that element.

[0022] The foregoing description illustrates and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A control method for audio, text, expression and body modalities alignment at frame level and driving 3D digital human real-time rendering, characterized in that, The method comprises the following steps: Establishing a unified time reference between the collection end and the service end, so that all media events are provided with global timestamps based on the time reference; Collecting audio at a fixed frame rate at the collection end, and pre-processing the collected audio; Performing timing recognition on the received audio with timestamps at the service end to obtain word-by-word / phoneme time mapping; Based on the phoneme time mapping, generating an initial viseme timeline as a baseline drive; Setting a compensation controller at the rendering end, which dynamically selects and combines compensation strategies based on real-time monitoring of recognition delay and network arrival delay, to maintain frame-level alignment of multi-modalities; The compensation strategies include: adaptive jitter buffer, short-term prediction frame filling, timeline interpolation or rearrangement, fine correction based on late information, and confidence-driven multi-modal fusion and error masking.

2. The control method of claim 1, wherein: Further comprising the following steps: Distributing the aligned timeline processed by the compensation controller to the 3D digital human rendering engine for rendering, and recording structured logs for backtracking and optimization.

3. The method of claim 1, wherein: The preprocessing includes: automatic gain control, echo cancellation, speech activity detection, and speech enhancement.

4. The method of claim 1, wherein: The adaptive jitter buffer mechanism implemented by the compensation controller is as follows: Based on the statistical characteristics of media arrival delay in the recent period of time, dynamically adjusting the buffer length at the rendering end to balance between network jitter and end-to-end delay, and using the buffer length as the time budget of the prediction and correction module.

5. The method of claim 1, wherein: The short-term prediction frame filling strategy adopted by the compensation controller is as follows: Using a lightweight prediction model to predict the viseme sequence in the short term based on the latest audio features and rendered viseme history at the end side, to temporarily drive rendering when the ASR result or alignment information is delayed; when the real delay alignment information arrives, preferentially use the real information for rollback correction.

6. The method of claim 1, wherein: The timeline interpolation or rearrangement strategy implemented by the compensation controller is as follows: When correcting the scheduled animation timeline or audio timing, use time interpolation method to rearrange the key frames, and perform high-quality resampling on the audio segment when needed.

7. The method of claim 1, wherein: The fine correction strategy based on late information implemented by the compensation controller is as follows: When receiving high-confidence alignment information that arrives with delay, based on short-term audio and phoneme / alignment information, calculate the optimal time offset within the local time window, and apply time correction in a step-by-step smoothing manner within several subsequent frames.

8. The method of claim 1, wherein: The confidence-driven multi-modal fusion and error masking strategy implemented by the compensation controller is as follows: Based on the confidence from ASR, phoneme alignment, prediction / lip reading model, and network / noise estimation, adaptively weight and fuse the results of each modality; When the weighted comprehensive confidence or network condition is lower than the preset threshold, trigger error masking or semi-synchronous rollback behavior, which includes: keeping the latest valid viseme and enabling the basic face loop animation to mask short-term asynchronization, while recording relevant events for subsequent optimization.

9. The control system for optimizing the recognition of recorded content according to claim 1, wherein: The method comprises: The collection module is configured to collect audio at a fixed frame rate, attach a global timestamp based on a unified time reference to each frame, and output a timestamped audio frame after pre-processing of the collected audio, wherein the pre-processing includes automatic gain control, echo cancellation, voice activity detection, and voice enhancement; The time synchronization module is configured to establish and maintain the unified time reference between the collection end and the service end, and to realize mapping of the collection end media clock and the service end clock and timestamp alignment; The timed recognition module is configured to receive the timestamped audio frame and perform voice recognition and pronunciation alignment, and output a word-by-word / phoneme time mapping and corresponding confidence; The baseline mapping module is configured to generate an initial mouth shape timeline according to the phoneme time mapping and a predefined or trained mapping relationship between phonemes and visemes; The compensation controller is arranged at the rendering end and communicates with the timed recognition module and the baseline mapping module; the compensation controller dynamically selects and combines a plurality of compensation sub-modules based on real-time monitored recognition delay, media arrival delay statistics, and correction information of the delay arrival to maintain modality alignment; The compensation sub-modules include: An adaptive jitter buffer sub-module configured to dynamically adjust a rendering buffer duration according to statistical characteristics of the media arrival delay, and provide a time budget for subsequent compensation; A prediction-driven short-term frame compensation sub-module configured to temporarily predict a short-term mouth shape based on end-side audio features and a history of rendered mouth shapes, and drive rendering before real alignment information arrives; An interpolation / rearrangement sub-module configured to perform time interpolation or rearrangement on scheduled animation keyframes, and perform high-quality resampling on an audio segment when necessary; A late fine correction sub-module configured to calculate a correction amount based on a local time window when high-confidence alignment information arrives late, and apply the correction to subsequent frames in a step-by-step smoothing strategy; A confidence-driven fusion and error masking sub-module configured to fuse confidence from ASR, phoneme alignment, prediction / lip reading model, and network / noise estimation, and form a driving decision accordingly; when confidence or network conditions are insufficient, trigger error masking or semi-synchronous fallback behavior, and generate a recovery prompt; A rendering module configured to receive the alignment timeline and action instructions output by the compensation controller, and drive real-time rendering of the mouth shape, facial expression, and body movement of a 3D digital person; A log and learning module configured to structure the timestamp, compensation decision, correction parameter, modality confidence, and network statistical information of an event, and use a data segment with low confidence or high alignment error as an active learning sample for subsequent offline labeling and model fine-tuning. The compensation controller can adaptively adjust the use and weight of each compensation sub-module according to the scene during runtime.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the control method for optimizing the recognition of the recorded content according to any one of claims 1 to 8. The processor executes the program to realize the control method for optimizing the recognition of the recorded content according to any one of claims 1 to 8.

Images