A virtual digital human multimedia teaching interaction method and system and a storage medium

Abstract

The application discloses a kind of virtual digital person multimedia teaching interaction method, system and storage medium, method includes: in unified time caliber and window basis, based on the same window data group calculates audio mouth shape time difference, mouth shape caption time difference, audio caption time difference, video mouth shape time difference, and compared with time difference threshold value output misplacement alarm result, realize the unified constraint to four kinds of key alignment relationship;When alarm triggers, according to misplacement alarm result selection degradation strategy and generate execution record, misplacement alarm result and execution record are formed evidence chain data package by summarizing;When the cumulative number in preset window number exceeds diffusion blocking threshold value, recording output switches to the video stream that reserved audio and caption are not superimposed mouth shape;Based on evidence chain data package generates rectification publishing stream and replaces recording and broadcast corresponding time segment and keeps hash check value when time sequence consistent index satisfies publishing threshold value, so that it realizes that handling action can be driven, diffusion can be inhibited and rectification can be replaced.

Description

Technical Field

[0001] This invention relates to the field of virtual digital human technology, specifically to a virtual digital human multimedia teaching interaction method, system, and storage medium. Background Technology

[0002] Virtual digital human multimedia teaching interaction has been applied to scenarios such as large-class live teaching, course recording, online Q&A, and exam review. It usually uses virtual digital humans as the teaching carrier to output audio-visual multimodal content such as voice explanation, lip-sync, facial expressions, subtitle prompts, and video overlay to the same teaching session. It may also simultaneously access the teacher's camera screen, the shared screen of the courseware, and the digital human rendering screen, and then complete the synthesis and forwarding on the platform side or edge nodes to meet the requirements of low latency and large-scale concurrent playback.

[0003] Existing implementations, in order to compress end-to-end latency, often divide audio processing, lip-syncing, facial expression generation, subtitle generation, and video rendering into parallel links for separate processing. Under weak network or congestion conditions, continuous playback mechanisms such as retransmission, jitter buffering, and tempo adjustment are introduced. However, because these links differ in processing queues, buffer depths, retransmission backoff strategies, and synthesized tempos, and because subtitle segments are data types covering a time range while audio and video frames are data types at discrete moments, inconsistencies in the "time position" of different data types at the same playback moment can easily occur. For example, within the same time range, audio frames may have progressed while lip-sync parameters still use the previous time position, or video frame tempo may have been adjusted while lip-sync parameters are still driven by the old time position. This results in temporal offsets between speech and lip-sync, lip-sync and subtitles, speech and subtitles, and video and lip-sync. These offsets typically do not manifest as black screens or stuttering but rather appear covertly as inconsistencies between lip-sync and semantics, leading to high user perception costs. Furthermore, in recorded broadcasts or screen recordings, these offsets are easily solidified into replayable segments, affecting the reliability of the teaching content presentation.

[0004] Therefore, a prominent problem with existing technologies is that, under conditions such as multi-link asynchronous processing, weak network retransmission, and edge synthesis, there is a lack of a unified time reference and cross-sequence timing consistency control mechanism for the same teaching session. It is difficult to organize audio frame sequences, lip-sync parameter sequences, facial expression parameter sequences, subtitle segment sequences, and video frame sequences into comparable peer data within an executable time window granularity, and to form a misalignment alarm result evidence chain data packet that can be used for handling. As a result, it is difficult to suppress the spread of misaligned segments in the recording output and carry out targeted correction and replacement in a timely manner.

[0005] Therefore, proposing a virtual digital human multimedia teaching interaction method, system, and storage medium to solve the difficulties existing in the prior art is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a virtual digital human multimedia teaching interaction method, system, and storage medium to address the shortcomings in the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a virtual digital human multimedia teaching interaction method, comprising:

[0008] Obtain the main time reference for the teaching session; generate unified timestamps for the full-dimensional audiovisual sequence based on the main time reference to obtain the aligned dataset;

[0009] A time window index is generated based on the aligned dataset. The time window index includes the association between the time window identifier and the corresponding data of each sequence. The corresponding data is extracted according to the time window identifier to form a data group with the same window. The missing count and duplicate count are counted to obtain the risk characteristics.

[0010] The timing consistency index is calculated based on the same data group, including at least audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference; the time difference is obtained by subtracting the unified timestamps of the corresponding two types of data; the time difference is compared with the time difference threshold, and a misalignment alarm result is output if the threshold is exceeded.

[0011] Based on the misalignment alarm results, a degradation strategy is selected and an execution record is generated; the misalignment alarm results and execution records are summarized to obtain an evidence chain data package; when the cumulative number of misalignment alarm results exceeds the diffusion blocking threshold within a preset window, the recording output is switched to a video stream that retains audio and subtitles and does not overlay lip movements; a correction release stream is generated based on the evidence chain data package, and when the timing consistency index of the correction release stream meets the release threshold, the corresponding time segment of the recording is replaced and the hash check value is retained.

[0012] Furthermore, the audiovisual full-dimensional sequence includes at least an audio frame sequence, a lip-sync parameter sequence, an facial expression and motion parameter sequence, a subtitle segment sequence, and a video frame sequence;

[0013] The master time base and the teaching session number are synchronously distributed to the edge synthesis nodes and the student terminals;

[0014] Based on the main time reference, the acquisition time or encapsulation time of the audio frame sequence and the video frame sequence are read and mapped to generate a unified timestamp.

[0015] Based on a unified timestamp, the corresponding audio frame time position or video frame time position is determined for the lip-sync parameter sequence, facial expression parameter sequence, and subtitle segment sequence, and then mapped to generate a unified timestamp. The audiovisual full-dimensional sequences are aggregated to form an aligned dataset.

[0016] Furthermore, based on the aligned dataset and using a unified timestamp as a time reference, the starting point and length of the time window are determined, and a time window index is generated. The time window index includes the association between the time window identifier and the audio frame sequence, lip-sync parameter sequence, facial expression parameter sequence, subtitle segment sequence, and video frame sequence.

[0017] Based on the time window index, extract the corresponding data of each sequence according to the time window identifier to form a data group with the same window;

[0018] Based on the same data group, the expected number and actual number of each sequence within the time window are calculated to obtain the missing count and duplicate count;

[0019] Risk characteristics are obtained by summarizing missing and duplicate counts by time window identifier.

[0020] Furthermore, corresponding data pairs are established based on the same window data group, namely, corresponding data pairs between audio frame sequences and lip-sync parameter sequences, corresponding data pairs between lip-sync parameter sequences and subtitle segment sequences, corresponding data pairs between audio frame sequences and subtitle segment sequences, and corresponding data pairs between video frame sequences and lip-sync parameter sequences.

[0021] Based on the corresponding data, unified timestamps were extracted from the audio frame sequence, lip-sync parameter sequence, and video frame sequence, and unified timestamps for the start and end points of the subtitle segment sequence were also extracted.

[0022] The time differences are obtained by subtracting the unified timestamps, including audio lip-sync time difference, lip-sync caption time difference, audio caption time difference, and video lip-sync time difference;

[0023] For the time range of the subtitle segment sequence, the lip-sync subtitle time difference and audio subtitle time difference are determined based on the unified timestamp at the start and end of the time range;

[0024] Within the same data group, the audio frame sequence and video frame sequence are sorted according to the same timestamp. The corresponding data of the audio frame sequence and the corresponding data of the video frame sequence that are closest to the same timestamp of the lip-sync parameter sequence are selected as the subtraction objects.

[0025] The timing of audio lip movements, lip-sync captions, audio captions, and video lip movements is summarized to obtain a timing consistency index, which is then associated with the time window identifier.

[0026] Each time difference is compared with the time difference threshold. When any time difference exceeds the corresponding time difference threshold, a misalignment alarm result is output. The misalignment alarm result includes the time window identifier, the threshold type, and the corresponding time difference.

[0027] Furthermore, based on the misalignment alarm results and read them in order of window number, a degradation strategy is selected from the degradation strategy set based on the threshold category. The degradation strategies include lip-sync parameter freezing strategy, lip-sync parameter back-off realignment strategy, subtitle delay alignment strategy, and synthesized video frame beat reconstruction strategy.

[0028] Execute the selected degradation strategy and generate an execution record. The execution record includes the trigger window number, threshold category, corresponding time difference, degradation strategy type, degradation start window number, degradation end window number, degradation execution time, and degradation output version number.

[0029] The results of misalignment alarms and execution records are summarized and matched to obtain the evidence chain data package. The evidence chain data package includes the teaching session number, window number range, misalignment alarm result list, execution record list, main time base parameters, degraded output version number list and generation time.

[0030] For audio frame sequences, subtitle segment sequences, and video frame sequences, the same timestamp and data length are sequentially concatenated within the window sequence number range, and the summary value is calculated to obtain audio frame sequence summary information, subtitle segment sequence summary information, and video frame sequence summary information, which are then written into the evidence chain data packet.

[0031] Furthermore, when the out-of-threshold category includes audio lip movements, a lip movement parameter freezing strategy or a lip movement parameter backoff realignment strategy is selected; when the out-of-threshold category includes lip movement subtitles or audio subtitles, a subtitle delay alignment strategy is selected; when the out-of-threshold category includes video lip movements, a synthesized video frame beat reconstruction strategy is selected.

[0032] The lip-sync parameter freezing strategy maintains the output of the lip-sync parameter sequence within a consecutive number of window sequences at the lip-sync parameters corresponding to the trigger window sequence number. The lip-sync parameter rollback and realignment strategy rolls back a preset number of window sequences from the trigger window sequence number and reselects the time position of the lip-sync parameter sequence so that the time position of the lip-sync parameter sequence is within the same window sequence number as the time position of the audio frame sequence. The subtitle delay alignment strategy delays the output time of the subtitle segment sequence by a preset number of delay windows so that the window sequence number covered by the subtitle segment sequence is consistent with the window sequence number covered by the audio frame sequence. The composite video frame beat reconstruction strategy redetermines the output beat of the video frame sequence within each window sequence number according to the time scale of the main time reference starting from the trigger window sequence number, so that the output quantity of the video frame sequence within each window sequence number is consistent with the time length corresponding to that window sequence number.

[0033] Furthermore, based on the misalignment alarm results and counting by window number within the statistical range of the scrolling window, the cumulative number of times within the preset number of windows is obtained;

[0034] The cumulative count is compared with the diffusion blocking threshold. When the cumulative count exceeds the diffusion blocking threshold, the recording output is switched.

[0035] Switch the recording output to a video stream that retains the audio frame sequence and subtitle segment sequence without superimposing the lip-sync parameter sequence. The video frame sequence is output according to the main time base, the subtitle segment sequence is output according to the subtitle delay alignment strategy, and the output time covers the window sequence range. The audio frame sequence is played back in the original output mode.

[0036] The starting window number, ending window number, and trigger time of the switch are recorded in the execution log, and the execution log is included in the evidence chain data packet.

[0037] Furthermore, the trigger window sequence range and the main time base parameter are extracted based on the evidence chain data packets;

[0038] Reorganize the audio frame sequence, subtitle segment sequence, and video frame sequence according to the trigger window sequence number range;

[0039] Using the main time base parameter as a time reference, the output time position of the lip shape parameter sequence within the trigger window sequence number range is redefined;

[0040] Calculate the timing consistency metrics of the corrective release stream within the trigger window sequence number range. The timing consistency metrics include audio lip-sync time difference, lip-sync caption time difference, audio caption time difference, and video lip-sync time difference.

[0041] The timing consistency index is compared with the release threshold. When the timing consistency index meets the release threshold, the corresponding time segment of the recording resource that corresponds to the trigger window sequence number range is replaced with the corresponding time segment of the correction release stream.

[0042] Calculate the hash verification value for the time segment corresponding to the replaced recording, and associate the hash verification value with the teaching session number, trigger window sequence number range, and downgraded output version number in the evidence chain data packet.

[0043] A virtual digital human multimedia teaching interaction system is provided to implement a virtual digital human multimedia teaching interaction method. The system includes:

[0044] The master time reference module is used to obtain the master time reference of the teaching session; and to generate a unified timestamp for the full-dimensional audiovisual sequence based on the master time reference to obtain the aligned dataset.

[0045] The window indexing module generates a time window index based on the aligned dataset. The time window index includes the association between the time window identifier and the corresponding data of each sequence. The corresponding data is extracted according to the time window identifier to form a data group with the same window. The missing count and duplicate count are counted to obtain the risk characteristics.

[0046] The consistency calculation module calculates the timing consistency index based on the same window data group, which includes at least audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference. The time difference is obtained by subtracting the unified timestamps of the corresponding two types of data. The time difference is compared with the time difference threshold, and a misalignment alarm result is output if the threshold is exceeded.

[0047] The correction module selects a degradation strategy and generates an execution record based on the misalignment alarm results; it summarizes the misalignment alarm results and execution records to obtain an evidence chain data packet; when the cumulative number of misalignment alarm results exceeds the diffusion blocking threshold within a preset window, the recording output is switched to a video stream that retains audio and subtitles and does not overlay lip movements; a correction release stream is generated based on the evidence chain data packet, and when the timing consistency index of the correction release stream meets the release threshold, the corresponding time segment of the recording is replaced and the hash check value is retained.

[0048] A computer-readable storage medium storing a computer program, which, when executed, implements a virtual digital human multimedia teaching interaction method.

[0049] The technical effects and advantages of the virtual digital human multimedia teaching interaction method, system, and storage medium provided by this invention are as follows:

[0050] By obtaining the master time reference for the teaching session and generating a unified timestamp for the audiovisual full-dimensional sequence based on the master time reference, an aligned dataset is obtained. This unifies the audio frame sequence, lip-sync parameter sequence, facial expression parameter sequence, subtitle segment sequence, and video frame sequence under the same time reference to express the time position. On this basis, a time window index is generated based on the aligned dataset, and the relationship between the corresponding data of each sequence is established according to the time window identifier. This eliminates the time reference split caused by the asynchronous links using different time calibers from the source. This allows the multimodal content within the same teaching session to obtain a common time caliber that can be directly compared. Any subsequent cross-sequence time position comparison can be completed on the unified timestamp and time window identifier, thereby solving the problems of lacking a unified time reference and difficulty in organizing comparable data across links, and providing an implementable basic data structure.

[0051] By employing a method that combines time window indexing, contiguous data sets, missing and duplicate counts, and risk features, the aligned dataset is reorganized into contiguous data sets within an executable window granularity. Missing and duplicate counts are then statistically analyzed for audio frame sequences, lip-sync parameter sequences, facial expression parameter sequences, subtitle segment sequences, and video frame sequences, yielding risk features that characterize external signs of retransmission backtracking, buffer adjustments, and tempo changes. On one hand, risk features are presented as the difference between expected and actual quantities, derived from objective statistics of contiguous data sets, independent of subjective judgment. On the other hand, subtitle segment sequences are incorporated into a unified window framework with statistical calibers addressing insufficient coverage / excessive overlap, resolving organizational difficulties caused by inconsistencies in the forms of frame-type and segment-type data. Thus, the problem of misalignment occurring covertly and being difficult to detect in advance is transformed into a risk characterization that can be continuously recorded and aggregated within the window boundaries, providing quantifiable input for subsequent alarms, degradation, and blocking.

[0052] Based on a unified time standard and windowed organization, this method calculates time-series consistency indicators based on data groups within the same window and derives a method by comparing various time differences with time difference thresholds to output misalignment alarm results. This achieves unified constraints on four key alignment relationships: audio lip movements, lip-sync subtitles, audio subtitles, and video lip movements. When an alarm occurs, a degradation strategy is selected based on the misalignment alarm results, and an execution record is generated. These records are then aggregated to form an evidence chain data package. When the cumulative number of occurrences exceeds a diffusion blocking threshold within a preset window, the recording output is switched to a video stream that retains audio and subtitles without overlapping lip movements. Finally, a correction release stream is generated based on the evidence chain data package, and when the release threshold is met, the corresponding time segment of the recording is replaced while retaining the hash verification value. Misalignment is windowed and fixed as an alarm result with the threshold category and corresponding time difference value. This can directly drive actions such as freezing lip-sync parameters, realigning lip-sync parameters, aligning subtitles with time delay, and reconstructing the beat of synthesized video frames, avoiding delays caused by relying solely on the playback end's perception. Secondly, the evidence chain data package associates and stores the misalignment alarm result, execution record, summary information, and hash check value, making the occurrence time, the type of offset triggered, the action taken, and the corresponding output version traceable facts. Thirdly, both recording output switching and recording segment replacement are triggered by threshold comparison and limited to the window sequence number range. This can reduce the probability of lip-sync misalignment segments being fixed and propagated during the misalignment propagation stage, and complete controllable replacement afterward with a correction release stream. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0054] Figure 1 This is a flowchart of a virtual digital human multimedia teaching interaction method according to the present invention;

[0055] Figure 2 This is a schematic diagram of a virtual digital human multimedia teaching interactive system according to the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1

[0058] like Figure 1 As shown in the figure, this embodiment of a virtual digital human multimedia teaching interaction method includes the following steps:

[0059] Step 1: Obtain the master time reference for the teaching session; generate a unified timestamp for the audiovisual full-dimensional sequence based on the master time reference to obtain the aligned dataset. The audiovisual full-dimensional sequence is a collection of multimodal sequences collected or generated for the same teaching session. It is generated by the acquisition end, edge synthesis node, or received by the student end, but its belonging object is always the same teaching session. The audiovisual full-dimensional sequence includes at least audio frame sequence, lip-sync parameter sequence, facial expression and action parameter sequence, subtitle segment sequence, and video frame sequence.

[0060] Specifically, the main time reference of the teaching session is obtained, and a one-to-one correspondence between the main time reference and the teaching session is established. During implementation, at the moment the teaching session is established, the acquisition end generates a session number to distinguish the teaching session, and generates the main time reference at the same time as generating the session number. The main time reference is used to characterize the time reference starting point and progress scale of various media data within the teaching session. In order to avoid the introduction of time reference splitting of multimodal data due to the deviation of local clocks of different devices, after generating the main time reference, the acquisition end forms a session-level time reference pair with the main time reference and the session number, and sends them to the edge synthesis node and the student end respectively. At the beginning of the acquisition or generation of audio frame sequence, lip-sync parameter sequence, facial expression parameter sequence, subtitle segment sequence, and video frame sequence, a reference relationship between the session number and the main time reference is established so that subsequent sequences all use the same main time reference as the time conversion basis.

[0061] Furthermore, the acquisition end maintains continuous advancement of the master time reference throughout the session, ensuring that the master time reference maintains a monotonically increasing time scale from the start to the end of the session. Even when frame rate fluctuations occur during the session due to screen switching, screen sharing being enabled or disabled, or changes in camera exposure, the master time reference remains the sole time reference, thus guaranteeing that subsequent unified timestamps have a consistent time origin and time scale. Through this method, the edge synthesis node and the student end share the same master time reference within the same teaching session, serving as the common input for generating unified timestamps for subsequent sequences.

[0062] Based on the master time reference, unified timestamps are generated for both audio and video frame sequences, and these unified timestamps are paired with their corresponding sequences. In practice, the acquisition end performs frame-by-frame processing on the audio to obtain an audio frame sequence. For each audio frame, the acquisition or encapsulation time is obtained and mapped to the time scale under the master time reference to obtain the unified timestamp corresponding to that audio frame. This unified timestamp includes at least the session number and the time scale value, used to characterize the audio frame's temporal position within the teaching session. The audio frame sequence carries the semantic content and rhythmic information of the teacher's explanation, serving as the source for lip-sync parameter sequences and subtitle segment sequences, and is also the primary carrier for students to perceive what was said. Audio is represented in frames rather than as entire audio segments because the teaching session is a continuous streaming output, and the audio's temporal sequence has the finest granularity and strongest continuity. When network jitter or buffer adjustments occur, changes in the temporal position at the audio frame level are reflected first, and the temporal progression of the speech content can be characterized without relying on the video's beat. In other words, the audio frame sequence is the foundational data for constructing a unified timeline and deriving lip-sync and subtitle temporal positions.

[0063] Simultaneously, the acquisition end performs frame processing on the video to obtain a video frame sequence. For each video frame, the acquisition time or encapsulation time of that video frame is obtained, and this time is mapped to the time scale under the main time reference to obtain the unified timestamp corresponding to that video frame. To ensure the comparability of the unified timestamps of the audio frame sequence and the video frame sequence, the same main time reference and the same time scale expression method are used during mapping, so that the audio frame sequence and the video frame sequence have a directly alignable time position description on the same time axis. The video frame sequence is the image carrier, including the courseware sharing screen, the teacher window screen, the virtual digital human rendering screen and its composite result, which is the basis of the final visual output seen by the student. In actual engineering, weak network retransmission, edge transcoding and compositing, and screen switching are more likely to cause changes in the beat and time axis of the video. Therefore, the video frame sequence must be used as the unified timestamp generation object to express which audio segment, which set of lip movements and facial expressions the student sees at a certain moment under the same main time reference. The reason for choosing frame sequences instead of video segments is that screen switching, exposure changes, and overlay layout changes often occur at the frame level, and only frame-level time positions can cover these dynamic changes.

[0064] Furthermore, when there are edge synthesis nodes performing synthesis processing on video frame sequences in the teaching session, the acquisition end sends the corresponding unified timestamp simultaneously when sending the video frame sequence to the edge synthesis node. When the edge synthesis node synthesizes the video frame sequence to generate a synthesized video frame sequence, it retains the time correlation between the synthesized video frame sequence and the main time reference, so that the synthesized video frame sequence continues to correspond to the unified timestamp under the same main time reference, thereby avoiding the introduction of an independent time axis in the synthesis processing.

[0065] Based on the master time reference and combined with the unified timestamps of audio and video frame sequences, unified timestamps are generated for lip-sync parameter sequences, facial expression parameter sequences, and subtitle segment sequences, forming an alignment dataset including five types of sequences. The lip-sync parameter sequence directly determines the correspondence between the virtual digital mouth movement and the pronunciation rhythm, and is the most critical data type for lip-sync consistency with semantics. Many misalignment events do not manifest as audio or video interruptions, but rather as a lack of synchronization between mouth movement and speech content. Therefore, the lip-sync parameter sequence needs to be included in the alignment scope as an independent object. The reason for choosing lip-sync parameters instead of just the rendered mouth image is that lip-sync parameters are an intermediate expression in the generation chain, which can be temporally characterized and established with audio frames before synthesis, avoiding inferring lip-sync solely from the pixel level of the image, reducing data volume and improving controllability, while also leaving a more direct operational object for subsequent downgrading and correction.

[0066] Facial expression and motion parameter sequences carry non-verbal visual cues such as facial expressions, head posture, and body movements, in addition to lip movements. They determine students' understanding of tone, emotion, emphasis, and direction. Even with lip-sync, misalignment between facial expressions / motions and the rhythm of speech or visuals can lead to misinterpretation. Especially in exam reviews and policy interpretations, the timing of facial expressions / motions and their correspondence with speech stress and pauses can affect students' grasp of key content. The reason for choosing facial expression / motion parameters over just the final rendered image is that these parameters are easier to characterize with a unified timestamp and establish a clear correspondence with audio and video frames. They also cover the visual semantic drift caused by edge compositing and weak network buffering due to timeline adjustments. (Subtitle segment sequence) Subtitles are the textual projection of audio content, serving as an important source of information for students watching in weak network conditions, noisy environments, or in silent mode. They are also the most easily cited and tagged evidence carriers in screen recordings. If subtitles are inconsistent with the audio or lip movements, users are more likely to perceive that the content has been rewritten or that the expression is inconsistent. This risk is greater than that of general playback delays. The reason for choosing subtitle segments rather than whole subtitles is that subtitle generation is usually a streaming output. It is necessary to express its coverage time range in segments in order to establish a beginning-end correspondence with the audio frame sequence and reflect the continuous progress of real-time teaching. Including the subtitle segment sequence in the unified timestamp generation range allows us to depict which segment of text corresponds to which segment of audio on the same timeline, thereby avoiding semantic misleading caused by subtitle drift.

[0067] During implementation, the lip-sync parameter sequence comes from the lip-sync change description data output by the audio driver link. When generating each lip-sync parameter, the time position of the corresponding audio frame is read, and the time scale value of that time position under the main time reference is used as the unified timestamp of the lip-sync parameter, so that the lip-sync parameter sequence establishes a direct correspondence with the audio frame sequence on the time axis. When the generation of lip-sync parameters and the generation of audio frames are processed in parallel, the main time reference is still used as a common reference, and the generation time of the lip-sync parameter is mapped to the main time reference to avoid the incomparability of time scales caused by the use of independent local clocks for the lip-sync parameter sequence. The facial expression and action parameter sequence originates from the data output of the facial expression and action generation link. When generating each facial expression and action parameter, the time position of the corresponding video frame or the time position of the corresponding audio frame is read, and a unified timestamp for the facial expression and action parameter is generated under the main time reference. This ensures that the facial expression and action parameter sequence and the video frame sequence or audio frame sequence form an aligned time position description on the same time axis. When the facial expression and action parameter is simultaneously constrained by both audio tone and video rhythm, the audio frame time position and the video frame time position are associated simultaneously under the main time reference with the same time scale, thereby ensuring that the facial expression and action parameter sequence has a consistent time assignment on the multimodal time axis.

[0068] The subtitle segment sequence originates from text segment data output by the speech recognition link. When generating each subtitle segment, the start and end range of the corresponding audio frame is determined, and the time scale value of this range under the main time reference is used as the unified timestamp of the subtitle segment, ensuring that the subtitle segment sequence maintains the same source time reference as the audio frame sequence on the timeline. When subtitle segments are generated word-by-word or in a streaming manner, the subtitle segment is split into multiple subtitle segment units, each of which generates a unified timestamp under the main time reference, ensuring that the subtitle segment sequence continuously covers the time interval corresponding to the audio content throughout the entire session. Finally, the audio frame sequence, lip-sync parameter sequence, facial expression parameter sequence, subtitle segment sequence, and synthesized video frame sequence with unified timestamps are aggregated, merged by session number, and sorted according to the time scale under the main time reference to obtain the aligned dataset.

[0069] The alignment dataset includes at least the data content of each sequence and the corresponding unified timestamp, enabling the alignment dataset to depict the temporal positional relationship between audio, lip movements, facial expressions, subtitles and video on the same timeline, and providing a unified data foundation for subsequent time window-based processing.

[0070] Step 2: Generate a time window index based on the aligned dataset. The time window index includes the association between the time window identifier and the corresponding data of each sequence. Extract the corresponding data according to the time window identifier to form a data group with the same window, and count the missing count and duplicate count to obtain the risk characteristics.

[0071] A time window index is generated based on the aligned dataset. This index includes the association between time window identifiers and corresponding data for each sequence. In implementation, the unified timestamp used in the aligned dataset is used as the sole time reference. The time window length is selected, and the starting point is determined. The selection of the time window length takes into account the frame intervals of the audio frame sequence, the frame intervals of the video frame sequence, and the coverage span of the subtitle segment sequence. This ensures that a single time window can simultaneously accommodate at least one audio frame time position and at least one video frame time position, and can cover the generation rhythm of lip-sync parameter sequences and facial expression parameter sequences. The reason for choosing to use the time window length instead of directly organizing by frame number is that the aligned dataset includes both frame-type data and segment-type data. Data is expressed in discrete time intervals, and fragment data is expressed in start and end ranges. The time window index can incorporate data of different forms into the same organizational framework with a unified time range. When generating the time window index, for each audio frame sequence data, lip-sync parameter sequence data, facial expression parameter sequence data, subtitle fragment sequence data, and video frame sequence data, the time window identifier to which it belongs is determined based on its unified timestamp. Among them, audio frame sequences and video frame sequences fall into a certain time window identifier according to their respective time positions, subtitle fragment sequences determine one or more time window identifiers they cover based on the overlap between their time range and the time window range, and lip-sync parameter sequences and facial expression parameter sequences fall into the corresponding time window identifiers based on their time positions.

[0072] After the above attribution is determined, a time window index is formed, so that each time window identifier is associated with the corresponding data of the audio frame sequence, the corresponding data of the lip shape parameter sequence, the corresponding data of the facial expression parameter sequence, the corresponding data of the subtitle segment sequence, and the corresponding data of the video frame sequence. This forms a relationship between the time window identifier and the corresponding data of each sequence, ensuring that subsequent extraction can be performed consistently according to the same time window identifier.

[0073] Data is extracted from time window identifiers to form contiguous data groups. During implementation, the time window index is used as the extraction basis. For each time window identifier, the corresponding data of the audio frame sequence, lip-sync parameter sequence, facial expression parameter sequence, subtitle segment sequence, and video frame sequence are located sequentially. The corresponding data of each sequence extracted under the same time window identifier are then aggregated into contiguous data groups. Each contiguous data group includes at least the audio frame sequence data set and the video frame sequence data set within that time window, and may include the lip-sync parameter sequence data set, facial expression parameter sequence data set, and subtitle segment sequence data set as needed. The reason for choosing contiguous data groups as the subsequent statistical object is that the risk of multimodal misalignment often manifests as missing, duplicate, or delayed data in different sequences within the same time period. If only observed within a single sequence, it is difficult to characterize the temporal organization of cross sequences. Contiguous data groups can juxtapose multiple sequence data within the same time range, thereby providing a unified statistical boundary for subsequent missing and duplicate counts.

[0074] Meanwhile, to ensure consistency in extraction, the extraction order is fixed as follows: first, extract the data corresponding to the audio frame sequence and the video frame sequence; then, extract the data corresponding to the lip-sync parameter sequence, the facial expression parameter sequence, and the subtitle segment sequence within the same time window. This is because the lip-sync parameter sequence and the subtitle segment sequence are time-dependent on the audio frame sequence, while the facial expression parameter sequence is time-dependent on either the video frame sequence or the audio frame sequence. Extracting the audio frame sequence and the video frame sequence first helps maintain time consistency within the same data group and reduces ambiguity.

[0075] For each peer dataset, missing and duplicate counts are statistically analyzed to obtain risk characteristics. During implementation, for each peer dataset, the expected and actual counts within a given time window are established for audio frame sequences, lip-sync parameter sequences, facial expression parameter sequences, subtitle segment sequences, and video frame sequences. The expected count is determined by the time window length and the time interval characteristics of the sequence in the aligned dataset, while the actual count is determined by the number of corresponding data entries extracted from the peer dataset. When the actual count is less than the expected count, the missing count for the corresponding sequence increases; the missing count characterizes the degree of gaps in the sequence data within that time window. Risk characteristics are determined when the actual count is greater than the expected count or when duplicate data with the same timestamp exists within the peer dataset. When the time window is reached, the repetition count of the corresponding sequence increases. The repetition count is used to characterize the degree to which the sequence data is reused or repeatedly reached within that time window. When calculating the missing count and the repetition count, the missing count is triggered by insufficient coverage for the subtitle segment sequence. That is, when the time window is not covered by any subtitle segment time range, the missing count is increased. The repetition count is triggered by excessive overlap. That is, when the time window is repeatedly covered by multiple subtitle segment time ranges and the text content is repeatedly carried, the repetition count is increased, thus adapting to the time range attributes of the subtitle segment sequence. After completing the missing count and repetition count statistics for each sequence, the missing count and repetition count are summarized according to the time window identifier to form a risk feature.

[0076] The risk features include at least the counts of missing audio frame sequences, the counts of duplicate audio frame sequences, the counts of missing lip-sync parameter sequences, the counts of duplicate lip-sync parameter sequences, the counts of missing facial expression parameter sequences, the counts of duplicate facial expression parameter sequences, the counts of missing subtitle segments, the counts of duplicate subtitle segments, the counts of missing video frame sequences, and the counts of duplicate video frame sequences. The reason for choosing the counts of missing and duplicate sequences as risk features is that factors such as weak network retransmission, buffer adjustment, and edge synthesis queuing often manifest as objective phenomena of missing or extra segments in the data within a time window. The counts of missing and duplicate sequences can be directly obtained from the same window data group without relying on subjective judgment. Furthermore, without introducing additional judgment steps, the organizational anomalies of multiple sequences can be condensed into risk features that can be used for subsequent processing using a unified time window identifier. This completes the coherent implementation of generating a time window index based on the aligned dataset, extracting corresponding data to form a same window data group according to the time window identifier, and obtaining risk features by counting the counts of missing and duplicate sequences.

[0077] For example, taking a teaching session as the object, step one has obtained an aligned dataset, which includes audio frame sequences, lip-sync parameter sequences, facial expression parameter sequences, subtitle segment sequences, and video frame sequences, and each data item has a uniform timestamp. Now, step two generates a time window index, setting the time window length to 200 milliseconds, with the window starting at the start time of the session, resulting in consecutive time window identifiers: window 1 covers 0 to 200 milliseconds, window 2 covers 200 to 400 milliseconds, and window 3 covers 400 to 600 milliseconds.

[0078] When generating a time window index based on the aligned dataset, each sequence of data is grouped into the window according to a unified timestamp: Assuming that within the second window, the audio frame sequence should have 10 frames, which are one frame every 20 milliseconds under normal circumstances; the video frame sequence should have 5 frames, which are one frame every 40 milliseconds under normal circumstances; the lip-sync parameter sequence usually corresponds to 10 lines according to the rhythm of the audio frames; the facial expression parameter sequence usually corresponds to 5 lines; and the subtitle segment sequence should cover the text content of the "second window" during this time period.

[0079] Data is extracted according to the time window identifier of the second window to form a contiguous data group: audio frames, lip-sync parameters, facial expression parameters, subtitle fragments, and video frames with a unified timestamp falling within the range of 200 to 400 milliseconds are extracted from the aligned dataset and aggregated into the contiguous data group of the second window. Assume the actual extraction results are as follows: 8 audio frames were extracted (2 frames are missing), 10 lip-sync parameters were extracted (the number is normal, but 2 of them have the same unified timestamp as the end of the previous window), 5 facial expression parameters were extracted (the number is normal), 0 subtitle fragments were extracted (this window has no subtitles covering it), and 6 video frames were extracted (1 frame is extra, and 1 of them has the same unified timestamp as the previous frame).

[0080] Risk characteristics are obtained by statistically analyzing missing and duplicate counts: For each data group in the second window, the audio frame sequence is statistically analyzed as follows: The expected number of frames is 10, but only 8 are actually present, resulting in a missing count of 2 and a duplicate count of 0. The expected number of lip-sync parameters is 10, but only 10 are actually present, with 2 frames having the same timestamp, resulting in a missing count of 0 and a duplicate count of 2. The expected number of facial expression parameters is 5, but only 5 are actually present, resulting in a missing count of 0 and a duplicate count of 0. The subtitle segment sequence is not covered within the window, resulting in a missing count of 1 and a duplicate count of 0. The expected number of video frame sequences is 5, but only 6 are actually present, with 1 frame having the same timestamp, resulting in a missing count of 0 and a duplicate count of 1. Finally, the risk characteristics of the second window can be represented as: 2 missing audio frames, 0 duplicate audio frames, 0 missing lip-sync frames, 2 duplicate lip-sync frames, 0 missing facial expressions, 0 duplicate facial expressions, 1 missing subtitle frame, 0 duplicate subtitle frames, 0 missing video frames, and 1 duplicate video frame. These risk characteristics, along with the time window identifier of the second window, are recorded for use in subsequent steps.

[0081] Step 3: Calculate the timing consistency index based on the same data group, including at least audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference; the time difference is obtained by subtracting the unified timestamps of the corresponding two types of data; compare the time difference with the time difference threshold, and output the misalignment alarm result if the threshold is exceeded.

[0082] Based on the peer data set, corresponding data pairs for calculating time difference are determined, and a unified timestamp is extracted for each corresponding data pair. The peer data set originates from the audio frame sequence corresponding data, lip-sync parameter sequence corresponding data, facial expression parameter sequence corresponding data, subtitle segment sequence corresponding data, and video frame sequence corresponding data extracted from the alignment dataset under the same time window identifier. The reason for using the peer data set as input is that it naturally limits the same time range, ensuring that the time position comparisons between audio frame sequences, lip-sync parameter sequences, subtitle segment sequences, and video frame sequences all occur within the same time window, thus avoiding time span interference introduced by cross-window comparisons. In this step, four types of corresponding data pairs are established: audio frame sequence corresponding data and lip-sync parameter sequence corresponding data form an audio-lip-sync corresponding data pair; lip-sync parameter sequence corresponding data and subtitle segment sequence corresponding data form a lip-sync subtitle corresponding data pair; audio frame sequence corresponding data and subtitle segment sequence corresponding data form an audio-subtitle corresponding data pair; and video frame sequence corresponding data and lip-sync parameter sequence corresponding data form a video-lip-sync corresponding data pair.

[0083] Audio frame sequence data is used to represent the temporal position of speech content, lip-sync parameter sequence data is used to represent the temporal position of mouth movement, subtitle segment sequence data is used to represent the temporal position of text content coverage, and video frame sequence data is used to represent the temporal position of image presentation. The reason for choosing the above four types of corresponding data pairs is that they respectively cover the four alignment relationships most prone to semantic misinterpretation: speech and lip-sync, lip-sync and subtitle, speech and subtitle, and image and lip-sync. Moreover, their temporal positions can all be expressed by a unified timestamp. To ensure the consistency of the source of subsequent time differences, a unified timestamp corresponding to the two types of data is extracted for each corresponding data pair. Specifically, a unified timestamp for a single time position is extracted for audio frame sequences and video frame sequences, a unified timestamp for a single time position is extracted for lip-sync parameter sequences, and a unified timestamp for a time range is extracted for subtitle segment sequences, including at least the unified timestamp at the beginning and end of the time range, so as to determine the coverage position of the subtitle segment sequence within the same time window.

[0084] Various time differences are obtained by subtracting the unified timestamps of the corresponding two types of data, and a time-series consistency index is formed accordingly. In this step, audio lip-sync time difference, lip-sync caption time difference, audio caption time difference, and video lip-sync time difference are calculated for each group of data. The audio lip-sync time difference is obtained by subtracting the unified timestamp of the data corresponding to the audio frame sequence from the unified timestamp of the data corresponding to the lip-sync parameter sequence, and is used to represent the offset between the speech time position and the lip-sync time position. The lip-sync caption time difference is obtained by subtracting the unified timestamp of the data corresponding to the lip-sync parameter sequence from the unified timestamp of the data corresponding to the caption segment sequence, and is used to represent the offset between the lip-sync time position and the caption coverage time position. The audio caption time difference is obtained by subtracting the unified timestamp of the data corresponding to the audio frame sequence from the unified timestamp of the data corresponding to the caption segment sequence, and is used to represent the offset between the speech time position and the caption coverage time position. The video lip-sync time difference is obtained by subtracting the unified timestamp of the data corresponding to the video frame sequence from the unified timestamp of the data corresponding to the lip-sync parameter sequence, and is used to represent the offset between the screen presentation time position and the lip-sync time position.

[0085] When the data corresponding to the subtitle segment sequence is expressed in terms of time range, the lip-sync subtitle time difference and the audio subtitle time difference are respectively formed by subtracting two sets of results using the unified timestamp of the start and end of the range of the subtitle segment sequence data. The subtraction result that can characterize the subtitle coverage position is selected as the time difference within the time window, thereby ensuring that the time range attribute of the subtitle segment sequence is included in the time difference calculation process.

[0086] Furthermore, to avoid the existence of multiple audio frames or multiple video frames within the same time window, which would lead to non-unique correspondences, the corresponding data of audio frame sequences and video frame sequences within the same window data group are arranged in chronological order according to a unified timestamp. The corresponding data of audio frame sequences and video frame sequences with the unified timestamp closest to the corresponding data of lip-sync parameter sequences are used as the subtraction objects, so that all four types of time differences come from the corresponding data with the closest time position within the same time window. This ensures that the time sequence consistency index reflects the instantaneous offset state within the time window.

[0087] After completing the calculation of the four types of time differences, the audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference are combined to form the time sequence consistency index corresponding to the data group in the same window, and a corresponding relationship is established with the time window identifier so that the misalignment alarm results can be output according to the window in the future.

[0088] The time difference is compared with the time difference threshold. If the threshold is exceeded, a misalignment alarm result is output. In this step, time difference thresholds are set for the four types of time differences. The setting of the four types of time difference thresholds is based on at least the frame interval of the audio frame sequence, the frame interval of the video frame sequence, the coverage granularity of the subtitle segment sequence, and the end-to-end presentation delay constraints of the teaching session in the alignment dataset. The audio lip-sync time difference threshold is used to constrain the allowable offset range of lip-sync relative to speech, the lip-sync subtitle time difference threshold is used to constrain the allowable offset range of lip-sync relative to subtitle, the audio subtitle time difference threshold is used to constrain the allowable offset range of subtitle relative to speech, and the video lip-sync time difference threshold is used to constrain the allowable offset range of lip-sync relative to the screen.

[0089] The time difference threshold can be provided by a preset threshold table. The preset threshold table provides corresponding thresholds at least according to the frame interval of the audio frame sequence, the frame interval of the video frame sequence, and the minimum coverage granularity of the subtitle segment sequence. It can further combine the network transmission link type of the teaching session and whether the edge synthesis node participates in the synthesis process to select the threshold level. In implementation, the timing consistency index of each classmate data group is compared item by item. When the audio lip-sync time difference exceeds the audio lip-sync time difference threshold, a misalignment alarm result including audio lip-sync trigger information is generated. When the lip-sync subtitle time difference exceeds the lip-sync subtitle time difference threshold, a misalignment alarm result including lip-sync subtitle trigger information is generated. When the audio subtitle time difference exceeds the audio subtitle time difference threshold, a misalignment alarm result including audio subtitle trigger information is generated; when the video lip-sync time difference exceeds the video lip-sync time difference threshold, a misalignment alarm result including video lip-sync trigger information is generated; when multiple (more than one) time differences exceed the threshold within the same window data group, the trigger information of each exceeding threshold item is merged and written into the same misalignment alarm result, so that the misalignment alarm result carries the time window identifier, the threshold type, and the corresponding time difference, thereby providing direct input for subsequent steps to process without changing the alignment dataset and the composition of the window data group; the threshold types include audio lip-sync, lip-sync subtitle, audio subtitle, and video lip-sync.

[0090] Step 4: Select a degradation strategy based on the misalignment alarm results and generate an execution record; summarize the misalignment alarm results and execution records to obtain an evidence chain data package; when the cumulative number of misalignment alarm results exceeds the diffusion blocking threshold within a preset window, the recording output is switched to a video stream that retains audio and subtitles and does not overlay lip movements; generate a correction release stream based on the evidence chain data package; when the timing consistency index of the correction release stream meets the release threshold, replace the corresponding time segment of the recording and retain the hash check value.

[0091] Based on the misalignment alarm results, a degradation strategy is selected and an execution record is generated. The misalignment alarm results are derived from the time difference threshold comparison output in step three, and include at least the window number, threshold category, corresponding time difference, and output time. During implementation, the misalignment alarm results are read sequentially according to the window number, and a degradation strategy is selected from the preset degradation strategy set according to the threshold category. The degradation strategies include at least the lip-sync parameter freezing strategy, the lip-sync parameter backtracking and realigning strategy, the subtitle delay alignment strategy, and the synthesized video frame beat reconstruction strategy. Each degradation strategy has directly executable inputs and outputs.

[0092] When the out-of-threshold category includes audio lip movements, either the lip movement parameter freezing strategy or the lip movement parameter backoff realignment strategy is selected; when the out-of-threshold category includes lip movement subtitles or audio subtitles, a subtitle delay alignment strategy is selected; when the out-of-threshold category includes video lip movements, a synthesized video frame beat reconstruction strategy is selected. The lip movement parameter freezing strategy is implemented by maintaining the output of the lip movement parameter sequence at the corresponding trigger window number for a consecutive number of window sequences. The number of consecutive windows to freeze is given by a preset number of frozen windows. For example, if the number of consecutive windows to freeze is 2, then after the trigger window number... The same lip-sync parameter is output within both window numbers. The lip-sync parameter backtracking and realigning strategy is executed by backtracking forward a preset number of windows from the trigger window number and reselecting the time position of the lip-sync parameter sequence, so that the time position of the lip-sync parameter sequence and the time position of the audio frame sequence are within the same window number. For example, the number of backtracking windows is 1. The subtitle delay alignment strategy is executed by delaying the output time of the subtitle segment sequence as a whole by a preset number of delay windows, so that the window number covered by the subtitle segment sequence is consistent with the window number covered by the audio frame sequence. For example, the number of delay windows is 1.

[0093] The execution method of the synthetic video frame beat reconstruction strategy is to redetermine the output beat of the video frame sequence within each window number according to the time scale of the main time base, starting from the trigger window number, so that the output quantity of the video frame sequence within each window number is consistent with the time length corresponding to that window number. The reason for selecting the degradation strategy according to the above threshold category is that the misalignment alarm result has given the corresponding relationship type of the offset occurrence. Directly driving the degradation strategy with the threshold category can avoid introducing additional judgment conditions and keep the degradation strategy and the misalignment alarm result in a one-to-one correspondence. At the same time as selecting and executing the degradation strategy, an execution record is generated. The execution record includes at least the trigger window number, the trigger threshold category, the corresponding time difference, the selected degradation strategy type, the degradation start window number, the degradation end window number, the degradation execution time, and the degradation output version number. The degradation output version number is used to distinguish the output content before and after degradation so as to align with the recording resources later.

[0094] The results of misalignment alarms and execution records are aggregated to obtain an evidence chain data package. During implementation, the misalignment alarm results generated within the same teaching session according to the window number are paired and aggregated with the corresponding execution records to form an evidence chain data package. The evidence chain data package includes at least the teaching session number, window number range, misalignment alarm result list, execution record list, main time base parameter, audio frame sequence summary information, subtitle segment sequence summary information, video frame sequence summary information, downgraded output version number list and generation time. The summary information is used to reduce storage size and maintain traceability. The summary information is obtained by sequentially concatenating the unified timestamp and data length of the corresponding sequence within the window number range and then calculating the summary value.

[0095] For example, within the same teaching session, if the evidence chain data package provides window numbers ranging from window 25 to window 32, and summary information for audio frame sequences, subtitle segment sequences, and video frame sequences needs to be generated, first, the unified timestamp and data length of each data item within that window number range are taken, and then concatenated into a text string with a fixed format in chronological order.

[0096] The audio frame sequence can be assembled into:

[0097] AUDIO T =5000|5020|5040|...|5140;AUDIO L =320|320|...|320, AUDIO is the type marker for the audio frame sequence, used to indicate that subsequent splicing content comes from the audio frame sequence.

[0098] Subtitle segments can be pieced together because they have start and end times:

[0099] SUB_T=5000-5280|5280-5600;SUB_L=24|18, SUB is the type marker for the subtitle segment sequence, used to indicate that the subsequent spliced ​​content comes from the subtitle segment sequence.

[0100] The video frame sequence can be assembled into:

[0101] VIDEO_T=5000|5040|5080|...|5200;VIDEO_L=14500|13980|...|15240, where VIDEO is the type marker for the video frame sequence, indicating that subsequent spliced ​​content comes from the video frame sequence; T is the time information segment marker, indicating that subsequent spliced ​​content has a unified timestamp; when the subtitle segment has a start and end range, T corresponds to the combination of the start and end unified timestamps; L is the length information segment marker, indicating that subsequent spliced ​​content has a data length; where the length of audio and video frames is expressed in bytes, and the length of subtitle segments is expressed in characters; "=" is the assignment separator, used to separate the segment marker from the specific spliced ​​content; ";" is the segment separator, used to separate the time information segment from the length information segment; "|" is the item separator, used to separate multiple items arranged in chronological order within the same segment; "-" is the range separator, used to connect the start and end unified timestamps of the subtitle segment.

[0102] The above-mentioned concatenated strings are then processed to obtain audio summary values, subtitle summary values, and video summary values, which are then written into the evidence chain data packet. This allows for the preservation of a traceable association between the sequence time position and length distribution within the window sequence number range with minimal storage.

[0103] The specific calculation method for the digest value of the above concatenated string is shown in the following example (this rule does not depend on external environment parameters, and any node will obtain the same digest value by calculating according to the same rule):

[0104] First, the concatenated string is converted into a byte sequence using a fixed character encoding, resulting in a sequence of byte values ​​arranged in order, with the byte values ​​ranging from 0 to 255. During the conversion, the numbers, separators, and order in the original concatenated string are not changed, ensuring that the same concatenated strings within the same window sequence number range obtain the same byte sequence.

[0105] Second, set the initial value of the digest accumulation value to 0, and set the multiplier to 131 and the modulus to 1000000007; then process each byte value one by one in the order of the byte sequence, and update the digest accumulation value according to the following rule: digest accumulation value = (digest accumulation value × multiplier + current byte value) modulo 1; after processing all byte values, the final digest accumulation value is obtained.

[0106] Third, the final summary cumulative value is converted into a decimal string as the summary value. To facilitate storage and comparison, the decimal string can be padded with zeros on the left to a fixed length (e.g., 10 or 12 digits) and written into the evidence chain data packet along with the window sequence range. This way, in subsequent tracing, consistency association can be completed simply by regenerating the spliced ​​string and calculating the summary value according to the same rule.

[0107] The reason for choosing to summarize the misalignment alarm results and execution records into an evidence chain data package is that the misalignment alarm results only describe what kind of threshold offset occurred, while the execution records describe what kind of handling action was taken and the range of the effective window number. The combination of the two can completely reflect the continuous link from triggering to handling, providing a locatable window number range and version information for subsequent recording switching and recording replacement. At the same time, retaining the main time base parameter in the evidence chain data package can ensure that the same main time base is still used as the time reference when generating the correction release stream, avoiding inconsistencies between the correction release stream and the original recording resource's time reference.

[0108] The recording output switching is controlled based on the comparison between the cumulative number of misalignment alarm results within a preset number of windows and the diffusion blocking threshold. In practice, misalignment alarm results are counted by window number within the scrolling window statistical range. The preset number of windows is the number of windows used for scrolling statistics. For example, a preset number of 10 indicates the number of misalignment alarm results within 10 consecutive window numbers. When this number exceeds the diffusion blocking threshold, the recording output switching is triggered. The diffusion blocking threshold, for example, is 3, indicating that 4 or more misalignment alarm results within 10 consecutive window numbers trigger the switch. After triggering, the recording output switches to a video stream that retains audio and subtitles without overlapping lip movements. The method for generating the video stream that retains audio and subtitles without overlapping lip movements is as follows:

[0109] The video frame sequence is still output according to the main time base, the subtitle segment sequence is still output according to the subtitle delay alignment strategy and covers the corresponding window number range, and the audio frame sequence is still played in the original output mode, but the mouth movements corresponding to the lip movement parameter sequence are not superimposed in the composite picture; the reason for choosing to switch to a video stream that retains audio and subtitles and does not superimpose lip movements when the diffusion blocking threshold is exceeded is that the misalignment of lip movement parameter sequence and speech is a presentation form that is easily interpreted by viewers as content differences. Removing lip movement superposition can reduce the probability of misaligned segments being solidified in the recording link. At the same time, retaining audio and subtitles can maintain the readability and listenability of teaching information. Moreover, the switching action is triggered by the threshold and limited by the window number range, which has an executable trigger condition; when the switching occurs, the switching start window number, switching end window number and trigger time are written into the execution record and synchronously included in the evidence chain data packet to maintain consistent archiving of recording resources and evidence chain data packets.

[0110] A corrective release stream is generated based on the evidence chain data packet. When the timing consistency index of the corrective release stream meets the release threshold, the corresponding time segment of the recorded broadcast is replaced while retaining the hash check value. During implementation, the trigger window sequence number range and the main time base parameter are extracted from the evidence chain data packet. The audio frame sequence, subtitle segment sequence, and video frame sequence are reorganized according to this window sequence number range. Within this window sequence number range, the output time position of the lip-sync parameter sequence is re-determined using the main time base as a time reference, ensuring that the audio lip-sync time difference between the lip-sync parameter sequence and the audio frame sequence within this window sequence number range does not exceed the release threshold. Simultaneously, the timing consistency index of the corrective release stream is calculated using the same window sequence number range. It includes at least audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference; when the timing consistency index of the corrective release stream meets the release threshold, the corresponding time segment of the recorded broadcast resource corresponding to the window sequence number range is replaced with the corresponding time segment of the corrective release stream, and a hash verification value is calculated for the replaced recorded broadcast corresponding time segment. The hash verification value is stored together with the teaching session number, window sequence number range, and downgraded output version number in the evidence chain data packet, so as to locate the source and version of the same replaced segment in the future; the release threshold can be given by a preset threshold table. For example, a release threshold of 60 milliseconds means that the release threshold is met when all four types of time differences are no greater than 60 milliseconds.

[0111] The principle behind setting the release threshold is as follows: using the unified timestamp under the main time base as the sole criterion, the audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference calculated window by window within the given window sequence number range of the evidence chain data packet are used as constraint objects, and uniform constraints are applied using milliseconds. The release threshold is given through a preset threshold table, which uses a combination of the frame duration of the audio frame sequence, the frame interval of the video frame sequence, and the minimum update interval of the subtitle segment sequence as indexes to output the corresponding release threshold, and the release threshold satisfies the lower bound rule of not being less than the basic time granularity; alternatively, the release threshold is obtained by multiplying the corresponding time difference threshold by a preset coefficient, which is a fixed value and given in the threshold table; when all four types of time differences for each window sequence number within the window sequence number range do not exceed the release threshold, the correction release stream is deemed to meet the release threshold and the corresponding time segment of the recording is replaced, while the hash check value is retained.

[0112] The reason for choosing to generate the correction release stream based on the evidence chain data package is that the evidence chain data package already includes the trigger window sequence number range, misalignment alarm results and execution records, which can directly limit the time range and required material range of the correction release stream, avoid the need to completely remake the entire recording, and use the release threshold as an objective condition when replacing the corresponding time segment of the recording.

[0113] Example 2

[0114] Please see Figure 2As shown, this embodiment provides a virtual digital human multimedia teaching interactive system. Details not shown are described in Embodiment 1. The system includes:

[0115] The master time reference module is used to obtain the master time reference of the teaching session; and to generate a unified timestamp for the full-dimensional audiovisual sequence based on the master time reference to obtain the aligned dataset.

[0116] The window index module generates a time window index based on the aligned dataset. The time window index includes the association between the time window identifier and the corresponding data of each sequence. The corresponding data is extracted according to the time window identifier to form a data group in the same window, and the missing count and duplicate count are counted to obtain the risk characteristics.

[0117] The consistency calculation module calculates timing consistency indicators based on the same window data group, including at least audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference; the time difference is obtained by subtracting the unified timestamps of the corresponding two types of data; the time difference is compared with the time difference threshold, and a misalignment alarm result is output if the threshold is exceeded.

[0118] The correction module selects a degradation strategy and generates an execution record based on the misalignment alarm results; it summarizes the misalignment alarm results and execution records to obtain an evidence chain data packet; when the cumulative number of misalignment alarm results exceeds the diffusion blocking threshold within a preset window, the recording output is switched to a video stream that retains audio and subtitles and does not overlay lip movements; a correction release stream is generated based on the evidence chain data packet, and when the timing consistency index of the correction release stream meets the release threshold, the corresponding time segment of the recording is replaced and the hash check value is retained.

[0119] Example 3

[0120] A computer-readable storage medium according to one embodiment of this application. The computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by a processor, a virtual digital human multimedia teaching interaction method according to an embodiment of this application, as described with reference to the above figures, can be executed. The storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0121] Furthermore, according to embodiments of this application, the processes described in the above-referenced flowcharts can be implemented as computer software programs. For example, this application provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be executed by a processor to perform instructions corresponding to the method steps provided in this application, such as a virtual digital human multimedia teaching interaction method. When this computer program is executed by a central processing unit (CPU), it performs the functions defined in the method of this application.

[0122] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual digital human multimedia teaching interaction method, characterized in that, include: Obtain the main time reference for the teaching session; Generate a unified timestamp for the full-dimensional audiovisual sequence based on the main time reference. The full-dimensional audiovisual sequence includes at least an audio frame sequence, a lip-sync parameter sequence, an facial expression parameter sequence, a subtitle segment sequence, and a video frame sequence to obtain an aligned dataset. A time window index is generated based on the aligned dataset. The time window index includes the association between the time window identifier and the corresponding data of each sequence. The time window identifier is the window number obtained by numbering the time windows according to the time order of the main time base. The corresponding data is extracted according to the time window identifier to form a data group in the same window. The missing count and duplicate count are counted to obtain the risk characteristics. The risk characteristics are associated with the time window identifier as a quantifiable input for subsequent alarms, degradation and blocking. The timing consistency index is calculated based on the same data group, including at least audio lip-sync time difference, lip-sync caption time difference, audio caption time difference, and video lip-sync time difference; the time difference is obtained by subtracting the unified timestamps of the corresponding two types of data; each time difference is compared with the corresponding time difference threshold, and a misalignment alarm result is output when any time difference exceeds the corresponding time difference threshold. The misalignment alarm result includes a time window identifier, the threshold category, and the corresponding time difference. Based on the misalignment alarm results and read them sequentially by window number, a degradation strategy is selected from the degradation strategy set based on the threshold category. The degradation strategies include lip-sync parameter freezing, lip-sync parameter backtracking and realignment, subtitle delay alignment, and composite video frame beat reconstruction. The selected degradation strategy is executed, and an execution record is generated. The execution record includes the trigger window number, threshold category, corresponding time difference, degradation strategy type, degradation start window number, degradation end window number, degradation execution time, and degradation output version number. The misalignment alarm results and execution records are summarized and matched to obtain the evidence. According to the chain data packet, the evidence chain data packet includes the teaching session number, window number range, misalignment alarm result list, execution record list, main time base parameter, downgraded output version number list and generation time; when the number of misalignment alarm results exceeds the diffusion blocking threshold within the preset window number, the recording output is switched to a video stream that retains the audio frame sequence and subtitle segment sequence without superimposing the lip-sync parameter sequence. The video frame sequence is output according to the main time base, the subtitle segment sequence is output according to the subtitle delay alignment strategy, and the audio frame sequence is played in the original output mode. Based on the evidence chain data packet, the trigger window sequence range and the main time reference parameter are extracted. The audio frame sequence, subtitle segment sequence, and video frame sequence are reorganized according to the trigger window sequence range. Using the main time reference parameter as a time reference, the output time position of the lip-sync parameter sequence within the trigger window sequence range is re-determined, generating a corrective release stream. The timing consistency index of the corrective release stream is calculated within the trigger window sequence range. The timing consistency index includes audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference. The timing consistency index is compared with the release threshold. When the timing consistency index meets the release threshold, the corresponding time segment of the recorded broadcast resource corresponding to the trigger window sequence range is replaced with the corresponding time segment of the corrective release stream. A hash check value is calculated for the replaced recorded broadcast corresponding time segment, and the hash check value is associated and stored with the teaching session number, trigger window sequence range, and downgraded output version number in the evidence chain data packet.

2. The virtual digital human multimedia teaching interaction method according to claim 1, characterized in that, The master time base and the teaching session number are synchronously distributed to the edge synthesis nodes and the student terminals; Based on the main time reference, the acquisition time or encapsulation time of the audio frame sequence and the video frame sequence are read and mapped to generate a unified timestamp. Based on a unified timestamp, the corresponding audio frame time position or video frame time position is determined for the lip-sync parameter sequence, facial expression parameter sequence, and subtitle segment sequence, and then mapped to generate a unified timestamp. The audiovisual full-dimensional sequences are aggregated to form an aligned dataset.

3. The virtual digital human multimedia teaching interaction method according to claim 2, characterized in that, Based on the aligned dataset and using a unified timestamp as a time reference, the window start point and time window length are determined, and a time window index is generated. The time window index includes the association between the time window identifier and the audio frame sequence, lip-sync parameter sequence, facial expression parameter sequence, subtitle segment sequence, and video frame sequence. Based on the time window index, extract the corresponding data of each sequence according to the time window identifier to form a data group with the same window; Based on the same data group, the expected number and actual number of each sequence within the time window are calculated to obtain the missing count and duplicate count; Risk characteristics are obtained by summarizing missing and duplicate counts according to time windows. The risk characteristics are then associated with time window identifiers.

4. The virtual digital human multimedia teaching interaction method according to claim 3, characterized in that, Based on the same window data group, establish corresponding data pairs between audio frame sequences and lip-sync parameter sequences, between lip-sync parameter sequences and subtitle segment sequences, between audio frame sequences and subtitle segment sequences, and between video frame sequences and lip-sync parameter sequences. Based on the corresponding data, unified timestamps were extracted from the audio frame sequence, lip-sync parameter sequence, and video frame sequence, and unified timestamps for the start and end points of the subtitle segment sequence were also extracted. Within the same data group, the audio frame sequence and video frame sequence are sorted by a unified timestamp. The data corresponding to the audio frame sequence with the unified timestamp closest to the lip-sync parameter sequence is selected as the audio lip-sync subtraction object, and the data corresponding to the video frame sequence with the unified timestamp closest to the lip-sync parameter sequence is selected as the video lip-sync subtraction object. The audio lip-sync time difference is obtained by subtracting the unified timestamp of the audio lip-sync object from the unified timestamp of the lip-sync parameter sequence; the video lip-sync time difference is obtained by subtracting the unified timestamp of the video lip-sync object from the unified timestamp of the lip-sync parameter sequence. Subtract the unified timestamp of the lip-sync parameter sequence from the unified timestamp of the start and end points of the subtitle segment sequence to obtain two sets of lip-sync subtitle subtraction results, and determine the lip-sync subtitle time difference based on the two sets of lip-sync subtitle subtraction results; The unified timestamp of the audio lip-sync subtraction object is subtracted from the unified timestamp of the start and end points of the subtitle segment sequence to obtain two sets of audio-subtitle subtraction results. The audio-subtitle time difference is determined based on the two sets of audio-subtitle subtraction results. The timing of audio lip movements, lip-sync captions, audio captions, and video lip movements is summarized to obtain a timing consistency index, which is then associated with the time window identifier. Each time difference is compared with the time difference threshold. When any time difference exceeds the corresponding time difference threshold, a misalignment alarm result is output. The misalignment alarm result includes the time window identifier, the threshold category, and the corresponding time difference.

5. The virtual digital human multimedia teaching interaction method according to claim 4, characterized in that, For audio frame sequences, subtitle segment sequences, and video frame sequences, the same timestamp and data length are sequentially concatenated within the window sequence number range, and the summary value is calculated to obtain audio frame sequence summary information, subtitle segment sequence summary information, and video frame sequence summary information, which are then written into the evidence chain data packet.

6. The virtual digital human multimedia teaching interaction method according to claim 5, characterized in that, When the out-of-threshold category includes audio lip movements, select the lip movement parameter freeze strategy or the lip movement parameter backoff realignment strategy; when the out-of-threshold category includes lip movement subtitles or audio subtitles, select the subtitle delay alignment strategy; when the out-of-threshold category includes video lip movements, select the synthesized video frame beat reconstruction strategy. The lip-sync parameter freezing strategy maintains the output of the lip-sync parameter sequence within a consecutive number of window sequences at the lip-sync parameters corresponding to the trigger window sequence number. The lip-sync parameter backtracking and realigning strategy backtracks forward a preset number of window sequences from the trigger window sequence number and reselects the time position of the lip-sync parameter sequence, ensuring that the time position of the lip-sync parameter sequence and the time position of the audio frame sequence are within the same window sequence number. The subtitle delay alignment strategy delays the output time of the subtitle segment sequence by a preset number of delay window sequences, ensuring that the window sequence number covered by the subtitle segment sequence is consistent with the window sequence number covered by the audio frame sequence. The composite video frame beat reconstruction strategy redetermines the output beat of the video frame sequence within each window sequence number based on the time scale of the main time reference, starting from the trigger window sequence number, ensuring that the output quantity of the video frame sequence within each window sequence number is consistent with the time length corresponding to that window sequence number.

7. The virtual digital human multimedia teaching interaction method according to claim 6, characterized in that, Based on the misalignment alarm results and counting by window number within the statistical range of the scrolling window, the cumulative number of times within the preset number of windows is obtained; The cumulative count is compared with the diffusion blocking threshold. When the cumulative count exceeds the diffusion blocking threshold, the recording output is switched. The starting window number, ending window number, and trigger time of the switch are recorded in the execution log, and the execution log is included in the evidence chain data packet.

8. A virtual digital human multimedia teaching interactive system, characterized in that, The system for implementing the virtual digital human multimedia teaching interaction method according to any one of claims 1-7 comprises: The master time reference module obtains the master time reference of the teaching session; generates a unified timestamp for the audiovisual full-dimensional sequence according to the master time reference, wherein the audiovisual full-dimensional sequence includes at least audio frame sequence, lip shape parameter sequence, facial expression and action parameter sequence, subtitle segment sequence and video frame sequence, and obtains an aligned dataset; The window index module generates a time window index based on the aligned dataset. The time window index includes the association between the time window identifier and the corresponding data of each sequence. The time window identifier is the window number obtained by numbering the time windows according to the time order of the main time base. The corresponding data is extracted according to the time window identifier to form a data group in the same window. The missing count and duplicate count are counted to obtain risk characteristics. The risk characteristics are associated with the time window identifier as a quantifiable input for subsequent alarms, degradation and blocking. The consistency calculation module calculates time-series consistency indicators based on the same window data group, including at least audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference; the time difference is obtained by subtracting the unified timestamps of the corresponding two types of data; each time difference is compared with the corresponding time difference threshold, and when any time difference exceeds the corresponding time difference threshold, a misalignment alarm result is output, and the misalignment alarm result includes a time window identifier, the threshold category, and the corresponding time difference; The correction module reads the misalignment alarm results sequentially according to the window number, selects a degradation strategy from the degradation strategy set based on the threshold category, and the degradation strategies include lip-sync parameter freezing strategy, lip-sync parameter backtracking and realignment strategy, subtitle delay alignment strategy, and composite video frame beat reconstruction strategy; executes the selected degradation strategy and generates an execution record, which includes the trigger window number, threshold category, corresponding time difference, degradation strategy type, degradation start window number, degradation end window number, degradation execution time, and degradation output version number; summarizes the misalignment alarm results and execution records and pairs them to obtain an evidence chain data packet, which includes the teaching session number, window number range, misalignment alarm result list, execution record list, main time base parameter, degradation output version number list, and generation time; when the cumulative number of misalignment alarm results exceeds the diffusion blocking threshold within a preset number of windows, the recording output is switched to a video stream that retains the audio frame sequence and subtitle segment sequence without superimposing the lip-sync parameter sequence, the video frame sequence is output according to the main time base, and the subtitle segment sequence is output according to... The subtitle delay alignment strategy outputs a time coverage window number range, and the audio frame sequence is played in its original output mode. Based on the evidence chain data packet, the trigger window number range and the main time reference parameter are extracted. The audio frame sequence, subtitle segment sequence, and video frame sequence are reorganized according to the trigger window number range. Using the main time reference parameter as a time reference, the output time position of the lip-sync parameter sequence within the trigger window number range is redefined, generating a corrective release stream. A timing consistency index for the corrective release stream is calculated within the trigger window number range. This timing consistency index includes audio lip-sync time difference, lip-sync subtitle time difference, audio subtitle time difference, and video lip-sync time difference. The timing consistency index is compared with a release threshold. When the timing consistency index meets the release threshold, the corresponding time segment in the recorded broadcast resources corresponding to the trigger window number range is replaced with the corresponding time segment in the corrective release stream. A hash check value is calculated for the replaced recorded broadcast corresponding time segment, and the hash check value is associated and stored with the teaching session number, trigger window number range, and downgraded output version number in the evidence chain data packet.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed, implements the virtual digital human multimedia teaching interaction method according to any one of claims 1-7.

Images