WebM Protocol Low-Latency Video and Audio Translation and Subtitle Optimization Methods and Systems

Abstract

This invention discloses a method and system for low-latency video and audio translation and subtitle optimization using the WebM protocol, belonging to the field of data processing technology. The method includes: parsing WebM audio and video streams; establishing a video track-audio track association list and constructing a dynamic causal graph; generating initial translated text based on audio frame sequences, video frame sequences, and the dynamic causal graph; extracting lip movement and action semantic data from video frames using a reverse generation model to generate complete translated text and embedding SimpleBlock elements; defining core subtitle genes, non-core genes, and position genes based on bitrate data, complete translated text, and user visual attention heatmaps; dynamically trimming non-core genes based on bitrate fluctuations; and adjusting position genes using heatmaps to generate adaptive subtitle data; and optimizing the edge weights of the dynamic causal graph and subtitle position genes by collecting feedback data through synchronized playback. This invention achieves robustness in audio and video translation and adaptability in subtitle display.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically a method and system for low-latency video and audio translation and subtitle optimization using the WebM protocol. Background Technology

[0002] Existing technologies for handling simultaneous audio and video translation suffer from the following problems: translation content may be interrupted or erroneous when network fluctuations cause audio packet loss; traditional subtitle generation methods cannot adaptively adjust to network conditions and user attention; and there is a lack of effective closed-loop optimization mechanisms to continuously improve translation and subtitle display quality. While WebM, as an open media file format based on the Matroska container format, supports VP9 video encoding and Opus audio encoding, its application in real-time translation and adaptive subtitles still requires further improvement. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a low-latency video and audio translation and subtitle optimization method and system for the WebM protocol. The method involves parsing WebM audio and video streams; establishing a video track-audio track association list and constructing a dynamic causal graph; generating initial translated text based on audio frame sequences, video frame sequences, and the dynamic causal graph; extracting lip movement and action semantic data from video frames using a reverse generation model to generate complete translated text and embedding SimpleBlock elements; defining core, non-core, and positional genes for subtitles based on bitrate data, complete translated text, and user visual attention heatmaps; dynamically cropping non-core genes based on bitrate fluctuations; and adjusting positional genes using heatmaps to generate adaptive subtitle data; and optimizing the edge weights of the dynamic causal graph and subtitle positional genes by collecting feedback data through synchronized playback. This invention achieves robustness in audio and video translation and adaptability in subtitle display.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] WebM protocol low-latency video and audio translation and subtitle optimization methods include:

[0006] Parse WebM audio and video streams to extract Track Link elements, bitrate data, VP9 video frame sequences, Opus audio frame sequences, and SimpleBlock elements;

[0007] Using Track Link elements, VP9 video frame sequences, and Opus audio frame sequences as input, a video track-audio track association list is established, and a timestamp-aligned dynamic causal graph is constructed.

[0008] Using Opus audio frame sequence, VP9 video frame sequence, SimpleBlock elements and dynamic causal graph as input, an initial translation text is generated. When packet loss is detected in the Opus audio frame sequence, lip movement and action semantic data of the corresponding VP9 video frame are extracted. The inverse generation model generates a complete translation text and embeds SimpleBlock elements. The output is a complete translation text and SimpleBlock elements with completion tags.

[0009] Based on bitrate data, complete translated text, and user visual attention heatmaps collected from the terminal, we define core genes, non-core genes, and position genes for subtitles. We dynamically trim non-core genes according to bitrate fluctuations and adjust position genes based on user visual attention heatmaps to generate adaptive subtitle data with timestamps.

[0010] The VP9 video frame sequence, Opus audio frame sequence, SimpleBlock elements with completion tags, and adaptive subtitle data with timestamps are played synchronously according to timestamps. Synchronization errors and user feedback are collected to optimize the edge weights and subtitle position genes of the dynamic causal graph.

[0011] Specifically, the parsing of WebM audio and video streams, extracting Track Link elements, bitrate data, VP9 video frame sequences, Opus audio frame sequences, and SimpleBlock elements, includes:

[0012] The WebM parser is used to read the input audio and video stream data by cluster, parse the segment information and track information elements at the container level, and identify and extract Track Link elements from the track information elements; the Track Link elements are used to describe the relationship between the video track and the audio track.

[0013] After completing the container structure parsing, the cluster data in the audio and video stream data are read sequentially, and the size and duration of each cluster are statistically analyzed in real time, and the bitrate data is calculated; the bitrate data includes the instantaneous bitrate and moving average bitrate of video and audio.

[0014] The data blocks within each cluster are demultiplexed to separate the VP9 video frame sequence and the Opus audio frame sequence. During the demultiplexing process, the original SimpleBlock structure corresponding to each data block is extracted synchronously and encapsulated into a complete SimpleBlock element. The VP9 video frame sequence contains time-stamped frame data and frame type identifiers. The Opus audio frame sequence contains time-stamped audio packets and encoding parameters. The SimpleBlock element is an audio frame encapsulation unit that contains the timestamp, track number, and data payload of the audio and video data blocks.

[0015] Specifically, the process of establishing the video track-audio track association list includes:

[0016] Using the Track Link element as input, parse the track link information defined therein, and establish a track pair that associates the track number of the video track with the track number of the audio track;

[0017] Using VP9 video frame sequences and Opus audio frame sequences as input, based on the track number carried in each frame of data, video frames and audio frames belonging to the same associated track pair are classified into the same processing group using associated track pairs.

[0018] For each processing group, sorting and alignment are performed based on the timestamps of the VP9 video frame sequence and Opus audio frame sequence within the group, using the time axis as the reference.

[0019] For each audio frame, search all video frames within a first predetermined time tolerance before and after it, establish a mapping relationship, and generate a video track-audio track association list; each item in the video track-audio track association list records the timestamp and data pointer of an audio frame, as well as the timestamps and data pointers of all video frames associated with it.

[0020] Specifically, the construction process of the timestamp-aligned dynamic causal graph includes:

[0021] Action recognition is performed on the VP9 video frame sequence to extract action feature vectors from consecutive video frames and generate a video action time sequence.

[0022] Event detection is performed on the Opus audio frame sequence, and event feature vectors in the audio frames are extracted, including volume changes and spectral feature changes, to generate an audio event time sequence.

[0023] Set the first time window threshold and the second time window threshold;

[0024] The starting timestamp of the action feature vector of each video is extracted from the video action time sequence, and the starting timestamp of the event feature vector of each audio is extracted from the audio event time sequence.

[0025] For each pair of video action and audio event, calculate the absolute value of their timestamp difference. If the absolute value is less than or equal to the first time window threshold, calculate the Pearson correlation coefficient between the action feature vector and the event feature vector. If the Pearson correlation coefficient is higher than the preset correlation coefficient threshold, it is determined to be a strong causal relationship.

[0026] If the absolute value is greater than the first time window threshold but less than or equal to the second time window threshold, and the Pearson correlation coefficient is positive, then it is determined to be a weak causal relationship.

[0027] Using each item in the video action time sequence and audio event time sequence as a node, and the strong causal relationship and weak causal relationship as directed edges, an initial causal graph is constructed. Based on the timestamp information in the video track-audio track association list, the nodes and directed edges in the initial causal graph are time-stamp aligned to form a time-stamp aligned dynamic causal graph.

[0028] Specifically, the step of generating initial translated text by taking Opus audio frame sequences, VP9 video frame sequences, SimpleBlock elements, and dynamic causal graphs as input includes:

[0029] The Opus audio frame sequence is decoded to obtain a continuous audio signal, which is then input into an automatic speech recognition engine to generate initial ASR text.

[0030] Using a dynamic causal graph as input, for a specified time period in the initial ASR text, it is verified whether there is a strong causal relationship within the specified time period. If there is, the corresponding video action semantic tags are extracted from the video action nodes that have a strong causal relationship with the current audio event node.

[0031] The semantic consistency of the video action semantic tags is compared with the text content of the corresponding time period in the initial ASR text. When ambiguity or conflict is found, the text is corrected using the video action semantic tags to generate a semantically consistent initial translation text.

[0032] Specifically, when packet loss is detected in the Opus audio frame sequence, the extraction of lip movement and action semantic data of the corresponding VP9 video frames for that time period includes:

[0033] The continuity of Opus audio frame sequences is monitored in real time. When a discontinuity in timestamps is detected, it is determined as a packet loss event, and the packet loss period is determined based on the interruption timestamp.

[0034] The video frames corresponding to the packet loss period are extracted from the VP9 video frame sequence. Face detection and lip region localization are performed on each video frame. The geometric features of the key points of the lips are extracted to form a lip movement feature sequence.

[0035] The lip movement feature sequence is input into a pre-trained lip movement recognition model, which outputs candidate text fragments. At the same time, video action semantic tags associated with all video action nodes during the packet loss period are retrieved from the dynamic causal graph. The video action semantic tags are used to supplement the contextual semantics and correct the logic of the candidate text fragments, generating lip movement and action semantic data.

[0036] Specifically, the step of generating the completed translated text via the reverse generation model and embedding SimpleBlock elements includes:

[0037] The lip movement and action semantic data are input into a pre-trained inverse generation model; the inverse generation model is a sequence generation model based on an encoder-decoder architecture, in which the encoder encodes the input data, the decoder generates a text sequence based on the encoding result, and the output is the completed translation text;

[0038] Using the initial translated text and the completed translated text as input, the completed translated text is merged and inserted into the corresponding position of the initial translated text according to the timestamp of the packet loss period corresponding to the completed translated text. Completion marks are added at the beginning and end positions of the merging and insertion to generate the complete translated text.

[0039] The complete translated text is written as user private data into the data payload of the SimpleBlock element, replacing or appending the original audio and video data to generate a SimpleBlock element with completion tags.

[0040] Specifically, the process of dynamically cropping non-core genes based on bitrate fluctuations and adjusting positional genes based on user visual attention heatmaps to generate timestamped adaptive subtitle data includes:

[0041] The system monitors the moving average bitrate in real time and generates a subtitle trimming trigger signal when the moving average bitrate is lower than a preset first bitrate threshold.

[0042] Based on the dynamic causal graph, non-core genes corresponding to weak causal relationships in the complete translated text are identified and trimmed to output semantically trimmed subtitle text.

[0043] The coordinates of user gaze points are extracted from the user visual attention heatmap collected from the terminal to form a gaze point coordinate sequence. The distribution of the gaze point coordinate sequence on the video screen is calculated to determine the visual focus area.

[0044] If the visual focus area is concentrated and stable, the edge position coordinates that avoid the visual focus area are output. If the visual focus area is scattered or has no focus, the default position coordinates are output. The output position coordinates are defined as the optimized position gene.

[0045] The semantically cropped subtitle text and optimized positional genes are bound to the corresponding timestamps of the VP9 video frame sequence and Opus audio frame sequence to generate time-stamped adaptive subtitle data.

[0046] Specifically, the step of synchronously playing VP9 video frame sequences, Opus audio frame sequences, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps according to timestamps, collecting synchronization errors and user feedback, and optimizing the edge weights and subtitle position genes of the dynamic causal graph includes:

[0047] Based on a unified timeline, VP9 video frame sequences, Opus audio frame sequences, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps are rendered and played synchronously.

[0048] During playback, by comparing the presentation timestamps of video frames, audio frames, and adaptive subtitle data with a unified timeline, audio-video synchronization error and word-sound synchronization error are calculated in real time and merged into a synchronization error dataset. At the same time, a feedback interface is provided on the playback interface to collect user feedback rating data on the subtitles, and the feedback rating data is associated with and stored with the synchronization error dataset according to time intervals.

[0049] Identify the time interval of the synchronization error, and adjust the weight of the edges in the corresponding time interval in the dynamic causal graph based on the identification results;

[0050] When the feedback score data is lower than the preset score threshold, location gene optimization is triggered.

[0051] The WebM protocol low-latency video and audio translation and subtitle optimization system includes: a stream parsing module, a causal modeling module, a translation and compensation module, a subtitle generation module, and a feedback optimization module;

[0052] The stream parsing module is used to deconstruct the original WebM audio and video streams into structured data;

[0053] The causal modeling module is used to analyze the spatiotemporal intrinsic relationship between video and audio, and to construct a timestamp-aligned dynamic causal graph.

[0054] The translation and compensation module is used to generate complete translated text;

[0055] The subtitle generation module is used to dynamically trim non-core genes based on dynamic causal graphs and bitrate data, and bind them with audio and video timestamps to generate the final adaptive subtitle data.

[0056] The feedback optimization module is used to present adaptive subtitle data and collect user feedback to optimize the dynamic causal graph and subtitle position genes.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] 1. This invention proposes a low-latency video and audio translation and subtitle optimization system based on the WebM protocol, and optimizes and improves its architecture, operation steps and processes. The system has the advantages of simple process, low investment and operating costs and low production costs.

[0059] 2. This invention proposes a low-latency video and audio translation and subtitle optimization method based on the WebM protocol. By constructing a dynamic causal graph and a reverse generation model, the accuracy and robustness of audio and video translation are improved. When the audio transmission is complete, the system utilizes the strong causal relationship between audio and video to optimize the translation quality. When audio packet loss occurs, the system extracts lip movement features and action semantics from video frames to generate complete translation text, effectively ensuring the continuity of the translated content. This dual guarantee mechanism ensures accurate and complete translation results under various network conditions.

[0060] 3. This invention proposes a low-latency video and audio translation and subtitle optimization method for the WebM protocol. It introduces a subtitle gene regulation mechanism to achieve adaptive optimization of subtitle display. The subtitle style is dynamically adjusted according to the real-time bitrate, and the display position is intelligently adjusted in combination with the user's visual attention heatmap. This ensures the effective transmission of core information while avoiding interference with the main visual content. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the WebM protocol low-latency video and audio translation and subtitle optimization method of the present invention;

[0062] Figure 2 This is a flowchart illustrating the principle of the WebM protocol-based low-latency video and audio translation and subtitle optimization method of the present invention.

[0063] Figure 3 This is a system architecture diagram for low-latency video and audio translation and subtitle optimization using the WebM protocol, as presented in this invention. Detailed Implementation

[0064] Example 1:

[0065] Please see Figure 1 and Figure 2 The present invention provides an embodiment of a low-latency video and audio translation and subtitle optimization method for the WebM protocol, the method comprising S1 to S5, including the following steps:

[0066] S1: Parse WebM audio and video streams, extract Track Link elements, bitrate data, VP9 video frame sequences, Opus audio frame sequences, and SimpleBlock elements;

[0067] S2: Using Track Link elements, VP9 video frame sequences, and Opus audio frame sequences as input, establish a video track-audio track association list and construct a timestamp-aligned dynamic causal graph;

[0068] S3: Taking Opus audio frame sequence, VP9 video frame sequence, SimpleBlock elements and dynamic causal graph as input, generate initial translation text. When packet loss is detected in the Opus audio frame sequence, extract the lip movement and action semantic data of the corresponding time period of the VP9 video frame. Generate the completed translation text through the inverse generation model and embed SimpleBlock elements. Output the complete translation text and SimpleBlock elements with completion tags.

[0069] S4: Based on bitrate data, complete translated text, and user visual attention heatmaps collected by the terminal, define the core genes, non-core genes, and position genes of the subtitles. Dynamically trim the non-core genes according to bitrate fluctuations and adjust the position genes based on the user visual attention heatmap to generate adaptive subtitle data with timestamps.

[0070] Furthermore, the user visual attention heatmap is generated by collecting user eye movement data from the terminal camera, with the sampling frequency synchronized with the video frame rate.

[0071] Furthermore, the core gene of the subtitle is the key semantic unit in the complete translated text that corresponds to a strong causal relationship, the non-core gene is the auxiliary semantic unit in the complete translated text that corresponds to a weak causal relationship or an irrelevant relationship, and the position gene is used to describe the display position coordinates of the subtitle in the video frame.

[0072] S5: Synchronize the VP9 video frame sequence, Opus audio frame sequence, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps according to timestamps, collect synchronization errors and user feedback, and optimize the edge weights and subtitle position genes of the dynamic causal graph.

[0073] In summary, the implementation logic of this application begins with deep analysis of WebM audio and video streams, extracting audio and video frame sequences, bitrates, and container metadata containing temporal relationships. Then, by analyzing the spatiotemporal correlation between video actions and audio events, a dynamic causal graph that characterizes the causal relationships of audio and video content is constructed. Based on this, speech recognition is performed to generate initial translated text. When audio packet loss is detected, the dynamic causal graph is used to retrieve the corresponding video frame lip movement features and action semantics. The lost translated text is then completed through a reverse generation model, achieving visual-driven translation error tolerance. Subsequently, based on real-time network bitrate and user visual attention heatmaps, non-core subtitle elements are dynamically trimmed and their display positions optimized, generating an adaptive subtitle data stream. Finally, during synchronized playback, errors and user feedback are collected to form a closed-loop optimization system, continuously adjusting the weights of the dynamic causal graph and the subtitle positioning strategy, thereby achieving low-latency, highly robust intelligent translation and subtitle experience.

[0074] From another perspective, this application primarily establishes a semantic bridge between audio and video content through a dynamic causal graph, and utilizes this bridge to enhance the robustness of translation and achieve adaptive subtitle generation. Specifically, the system first parses the streaming media and extracts basic units; then it analyzes the spatiotemporal and statistical correlation between video actions and audio events to construct a dynamic causal graph that characterizes their inherent causal relationships. This dynamic causal graph has two main functions: first, it provides visual context to improve accuracy during audio translation and drives a reverse generation model for visual completion when audio packets are lost; second, it serves as the basis for intelligent subtitle cropping, combining real-time bitrate and user gaze to generate adaptive subtitle data. Finally, the system forms a closed loop by synchronizing playback errors and user feedback data to continuously optimize the accuracy of the dynamic causal graph and the subtitle display strategy, thereby achieving low-latency, high-accuracy, and highly adaptive audio and video translation and subtitle experience.

[0075] For example, in a video conference scenario, a speaker is explaining product features while using gestures. After parsing the audio and video streams, the system constructs a dynamic causal graph, establishing a strong causal relationship between a finger pointing to a specific chart on the screen and the audio event, i.e., "Now let's look at this data." During transmission, if a brief audio segment describing key features is lost, the system immediately detects the packet loss and extracts the speaker's lip movement features from the corresponding video frame. Simultaneously, it retrieves the semantic meaning of the finger pointing to the chart from the dynamic causal graph. Based on visual information, the reverse generation model successfully generates a complete translation of "peak performance improved by 20%." At the same time, due to network fluctuations and reduced bitrate, the system automatically trims non-core modifiers such as "as you can see" from the subtitles, retaining only the core information "peak performance improved by 20%." Based on the heatmap of the area where the user is currently looking at the chart, the system intelligently adjusts the subtitles to the blank space at the top of the screen to avoid obstruction. After the meeting, the system fine-tunes the weights of the causal relationship between gestures and explanations based on the synchronous data of the meeting and the lack of user feedback, making the recognition more accurate.

[0076] The process of parsing the WebM audio and video streams and extracting Track Link elements, bitrate data, VP9 video frame sequences, Opus audio frame sequences, and SimpleBlock elements includes:

[0077] A1: Use the WebM parser to read the input audio and video stream data by cluster, parse the segment information and track information elements at the container level, and identify and extract Track Link elements from the track information elements; the Track Link elements are used to describe the association between the video track and the audio track;

[0078] Furthermore, track information usually follows segment information, forming a container within a container, which can be understood as a nested logic that encapsulates the definitions of all tracks.

[0079] Furthermore, the specific steps of A1 include:

[0080] (1) Use the WebM parser to read the input audio and video stream data. First, locate and parse the segment information element at the container level. After successful parsing, extract the segment unique identifier from the segment information element and use it as the global context of the entire media segment.

[0081] (2) After completing the segment information parsing, the track information element at the container level is located and parsed. All track entry sub-elements within the track information element are traversed. For each track entry, its track number, track type code, and track unique identifier are read in sequence to construct a track list containing all basic track attributes.

[0082] (3) Check each track entry sub-element in the track list in turn, and search for the existence of track link sub-elements within each track entry sub-element;

[0083] (4) When a track link sub-element is found in any track entry sub-element, the track link sub-element is parsed, the unique identifier of the link track contained therein is extracted, and then the unique identifier of the current track is combined with the extracted unique identifier of the link track to form a track association pair and recorded.

[0084] (5) After completing the traversal of the entire track list, summarize and encapsulate all the recorded track associations to finally generate Track Link elements that describe the relationship between video tracks and audio tracks.

[0085] A2: After completing the container structure parsing, the cluster data in the audio and video stream data are read sequentially, and the size and duration of each cluster are statistically analyzed in real time, and the bitrate data is calculated; the bitrate data includes the instantaneous bitrate and moving average bitrate of video and audio. The real-time statistical process is implemented through statistical analysis methods, and statistical analysis is the existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0086] A3: Demultiplex the data blocks within each cluster to separate the VP9 video frame sequence and the Opus audio frame sequence. During the demultiplexing process, the original SimpleBlock structure corresponding to each data block is extracted synchronously and encapsulated into a complete SimpleBlock element. The VP9 video frame sequence contains time-stamped frame data and frame type identifiers. The Opus audio frame sequence contains time-stamped audio packets and encoding parameters. The SimpleBlock element is an audio frame encapsulation unit, containing the timestamp, track number, and data payload of the audio and video data blocks.

[0087] Furthermore, the specific steps in A3 include:

[0088] (1) Obtain the data clusters sequentially parsed from the WebM audio and video streams, read the cluster timestamp contained in the header of the data cluster, and use it as the common time reference for all data blocks in this cluster;

[0089] (2) Read each data block in the data cluster one by one. For each data block, parse its block header, extract the track number used to identify the track to which it belongs and the block time code relative to the cluster timestamp, and calculate the absolute timestamp of the data block based on the cluster timestamp and the block time code.

[0090] (3) Using the extracted track number, absolute timestamp, and data block payload data as input, query the pre-stored track list to determine the track type: If it is a video track, use the payload data as VP9 video frame data, parse its header to obtain the frame type identifier, combine it into a timestamped VP9 video frame, and append it to the VP9 video frame sequence; if it is an audio track, use the payload data as an Opus audio packet, combine it with the encoding parameters to form a timestamped Opus audio frame, and append it to the Opus audio frame sequence.

[0091] (4) While processing each data block, the absolute timestamp, track number and a data pointer pointing to the corresponding frame data stored in the VP9 video frame sequence or Opus audio frame sequence are used as input to encapsulate a structured SimpleBlock element.

[0092] The process of establishing the video track-audio track association list includes:

[0093] B1: Taking the Track Link element as input, parse the track link information defined therein, and establish the associated track pairs between the track numbers of the video track and the track numbers of the audio track;

[0094] Furthermore, the specific steps for B1 include:

[0095] (1) Parse the internal structure of the Track Link element, extract all original link relationships represented by track unique identifiers, and generate a list of original track link relationships. Each item in the list of original track link relationships contains a source track unique identifier and a target track unique identifier.

[0096] (2) Traverse each track entry in the track list, read the track unique identifier and track number of each entry, and construct a track mapping dictionary; the track mapping dictionary uses the track unique identifier as the key and the corresponding track number as the value;

[0097] (3) Traverse the original track link relationship list. For each item in the original track link relationship list, use the unique identifier of the source track contained therein to query the track mapping dictionary to obtain the source track number, and use the unique identifier of the target track to query the target track number. Combine the queryed pair of source track numbers and target track numbers into an associated track pair.

[0098] (4) Summarize all generated associated track pairs to form the final associated track pair set; the associated track pair set clearly defines the track number correspondence between all video tracks and audio tracks that have a link relationship.

[0099] B2: Taking the VP9 video frame sequence and Opus audio frame sequence as input, based on the track number carried in each frame of data, the video frames and audio frames belonging to the same associated track pair are classified into the same processing group using associated track pairs.

[0100] B3: For each processing group, sort and align the VP9 video frame sequence and Opus audio frame sequence based on their respective timestamps and time axis.

[0101] B4: For each audio frame, search all video frames within a first predetermined time tolerance before and after it, establish a mapping relationship, and generate a video track-audio track association list; each item in the video track-audio track association list records the timestamp and data pointer of an audio frame, as well as the timestamps and data pointers of all video frames associated with it.

[0102] Furthermore, the specific steps of B4 include:

[0103] (1) Taking the video frame sequence and audio frame sequence classified by track number and the set of associated track pairs as input, extract the corresponding video frame sequence and audio frame sequence to be associated according to the track number specified in the associated track pair, and ensure that both sequences are arranged in ascending order of their timestamps;

[0104] (2) According to the audio frame sequence to be associated, each audio frame is traversed in turn. For the currently traversed audio frame, a first predetermined time tolerance is extended forward and backward with its timestamp as the center, and a time search window with the audio frame as the core is calculated.

[0105] (3) Taking the obtained sequence of video frames to be associated and the time search window determined for the current audio frame as input, in the sequence of video frames to be associated, the first video frame with a timestamp greater than or equal to the start time of the time search window is located using the binary search method. Starting from this position, the sequence of video frames to be associated is traversed sequentially, and all video frames with timestamps less than or equal to the end time of the time search window are filtered out to form a list of candidate associated video frames corresponding to the current audio frame. The binary search method is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0106] (4) Encapsulate the currently traversed audio frame, its timestamp and data pointer, and the list of candidate associated video frames selected into an audio-video association item. This audio-video association item fully records the mapping relationship between an audio frame and all the video frames that are associated with it in time.

[0107] (5) After processing all audio frames of the audio frame sequence to be associated, all audio and video association items generated for each audio frame will be summarized to form the final video track-audio track association list.

[0108] The construction process of the timestamp-aligned dynamic causal graph includes:

[0109] C1: Perform action recognition on the VP9 video frame sequence, extract action feature vectors from consecutive video frames, and generate a video action time sequence;

[0110] Furthermore, the specific steps of C1 include:

[0111] (1) Obtain the VP9 video frame sequence, which contains compressed frame data and its timestamps;

[0112] (2) Call the VP9 decoder to decode each frame of compressed data to obtain the original bitmap image. Then, perform size adjustment and pixel value normalization on each bitmap image to generate a standardized image with uniform specifications.

[0113] (3) Input each standardized image into a pre-trained deep convolutional neural network, read the activation values ​​of neurons in the feature layer of the deep convolutional neural network, arrange them into a one-dimensional numerical array, and generate a deep feature vector representing the visual content of a single frame. The deep convolutional neural network is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0114] (4) Cache the continuous depth feature vectors in a queue, calculate the difference between the depth feature vector of the current frame and the depth feature vector of the previous frame, and generate a motion feature vector that represents the trend of visual content change between consecutive frames.

[0115] (5) The depth feature vector generated for the current frame and the motion feature vector generated for the current frame are concatenated and fused into a composite motion feature vector that contains both static scene information and dynamic change information.

[0116] (6) For each frame in the VP9 video frame sequence, the timestamp of the obtained frame is bound to the composite action feature vector generated for the frame. The feature vectors bound to all frames are arranged in the order of the timestamps to finally generate an ordered video action time sequence.

[0117] C2: Perform event detection on the Opus audio frame sequence, extract event feature vectors from the audio frames, including volume changes and spectral feature changes, and generate an audio event time sequence;

[0118] Furthermore, the specific steps of C2 include:

[0119] (1) Obtain the Opus audio frame sequence; the Opus audio frame sequence contains compressed audio packets and their timestamps;

[0120] (2) Call the Opus decoder to decode each frame of compressed audio packets, restore the compressed code to a linear pulse code modulation audio signal, and obtain the original time-domain audio signal.

[0121] (3) Divide the continuous original time-domain audio signal into multiple short-time analysis windows, calculate the sum of squares of the amplitudes of all samples in each short-time analysis window, obtain the short-time energy value of each time segment, detect the sharp rise point of the energy value by comparing the short-time energy values ​​of the continuous analysis windows, and record the intensity of the volume change within each audio frame time segment.

[0122] (4) Apply Fast Fourier Transform to the time-domain audio signal within each short-time analysis window to transform it from the time domain to the frequency domain, obtain the spectrum of each time segment, and calculate the spectral centroid and spectral spread of each spectrum. By analyzing the changes in the spectral centroid and spectral spread of consecutive short-time analysis windows, calculate the change in spectral characteristics within each audio frame time period. Here, Fast Fourier Transform is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0123] (5) Combine the volume change intensity and the obtained spectral feature changes into a multi-dimensional numerical array to generate an audio event feature vector representing the comprehensive auditory events in that time period;

[0124] (6) For each frame in the Opus audio frame sequence, the timestamp of the acquired audio frame is bound to the audio event feature vector. The feature vectors bound to all audio frames are arranged in chronological order of the timestamps to finally generate an ordered audio event time sequence.

[0125] C3: Set the threshold for the first time window and the threshold for the second time window;

[0126] Furthermore, the first time window threshold is used to define the maximum time offset that may constitute a strong causal relationship between video actions and audio events; the second time window threshold is greater than the first time window threshold and is used to define the maximum time offset that constitutes a weak causal relationship.

[0127] Furthermore, the first time window threshold ranges from 0 to 200 milliseconds, and the second time window threshold ranges from 200 milliseconds to 500 milliseconds.

[0128] C4: Extract the start timestamp of the action feature vector of each video from the video action time sequence, and at the same time, extract the start timestamp of the event feature vector of each audio from the audio event time sequence;

[0129] C5: For each pair of video action and audio event, calculate the absolute value of their timestamp difference. If the absolute value is less than or equal to the first time window threshold, calculate the Pearson correlation coefficient between the action feature vector and the event feature vector. If the Pearson correlation coefficient is higher than the preset correlation coefficient threshold, it is determined to be a strong causal relationship. The Pearson correlation coefficient is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0130] C6: If the absolute value is greater than the first time window threshold but less than or equal to the second time window threshold, and the Pearson correlation coefficient is positive, then it is determined to be a weak causal relationship;

[0131] Furthermore, in this invention, determining a strong causal relationship requires that the Pearson correlation coefficient be higher than a preset correlation coefficient threshold. This is because it requires not only that the changes in their features be statistically correlated, but also that this correlation be strong enough to capture causal events with a very high degree of certainty. However, when determining a weak causal relationship, it is not necessary for the features to have a strong statistical correlation, as this would miss many meaningful but delayed associations. Therefore, the criterion is relaxed to simply require a positive correlation, indicating that the trends in feature changes are consistent, and although the synchronicity is not strong, they may still belong to different stages of the same macroscopic event.

[0132] C7: Using each item in the video action time sequence and audio event time sequence as a node, and the strong causal relationship and weak causal relationship as directed edges, an initial causal graph is constructed. Based on the timestamp information in the video track-audio track association list, the nodes and directed edges in the initial causal graph are time-stamp aligned to form a time-stamp aligned dynamic causal graph. The process of constructing the graph under the premise of known nodes and edges is prior art in this field and is not an inventive solution of this application, and will not be described in detail here.

[0133] The process of generating initial translated text by taking Opus audio frame sequences, VP9 video frame sequences, SimpleBlock elements, and dynamic causal graphs as input includes:

[0134] D1: Decode the Opus audio frame sequence to obtain a continuous audio signal, and input the audio signal into the automatic speech recognition engine to generate the initial ASR text;

[0135] Furthermore, the specific steps of D1 include:

[0136] (1) Obtain the Opus audio frame sequence, call the Opus decoder to decode each frame of compressed audio packet, restore the compressed code to the standard pulse code modulation audio sample, and then splice all the decoded discrete audio samples according to their timestamp order to reconstruct a complete and continuous time domain audio signal stream.

[0137] (2) The output time-domain audio signal stream is preprocessed, including applying a digital filter to suppress background noise, performing gain control to make the volume stable, and performing a resampling operation to ensure that the signal format is uniform. After processing, a purified preprocessed audio signal that meets the input requirements of the automatic speech recognition engine is obtained. The digital filter is a prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0138] (3) Based on the output preprocessed audio signal, the automatic speech recognition engine first divides the continuous preprocessed audio signal into short time frames. For each frame signal, the acoustic features are calculated using Mel frequency cepstral coefficients. This process transforms the audio waveform into a set of stable numerical sequences that can characterize phoneme attributes and generates an acoustic feature vector sequence. Mel frequency cepstral coefficients are existing technology in this field and are not an inventive solution of this application. They will not be described in detail here.

[0139] (4) Input the acoustic feature vector sequence into the pre-trained acoustic model. The acoustic model analyzes the acoustic feature vector sequence and outputs a preliminary phoneme sequence. Then, call the pronunciation dictionary to map the phoneme sequence into a candidate word sequence. The acoustic model is existing technology in this field and is not an inventive solution of this application. It will not be described in detail here.

[0140] (5) Input the candidate word sequence into the language model. The language model scores the candidate word sequence based on its language knowledge base, selects the optimal word sequence that is most coherent in the context and most in line with language habits, and performs post-processing and text normalization on the optimal word sequence to generate the final initial ASR text. The language model is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0141] D2: Using the dynamic causal graph as input, for a specified time period in the initial ASR text, verify whether there is a strong causal relationship within the specified time period. If there is, extract the corresponding video action semantic tags from the video action nodes that have a strong causal relationship with the current audio event node.

[0142] Furthermore, the specific steps of D2 include:

[0143] (1) Obtain the initial ASR text, select the specific text segment to be verified, read the start and end timestamps corresponding to the text segment, and define a time query window for querying in the graph;

[0144] (2) Obtain the timestamp-aligned dynamic causal graph and the determined time query window, traverse all edges in the dynamic causal graph, and filter out edges that simultaneously meet the following two conditions: the weight of the edge is greater than or equal to the strong relation threshold, and the timestamps of the audio and video nodes connected to the edge all fall within the time query window; classify all edges that meet the conditions into a set of strong causal relation edges.

[0145] (3) Traverse each edge in the selected set of strong causal relationship edges, extract video action nodes from the two nodes connected by the edge, read the predefined video action semantic tags from each extracted video action node, summarize all non-repeating semantic tags, and form a candidate semantic tag set.

[0146] (4) Based on the frequency of each tag in the candidate semantic tag set and the weight strength of its corresponding edge, a comprehensive analysis is conducted to select the most representative dominant video action semantic tag. This tag is used as the visual context basis for verifying and enhancing the text.

[0147] D3: Compare the semantic consistency of the video action semantic tags with the text content of the corresponding time period in the initial ASR text. When ambiguity or conflict is found, use the video action semantic tags to correct the text and generate a semantically consistent initial translation text.

[0148] Furthermore, the specific steps of D3 include:

[0149] (1) Obtain the semantic tags of the dominant video action and the text content of the corresponding time period in the initial ASR text. Use the pre-trained language model to encode the text content and the semantic tags of the dominant video action into high-dimensional vectors respectively, and generate text semantic vectors representing text semantics and tag semantic vectors representing visual semantics.

[0150] (2) Calculate the cosine similarity between the text semantic vector and the tag semantic vector to obtain the semantic similarity value. Compare the semantic similarity value with the preset similarity threshold. If the semantic similarity value is lower than the preset similarity threshold, it is determined that there is a semantic conflict. Cosine similarity is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0151] (3) When a semantic conflict is determined, the conflicting text content and the dominant video action semantic tag are used as input to analyze the specific type of conflict, whether it is based on unclear reference or complete semantic inconsistency, and generate a specific text correction strategy based on the analysis results.

[0152] (4) Call the text generation model, integrate visual semantic information into the text generation process according to the text correction strategy, and output the corrected text segment to ensure that the corrected text is semantically consistent with the video tag and grammatically fluent.

[0153] (5) Replace the conflicting parts in the initial ASR text with the corrected text segments and perform context connection processing to finally generate an initial translated text that is semantically consistent with the visual content across the entire timeline.

[0154] Furthermore, the preset similarity threshold is determined in the following way:

[0155] (1) Construct a test dataset containing multiple audio and video scenarios, wherein the test dataset contains known semantically consistent sample pairs and semantically conflicting sample pairs;

[0156] (2) Using the language model to process the test dataset, calculate the cosine similarity of the semantically consistent sample pairs and the semantically conflicting sample pairs respectively, and obtain the positive sample similarity distribution and the negative sample similarity distribution;

[0157] (3) Based on the positive sample similarity distribution and the negative sample similarity distribution, with the goal of maximizing the F1 score, the optimal similarity threshold range is determined on the validation set. The formula for calculating the F1 score is the prior art in this field and is not an inventive solution of this application. It will not be elaborated here.

[0158] (4) Based on the tolerance for false alarms and false negatives in actual applications, fine-tune within the similarity threshold range, and finally set the preset similarity threshold.

[0159] When packet loss is detected in the Opus audio frame sequence, the lip movement and action semantic data of the corresponding VP9 video frames for that time period are extracted, including:

[0160] E1: Real-time monitoring of the continuity of Opus audio frame sequences. When a discontinuous timestamp is detected, it is determined as a packet loss event, and the packet loss period is determined based on the interruption timestamp.

[0161] E2: Extract the video frames corresponding to the packet loss period from the VP9 video frame sequence, perform face detection and lip region localization on each video frame, extract the geometric features of lip key points, and form a lip movement feature sequence.

[0162] Furthermore, the specific steps of E2 include:

[0163] (1) Obtain the determined packet loss period and VP9 video frame sequence; the packet loss period includes a start timestamp and an end timestamp;

[0164] (2) In the VP9 video frame sequence, find the first frame whose timestamp is greater than or equal to the start timestamp of the packet loss period, and select subsequent video frames in sequence until the timestamp of any frame exceeds the end timestamp of the packet loss period, and form a subset of video frames to be processed from all selected video frames.

[0165] (3) Call the VP9 decoder to decode each frame of compressed data in the video frame subset to be processed, obtain the original bitmap image, and then perform size adjustment, color space conversion and contrast enhancement on each bitmap image to generate a pre-processed image with uniform specifications.

[0166] (4) Input the pre-processed image into the pre-trained face detection model. The face detection model outputs the bounding box coordinates of one or more face regions in the image. Select the main face region and use the face detection model to locate multiple key points that identify the lip contour in the region. Based on these key points, crop the lip region of interest image from the pre-processed image. The face detection model is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0167] (5) Calculate the tilt angle of the head based on the facial key points, and perform rotation correction on the region of interest image of the lips to eliminate the influence of head posture. Then scale the image to a fixed size to generate a spatially normalized standardized lip image.

[0168] (6) Based on the output standardized lip image, use a fine lip key point detection model to predict the detailed key point coordinates of the lip contour. Based on the detailed key point coordinates, calculate the geometric measures representing the lip shape and movement, including lip opening, width, aspect ratio and key point movement speed relative to the previous frame. Combine the geometric measures into a numerical lip movement feature vector. The lip key point detection model is constructed based on a deep convolutional neural network.

[0169] (7) For each frame in the subset of video frames to be processed, its timestamp is bound to the lip movement feature vector calculated for that frame. The feature vectors bound to all frames are arranged in chronological order of timestamps to finally generate an ordered lip movement feature sequence.

[0170] E3: Input the lip movement feature sequence into the pre-trained lip movement recognition model. The lip movement recognition model outputs a candidate text segment. At the same time, retrieve the video action semantic tags associated with all video action nodes during the packet loss period from the dynamic causal graph. Use the video action semantic tags to supplement the contextual semantics and correct the logic of the candidate text segment, and generate lip movement and action semantic data.

[0171] Furthermore, the specific steps of E3 include:

[0172] (1) Obtain the lip movement feature sequence and input the lip movement feature sequence into the pre-trained lip movement recognition model. The lip movement recognition model is based on the sequence-to-sequence architecture, maps the lip movement features into text, and outputs a preliminary candidate text fragment. The lip movement recognition model is built based on an end-to-end deep learning model, and the deep learning model is the existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0173] (2) In the dynamic causal graph, retrieve all video action nodes whose timestamps fall within the packet loss period, extract the attached video action semantic tags from these video action nodes, and form an action semantic tag set;

[0174] (3) Analyze the tags in the action semantic tag set. If there is a single dominant tag, use it directly as the dominant action semantic description. If there are multiple tags, analyze their logical relationship and merge them to generate a coherent comprehensive action semantic description.

[0175] (4) Compare the candidate text fragments with the action semantic descriptions for logical consistency. When ambiguity or conflict is found, perform semantic constraints, word substitution or context supplementation on the candidate text fragments based on the action semantic descriptions to generate the verified and corrected text fragments.

[0176] (5) The generated text fragments after verification and correction are associated and packaged with the action semantic description to form a lip movement and action semantic data containing visual recognition results and action semantic context.

[0177] The process of generating complete translated text and embedding SimpleBlock elements via a reverse generation model includes:

[0178] F1: Input lip movement and action semantic data into a pre-trained inverse generation model; the inverse generation model is a sequence generation model based on an encoder-decoder architecture, in which the encoder encodes the input data, the decoder generates a fluent text sequence based on the encoding result, and the output is a completed translation text. The sequence generation model is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0179] F2: Using the initial translation text and the completed translation text as input, based on the timestamp of the packet loss period corresponding to the completed translation text, merge and insert it into the corresponding position of the initial translation text, and add completion markers at the start and end positions of the merge and insertion to generate the complete translation text;

[0180] Furthermore, the specific steps of F2 include:

[0181] (1) Obtain the initial translation text and the completed translation text; both the initial translation text and the completed translation text are timestamped;

[0182] (2) Read the start and end timestamps of the packet loss period corresponding to the completed translation text, find the last word in the initial translation text whose timestamp is less than or equal to the start timestamp of the packet loss period, and the first word whose timestamp is greater than or equal to the end timestamp of the packet loss period, and define the text interval between these two words as the text segment to be replaced;

[0183] (3) Taking the determined text segment to be replaced, its immediate preceding and following contexts, and the completed translation text as input, analyze the grammatical and semantic connection between the preceding context and the beginning of the completed translation text, adjust the capitalization or conjunctions of the beginning part of the completed translation text, analyze the connection between the end of the completed translation text and the following context, make adjustments, and generate a preprocessed completed text that is smoothly connected with the context.

[0184] (4) Remove the text segment to be replaced from the initial translated text and insert the preprocessed completed text precisely into the position to generate an intermediate complete text with complete content but no tagging;

[0185] (5) Add a start completion marker at the beginning of the preprocessed text and an end completion marker at the end to generate a tokenized complete text with completion markers;

[0186] (6) Perform a final consistency check on the tokenized complete text with completion marks to ensure that the timestamps are continuous and the meaning is coherent, and output the complete translated text that has passed the check and integrates the original content and the completion content.

[0187] F3: Write the complete translated text as user private data into the data payload of the SimpleBlock element, replacing or appending it to the original audio and video data, and generating a SimpleBlock element with completion tags.

[0188] Furthermore, the specific steps for F3 include:

[0189] (1) Obtain the complete translation text, which includes text content with timestamps and completion marks. Serialize the text content, timestamp information and completion marks according to a predefined structured format and encode them into a byte sequence to generate a self-contained private text data block.

[0190] (2) Obtain the original SimpleBlock element, parse the header information of the original SimpleBlock element, obtain its track number, timestamp and other key metadata, determine whether to replace the original audio and video data load or to append the new data to it, and prepare a SimpleBlock structure to be modified accordingly.

[0191] (3) Write the private text data block into the data payload area of ​​the SimpleBlock structure to be modified, and generate an intermediate SimpleBlock element whose data payload has been updated.

[0192] (4) Update the header information of the intermediate SimpleBlock element according to the new data load size and content, including recalculating the load length field and setting the corresponding private data identifier flag. Perform syntax and format verification on the updated element to ensure that it conforms to the WebM container specification and generate a SimpleBlock element that passes the verification.

[0193] (5) After the verification is passed, the SimpleBlock element that passes the verification is the final SimpleBlock element with completion mark.

[0194] The process of dynamically cropping non-core genes based on bitrate fluctuations and adjusting positional genes based on user visual attention heatmaps to generate timestamped adaptive subtitle data includes:

[0195] G1: Real-time monitoring of the moving average bitrate in the bitrate data; when the moving average bitrate is lower than the preset first bitrate threshold, a subtitle trimming trigger signal is generated.

[0196] Furthermore, the specific steps of G1 include:

[0197] (1) Obtain the bitrate data obtained by real-time parsing. At the same time, the system maintains a first-in-first-out bitrate sampling buffer, continuously storing the newly arrived instantaneous bitrate value and its duration into the buffer, and removing old data that exceeds the predetermined time window, so as to keep the bitrate sampling buffer always containing valid bitrate samples within the most recent time window.

[0198] (2) For each valid bitrate sample in the bitrate sampling buffer, multiply its instantaneous bitrate value by its corresponding data duration to obtain the weighted data amount of the sample. Sum the weighted data amounts of all samples to obtain the total data amount in the window. At the same time, sum the data durations of all samples to obtain the total window time. Then divide the total data amount in the window by the total window time to obtain the time-weighted moving average bitrate.

[0199] (3) Compare the time-weighted moving average bit rate with the first bit rate threshold; if the moving average bit rate is lower than the first bit rate threshold, generate a preliminary trigger condition fulfillment signal;

[0200] (4) When the initial trigger condition is met, a delay confirmation timer is started. During the operation of the delay confirmation timer, a new moving average bit rate is continuously calculated and compared with the first bit rate threshold. If the moving average bit rate calculated at any time during the operation of the delay confirmation timer rises back to the first bit rate threshold or above, it is determined to be network jitter and the initial trigger condition is met signal is cleared. If the moving average bit rate is continuously lower than the first bit rate threshold during the entire operation of the delay confirmation timer, a confirmed continuous low bit rate signal is generated.

[0201] (5) With the generated and confirmed continuous low bit rate signal as input, the system officially generates and outputs a high-level subtitle trimming trigger signal, and at the same time sets the internal state of the system to triggered. Meanwhile, the high-level subtitle trimming trigger signal will be latched until the sliding average bit rate is detected to be continuously higher than a higher recovery bit rate threshold before it will be cleared.

[0202] G2: Identify non-core genes corresponding to weak causal relationships in the complete translated text based on the dynamic causal graph and trim them to output semantically trimmed subtitle text;

[0203] Furthermore, the specific steps of G2 include:

[0204] (1) Obtain a dynamic causal graph with complete translated text and timestamp alignment;

[0205] (2) Traverse each word or phrase unit in the complete translated text, locate the video action node and audio event node that overlaps with or is closest to the time on the time axis of the dynamic causal graph according to its timestamp, query whether there is an edge between the video action node and the audio event node, and label the semantic gene type of each text unit according to the weight type of the edge, and generate text units with gene type labels.

[0206] (3) Select the set of non-core gene units that are labeled as having weak causal relationship or no relationship from all text units with gene type labels. Sort the text units in the set of non-core gene units according to the preset priority principle, including causal relationship strength, syntactic function importance and information redundancy, and generate an ordered non-core gene trimming priority queue.

[0207] (4) Starting from the head of the non-core gene pruning priority queue, take out the non-core gene units to be pruned in sequence, simulate removing non-core gene units from text units with gene type labels, and check the syntactic correctness, semantic coherence and consistency with strong causal relationships in the graph of the remaining text units with gene type labels; if the check passes, pruning is performed to generate the current pruning intermediate text; if the check fails, the non-core gene unit is retained; repeat this process until the pruning target is reached or the queue is exhausted, and output the pruned primary text.

[0208] (5) Perform grammatical and fluency analysis on the primary text, and make the minimum necessary corrections, including adding conjunctions, adjusting articles, and correcting verb forms, to ensure that the text is fluent and natural, and generate the grammatically reorganized text, which is the final semantically trimmed subtitle text.

[0209] G3: Extract the coordinates of the user's gaze point from the user's visual attention heatmap collected from the terminal, form a gaze point coordinate sequence, calculate its distribution on the video screen based on the gaze point coordinate sequence, and determine a visual focus area;

[0210] Furthermore, the user's gaze coordinates are extracted from the user's visual attention heatmap collected from the terminal to form a gaze coordinate sequence, including:

[0211] (1) The raw eye movement data stream is continuously collected from the sensor of the terminal device. The raw eye movement data stream contains high-frequency eye feature sampling points. The raw eye movement data stream is filtered in real time to smooth the trajectory and remove noise points, and a stable pre-processed eye movement signal is generated.

[0212] (2) The stable preprocessed eye movement signal is input into the pre-calibrated eye movement screen coordinate mapping model. The eye movement screen coordinate mapping model converts the eye movement angle into screen pixel coordinates and outputs a continuous, time-varying screen gaze coordinate stream. The eye movement screen coordinate mapping model can be understood as a regression mapping, and its construction process is as follows:

[0213] A second-order polynomial regression model is used to establish a nonlinear mapping relationship from eye movement feature space to screen pixel coordinate space. The input of the second-order polynomial regression model is the preprocessed coordinates of the pupil centers of both eyes and the corneal reflection spot vector, and the output is the two-dimensional pixel coordinates on the screen. The second-order polynomial regression model is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0214] Model parameters are determined in real time through the user calibration process. During calibration, the user needs to gaze at N calibration points with known coordinates on the screen in sequence. When each point is gazed at stably, the system synchronously records the screen standard coordinates of that point and the corresponding multiple sets of original eye movement feature data to form a calibration dataset.

[0215] After calibration, the system uses the least squares method to perform regression analysis on the calibration dataset, calculates specific coefficients in the second-order polynomial regression model, and generates an eye-tracking-screen coordinate mapping function with defined parameters specific to the current user and the current head position, which serves as the eye-tracking-screen coordinate mapping model. The least squares method is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0216] (3) Using the output screen gaze coordinate stream as input, analyze the movement speed and spatial clustering of coordinate points in real time, classify low-speed and spatially clustered point clusters as gaze events, and record their spatial centroid coordinates, start timestamp and end timestamp, and classify high-speed and linear movement segments as saccade events.

[0217] (4) Extract the spatial centroid coordinates and timestamp of each gaze event, and arrange the coordinates of all gaze events in chronological order of the timestamps to generate a gaze point coordinate sequence;

[0218] (5) Synchronize the timestamp of each point in the gaze coordinate sequence based on a unified time reference to make it precisely aligned with the time axis of the video frame and subtitle data, and output the final gaze coordinate sequence with synchronized timestamps and time axis alignment.

[0219] Furthermore, the visual focus area is determined by calculating the centroid of the user's visual attention heatmap and expanding a rectangular area of ​​preset width and height around the centroid. The formula for calculating the centroid is prior art in this field and is not an inventive solution of this application, so it will not be elaborated here.

[0220] G4: If the visual focus area is concentrated and stable, output an edge position coordinate that avoids the visual focus area; if the visual focus area is scattered or has no focus, output a default position coordinate and define the output position coordinate as the optimized position gene.

[0221] Furthermore, the specific steps of G4 include:

[0222] (1) Obtain the generated time-axis aligned gaze point coordinate sequence, calculate the centroid coordinates of all gaze point coordinates in the screen space, and calculate the standard deviation of all gaze point coordinates relative to the centroid in the X-axis and Y-axis directions.

[0223] (2) Compare the standard deviations of the X-axis and Y-axis with the preset dispersion thresholds respectively; if the standard deviations in both directions are lower than the dispersion thresholds, the visual focus area is determined to be concentrated and the centroid coordinates are recorded; otherwise, the visual focus area is determined to be dispersed.

[0224] Furthermore, the dispersion threshold is determined by collecting eye-tracking data from users in a controlled experimental environment when performing tasks requiring focused attention and free browsing tasks, forming a sample set containing known attention states. The standard deviation distribution of the samples is then obtained through statistical analysis, and an ROC curve is plotted. Finally, the dispersion threshold is selected based on the principle of maximizing the exponent on the ROC curve.

[0225] (3) When the visual focus area is determined to be concentrated, the gaze point coordinate sequence aligned with the time axis and the obtained centroid coordinates are used as inputs. The gaze point coordinate sequence is divided into continuous time windows, and the centroid coordinates of the gaze point in each time window are calculated. At the same time, the displacement between the centroids of the continuous time windows is calculated. If the displacement of multiple consecutive time windows is less than the preset stability threshold, the visual focus area is determined to be a stable visual focus area.

[0226] Furthermore, the stability threshold is set as follows: Based on prior knowledge of visual attention duration in cognitive psychology, a preliminary lower limit for stability time is set, such as 1.5 seconds; eye movement data is collected under scenarios of stable gaze and eye movement, and the statistical distribution of centroid displacement in continuous time windows is analyzed. ROC curve analysis is used to determine the optimal displacement threshold to distinguish between the two states, which is set as the stability threshold. For example, the stability threshold is equal to 1% of the screen width; based on the preliminary lower limit for stability time and the selected time window size, such as 0.5 seconds, the required number of consecutive windows M is calculated, such as M=3; finally, a stability criterion is formed where the displacement of M consecutive time windows is less than the stability threshold; according to the system's requirements for response speed, the stability threshold and the number of consecutive windows M are fine-tuned to achieve a balance between avoiding interference and maintaining subtitle stability, thus obtaining the stability threshold.

[0227] (4) When a stable visual focus area is determined, the obtained centroid coordinates are used as input. Based on the position of the centroid coordinates relative to the center of the screen, the screen quadrant to which it belongs is determined. The quadrant diagonally opposite to this quadrant is selected as the candidate avoidance quadrant. Within the candidate avoidance quadrant, an edge position coordinate that satisfies the minimum safe distance from both the screen edge and the visual focus area is calculated. This coordinate is used as the avoidance position coordinate, and the output avoidance position coordinate is the final result.

[0228] (5) When the visual focus area is determined to be scattered, or the visual focus area is determined to be unstable, or the gaze point coordinate sequence is empty, the system determines that there is no significant visual focus. In this case, the system reads the default position coordinates from the preset configuration and outputs the default position coordinates as the final result.

[0229] G5: Binds the semantically cropped subtitle text, the optimized positional gene, and the corresponding timestamps of the VP9 video frame sequence and the Opus audio frame sequence to generate time-stamped adaptive subtitle data.

[0230] The process of synchronously playing VP9 video frame sequences, Opus audio frame sequences, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps according to timestamps, collecting synchronization errors and user feedback, and optimizing the edge weights and subtitle position genes of the dynamic causal graph includes:

[0231] H1: Based on a unified timeline, VP9 video frame sequences, Opus audio frame sequences, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps are rendered and played synchronously. The rendering process is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0232] H2: During playback, by comparing the presentation timestamps of video frames, audio frames, and adaptive subtitle data with timestamps with a unified timeline, audio-video synchronization error and word-sound synchronization error are calculated in real time and merged into a synchronization error dataset. At the same time, a feedback interface is provided on the playback interface to collect user feedback rating data on subtitles, and the feedback rating data is associated with and stored with the synchronization error dataset according to time intervals.

[0233] H3: Identify time intervals where synchronization errors are consistently high. Based on the identification results, adjust the weights of the edges within the corresponding time intervals in the dynamic causal graph. The adjustment method is to reduce the weights of the corresponding edges.

[0234] H4: When the feedback score data is lower than the preset score threshold, location gene optimization is triggered.

[0235] Example 2:

[0236] Please see Figure 3 Another embodiment of the present invention provides a WebM protocol low-latency video and audio translation and subtitle optimization system, comprising:

[0237] Stream parsing module, causal modeling module, translation and compensation module, subtitle generation module, feedback optimization module;

[0238] The stream parsing module is used to deconstruct the raw WebM audio and video streams into structured data;

[0239] The causal modeling module is used to analyze the spatiotemporal intrinsic relationship between video and audio, and to construct a timestamp-aligned dynamic causal graph.

[0240] The translation and compensation module is used to generate complete translated text;

[0241] The subtitle generation module is used to dynamically trim non-core genes based on dynamic causal graphs and bitrate data, and strictly bind them with audio and video timestamps to generate the final adaptive subtitle data;

[0242] The feedback optimization module is used to ultimately present adaptive caption data and collect user feedback to optimize the dynamic causal graph and caption position genes.

[0243] The stream parsing module includes: a container parsing unit, a bitrate analysis unit, a demultiplexing unit, and a metadata encapsulation unit;

[0244] The container parsing unit is used to parse the WebM container format and read segment information and track information.

[0245] The bitrate analysis unit is used to analyze the size and arrival time of data clusters in real time, and to calculate the instantaneous bitrate and the moving average bitrate.

[0246] The demultiplexing unit is used to separate intertwined audio and video data streams according to the track number of the data block;

[0247] Metadata encapsulation unit, used to extract and encapsulate SimpleBlock elements in WebM.

[0248] The causal modeling module includes: a spatiotemporal alignment unit, a feature extraction unit, a causal inference unit, and a graph construction unit;

[0249] The spatiotemporal alignment unit is used to create an accurate video track-audio track association list based on the Track Link element and timestamp, ensuring that video frames and audio frames correspond correctly on the timeline;

[0250] The feature extraction unit is used to extract features from video frame sequences by action recognition and to extract features from audio frame sequences by event detection.

[0251] The causal reasoning unit is used to calculate the temporal proximity and statistical correlation between audio and video features, and to determine strong and weak causal relationships based on preset thresholds.

[0252] The graph construction unit is used to construct and maintain a dynamic causal graph with timestamps, using audio and video frames as nodes and causal relationships as edges.

[0253] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

Claims

1. A method for low-latency video and audio translation and subtitle optimization using the WebM protocol, characterized in that: include: Parse WebM audio and video streams to extract Track Link elements, bitrate data, VP9 video frame sequences, Opus audio frame sequences, and SimpleBlock elements; Using Track Link elements, VP9 video frame sequences, and Opus audio frame sequences as input, a video track-audio track association list is established, and a timestamp-aligned dynamic causal graph is constructed. Using Opus audio frame sequence, VP9 video frame sequence, SimpleBlock elements and dynamic causal graph as input, an initial translation text is generated. When packet loss is detected in the Opus audio frame sequence, lip movement and action semantic data of the corresponding VP9 video frame are extracted. The inverse generation model generates a complete translation text and embeds SimpleBlock elements. The output is a complete translation text and SimpleBlock elements with completion tags. Based on bitrate data, complete translated text, and user visual attention heatmaps collected from the terminal, we define core genes, non-core genes, and position genes for subtitles. We dynamically trim non-core genes according to bitrate fluctuations and adjust position genes based on user visual attention heatmaps to generate adaptive subtitle data with timestamps. The VP9 video frame sequence, Opus audio frame sequence, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps are played synchronously according to timestamps. Synchronization errors and user feedback are collected to optimize the edge weights and subtitle position genes of the dynamic causal graph. The construction process of the timestamp-aligned dynamic causal graph includes: Action recognition is performed on the VP9 video frame sequence to extract action feature vectors from consecutive video frames and generate a video action time sequence. Event detection is performed on the Opus audio frame sequence, and event feature vectors in the audio frames are extracted, including volume changes and spectral feature changes, to generate an audio event time sequence. Set the first time window threshold and the second time window threshold; The starting timestamp of the action feature vector of each video is extracted from the video action time sequence, and the starting timestamp of the event feature vector of each audio is extracted from the audio event time sequence. For each pair of video action and audio event, calculate the absolute value of their timestamp difference. If the absolute value is less than or equal to the first time window threshold, calculate the Pearson correlation coefficient between the action feature vector and the event feature vector. If the Pearson correlation coefficient is higher than the preset correlation coefficient threshold, it is determined to be a strong causal relationship. If the absolute value is greater than the first time window threshold but less than or equal to the second time window threshold, and the Pearson correlation coefficient is positive, then it is determined to be a weak causal relationship. Using each item in the video action time sequence and audio event time sequence as a node, and the strong causal relationship and weak causal relationship as directed edges, an initial causal graph is constructed. Based on the timestamp information in the video track-audio track association list, the nodes and directed edges in the initial causal graph are time-stamp aligned to form a time-stamp aligned dynamic causal graph. The process of dynamically cropping non-core genes based on bitrate fluctuations and adjusting positional genes based on user visual attention heatmaps to generate timestamped adaptive subtitle data includes: The system monitors the moving average bitrate in real time and generates a subtitle trimming trigger signal when the moving average bitrate is lower than a preset first bitrate threshold. Based on the dynamic causal graph, non-core genes corresponding to weak causal relationships in the complete translated text are identified and trimmed to output semantically trimmed subtitle text. The coordinates of user gaze points are extracted from the user visual attention heatmap collected from the terminal to form a gaze point coordinate sequence. The distribution of the gaze point coordinate sequence on the video screen is calculated to determine the visual focus area. If the visual focus area is concentrated and stable, the edge position coordinates that avoid the visual focus area are output. If the visual focus area is scattered or has no focus, the default position coordinates are output. The output position coordinates are defined as the optimized position gene. The semantically cropped subtitle text and optimized positional genes are bound to the corresponding timestamps of the VP9 video frame sequence and Opus audio frame sequence to generate time-stamped adaptive subtitle data.

2. The WebM protocol low-latency video and audio translation and subtitle optimization method as described in claim 1, characterized in that, The process of parsing the WebM audio and video streams and extracting Track Link elements, bitrate data, VP9 video frame sequences, Opus audio frame sequences, and SimpleBlock elements includes: The WebM parser is used to read the input audio and video stream data by cluster, parse the segment information and track information elements at the container level, and identify and extract Track Link elements from the track information elements; the Track Link elements are used to describe the relationship between the video track and the audio track. After completing the container structure parsing, the cluster data in the audio and video stream data are read sequentially, and the size and duration of each cluster are statistically analyzed in real time, and the bitrate data is calculated; the bitrate data includes the instantaneous bitrate and moving average bitrate of video and audio. The data blocks within each cluster are demultiplexed to separate the VP9 video frame sequence and the Opus audio frame sequence. During the demultiplexing process, the original SimpleBlock structure corresponding to each data block is extracted synchronously and encapsulated into a complete SimpleBlock element. The VP9 video frame sequence contains time-stamped frame data and frame type identifiers. The Opus audio frame sequence contains time-stamped audio packets and encoding parameters. The SimpleBlock element is an audio frame encapsulation unit that contains the timestamp, track number, and data payload of the audio and video data blocks.

3. The WebM protocol low-latency video and audio translation and subtitle optimization method as described in claim 2, characterized in that, The process of establishing the video track-audio track association list includes: Using the Track Link element as input, parse the track link information defined therein, and establish a track pair that associates the track number of the video track with the track number of the audio track; Using VP9 video frame sequences and Opus audio frame sequences as input, based on the track number carried in each frame of data, video frames and audio frames belonging to the same associated track pair are classified into the same processing group using associated track pairs. For each processing group, sorting and alignment are performed based on the timestamps of the VP9 video frame sequence and Opus audio frame sequence within the group, using the time axis as the reference. For each audio frame, search all video frames within a first predetermined time tolerance before and after it, establish a mapping relationship, and generate a video track-audio track association list; each item in the video track-audio track association list records the timestamp and data pointer of an audio frame, as well as the timestamps and data pointers of all video frames associated with it.

4. The WebM protocol low-latency video and audio translation and subtitle optimization method as described in claim 3, characterized in that, The process of generating initial translated text by taking Opus audio frame sequences, VP9 video frame sequences, SimpleBlock elements, and dynamic causal graphs as input includes: The Opus audio frame sequence is decoded to obtain a continuous audio signal, which is then input into an automatic speech recognition engine to generate initial ASR text. Using a dynamic causal graph as input, for a specified time period in the initial ASR text, it is verified whether there is a strong causal relationship within the specified time period. If there is, the corresponding video action semantic tags are extracted from the video action nodes that have a strong causal relationship with the current audio event node. The semantic consistency of the video action semantic tags is compared with the text content of the corresponding time period in the initial ASR text. When ambiguity or conflict is found, the text is corrected using the video action semantic tags to generate a semantically consistent initial translation text.

5. The WebM protocol low-latency video and audio translation and subtitle optimization method as described in claim 4, characterized in that, When packet loss is detected in the Opus audio frame sequence, the lip movement and action semantic data of the corresponding VP9 video frames for that time period are extracted, including: The continuity of Opus audio frame sequences is monitored in real time. When a discontinuity in timestamps is detected, it is determined as a packet loss event, and the packet loss period is determined based on the interruption timestamp. The video frames corresponding to the packet loss period are extracted from the VP9 video frame sequence. Face detection and lip region localization are performed on each video frame. The geometric features of the key points of the lips are extracted to form a lip movement feature sequence. The lip movement feature sequence is input into a pre-trained lip movement recognition model, which outputs candidate text fragments. At the same time, video action semantic tags associated with all video action nodes during the packet loss period are retrieved from the dynamic causal graph. The video action semantic tags are used to supplement the contextual semantics and correct the logic of the candidate text fragments, generating lip movement and action semantic data.

6. The WebM protocol low-latency video and audio translation and subtitle optimization method as described in claim 5, characterized in that, The process of generating complete translated text and embedding SimpleBlock elements via a reverse generation model includes: The lip movement and action semantic data are input into a pre-trained inverse generation model; the inverse generation model is a sequence generation model based on an encoder-decoder architecture, in which the encoder encodes the input data, the decoder generates a text sequence based on the encoding result, and the output is the completed translation text; Using the initial translated text and the completed translated text as input, the completed translated text is merged and inserted into the corresponding position of the initial translated text according to the timestamp of the packet loss period corresponding to the completed translated text. Completion marks are added at the beginning and end positions of the merging and insertion to generate the complete translated text. The complete translated text is written as user private data into the data payload of the SimpleBlock element, replacing or appending the original audio and video data to generate a SimpleBlock element with completion tags.

7. The WebM protocol low-latency video and audio translation and subtitle optimization method as described in claim 6, characterized in that, The process of synchronously playing VP9 video frame sequences, Opus audio frame sequences, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps according to timestamps, collecting synchronization errors and user feedback, and optimizing the edge weights and subtitle position genes of the dynamic causal graph includes: Based on a unified timeline, VP9 video frame sequences, Opus audio frame sequences, SimpleBlock elements with completion markers, and adaptive subtitle data with timestamps are rendered and played synchronously. During playback, by comparing the presentation timestamps of video frames, audio frames, and adaptive subtitle data with a unified timeline, audio-video synchronization error and word-sound synchronization error are calculated in real time and merged into a synchronization error dataset. At the same time, a feedback interface is provided on the playback interface to collect user feedback rating data on the subtitles, and the feedback rating data is associated with and stored with the synchronization error dataset according to time intervals. Identify the time interval of the synchronization error, and adjust the weight of the edges in the corresponding time interval in the dynamic causal graph based on the identification results; When the feedback score data is lower than the preset score threshold, location gene optimization is triggered.

8. A WebM protocol low-latency video and audio translation and subtitle optimization system, used to implement the WebM protocol low-latency video and audio translation and subtitle optimization method as described in any one of claims 1-7, characterized in that, include: Stream parsing module, causal modeling module, translation and compensation module, subtitle generation module, feedback optimization module; The stream parsing module is used to deconstruct the original WebM audio and video streams into structured data; The causal modeling module is used to analyze the spatiotemporal intrinsic relationship between video and audio, and to construct a timestamp-aligned dynamic causal graph. The translation and compensation module is used to generate complete translated text; The subtitle generation module is used to dynamically trim non-core genes based on dynamic causal graphs and bitrate data, and bind them with audio and video timestamps to generate the final adaptive subtitle data. The feedback optimization module is used to present adaptive subtitle data and collect user feedback to optimize the dynamic causal graph and subtitle position genes.

Images