Cross-platform video adaptation response method and system

Abstract

The invention discloses a cross-platform video adaptation response method and system, and belongs to the technical field of video adaptation, and the method specifically comprises the steps: separating an input video audio stream, converting an audio signal into a target language voice and corresponding timestamp subtitle data through a translation model with rhythm maintenance, and transmitting the target language voice and the corresponding timestamp subtitle data to a server; fusing the subtitle data with the video frame, automatically determining a subtitle position based on content area analysis, generating a multi-language subtitle display layer, packaging the multi-language subtitle display layer into a WebM container, performing cross-terminal playing compatibility through adaptive code rate coding, and when the subtitle data is distributed to different terminals, performing cross-terminal playing compatibility on the multi-language subtitle display layer. Performing dynamic adjustment on subtitle synchronization and packaging parameters according to playing feedback, so that video and audio streams are adapted and responded in real time on different platforms; according to the invention, continuous, stable and multi-language compatible video playing can be realized in a multi-platform environment, and the consistency and expandability of contents in cross-device distribution and playing are ensured.

Description

Technical Field

[0001] This invention belongs to the field of video adaptation technology, specifically a cross-platform video adaptation response method and system. Background Technology

[0002] With the widespread dissemination of video content across the internet, mobile terminals, and smart devices, the video playback environment has become highly fragmented. Different operating systems, hardware performance, and network conditions can easily lead to issues such as uneven image quality, decoding incompatibility, and subtitle misalignment when playing videos across platforms. Traditional video adaptation methods often rely on fixed bitrates or single encapsulation standards, making it difficult to meet the dynamic needs of multiple platforms and scenarios.

[0003] Existing cross-platform video processing solutions typically combine transcoding and distribution, but often lack control over the synchronization between subtitles, audio, and video. Especially in scenarios involving multilingual subtitle fusion and multi-terminal decoding, the common solution is simply to treat subtitles as external files. This can easily lead to loss of timing synchronization when there are network fluctuations or when switching terminals, failing to guarantee a consistent viewing experience for users across different platforms. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a cross-platform video adaptation and response method and system. It can flexibly support multi-bitrate, multi-language, and multi-dimensional index management at the encapsulation level, and provide an adaptive response mechanism during distribution and playback, enabling continuous playback and unified display of videos under different network bandwidths, device performance, and multilingual environments.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Cross-platform video adaptation response methods include:

[0007] The input video and audio streams are separated, and a prosody-preserving translation model is used to convert the audio signal into the target language speech and corresponding timestamp subtitle data.

[0008] The subtitle data is fused with video frames, and the subtitle position is automatically determined based on content area analysis to generate a multilingual subtitle display layer;

[0009] The multilingual subtitle display layer is encapsulated in a WebM container and cross-terminal playback compatibility is achieved through adaptive bitrate encoding;

[0010] When distributed to different terminals, the subtitle synchronization and encapsulation parameters are dynamically adjusted based on playback feedback, so that the video and audio streams can be adapted and responded to in real time on different platforms.

[0011] Specifically, the separation of the input video and audio streams, and the conversion of the audio signal into target language speech and corresponding timestamp subtitle data using a prosody-preserving translation model, includes:

[0012] The input video and audio streams are separated into multi-channel signals to obtain independent video and audio data streams, and a timestamp index is added to the audio data stream.

[0013] The audio data stream is sliced ​​to extract speech content features, speech prosody features and background noise interference features, and the corresponding timestamp indexes are marked.

[0014] A cross-language mapping dictionary is built based on speech content features, and a prosodic prediction sequence is built by combining speech prosodic features to form a multi-dimensional input matrix for translation;

[0015] The multidimensional input matrix is ​​input into a translation model with prosody preservation, and the output is a target language speech segment and its corresponding semantic index sequence.

[0016] The target language speech segments are re-matched according to the timestamp index and semantic index sequence, and audio data and synchronized subtitle data in the target language are generated after the matching is completed.

[0017] Specifically, the target language speech segments are re-matched according to the timestamp index and semantic index sequence, and after the matching is completed, audio data and synchronized subtitle data in the target language are generated, including:

[0018] The target language speech segments are arranged sequentially according to their timestamp indices, and a preliminary speech segment sequence is established.

[0019] The speech segment sequence is bidirectionally compared with the semantic index sequence. During the comparison process, the semantic tags with ambiguity are prioritized to obtain a matching relationship table.

[0020] The speech segments are reconstructed according to the matching relationship table to generate a continuous target language audio stream, and synchronization markers corresponding to the semantic index are inserted during the reconstruction process.

[0021] Based on the synchronization marker, subtitle data corresponding to the target language audio stream is generated, and the subtitle data is bound to the target language audio stream to form target language audio data and synchronized subtitle data.

[0022] Specifically, the subtitle data is fused with video frames, and the subtitle position is automatically determined based on content region analysis to generate a multilingual subtitle display layer, including:

[0023] The video frames are segmented to extract the main target region and background region, and a location index is established for each region.

[0024] The synchronized subtitle data is processed by language layering, and subtitles of different languages ​​are independently encoded into vectorized symbol units to generate subtitle units, and a corresponding relationship table is formed with the position index.

[0025] Based on the aforementioned correspondence table, content area analysis is performed to determine candidate display areas for subtitles, and overlapping candidate display areas are prioritized.

[0026] The subtitle units are mapped to the highest priority candidate display area, and the subtitle line spacing and scaling ratio are dynamically adjusted during the mapping process to generate a video frame sequence containing the multilingual subtitle display layer;

[0027] Timestamp information is added to the video frame sequence containing the multilingual subtitle display layer to form a multilingual subtitle fusion result corresponding to the original video frame.

[0028] Specifically, based on the aforementioned correspondence table, content region analysis is performed to determine candidate display regions for subtitles, and overlapping candidate display regions are prioritized, including:

[0029] The region index of the video frame is subjected to saliency detection to identify the visually dominant region and mark it as a subtitle-disabled region;

[0030] Outside the subtitle disabled area, multiple candidate display areas are generated based on the language layer corresponding to the subtitles, and an initial priority value is assigned to each candidate display area;

[0031] Overlap detection is performed on the candidate display areas. When two or more candidate display areas overlap, their priorities are recalculated.

[0032] The recalculated candidate display areas are sorted by priority to form a sequence of candidate areas for subtitle mapping.

[0033] Specifically, the vectorized symbol units are mapped to the highest priority candidate display areas, and the subtitle line spacing and scaling ratio are dynamically adjusted during the mapping process to generate a video frame sequence containing the multilingual subtitle display layer, including:

[0034] The subtitle units are loaded into the highest priority candidate display area according to the language hierarchy, and a mapping table between subtitle units and area indexes is established;

[0035] During the mapping process, the initial layout parameters are calculated based on the number of subtitle characters and the size of the display area, and the corresponding line spacing and scaling settings are generated.

[0036] The initial layout parameters are dynamically adjusted, and when there is a risk of overlapping multilingual subtitles, the subtitle units are scaled and separated layer by layer and the line spacing is allocated hierarchically.

[0037] Based on the dynamic adjustment results, a video frame sequence containing the multilingual subtitle display layer is generated.

[0038] Specifically, the multilingual subtitle display layer is encapsulated in a WebM container, and cross-terminal playback compatibility is achieved through adaptive bitrate encoding, including:

[0039] The multilingual subtitle display layer is hierarchically categorized with the corresponding video frames and target language audio streams, and an independent encapsulation index is established for each layer.

[0040] The encapsulation index is input into the WebM container building module, which allocates independent tracks for data at each layer in the container structure and generates a cross-layer reference table.

[0041] Multi-rate encoding is performed on the data at each layer, and multiple bitrate versions of encapsulated data are generated according to the preset network bandwidth tiering standard;

[0042] The encapsulated data of the various bitrate versions are adaptively sorted. When the terminal's decoding capability is insufficient or the network conditions fluctuate, the corresponding bitrate version is switched through the cross-layer reference table.

[0043] The sorted and encapsulated data is output as a WebM stream, and a terminal identification tag is attached to the WebM stream for cross-platform access and playback.

[0044] Specifically, the process of performing multi-rate encoding on data at each layer and generating encapsulated data with multiple rate versions according to a preset network bandwidth tiering standard includes:

[0045] The data from each layer is sampled and segmented to form multiple time-series segments;

[0046] The time-series segments are divided into corresponding multi-level bit rate sets according to a preset network bandwidth grading standard;

[0047] Parallel encoding processing is performed on the bitrate set, and different compression ratios are applied to the subtitle layer, audio layer and video layer during the encoding process to generate segment data with multiple bitrate versions;

[0048] The fragment data is merged according to the in-layer mapping table, and encapsulated data containing multiple bitrate versions is output.

[0049] Specifically, the encapsulated data for the various bitrate versions is adaptively sorted. When the terminal's decoding capability is insufficient or network conditions fluctuate, the system switches to the corresponding bitrate version through a cross-layer reference table, including:

[0050] A performance index table is established for the encapsulated data of the various bitrate versions, and sorting weights are assigned to each version based on bitrate and decoding complexity;

[0051] The performance index table is compared with the cross-layer reference table to generate a comprehensive sorting sequence that includes the subtitle layer, audio layer and video layer;

[0052] When insufficient terminal decoding capability or decreased network bandwidth is detected, the second highest bitrate version is called from the comprehensive sorting sequence, while maintaining the synchronous mapping relationship between the subtitle layer and the audio layer.

[0053] When network or terminal conditions are detected to have recovered, the comprehensive sorting sequence is reloaded, and a higher bitrate version is switched based on the sorting weight.

[0054] A cross-platform video adaptation and response system is used to implement the cross-platform video adaptation and response method, including: a signal conversion module, a subtitle display module, an encoding module, and a dynamic adaptation module;

[0055] The signal conversion module is used to separate the input video and audio streams and, using a translation model with prosody preservation, convert the audio signal into target language speech and corresponding timestamp subtitle data.

[0056] The subtitle display module is used to fuse the subtitle data with video frames, automatically determine the subtitle position based on content area analysis, and generate a multilingual subtitle display layer.

[0057] The encoding module is used to encapsulate the multilingual subtitle display layer into a WebM container and achieve cross-terminal playback compatibility through adaptive bitrate encoding;

[0058] The dynamic adaptation module is used to dynamically adjust the subtitle synchronization and encapsulation parameters based on playback feedback when distributing to different terminals, so that the video and audio streams can be adapted and responded to in real time on different platforms.

[0059] Compared with the prior art, the beneficial effects of the present invention are:

[0060] This invention proposes a cross-platform video adaptation response method and system. By processing video, audio, and subtitle data in layers and establishing a multi-bitrate and cross-layer index management mechanism in the WebM container, it achieves adaptive calling and switching under different terminal environments. The overall method can automatically match the appropriate data layer and maintain the synchronization of audio, video, and subtitles even when there are differences in network bandwidth, device performance, and language preferences. This enables continuous, stable, and multilingual compatible video playback in multi-platform environments, ensuring the consistency and scalability of content in cross-device distribution and playback. Attached Figure Description

[0061] Figure 1 Flowchart of the cross-platform video adaptation response method provided by the present invention;

[0062] Figure 2 The packaging and adaptation flowchart provided for this invention;

[0063] Figure 3 This is a diagram illustrating the cross-platform video adaptation and response system architecture provided by the present invention. Detailed Implementation

[0064] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0067] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0068] Example 1:

[0069] Please see Figure 1 and Figure 2 The present invention provides an embodiment of a cross-platform video adaptation response method, which includes the following specific steps:

[0070] Step S1: Separate the input video and audio streams, and use a translation model with prosody preservation to convert the audio signal into target language speech and corresponding timestamp subtitle data.

[0071] The specific steps of step S1 are as follows:

[0072] Step S101: Perform multi-channel signal separation on the input video and audio streams to obtain independent video and audio data streams, and attach a timestamp index to the audio data stream.

[0073] In this embodiment, a multi-channel signal separation method is used to deconstruct the input video and audio streams. Specifically, the input video and audio streams are typically encapsulated as composite bitstreams, where video and audio data are interleaved in time. The time-series parsing module first performs frame-level segmentation on the input stream to identify the boundaries between video and audio frames. Then, through a channel parsing algorithm, the video data frames are output separately as video data streams according to their image coding features, while the audio data frames are extracted according to their acoustic coding format to form independent audio data streams. During this process, to ensure subsequent audio and video synchronization, each audio data frame needs to be compared with the timeline of the original input stream during separation, and a corresponding timestamp index is added.

[0074] Step S102: Perform feature slicing on the audio data stream, extract speech content features, speech prosody features and background noise interference features respectively, and label them with timestamp indexes.

[0075] In this embodiment, the continuous audio signal is divided into processable time segments by segmenting the audio data stream, and feature information at different levels is extracted from each segment. Specifically, firstly, the audio segments are framed and windowed using short-time analysis methods to preserve local temporal features. Secondly, semantically relevant speech content features, such as acoustic spectrum distribution and speech fundamental frequency sequence, are extracted from the framed data to reflect language information. At the same time, audio energy changes, pause intervals, and syllable lengths are statistically analyzed to obtain speech prosodic features reflecting speech rate and rhythm. Furthermore, background noise interference features are extracted by separating non-speech components to identify environmental noise or accompanying sound effects. Finally, the above three types of features are combined with the corresponding time segment index to form a multi-dimensional feature sequence with timestamp annotations.

[0076] Step S103: Establish a cross-language mapping dictionary based on speech content features, and combine speech prosody features to establish a prosody prediction sequence, forming a multi-dimensional input matrix for translation.

[0077] In this embodiment, a multi-dimensional translation input framework is constructed using extracted speech content features and speech prosody features. Specifically, firstly, based on speech content features, the corresponding acoustic feature parameters in the audio segment are mapped to language units. By comparing with existing cross-language training corpora, a mapping dictionary that can be used for correspondence conversion between different languages ​​is gradually established. This mapping dictionary not only covers the correspondence of basic words but also includes phrase-level semantic matching to achieve semantic continuity in subsequent translation. Subsequently, speech prosody features are modeled separately, and a prosody prediction sequence is generated based on elements such as pause duration, syllable extension, and intensity changes, thereby forming a rhythmic trajectory that is independent of the content features but maintains a corresponding time axis. Finally, the results of the speech content mapping dictionary and the prosody prediction sequence are aligned and integrated according to timestamps to construct a multi-dimensional input matrix containing semantic and prosodic dimensions.

[0078] Step S104: Input the multidimensional input matrix into the prosody-preserving translation model and output the target language speech segment and its corresponding semantic index sequence.

[0079] In this embodiment, a translation model with a prosody preservation mechanism is used to process the multidimensional input matrix, thereby simultaneously achieving cross-language conversion of semantic content and continuation of prosodic features. Specifically, the multidimensional input matrix, consisting of a cross-language mapping dictionary and a prosodic prediction sequence, is first input into the translation model, which includes two parallel paths: a semantic decoding unit and a prosodic alignment unit. The semantic decoding unit completes the mapping from the source language to the target language based on the content feature vector, outputting a preliminary sequence of target language phonetic symbols. The prosodic alignment unit then performs pause insertion, pitch adjustment, and duration correction on the generated target language phonetic symbols based on the predicted rhythm trajectory, so that the rhythm and intonation are consistent with the original speech. Subsequently, the results of the two paths are synchronously fused to generate target language speech segments with temporal markers, and a semantic index sequence is added to each segment.

[0080] Step S105: Rematch the target language speech segments according to the timestamp index and semantic index sequence, and generate target language audio data and synchronized subtitle data after the matching is completed.

[0081] The specific steps of step S105 are as follows:

[0082] Step S1051: Arrange the target language speech segments sequentially according to the timestamp index and establish a preliminary speech segment sequence.

[0083] In this embodiment, the generated target language speech segments are temporally reorganized to ensure the matching basis for subsequent semantic indexing. Specifically, the timestamp indexes attached to each speech segment are first extracted, and all segments are sorted according to the order of the timestamps. During the sorting process, for segments with overlapping or gapd timestamps, insertion or merging is performed by comparing their original semantic tags with the time span of adjacent segments to maintain the continuity between segments. Subsequently, the sorted and corrected segments are connected sequentially to establish a preliminary speech segment sequence, which has a linear structure on the time axis.

[0084] Step S1052: Perform a bidirectional comparison between the speech segment sequence and the semantic index sequence. During the comparison process, prioritize the semantic tags that are ambiguous to obtain a matching relationship table.

[0085] In this embodiment, a stable correspondence is established through bidirectional comparison between the speech segment sequence and the semantic index sequence. Specifically, firstly, the timestamp information of the speech segment is used to perform a preliminary match with the timestamps in the semantic index sequence to obtain a possible set of one-to-one correspondences. Subsequently, during the comparison process, when a speech segment corresponds to multiple semantic tags, or a semantic tag can correspond to multiple speech segments, it is considered an ambiguous case and sorted according to a pre-set priority rule. This rule usually considers factors such as semantic completeness, contextual continuity, and segment duration to determine the optimal correspondence. Finally, the results after ambiguity resolution and priority filtering are organized into a matching relationship table.

[0086] Step S1053: Reconstruct the speech segments according to the matching relationship table to generate a continuous target language audio stream, and insert synchronization markers corresponding to the semantic index during the reconstruction process.

[0087] In this embodiment, a matching table is used to reconstruct speech segments temporally and semantically to generate a continuous target language audio stream. Specifically, firstly, based on the correspondence between the segments and semantic indices determined in the matching table, the scattered speech segments are spliced ​​together in chronological order, and gaps in the segments are smoothed using the duration difference of neighboring indices, thereby avoiding unnecessary breaks in the audio stream. Subsequently, during the splicing process, synchronization markers are embedded in the audio stream according to the position of the semantic indices. These synchronization markers are aligned with the subtitle generation and timeline, enabling unified retrieval and scheduling of audio and semantic content in subsequent stages. Finally, the audio stream, after reconstruction and the insertion of synchronization markers, forms a continuous target language output trajectory.

[0088] Step S1054: Generate corresponding subtitle data for the target language audio stream based on the synchronization marker, and bind the subtitle data with the target language audio stream to form target language audio data and synchronized subtitle data.

[0089] In this embodiment, the synchronization markers inserted in the previous step are used to establish a one-to-one correspondence between the audio stream and the subtitle data, thereby generating a bindable multimodal output. Specifically, firstly, the synchronization markers embedded in the target language audio stream are parsed, and corresponding semantic segments are divided on the timeline based on these markers. Then, the semantic index is mapped to the divided time segments to generate subtitle text units that match the audio content, and timestamps are retained during the generation process to ensure timing accuracy. Furthermore, the subtitle text units and audio stream segments are aligned at the timestamp level to achieve cross-modal bidirectional binding. Finally, the bound subtitle data and the target language audio stream are combined into a unified data structure.

[0090] Step S2: The subtitle data is fused with the video frame, and the subtitle position is automatically determined based on content area analysis to generate a multilingual subtitle display layer.

[0091] The specific steps of step S2 are as follows:

[0092] Step S201: Perform region segmentation on the video frame, extract the main target region and background region, and establish a location index for each region.

[0093] In this embodiment, image segmentation and region labeling methods are used to divide the input video frame into sub-regions with different semantic attributes, providing a recognizable spatial index for subsequent subtitle mapping. Specifically, firstly, pixel-level feature extraction is performed on the video frame, and combined with color histograms, edge gradients, and motion vector information, foreground objects and background regions are initially distinguished. Then, clustering or region growing methods are used to refine the foreground portion, extracting the main target regions, such as people, objects, or scene subjects, while maintaining spatial separation from the background. Based on this, a unique spatial location index is assigned to each segmented region. This index typically consists of an intra-frame coordinate range and a region number, reflecting the region's position and boundaries in the image. Through the above processing, each video frame not only contains visual content but also possesses a region index structure.

[0094] Step S202: Perform language layering processing on the synchronized subtitle data, independently encode different language subtitles into vectorized symbol units, generate subtitle units, and form a correspondence table with the position index.

[0095] In this embodiment, multilingual subtitles are transformed into structured units that are easy to call across platforms through layering and symbolization processing at the language dimension. Specifically, the input subtitle data is first distinguished according to language category, and an independent subtitle layer is generated for each language to avoid information interference when multiple languages ​​are presented in parallel. Then, within each subtitle layer, the text content is parsed into basic symbol units, and the subtitles are transformed from traditional bitmap representations into scalable vector symbol units through font outline extraction and character vectorization encoding, thus obtaining subtitle units and ensuring clarity at different resolutions. Furthermore, the subtitle units in each subtitle layer are mapped to the spatial position index of the video frame to form a relationship table. This relationship table clearly identifies the mapping position and hierarchical affiliation of each language subtitle unit within the video frame. Through the above processing, multilingual subtitles are associated with the video spatial structure in an independent and scalable form.

[0096] Step S203: Based on the correspondence table, perform content area analysis to determine the candidate display areas for subtitles, and prioritize the candidate display areas that overlap.

[0097] The specific steps of step S203 are as follows:

[0098] Step S2031: Perform saliency detection on the region index of the video frame, identify the visually dominant region, and mark it as a subtitle-disabled region.

[0099] In this embodiment, a saliency analysis method is used to filter the region indexes in the video frame to determine the key visual regions that subtitles should avoid. Specifically, firstly, saliency calculation is performed on the segmented video frame regions. This calculation integrates multi-dimensional features such as brightness contrast, color difference, edge intensity, and motion vector amplitude to evaluate the importance of each region in visual attention. Then, regions with saliency scores higher than a threshold are identified as visually dominant regions. These regions typically contain human faces, moving subjects, or core elements of the scene. After identification, these regions are marked as subtitle-prohibited areas by indexing, thereby preventing subtitles from obscuring key video information in the subsequent subtitle layout stage. Through the above processing, a visual priority distribution that can be referenced by the subtitle fusion algorithm is formed in the spatial structure of the video frame.

[0100] Step S2032: Outside the subtitle disabled area, multiple candidate display areas are generated based on the language layer corresponding to the subtitle, and an initial priority value is assigned to each candidate display area.

[0101] In this embodiment, selectable display positions for multilingual subtitles are constructed within the available area of ​​the video frame, and an initial priority sequence is established. Specifically, firstly, outside the subtitle prohibited area determined by saliency detection, several free spaces are divided according to the geometric boundaries of the region index; these free spaces serve as candidate display areas. Subsequently, combining the language layering information of the subtitle data, subtitle units of different languages ​​are mapped to these candidate display areas respectively, ensuring that subtitles of each language can be displayed independently without interfering with each other. Then, based on the correspondence between the candidate display areas and the subtitle language layers, an initial priority value is assigned to each candidate display area. The setting of this priority value usually takes into account factors such as the distance between the area and the main video content, the size of the area, and the number of subtitle layers. Through the above processing, multiple sets of candidate display areas and their initial priorities are formed.

[0102] Step S2033: Perform overlap detection on the candidate display areas. When two or more candidate display areas overlap, recalculate their priority.

[0103] In this embodiment, spatial overlap analysis is used to detect conflicts in candidate display areas, and their priorities are adjusted when overlap is detected. Specifically, firstly, the geometric boundary information recorded in the region index is used to cross-compare all candidate display areas one by one to determine whether they have overlapping or partial overlap in the intra-frame coordinate system. When two or more regions are detected to overlap, these regions are extracted to form a conflict set. Then, in the conflict set, a weighted evaluation is performed based on factors such as region area, spatial distance from the main video content, priority of the subtitle language it carries, and number of adjacent subtitle layers, and the priority value of each region is recalculated. Finally, an updated priority sequence of candidate display areas is generated. In this way, the rationality and hierarchical order of the subtitle display position are ensured during the region layout stage.

[0104] Step S2034: Sort the recalculated candidate display areas according to priority and form a sequence of candidate areas available for subtitle mapping.

[0105] In this embodiment, the candidate display regions after overlap detection and priority update are serialized to generate a spatial arrangement that can be directly used for subtitle mapping. Specifically, the latest priority value of each candidate display region is first extracted, and the regions are sorted from high to low according to their numerical values. During the sorting process, if the priority values ​​are the same, an alternative judgment rule is introduced, such as differentiating the ranking based on the region area, the distance between the region position and the frame boundary, or the relative relationship with the main target region. Then, the sorted candidate display regions are arranged sequentially to form an ordered candidate region sequence. The order of the regions in this sequence represents the selection priority during subtitle mapping. Through this serialization operation, the subtitle mapping process can follow a deterministic order among multiple candidate positions.

[0106] Step S204: Map the subtitle unit to the candidate display area with the highest priority, and dynamically adjust the subtitle line spacing and scaling ratio during the mapping process to generate a video frame sequence containing the multilingual subtitle display layer.

[0107] The specific steps of step S204 are as follows:

[0108] Step S2041: Load the subtitle units into the candidate display area with the highest priority according to the language layer order, and establish a mapping table between subtitle units and area indexes.

[0109] In this embodiment, the spatial loading of subtitle units and the establishment of corresponding relationships are completed by combining the language layering order with the priority of candidate display areas. Specifically, the vectorized subtitle units are first grouped according to the language layering structure, with each language layer maintaining its original word order internally. Then, the sorted candidate display area sequence is called, and subtitle units are allocated sequentially starting from the highest priority area, so that the subtitles are mapped to the most suitable display position first. During the allocation process, the timestamp, language layer information and area index of the subtitle unit are bound together, and a mapping table is generated accordingly. The mapping table clearly records the correspondence between the subtitle unit and the spatial position of the video frame. In this way, subtitles in different languages ​​can enter the appropriate display area sequentially under a predetermined priority, forming a structured mapping basis.

[0110] Step S2042: During the mapping process, calculate the initial layout parameters based on the number of subtitle characters and the size of the display area, and generate the corresponding line spacing and scaling settings.

[0111] In this embodiment, by quantifying the spatial relationship between the subtitle text and the display area, adaptive layout parameters are generated to ensure the complete visual presentation of the subtitles. Specifically, firstly, the number of characters in the current subtitle unit to be mapped is counted, and the required number of lines is determined in combination with the language layering. Then, the geometric dimension parameters of the display area, including width and height, are called to calculate the maximum available layout space of the subtitle unit in the area. Based on this, the number of characters that each line can accommodate and the line spacing are derived, and a scaling factor is generated according to the ratio between the total number of characters and the area boundary, so that the subtitles can be evenly distributed without exceeding the area boundary. The final line spacing and scaling ratio parameters provide initial settings for subsequent dynamic adjustments, so that the subtitle layout has a controllable baseline structure.

[0112] Step S2043: Dynamically adjust the initial layout parameters. When there is a risk of overlapping multilingual subtitles, perform hierarchical scaling and line spacing of the subtitle units.

[0113] In this embodiment, a hierarchical dynamic adjustment mechanism is introduced during the subtitle layout process to avoid overlapping of multilingual subtitles in the same area. Specifically, the layout position of each language subtitle unit in the candidate display area is first detected. When the detection result indicates that there is a risk of overlap, the dynamic adjustment process is triggered. Then, according to the priority order of the language layers, the original scaling ratio and line spacing of high-priority subtitles are maintained, while low-priority subtitles are scaled sequentially to reduce their occupied space. On this basis, the line spacing between different language layers is redistributed so that each subtitle layer forms a hierarchical separation in the vertical direction, avoiding visual overlap conflicts. Finally, the adjusted layout parameters are written into the mapping table so that the subtitle units can achieve multi-level distribution while maintaining temporal consistency.

[0114] Step S2044: Based on the dynamic adjustment results, generate a video frame sequence containing the multilingual subtitle display layer.

[0115] In this embodiment, dynamically arranged multilingual subtitles are fused with video frames to form a continuously output video frame sequence. Specifically, firstly, the scaling ratio and line spacing parameters of each subtitle unit are mapped to the corresponding candidate display area, and different language subtitles are rendered as independent display layers according to the adjusted hierarchical order. Then, the subtitle display layer is superimposed on the original video frames frame by frame, maintaining strict alignment between the subtitle layer and the video frame timestamp during the superposition process to ensure the consistency of subtitle and audio semantics. Furthermore, the superimposed results are arranged sequentially into a frame sequence, so that each frame contains the corresponding multilingual subtitle display layer. The final video frame sequence can serve as a complete video output channel.

[0116] Step S205: Add timestamp information to the video frame sequence containing the multilingual subtitle display layer to form a multilingual subtitle fusion result corresponding to the original video frame.

[0117] In this embodiment, a timestamp binding mechanism is used to establish a one-to-one correspondence between the generated subtitle display layer frame sequence and the original video frames on the timeline, thereby forming a unified multilingual subtitle fusion result. Specifically, firstly, the generation order information of each frame in the generated subtitle display layer frame sequence is read and compared with the decoding timestamp of the original video frame to ensure that the two are perfectly matched in time. Subsequently, during the matching process, a timestamp identifier consistent with its corresponding video frame is attached to each subtitle frame. This identifier can serve as a unified retrieval index in the playback or encapsulation stage. Furthermore, a synchronization index table is established for all subtitle display layer frames and the original video frames to achieve bidirectional correspondence at the frame level. The final subtitle fusion result not only includes video content but also embeds a timestamp-controlled multilingual subtitle layer at the structural level.

[0118] Step S3: Encapsulate the multilingual subtitle display layer into a WebM container and use adaptive bitrate encoding for cross-terminal playback compatibility.

[0119] like Figure 2 As shown, the specific steps of step S3 are as follows:

[0120] Step S301: The multilingual subtitle display layer is hierarchically classified with the corresponding video frames and target language audio streams, and an independent encapsulation index is established for each layer.

[0121] In this embodiment, multimodal data is categorized using a hierarchical management approach to enable flexible index retrieval in the subsequent encapsulation stage. Specifically, the generated multilingual subtitle display layer, corresponding video frames, and target language audio streams are first classified according to data type, with each type of data stored independently and maintained consistent with the original timeline. Subsequently, after classification, an independent encapsulation index is created for each type of data. This index not only includes timestamp identifiers but also records hierarchical attributes and data type tags, thereby logically distinguishing the subtitle layer, video layer, and audio layer from each other. Furthermore, these encapsulation indexes are organized into an index table, enabling rapid location of specific data segments during retrieval. Through the above processing, a multimodal data classification structure based on layers is formed.

[0122] Step S302: Input the encapsulation index into the WebM container building module, allocate independent tracks for each layer of data in the container structure, and generate a cross-layer reference table.

[0123] In this embodiment, a containerization encapsulation mechanism is used to maintain the independence and relevance of data at different levels within a unified data structure. Specifically, the encapsulation index generated in step S301 is first input into the WebM container building module. Based on the data type and timestamp information marked by the encapsulation index, this module allocates independent tracks for the subtitle layer, video layer, and audio layer, ensuring that data of different modalities have separate storage channels within the container. Subsequently, after the track allocation is completed, a cross-layer reference table is generated. This reference table cross-maps the timestamps and index numbers in each track, enabling subtitle frames to directly reference corresponding video frames, and audio clips to maintain alignment with subtitle markers. Through this structured design, the WebM container ensures the integrity of data within each layer while also possessing cross-layer synchronized index relationships.

[0124] Step S303: Perform multi-rate encoding on the data of each layer, and generate encapsulated data of multiple rate versions according to the preset network bandwidth grading standard.

[0125] The specific steps of step S303 are as follows:

[0126] Step S3031: Sample and segment the data of each layer to form multiple time segments.

[0127] Step S3032: Generate a corresponding multi-level bit rate set for the time segment according to the preset network bandwidth grading standard.

[0128] In this embodiment, the timing segments are divided into multiple bitrate versions using bandwidth tiering rules, thereby generating a bitrate set adaptable to different network environments. Specifically, the audio, video, and subtitle timing segments obtained in step S3031 are first read, and their original bitrates and segment durations are analyzed. Then, according to the preset network bandwidth tiering standard, the available bandwidth is divided into several level intervals, such as low speed, medium speed, and high speed, with each level corresponding to a target bitrate range. Based on this, the original segments are transcoded or resampled to generate different bitrate versions, and segments belonging to the same level are grouped together to form a multi-level bitrate set. Finally, each set contains several timing-consistent segment versions, which can be dynamically selected according to the actual bandwidth situation during playback. Through this process, video, audio, and subtitle data all obtain multi-level bitrate representations.

[0129] Step S3033: Perform parallel encoding processing on the bitrate set. During the encoding process, apply different compression ratios to the subtitle layer, audio layer and video layer respectively to generate segment data with multiple bitrate versions.

[0130] In this embodiment, a parallel processing mechanism is used to perform layered encoding of the bitrate set to ensure that different modal data are adaptable in a multi-bitrate environment. Specifically, the multi-level bitrate set generated in step S3032 is first split into layers, so that the video layer, audio layer, and subtitle layer enter the encoding queue independently. Then, during the parallel encoding process, different compression ratios are set for different layers of data. For example, the video layer uses a high compression ratio based on resolution and motion complexity to save bandwidth, the audio layer uses a medium compression ratio based on sampling rate and acoustic features to maintain intelligibility, and the subtitle layer uses a low compression ratio due to its smaller data volume to ensure the clarity of the vector form. Under the control of multi-threaded parallel computing, each layer of segments is encoded simultaneously and generates the corresponding multi-bitrate version. The final output is a set of segment data that are consistent in timing but differ in bandwidth requirements and decoding complexity.

[0131] Step S3034: Merge the fragment data according to the in-layer mapping table and output encapsulated data containing multiple bitrate versions.

[0132] In this embodiment, multi-bitrate segments are systematically integrated through intra-layer indexing relationships to generate unified encapsulation data that can be used for subsequent calls. Specifically, firstly, the subtitle, audio, and video segment data output in step S3033 are retrieved according to their respective intra-layer mapping tables to ensure that the multi-bitrate version of each time segment is consistent with the original timing in terms of index. Then, the segments of different layers are aligned according to the timestamp order, and the bitrate classification information within each layer is preserved during the alignment process so that they can be switched as needed during calls. Furthermore, the aligned segments are merged and packaged into an encapsulation set containing multiple bitrate versions, and cross-layer indexes are embedded in the set to maintain the synchronization relationship between subtitles, audio, and video. The final output encapsulation data not only has complete temporal continuity but also retains the multi-bitrate structure.

[0133] Step S304: Adaptively sort the encapsulated data of the various bitrate versions. When the terminal's decoding capability is insufficient or network conditions fluctuate, switch to the corresponding bitrate version through the cross-layer reference table.

[0134] The specific steps of step S304 are as follows:

[0135] Step S3041: Establish a performance index table for the encapsulated data of the multiple bitrate versions, and assign sorting weights to each version according to the bitrate and decoding complexity.

[0136] In this embodiment, a performance metric table that can be dynamically selected is established by quantitatively evaluating the encapsulation data of multiple bitrate versions. Specifically, firstly, key parameters of each version of the encapsulation data are extracted, including average bitrate, peak bandwidth requirement, frame structure complexity, and corresponding decoding computation. Then, based on the comprehensive relationship between bitrate and decoding complexity, an evaluation model is constructed, mapping different versions to comparable metric dimensions. On this basis, a ranking weight is assigned to each version, which reflects the adaptation priority of the version under different network conditions and terminal performance. Furthermore, the ranking weights of all versions are organized into a performance metric table, which not only contains numerical weights but also retains the hierarchical labels and timestamp indexes corresponding to each version. Through the above process, a performance metric table that can be used for subsequent ranking and adaptive switching is obtained.

[0137] Step S3042: Compare the performance index table with the cross-layer reference table to generate a comprehensive sorting sequence that includes the subtitle layer, audio layer and video layer.

[0138] In this embodiment, a sorting sequence applicable across multimodal data is generated through a joint comparison of performance metrics and cross-layer reference relationships. Specifically, the performance metric table formed in step S3041 is first used as the basic input, and the sorting weights of each bitrate version and their corresponding layer identifiers are extracted. Then, this information is compared item by item with the cross-layer reference table to ensure that a cross-accessible index relationship is established between the subtitle layer, audio layer, and video layer under the same timestamp. During the comparison process, inconsistencies or missing mapping entries are corrected to keep each layer logically synchronized. Furthermore, the multi-layer data after comparison is comprehensively arranged according to the sorting weights to generate a unified cross-layer sorting sequence. This sequence reflects performance priority and maintains the temporal consistency between subtitles, audio, and video. The final comprehensive sorting sequence serves as a global reference for subsequent adaptive switching.

[0139] Step S3043: When insufficient terminal decoding capability or decreased network bandwidth is detected, the second highest bitrate version is called from the comprehensive sorting sequence, and the synchronous mapping relationship between the subtitle layer and the audio layer is maintained.

[0140] In this embodiment, the network and terminal status are monitored in real time during playback, and dynamic switching is performed based on the priority results of the comprehensive sorting sequence. Specifically, the current status of the terminal is first determined by the bandwidth monitoring module and decoding load analysis. When the detection result shows that the decoding capability is insufficient or the available bandwidth is lower than the threshold, a bitrate degradation strategy is triggered. Then, the entry corresponding to the current timestamp is retrieved in the comprehensive sorting sequence, and the bitrate version with the second highest sorting weight is called to ensure that the content continuity is maintained while reducing the data load. During the calling process, the synchronous mapping relationship between the subtitle layer and the audio layer to the video layer is maintained, that is, the timestamp is compared and aligned through the cross-layer reference table so that the subtitle text and voice data still maintain a one-to-one correspondence with the video frame. Through this process, although the playback stream generated under resource-constrained conditions has been switched to a lower bitrate version, the temporal consistency of multimodal data can still be maintained.

[0141] Step S3044: When network or terminal conditions are detected to have recovered, reload the comprehensive sorting sequence and switch to a higher bitrate version according to the sorting weight.

[0142] In this embodiment, when terminal decoding resources or network bandwidth are restored, a comprehensive sorting sequence is used to switch up the bitrate and restore higher quality data transmission. Specifically, firstly, bandwidth detection and decoding load monitoring confirm that the current network conditions and hardware performance are higher than a preset threshold, meeting the decoding and transmission requirements of the higher bitrate version. Then, the comprehensive sorting sequence is reloaded, and available multi-bitrate versions are retrieved at the corresponding timestamp positions. In the retrieval results, the versions are compared from high to low according to their sorting weights, and the bitrate version with the highest priority and matching the current state is selected as the switching target. Furthermore, the video layer, audio layer, and subtitle layer are synchronously corrected through a cross-layer reference table to ensure that the timestamp alignment of different modal data does not shift during the switching process. Finally, the output playback stream smoothly transitions from the lower bitrate version to the higher bitrate version, so that the overall media quality is restored after the conditions improve.

[0143] Step S305: Output the sorted encapsulated data as a WebM stream, and attach a terminal identification tag to the WebM stream for cross-platform access and playback.

[0144] In this embodiment, multi-bitrate encapsulated data that has undergone sorting and adaptive switching is uniformly output as a WebM stream, and a terminal identification tag is attached to the output end to achieve cross-platform compatible calling. Specifically, the sorted encapsulated data is first formatted and packaged, and the data of the video layer, audio layer, and subtitle layer are aligned according to the timestamp index to generate a streaming structure that conforms to the WebM container specification. Subsequently, during the output process, a terminal identification tag is attached to the WebM stream. This tag contains metadata such as device type, operating system environment, and decoding capability level, which is used to achieve differentiated scheduling at the distribution end and the playback end. Furthermore, by combining the tag with the index table, the playback terminal can automatically select the appropriate bitrate version and subtitle layer when calling, ensuring that different platforms can correctly parse the same data stream. The final WebM stream has cross-platform callability and playback consistency.

[0145] Figure 2 The diagram illustrates the workflow and internal logic of the multilingual subtitle display layer during encoding and encapsulation within the WebM container. Specifically, subtitle data, audio frames, and video frames serve as input data sources. Before entering the WebM container, these data undergo sampling and segmentation processing, dividing the continuous signal into multiple independently operable time segments. Subsequently, these time segments are mapped to multiple sets of bitrates at different levels according to the network bandwidth tiering rules, thus providing selection space for subsequent multi-bitrate adaptation.

[0146] After the bitrate is divided, the various types of data enter the encapsulation and encoding process. This unit compresses the subtitle layer, audio layer and video layer respectively, and generates segment data in multiple bitrate versions. The encoded data is then merged according to mapping rules to form a multi-bitrate encapsulated dataset.

[0147] Meanwhile, the WebM container building module assigns independent tracks to the video, audio, and subtitle layers based on the input encapsulation index, and achieves timing alignment and logical association between different tracks through cross-layer reference tables, thereby ensuring that subtitles, audio, and video remain consistent during playback.

[0148] Finally, the encapsulated data processed above, together with the cross-layer reference information, is output as a WebM stream. This output not only has cross-terminal compatibility, but also supports adaptive switching according to different playback environments, thereby achieving real-time adaptation and stable playback under cross-platform conditions.

[0149] Step S4: When distributing to different terminals, dynamically adjust the subtitle synchronization and encapsulation parameters based on playback feedback to enable real-time adaptation and response of video and audio streams on different platforms.

[0150] In this embodiment, a terminal feedback mechanism is used to monitor the cross-platform playback status in real time, and the subtitle synchronization and encapsulation parameters are dynamically corrected based on the feedback results to ensure consistent adaptation across different devices. Specifically, during the distribution process, playback feedback information is first collected from various terminals, including indicators such as subtitle latency, audio and video decoding rate, frame loss rate, and buffer duration. Then, the feedback data is analyzed to determine whether there is an offset between the subtitles and audio, or whether the encapsulation parameters are overloaded in the current network environment. When a difference is found, the synchronization correction module is used to fine-tune the subtitle timestamp to realign it with the audio stream, and the bitrate configuration or buffer window size of the encapsulation layer is adjusted according to the feedback results. Furthermore, the corrected parameters are written back to the flow control module at the distribution end so that subsequent data streams can be continuously output according to the updated parameters. Through the above process, video, audio, and subtitles can achieve real-time adaptation and dynamic response on different platforms, thereby ensuring playback stability and timing consistency in a multi-terminal environment.

[0151] Example 2:

[0152] Please see Figure 3 Another embodiment of the present invention provides a cross-platform video adaptation response system, comprising: a signal conversion module, a subtitle display module, an encoding module, and a dynamic adaptation module;

[0153] The signal conversion module is used to separate the input video and audio streams and, using a translation model with prosody preservation, convert the audio signal into target language speech and corresponding timestamp subtitle data.

[0154] The subtitle display module is used to fuse the subtitle data with video frames, automatically determine the subtitle position based on content area analysis, and generate a multilingual subtitle display layer.

[0155] The encoding module is used to encapsulate the multilingual subtitle display layer into a WebM container and achieve cross-terminal playback compatibility through adaptive bitrate encoding;

[0156] The dynamic adaptation module is used to dynamically adjust the subtitle synchronization and encapsulation parameters based on playback feedback when distributing to different terminals, so that the video and audio streams can be adapted and responded to in real time on different platforms.

[0157] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0158] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-platform video adaptation response method, characterized in that, include: The input video and audio streams are separated, and a prosody-preserving translation model is used to convert the audio signal into the target language speech and corresponding timestamp subtitle data. The subtitle data is fused with video frames, and the subtitle position is automatically determined based on content area analysis to generate a multilingual subtitle display layer; The multilingual subtitle display layer is encapsulated in a WebM container and cross-terminal playback compatibility is achieved through adaptive bitrate encoding; When distributed to different terminals, the subtitle synchronization and encapsulation parameters are dynamically adjusted based on playback feedback, so that the video and audio streams can be adapted and responded to in real time on different platforms.

2. The cross-platform video adaptation response method as described in claim 1, characterized in that, The process of separating the input video and audio streams and using a prosody-preserving translation model to convert the audio signal into target language speech and corresponding timestamp subtitle data includes: The input video and audio streams are separated into multi-channel signals to obtain independent video and audio data streams, and a timestamp index is added to the audio data stream. The audio data stream is sliced ​​to extract speech content features, speech prosody features and background noise interference features, and the corresponding timestamp indexes are marked. A cross-language mapping dictionary is built based on speech content features, and a prosodic prediction sequence is built by combining speech prosodic features to form a multi-dimensional input matrix for translation; The multidimensional input matrix is ​​input into a translation model with prosody preservation, and the output is a target language speech segment and its corresponding semantic index sequence. The target language speech segments are re-matched according to the timestamp index and semantic index sequence, and audio data and synchronized subtitle data in the target language are generated after the matching is completed.

3. The cross-platform video adaptation response method as described in claim 2, characterized in that, The target language speech segments are re-matched according to the timestamp index and semantic index sequence, and after the matching is completed, audio data and synchronized subtitle data in the target language are generated, including: The target language speech segments are arranged sequentially according to their timestamp indices, and a preliminary speech segment sequence is established. The speech segment sequence is bidirectionally compared with the semantic index sequence. During the comparison process, the semantic tags with ambiguity are prioritized to obtain a matching relationship table. The speech segments are reconstructed according to the matching relationship table to generate a continuous target language audio stream, and synchronization markers corresponding to the semantic index are inserted during the reconstruction process. Based on the synchronization marker, subtitle data corresponding to the target language audio stream is generated, and the subtitle data is bound to the target language audio stream to form target language audio data and synchronized subtitle data.

4. The cross-platform video adaptation response method as described in claim 3, characterized in that, The subtitle data is fused with video frames, and the subtitle position is automatically determined based on content region analysis to generate a multilingual subtitle display layer, including: The video frames are segmented to extract the main target region and background region, and a location index is established for each region. The synchronized subtitle data is processed by language layering, and subtitles of different languages ​​are independently encoded into vectorized symbol units to generate subtitle units, and a corresponding relationship table is formed with the position index. Based on the aforementioned correspondence table, content area analysis is performed to determine candidate display areas for subtitles, and overlapping candidate display areas are prioritized. The subtitle units are mapped to the highest priority candidate display area, and the subtitle line spacing and scaling ratio are dynamically adjusted during the mapping process to generate a video frame sequence containing the multilingual subtitle display layer; Timestamp information is added to the video frame sequence containing the multilingual subtitle display layer to form a multilingual subtitle fusion result corresponding to the original video frame.

5. The cross-platform video adaptation response method as described in claim 4, characterized in that, Based on the aforementioned correspondence table, content region analysis is performed to determine candidate display regions for subtitles, and overlapping candidate display regions are prioritized, including: The region index of the video frame is subjected to saliency detection to identify the visually dominant region and mark it as a subtitle-disabled region; Outside the subtitle disabled area, multiple candidate display areas are generated based on the language layer corresponding to the subtitles, and an initial priority value is assigned to each candidate display area; Overlap detection is performed on the candidate display areas. When two or more candidate display areas overlap, their priorities are recalculated. The recalculated candidate display areas are sorted by priority to form a sequence of candidate areas for subtitle mapping.

6. The cross-platform video adaptation response method as described in claim 5, characterized in that, The subtitle units are mapped to the highest priority candidate display area, and the subtitle line spacing and scaling ratio are dynamically adjusted during the mapping process to generate a video frame sequence containing the multilingual subtitle display layer, including: The subtitle units are loaded into the highest priority candidate display area according to the language hierarchy, and a mapping table between subtitle units and area indexes is established; During the mapping process, the initial layout parameters are calculated based on the number of subtitle characters and the size of the display area, and the corresponding line spacing and scaling settings are generated. The initial layout parameters are dynamically adjusted, and when there is a risk of overlapping multilingual subtitles, the subtitle units are scaled and separated layer by layer and the line spacing is allocated hierarchically. Based on the dynamic adjustment results, a video frame sequence containing the multilingual subtitle display layer is generated.

7. The cross-platform video adaptation response method as described in claim 6, characterized in that, The multilingual subtitle display layer is encapsulated in a WebM container and cross-terminal playback compatibility is achieved through adaptive bitrate encoding, including: The multilingual subtitle display layer is hierarchically categorized with the corresponding video frames and target language audio streams, and an independent encapsulation index is established for each layer. The encapsulation index is input into the WebM container building module, which allocates independent tracks for data at each layer in the container structure and generates a cross-layer reference table. Multi-rate encoding is performed on the data at each layer, and multiple bitrate versions of encapsulated data are generated according to the preset network bandwidth tiering standard; The encapsulated data of the various bitrate versions are adaptively sorted. When the terminal's decoding capability is insufficient or the network conditions fluctuate, the corresponding bitrate version is switched through the cross-layer reference table. The sorted and encapsulated data is output as a WebM stream, and a terminal identification tag is attached to the WebM stream for cross-platform access and playback.

8. The cross-platform video adaptation response method as described in claim 7, characterized in that, The process of performing multi-rate encoding on data at each layer and generating encapsulated data with multiple rate versions according to a preset network bandwidth tiering standard includes: The data from each layer is sampled and segmented to form multiple time-series segments; The time-series segments are divided into corresponding multi-level bit rate sets according to a preset network bandwidth grading standard; Parallel encoding processing is performed on the bitrate set, and different compression ratios are applied to the subtitle layer, audio layer and video layer during the encoding process to generate segment data with multiple bitrate versions; The fragment data is merged according to the in-layer mapping table, and encapsulated data containing multiple bitrate versions is output.

9. The cross-platform video adaptation response method as described in claim 8, characterized in that, The encapsulated data for the various bitrate versions is adaptively sorted. When the terminal's decoding capability is insufficient or network conditions fluctuate, the system switches to the corresponding bitrate version through a cross-layer reference table, including: A performance index table is established for the encapsulated data of the various bitrate versions, and sorting weights are assigned to each version based on bitrate and decoding complexity; The performance index table is compared with the cross-layer reference table to generate a comprehensive sorting sequence that includes the subtitle layer, audio layer and video layer; When insufficient terminal decoding capability or decreased network bandwidth is detected, the second highest bitrate version is called from the comprehensive sorting sequence, while maintaining the synchronous mapping relationship between the subtitle layer and the audio layer. When network or terminal conditions are detected to have recovered, the comprehensive sorting sequence is reloaded, and a higher bitrate version is switched based on the sorting weight.

10. A cross-platform video adaptation and response system, used to implement the cross-platform video adaptation and response method according to any one of claims 1-9, characterized in that, include: Signal conversion module, subtitle display module, encoding module, and dynamic adaptation module; The signal conversion module is used to separate the input video and audio streams and, using a translation model with prosody preservation, convert the audio signal into target language speech and corresponding timestamp subtitle data. The subtitle display module is used to fuse the subtitle data with video frames, automatically determine the subtitle position based on content area analysis, and generate a multilingual subtitle display layer. The encoding module is used to encapsulate the multilingual subtitle display layer into a WebM container and achieve cross-terminal playback compatibility through adaptive bitrate encoding; The dynamic adaptation module is used to dynamically adjust the subtitle synchronization and encapsulation parameters based on playback feedback when distributing to different terminals, so that the video and audio streams can be adapted and responded to in real time on different platforms.

Images