Intelligent collaborative distribution method and system for broadcast television multi-modal content

Abstract

The application relates to a broadcast television multi-modal content intelligent collaborative distribution method and system. The method comprises the following steps: acquiring a video stream, an audio stream and a native subtitle stream of a broadcast television program, performing cross-modal semantic alignment analysis, obtaining a semantic anchor point set and a semantic time axis, combining historical transmission data of a heterogeneous distribution path to predict path network delay parameters, using a pre-trained reinforcement learning model to dynamically adjust the playing time sequence of multi-modal content, generating perceptually synchronized semantic anchor points, decoupling a content adaptation task into a local executable task and a remote collaborative task according to real-time resource state data of a target terminal, and finally generating a task scheduling instruction through collaborative scheduling optimization and sending the task scheduling instruction to the target terminal. The method can realize intelligent time sequence synchronization and collaborative task distribution of broadcast television content.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of broadcast television content distribution and artificial intelligence, and in particular relates to a method and system for intelligent collaborative distribution of multimodal broadcast television content. Background Technology

[0002] With the deep integration of broadcasting technology and artificial intelligence, intelligent content distribution technology has emerged, targeting multiple terminals and multiple modalities. This technology extracts multimodal features from video, audio, and subtitles to establish cross-platform adaptation models, aiming to achieve automated, precise, and collaborative distribution of program content across cable television, mobile devices, and the internet.

[0003] In traditional technologies, multi-terminal content distribution mainly relies on static transcoding and adaptation methods based on physical layers such as container format, resolution, and bitrate. It passively responds to parameters reported by the terminals and employs relatively independent distribution strategies across each terminal. For audio-visual subtitle synchronization, existing methods are mostly based on timestamp alignment mechanisms, such as RTP or RTCP protocols, compensating for latency at the receiving end through buffering and catching up.

[0004] However, current traditional adaptation and synchronization methods have significant limitations: First, the adaptation process lacks an understanding of content semantics, resulting in a clunky experience on new terminals such as portrait and foldable screens; second, the system cannot predict and proactively optimize for network fluctuations, device load, and user status; third, the distribution strategies for each terminal are isolated, lacking a global collaborative perspective and making it difficult to achieve efficient resource linkage; fourth, in real-time scenarios such as live streaming, the timestamp-dependent synchronization mechanism is susceptible to network jitter and processing latency, making it difficult to guarantee the perceptual consistency of audio, video, and subtitles, especially in heterogeneous streaming scenarios such as AI-generated real-time subtitles. Therefore, existing methods still have significant shortcomings in terms of intelligence, collaboration, and accuracy, hindering the improvement of the next-generation broadcast content distribution experience. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for intelligent collaborative distribution of multimodal broadcast television content to address the aforementioned technical problems.

[0006] Firstly, this application provides a method for intelligent collaborative distribution of multimodal broadcast television content, including:

[0007] S101. Obtain the original multimodal data of broadcast television programs, and perform cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline; wherein, the original multimodal data includes video stream, audio stream and native subtitle stream;

[0008] S102. Obtain historical transmission data of heterogeneous distribution paths, and perform time series analysis on the historical transmission data to obtain the path network delay parameters of each heterogeneous distribution path in the future distribution period.

[0009] S103. Based on the semantic anchor point set, semantic time axis and path network delay parameters, the playback timing of the original multimodal data is adjusted by combining a pre-trained reinforcement learning model to obtain the semantic anchor points for perceptual synchronization.

[0010] S104. Obtain real-time resource status data input from the target terminal. Based on the semantic anchor point and semantic time axis of perception synchronization, perform semantic granularity division and adaptation requirement parsing on the original multimodal data in sequence, generate content adaptation tasks corresponding to the original multimodal data, and decouple the content adaptation tasks into local executable tasks and remote collaborative tasks based on the resource status data.

[0011] S105. Perform collaborative scheduling optimization analysis on local executable tasks and remote collaborative tasks, generate corresponding task scheduling instructions, and send the task scheduling instructions to the target terminal; wherein, the task scheduling instructions are used to instruct the target terminal to execute local executable tasks and initiate collaborative processing requests for remote collaborative tasks to the edge server or cloud server.

[0012] In one embodiment, cross-modal semantic alignment analysis is performed on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline, including:

[0013] S201. Extract features from the video stream, audio stream, and native subtitle stream to obtain video spatiotemporal feature sequence, audio acoustic feature sequence, and subtitle semantic feature sequence;

[0014] S202. Based on a cross-modal multi-head attention mechanism, cross-modal semantic interaction calculations are performed on the video spatiotemporal feature sequence, audio acoustic feature sequence, and subtitle semantic feature sequence to generate an audio-video semantic association weight matrix and a video subtitle semantic association weight matrix. The video semantic association weight matrix is ​​used to characterize the semantic association strength between the video spatiotemporal feature sequence and the audio acoustic feature sequence; the video subtitle semantic association weight matrix is ​​used to characterize the semantic association strength between the video spatiotemporal feature sequence and the subtitle semantic feature sequence.

[0015] S203. Perform temporal-dimensional aggregation calculations on the audio-visual semantic association weight matrix and the video subtitle semantic association weight matrix respectively to obtain the association strength sequence corresponding to each time point, and select the time points that simultaneously meet the peak condition from the association strength sequence according to the preset peak threshold as the synchronization time points.

[0016] S204. Construct a triplet by combining the synchronization time point with the corresponding video frame index and subtitle text index, and use the structured set composed of each triplet as the semantic anchor set.

[0017] S205. Perform real-time speech recognition on the audio acoustic feature sequence and audio stream to obtain the generation completion time of the subtitle text unit in the original subtitle stream. Combine the time correspondence in the semantic anchor set to perform offset correction on the generation completion time based on semantic coherence, and obtain the optimal presentation time point of each subtitle text unit.

[0018] S206. Based on the optimal presentation time and generation completion time of each subtitle text unit, construct an ordered data structure in chronological order and output it as a semantic timeline.

[0019] In one embodiment, based on the optimal presentation time and generation completion time of each subtitle text unit, an ordered data structure is constructed in chronological order, and the output is a semantic timeline, including:

[0020] S301. Perform vectorization encoding on the semantic features to obtain the initial semantic feature vector, and map the initial semantic feature vector to a preset high-dimensional semantic space to obtain the semantic feature vector corresponding to each subtitle text unit.

[0021] S302. Calculate the semantic similarity between the semantic feature vectors of adjacent subtitle text units to obtain a semantic similarity sequence, and perform minimum value detection on the semantic similarity sequence according to the preset mutation detection threshold to obtain semantic mutation points.

[0022] S303. Take the best presentation time point corresponding to the semantic mutation point as the time axis segmentation node, and combine it with the generation completion time corresponding to the semantic mutation point to obtain the time span range of each time axis segmentation node.

[0023] S304. Linearly concatenate the time axis segmentation nodes and time span intervals in chronological order to obtain the initial time series framework, and fill the semantic feature vectors into the corresponding time span intervals in the initial time series framework to generate the semantic time axis.

[0024] In one embodiment, the reinforcement learning model includes a temporal policy decision layer, a dynamic compensation calculation layer, and a temporal synchronization correction layer. Based on the semantic anchor set, semantic timeline, and path network delay parameters, the model, combined with a pre-trained reinforcement learning model, adjusts the playback timing of the original multimodal data to obtain perceptually synchronized semantic anchors, including:

[0025] S401. Extract the semantic feature vector and temporal dynamic features corresponding to the semantic anchor point set and semantic time axis, and concatenate the semantic feature vector, temporal dynamic features and path network delay parameters in multiple dimensions to construct a joint state feature vector; wherein, the joint state feature vector is used to characterize the current network transmission environment and content playback status;

[0026] S402. The joint state feature vector is nonlinearly mapped by the deep feature extraction network of the time-series policy decision layer to obtain the high-dimensional hidden feature vector of the joint state feature vector.

[0027] S403. Through a hierarchical reinforcement learning network in the temporal strategy decision layer, a global temporal adjustment vector is obtained by making a global trend decision on the high-dimensional latent feature vector. A local adjustment step size is also obtained by calculating the local step size. The global temporal adjustment vector and the local adjustment step size are fused to output a temporal optimization control strategy for the current distribution period. Among them, the high-dimensional latent feature vector is used to characterize the deep correlation features of the original multimodal data in the semantic, temporal, and network state dimensions; the global temporal adjustment vector is used to characterize the optimization direction and overall offset trend of the playback timing under the current network transmission environment; and the local adjustment step size is used to quantify the specific latency compensation magnitude of each semantic anchor point.

[0028] S404. The timing optimization control strategy is analyzed by the strategy network of the dynamic compensation calculation layer. The strategy feature vector corresponding to each semantic anchor point is extracted, the strategy feature vector is mapped to a scalar value, and the scalar value is used as the basic adjustment coefficient of each semantic anchor point.

[0029] S405. Based on the semantic feature vectors corresponding to each semantic anchor point, the importance of each semantic anchor point is quantitatively evaluated to obtain the semantic importance weight of each semantic anchor point.

[0030] S406. Substitute the basic adjustment coefficient, semantic importance weight, and path network delay parameters into the preset formula to calculate the dynamic delay compensation amount for each semantic anchor point.

[0031] S407. The initial playback timestamp and the corresponding dynamic delay compensation amount are superimposed through the timing synchronization correction layer to obtain the preliminary compensation timestamp;

[0032] S408. Based on the semantic timeline and the temporal correlation between each semantic anchor point determined based on the semantic feature vector, perform contextual temporal smoothness detection on the preliminary compensation timestamp and generate temporal smoothness constraints.

[0033] S409. Based on the time series smoothness constraint, the initial compensation timestamp is fine-tuned and corrected using an iterative optimization algorithm to obtain the final compensation timestamp.

[0034] S410. Update the timestamp attribute of the corresponding semantic anchor point according to the final compensation timestamp to obtain the corrected semantic anchor point, and use the corrected semantic anchor point as the semantic anchor point for perception synchronization.

[0035] In one embodiment, the basic adjustment coefficient, semantic importance weight, and path network delay parameters are substituted into a preset formula for calculation to obtain the dynamic delay compensation amount for each semantic anchor point, including:

[0036] The expression for dynamic delay compensation is:

[0037]

[0038] in, The first semantic anchor in the set Index of semantic anchor points Indicates that for the first The dynamic latency compensation amount calculated for each semantic anchor point. The policy network of the reinforcement learning model is represented as the first... The basic adjustment coefficients for the output of each semantic anchor point Indicates the first The semantic importance weight of each semantic anchor point Indicates the first term used by the target terminal Predicting network latency jitter for each distribution path. Indicates the target terminal using the first The predicted average latency of each distribution path, and They represent the first The and the first The initial playback timestamp of each semantic anchor point Represents a timescale reference constant. This represents the fidelity scaling factor, used to control the sensitivity of the compensation amount to the original interval between anchor points. This represents the hyperbolic tangent function, used as a saturation function.

[0039] In one embodiment, the method further includes:

[0040] S501. Obtain the local model update data uploaded by each target terminal, and apply random noise perturbation that meets the differential privacy requirements to the local model update data to obtain noisy model update data.

[0041] S502. Perform secure aggregation calculations on the noisy model update data to generate a global model update gradient;

[0042] S503. Iteratively update the pre-trained reinforcement learning model based on the global model update gradient to update the reinforcement learning model.

[0043] Secondly, this application also provides a smart collaborative distribution system for multimodal broadcast television content, comprising:

[0044] The alignment analysis module is used to acquire the raw multimodal data of broadcast television programs and perform cross-modal semantic alignment analysis on the raw multimodal data to obtain a set of semantic anchor points and a semantic timeline; the raw multimodal data includes video streams, audio streams and native subtitle streams;

[0045] The latency parameter acquisition module is used to acquire historical transmission data of heterogeneous distribution paths, perform time series analysis on the historical transmission data, and obtain the path network latency parameters of each heterogeneous distribution path in the future distribution period.

[0046] The timing synchronization module is used to adjust the playback timing of the original multimodal data based on the semantic anchor point set, semantic timing axis and path network delay parameters, combined with a pre-trained reinforcement learning model, to obtain the semantic anchor points for perceptual synchronization.

[0047] The adaptation task generation module is used to obtain real-time resource status data input from the target terminal. Based on the semantic anchor point and semantic time axis of perception synchronization, the original multimodal data is sequentially divided into semantic granularity and the adaptation requirements are parsed to generate content adaptation tasks corresponding to the original multimodal data. Based on the resource status data, the content adaptation tasks are decoupled into locally executable tasks and remote collaborative tasks.

[0048] The instruction generation module is used to perform collaborative scheduling optimization analysis on local executable tasks and remote collaborative tasks, generate corresponding task scheduling instructions, and send the task scheduling instructions to the target terminal. The task scheduling instructions are used to instruct the target terminal to execute local executable tasks and initiate collaborative processing requests for remote collaborative tasks to edge servers or cloud servers.

[0049] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0050] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0051] The aforementioned intelligent collaborative distribution method and system for multimodal content in broadcast television acquires the original multimodal data of broadcast television programs and performs cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline. The original multimodal data includes video streams, audio streams, and native subtitle streams. Historical transmission data of heterogeneous distribution paths is acquired, and time-series analysis is performed on the historical transmission data to obtain path network delay parameters for each heterogeneous distribution path during future distribution periods. Based on the set of semantic anchor points, the semantic timeline, and the path network delay parameters, a pre-trained reinforcement learning model is used to adjust the playback timing of the original multimodal data to obtain perceptually synchronized semantic anchor points. The method involves acquiring real-time resource status data from the target terminal, and then, based on the semantic anchor points and semantic timeline of the perception synchronization, sequentially performing semantic granularity segmentation and adaptation requirement parsing on the original multimodal data. This generates content adaptation tasks corresponding to the original multimodal data, and decouples these tasks into locally executable tasks and remote collaborative tasks based on the resource status data. The method also performs collaborative scheduling optimization analysis on the locally executable tasks and remote collaborative tasks, generating corresponding task scheduling instructions, which are then sent to the target terminal. These task scheduling instructions instruct the target terminal to execute the locally executable tasks and initiate collaborative processing requests for remote collaborative tasks to edge servers or cloud servers. This method enables intelligent time-series synchronization and collaborative task distribution of broadcast television content. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating an intelligent collaborative distribution method for multimodal broadcast television content, provided as an exemplary embodiment of this application;

[0054] Figure 2 This is a schematic diagram of the structure of a broadcast television multimodal content intelligent collaborative distribution system provided as an exemplary embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] In one embodiment, such as Figure 1As shown, a method for intelligent collaborative distribution of multimodal broadcast television content is provided. This embodiment illustrates the application of this method to a smart terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through the interaction between the smart terminal and the server. In this embodiment, the method includes the following steps:

[0057] S101. Obtain the original multimodal data of broadcast television programs, and perform cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline; wherein, the original multimodal data includes video stream, audio stream and native subtitle stream.

[0058] The semantic anchor set can be a structured set constructed by the smart terminal through cross-modal semantic alignment analysis, which identifies time points in the video stream, audio stream and native subtitle stream that are highly semantically related, and constructs them with the corresponding video frame index and subtitle text index, respectively. This set is used to mark the core semantic synchronization benchmark of multimodal data.

[0059] A semantic timeline can be an ordered temporal framework generated by a smart terminal by integrating the semantic association time information of various modal data in chronological order.

[0060] Specifically, smart terminals can acquire raw multimodal data in real time through signal reception or network interfaces. Secondly, smart terminals can employ cross-modal semantic alignment technology: First, the core features of video, audio, and subtitles can be extracted separately; then, the smart terminal can map the core features of video, audio, and subtitles to a unified semantic vector space through cross-modal feature embedding, and use an attention interaction mechanism to calculate the semantic association strength between modalities, thereby identifying key time points where video images, audio content, and subtitle text are highly semantically consistent.

[0061] Furthermore, smart terminals can generate a set of semantic anchor points by establishing structured associations between key nodes that are highly consistent in semantic expression across different modalities and their corresponding video frame indices and subtitle text indices. Even further, smart terminals can integrate the semantic association time information of each node in chronological order to construct an ordered semantic timeline, serving as a semantic perception framework for subsequent time-series adjustments and content adaptation.

[0062] S102. Obtain historical transmission data of heterogeneous distribution paths, and perform time series analysis on the historical transmission data to obtain the path network delay parameters of each heterogeneous distribution path in the future distribution period.

[0063] Among them, heterogeneous distribution paths can be a collection of multiple logical or physical channels with different transmission characteristics provided by different network technologies, transmission protocols or network operators between the target terminal and the content source.

[0064] The path network delay parameter can be a set of core parameters that characterize the transmission delay characteristics of heterogeneous distribution paths in future distribution periods, obtained by intelligent terminals through time series analysis to mine the transmission patterns of heterogeneous distribution paths.

[0065] Optionally, the path network delay parameters may include, but are not limited to, the predicted average delay, the delay jitter amplitude, and the peak delay.

[0066] Specifically, the smart terminal can acquire historical transmission data of each heterogeneous distribution path in real time through its built-in network data acquisition module. Secondly, the smart terminal can use time series analysis technology to preprocess the historical transmission data, identifying and removing interfering data using the 3σ principle and the isolated forest algorithm, and performing data standardization to unify the data dimensions of different paths. Subsequently, the smart terminal can perform trend analysis and fluctuation feature extraction on the historical transmission data to uncover the long-term transmission delay variation patterns over time. By calculating statistical indicators such as variance, standard deviation, and skewness within a sliding time window, the strength and distribution characteristics of delay fluctuations can be quantified, thereby constructing a time series model that reflects the transmission patterns of heterogeneous paths. Finally, the path network delay parameters for each heterogeneous distribution path in future distribution periods can be obtained.

[0067] Optionally, the temporal pattern model can be a modular temporal prediction model adapted to the transmission characteristics of heterogeneous distribution paths. The input can be standardized historical transmission core data, and the output can be a set of path network delay parameters for each heterogeneous distribution path in the future distribution period. The temporal pattern model includes a feature fusion layer, a temporal dependency capture layer, and a parameter prediction layer. The feature fusion layer is used to integrate multi-dimensional temporal features and path attribute features. The temporal dependency capture layer captures the temporal evolution logic of transmission delay by mining the correlation between long-term temporal change patterns and short-term fluctuations in historical data. The parameter prediction layer outputs quantified delay parameters based on the aforementioned features and patterns.

[0068] S103. Based on the semantic anchor set, semantic time axis and path network delay parameters, the playback timing of the original multimodal data is adjusted by combining a pre-trained reinforcement learning model to obtain the semantic anchors for perceptual synchronization.

[0069] Among them, the semantic anchor point for perception synchronization can be a set of structured anchor points generated by the intelligent terminal after dynamically optimizing and correcting the timestamp of the original semantic anchor point by combining network transmission characteristics and multimodal semantic association.

[0070] The reinforcement learning model can be an intelligent decision-making model for multimodal content temporal synchronization optimization. The input can be the structured information of the semantic anchor set, the temporal dynamic features of the semantic time axis, and the joint state feature vector formed by fusing the path network delay parameters. The output can be the temporal optimization control strategy and dynamic delay compensation amount for each semantic anchor. The reinforcement learning model adopts a hierarchical modular architecture.

[0071] Specifically, the smart terminal can extract the core features of the semantic anchor set and semantic time axis, and then perform multi-dimensional fusion and standardization of the core features of the semantic anchor set and semantic time axis with the path network latency parameters to construct a joint state feature vector representing the current network state and content playback state. Secondly, the smart terminal can input the joint state feature vector into a pre-trained reinforcement learning model. Through the collaborative processing of the model's hierarchical modules, a timing optimization control strategy and dynamic latency compensation amount are generated. Subsequently, the original semantic anchor timestamps are corrected based on the dynamic latency compensation amount, and synchronization verification is completed in conjunction with semantic coherence constraints. Finally, the perceptually synchronized semantic anchors are obtained.

[0072] S104. Obtain real-time resource status data input from the target terminal. Based on the semantic anchor point and semantic time axis of perception synchronization, perform semantic granularity division and adaptation requirement parsing on the original multimodal data in sequence, generate content adaptation tasks corresponding to the original multimodal data, and decouple the content adaptation tasks into locally executable tasks and remote collaborative tasks based on the resource status data.

[0073] Among them, real-time resource status data can be a comprehensive data set representing the target terminal's own operating capabilities and available resources, reported in real time.

[0074] Optionally, real-time resource status data may include, but is not limited to, terminal hardware computing power, memory usage, remaining storage space, current network connection stability, and local processing module load status.

[0075] Content adaptation tasks can be a set of targeted adaptation processing tasks generated by intelligent terminals after performing semantic-level granular decomposition of the original multimodal data and terminal adaptation requirement analysis based on the semantic anchor points and semantic time axis of perception synchronization.

[0076] Specifically, the smart terminal can receive and verify real-time resource status data reported by the target terminal, extract core resource features, and, using the semantic anchor point of perception synchronization as the splitting benchmark and combining the temporal logic of the semantic time axis, perform semantic granularity segmentation on the original multimodal data to obtain content fragments with independent semantic integrity. Secondly, the smart terminal can analyze adaptation requirements based on the target terminal's device specifications and usage scenarios to generate content adaptation tasks. Finally, the smart terminal can match and evaluate the requirements of the content adaptation tasks with the real-time resource status data, determining tasks that can be carried out locally as locally executable tasks, and classifying those exceeding the carrying capacity as remote collaborative tasks.

[0077] S105. Perform collaborative scheduling optimization analysis on local executable tasks and remote collaborative tasks, generate corresponding task scheduling instructions, and send the task scheduling instructions to the target terminal; wherein, the task scheduling instructions are used to instruct the target terminal to execute local executable tasks and initiate collaborative processing requests for remote collaborative tasks to the edge server or cloud server.

[0078] Specifically, the intelligent terminal extracts the core attribute information of locally executable tasks and remote collaborative tasks, and combines this information with the path network latency parameters of the heterogeneous distribution path, the real-time resource status of the target terminal, and the available resource overview of the edge server or cloud server to perform collaborative scheduling decision analysis. The collaborative scheduling decision analysis aims to comprehensively optimize task processing efficiency, minimize end-to-end latency, and improve the overall system resource utilization rate, determining the execution priority and execution sequence of each task, as well as the optimal service node and transmission link for remote collaborative tasks. Finally, a task scheduling instruction is generated, including the local task execution command, the service object identifier of the remote collaborative task, task priority, and execution sequence information, and sent to the target terminal through a secure communication link.

[0079] Optionally, core attribute information may include, but is not limited to, task complexity, processing time limit, and resource requirements.

[0080] The aforementioned intelligent collaborative distribution method for multimodal broadcast television content establishes a set of semantic anchor points and a semantic timeline through cross-modal semantic alignment analysis on the intelligent terminal, achieving deep semantic association capture of video, audio, and subtitles, overcoming the shortcomings of traditional timestamp alignment in semantic coherence. By predicting path network latency parameters through time-series analysis based on historical transmission data, it can proactively assess network conditions and provide accurate network environment representations for timing synchronization. Dynamic adjustment of playback timing using a pre-trained reinforcement learning model maintains perceptual consistency of multimodal content under varying network conditions. By semantically dividing and decoupling content adaptation tasks based on the resource status of the target terminal, it achieves reasonable scheduling of local and remote collaborative tasks, improving system resource utilization and task execution efficiency. Finally, it generates task scheduling instructions through collaborative scheduling optimization, supporting efficient linkage between the terminal and edge cloud. In summary, this method achieves systematic integration in semantic understanding, network awareness, dynamic timing optimization, and resource collaboration, significantly improving the synchronization accuracy, adaptability, and collaborative capabilities of broadcast television content in complex distribution environments.

[0081] In one embodiment, performing cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline may include the following steps:

[0082] S201. Extract features from the video stream, audio stream, and native subtitle stream to obtain video spatiotemporal feature sequence, audio acoustic feature sequence, and subtitle semantic feature sequence;

[0083] Among them, the video spatiotemporal feature sequence can be an ordered set of features obtained by the smart terminal after extracting features from the continuous frames of the video stream, which integrates intra-frame spatial structure information and inter-frame temporal motion information.

[0084] An audio acoustic feature sequence can be an ordered set of vectors representing the physical properties and auditory perception features of sound extracted by a smart terminal after the audio stream is segmented and processed according to a preset time window.

[0085] The semantic feature sequence of subtitles can be an ordered set of features that reflects the semantic connotation and contextual relationship of the text after the smart terminal performs semantic encoding processing on the text units in the native subtitle stream.

[0086] For example, a smart terminal can employ a general feature extraction technique adapted to the characteristics of multimodal data. Targeting the dynamic temporal and spatial structural attributes of video streams, it can first capture the spatial features of consecutive video frames frame by frame to obtain core spatial information. Secondly, the smart terminal can mine the temporal dimension features in adjacent frames and consecutive frame sequences based on inter-frame correlation extraction techniques. Furthermore, it can organically integrate spatial and temporal features through spatiotemporal feature fusion processing, outputting an ordered video spatiotemporal feature sequence that combines spatial representation capabilities with temporal correlation information. For audio streams, the smart terminal can segment continuous audio signals according to a preset time window and extract core acoustic parameters that characterize the essence of sound through feature analysis methods adapted to the physical properties and auditory perception characteristics of audio signals. These parameters are then integrated in an orderly manner to generate an audio acoustic feature sequence.

[0087] Furthermore, smart terminals can break down the subtitle text of the native subtitle stream into units to obtain independent semantic expression units. Through a text semantic encoding processing mechanism, each text unit is converted into a vector form that can reflect semantic connotations. At the same time, the contextual semantic relationships between adjacent text units are captured, and finally an ordered sequence of subtitle semantic features is generated.

[0088] S202. Based on the cross-modal multi-head attention mechanism, cross-modal semantic interaction calculation is performed on the video spatiotemporal feature sequence, audio acoustic feature sequence, and subtitle semantic feature sequence to generate an audio-video semantic association weight matrix and a video subtitle semantic association weight matrix. Among them, the video semantic association weight matrix is ​​used to characterize the semantic association strength between the video spatiotemporal feature sequence and the audio acoustic feature sequence; the video subtitle semantic association weight matrix is ​​used to characterize the semantic association strength between the video spatiotemporal feature sequence and the subtitle semantic feature sequence.

[0089] Among them, the cross-modal multi-head attention mechanism can be modified based on the multi-head attention module in the Transformer architecture. The core feature of the cross-modal multi-head attention mechanism is that it allows the query vector, key vector and value vector to come from different modal feature sequences, and calculates the semantic association between different modal pairs in parallel by pre-setting the division of labor of the attention heads.

[0090] For example, the smart terminal can perform dimensional unification and temporal alignment preprocessing on the video spatiotemporal feature sequence, audio acoustic feature sequence, and subtitle semantic feature sequence. Secondly, the smart terminal can use a cross-modal multi-head attention mechanism to split multiple parallel attention heads to capture cross-modal semantic associations in different dimensions. Some attention heads focus on mining the semantic fit points between video spatiotemporal features and audio acoustic features, calculating multi-dimensional association scores between them. Another set of attention heads focuses on the semantic matching relationship between video spatiotemporal features and subtitle semantic features, quantifying the correspondence strength between the image content and the text meaning. After concatenation and fusion of the calculation results from all attention heads, a scaling dot product attention mechanism is used to normalize and weight the association strength, generating audio-video semantic association weight matrices and video-subtitle semantic association weight matrices, respectively.

[0091] The expression for the audio-visual semantic association weight matrix is ​​as follows:

[0092]

[0093] In the formula, This represents the semantic association weight matrix between audio and video. This represents the query matrix in audio / video attention. Indicates a query. The query matrix in audio-video attention is used for audio-video cross-modal association calculation. This represents the key projection matrix in audio / video attention. Indicates key, This represents the transpose of the audio feature sequence after transformation by the key projection matrix. Indicates the scaling factor. Represents a video feature sequence. Represents an audio feature sequence. The function is used to normalize the association scores into a probability distribution and generate a weight matrix.

[0094] The expression for the semantic association weight matrix of video subtitles is as follows:

[0095]

[0096] In the formula, This represents the semantic association weight matrix of video subtitles. This represents the query matrix in video caption attention. Indicates a query. This indicates that the query matrix in video caption attention is specifically used for video-caption cross-modal association calculation. This represents the key projection matrix in video subtitle attention. Indicates key, This represents the transpose of the subtitle feature sequence after transformation by the key projection matrix. Indicates the scaling factor. Represents a video feature sequence. Represents the subtitle feature sequence, The function is used to normalize the association scores into a probability distribution and generate a weight matrix.

[0097] S203. Perform temporal dimension aggregation calculations on the audio-visual semantic association weight matrix and the video subtitle semantic association weight matrix respectively to obtain the association strength sequence corresponding to each time point, and select the time points that simultaneously meet the peak condition from the association strength sequence according to the preset peak threshold as the synchronization time points.

[0098] The correlation strength sequence at each time point can be an ordered numerical sequence arranged in chronological order, obtained by the smart terminal performing time-dimensional correlation strength aggregation calculations on the audio-visual semantic correlation weight matrix and the video subtitle semantic correlation weight matrix respectively.

[0099] For example, given the audio-visual semantic association weight matrix and the video subtitle semantic association weight matrix, the smart terminal can use a temporal-dimensional association strength aggregation method to perform feature integration and quantization calculation on the matrix elements corresponding to each time point. This transforms the multidimensional parameters reflecting the semantic association between modalities at the same time point into a single association strength value, and generates two independent association strength sequences in chronological order. Each sequence's elements quantify the degree of semantic association between the two modalities at the corresponding time point. Secondly, the smart terminal can retrieve a preset peak threshold and perform peak detection on the two association strength sequences, identifying peak time points in each sequence whose values ​​exceed the threshold. It then filters out time points that simultaneously exist in both sequence peak sets through temporal consistency verification, and finally uses these time points as synchronization time points.

[0100] S204. Construct a triplet by combining the synchronization time point with the corresponding video frame index and subtitle text index, and use the structured set composed of each triplet as the semantic anchor set.

[0101] For example, the smart terminal can associate each identified synchronization time point with a corresponding frame in the original video stream using a preset timing mapping rule. Specifically, the smart terminal calculates the precise video frame number corresponding to each synchronization time point based on the frame rate and timestamp information of the video stream and records it as a video frame index. Secondly, after parsing the timing metadata of the native subtitle stream, the smart terminal maps the same synchronization time point to subtitle text units appearing before and after that time point and records the unique position identifier of that text unit in the subtitle sequence as a subtitle text index.

[0102] Furthermore, the smart terminal can combine each synchronization time point, its corresponding video frame index, and subtitle text index to construct a triplet data unit with a fixed format. Finally, the smart terminal traverses all identified synchronization time points, generates corresponding triplets, and organizes and stores all triplets according to chronological order or specific rules to generate a complete, ordered, and efficiently queried structured data set, namely the semantic anchor set.

[0103] S205. Perform real-time speech recognition on the audio acoustic feature sequence and audio stream to obtain the generation completion time of the subtitle text unit in the original subtitle stream. Combine the time correspondence in the semantic anchor set to perform offset correction on the generation completion time based on semantic coherence, and obtain the optimal presentation time point of each subtitle text unit.

[0104] For example, a smart terminal can perform online speech recognition processing on audio acoustic feature sequences and raw audio stream data. The online speech recognition processing searches for the text sequence that is temporally aligned with the audio signal and has the highest probability within the observed sequence composed of the audio feature sequences, thereby outputting recognized text segments arranged in chronological order and their corresponding time intervals. Secondly, for each subtitle text unit in the native subtitle stream, the smart terminal can calculate the semantic similarity between the subtitle text unit and each recognized text segment, then match the semantic similarity with the recognized segment with the highest similarity, and record the end timestamp of that recognized segment as the generation completion time of the subtitle text unit.

[0105] Furthermore, given a set of semantic anchor points, the smart terminal can analyze the semantic role of each subtitle text unit within its context and examine the deviation between the generation completion time of each subtitle text unit and the standard synchronization time marked by neighboring semantic anchor points. Finally, based on preset semantic coherence principles, the smart terminal can perform minor forward or backward offset corrections on the generation completion times with significant deviations, calculating the optimal presentation time for each subtitle text unit that perceptually aligns with the video plot and audio rhythm.

[0106] S206. Based on the optimal presentation time and generation completion time of each subtitle text unit, construct an ordered data structure in chronological order and output it as a semantic timeline.

[0107] For example, the smart terminal can create an ordered data structure that supports timestamp indexing as a container for the semantic timeline. Secondly, the smart terminal can traverse all subtitle text units. For each subtitle text unit, the optimal presentation time determined in step S205 is used as the main timestamp, and its original generation completion time is used as the reference timestamp. These are then encapsulated with the semantic feature representation corresponding to the subtitle text unit to generate a structured timeline data item.

[0108] Furthermore, the smart terminal can insert all data items into the aforementioned ordered container according to the chronological order of their master timestamps, generating a timeline sequence. Finally, the smart terminal can serialize this sequence, outputting a semantic timeline that includes complete time information and semantic associations, which can be directly accessed by subsequent modules.

[0109] In this embodiment, the intelligent terminal establishes a high-precision semantic timing benchmark for subsequent intelligent distribution by autonomously extracting cross-modal features, calculating semantic associations, and aligning time points on the edge.

[0110] In one embodiment, based on the optimal presentation time and generation completion time of each subtitle text unit, an ordered data structure is constructed in chronological order, and the output is a semantic timeline, which may include the following steps:

[0111] S301. Perform vectorization encoding on the semantic features to obtain the initial semantic feature vector, and map the initial semantic feature vector to the preset high-dimensional semantic space to obtain the semantic feature vector corresponding to each subtitle text unit.

[0112] The initial semantic feature vector can be the original vector representation obtained by the smart terminal after performing basic semantic encoding on the subtitle text unit, which is used to initially reflect the literal meaning and basic semantic distribution of the text.

[0113] For example, the smart terminal can perform standardized text processing on each subtitle text unit, including word segmentation, lemmatization, and punctuation filtering, and call a pre-loaded lightweight semantic encoder to encode the processed word sequence, outputting a fixed-dimensional vector as the initial semantic feature vector of the text unit. Secondly, the smart terminal can load a pre-trained high-dimensional semantic mapping model, input each initial semantic feature vector into the model, and project it into a higher-dimensional vector space with more detailed semantic discrimination through nonlinear transformation, thereby obtaining the deep semantic feature vector corresponding to each subtitle text unit.

[0114] Specifically, the input of the pre-trained high-dimensional semantic mapping model can be the initial semantic feature vector corresponding to the subtitle text unit, and the output can be the deep semantic feature vector corresponding to the same subtitle text unit. The high-dimensional semantic mapping model can include an input projection layer, a nonlinear transformation layer, a residual connection-feature fusion layer, and an output projection layer.

[0115] In the processing flow of a high-dimensional semantic mapping model, the data flow can be executed in the following steps:

[0116] S3011, Input the initial semantic feature vector In the input projection layer, through a linear transformation matrix Features from the original dimension Elevating to higher-dimensional hidden space Output projection vector ;in, This represents the initial semantic feature vector. The dimension of the original semantic feature vector. The dimension representing the higher-dimensional latent space. This represents the linear transformation weight matrix of the input projection layer. This represents the high-dimensional projection vector obtained after the input projection layer transformation.

[0117] S3012, Input projection vector The nonlinear transformation layer introduces nonlinearity through an activation function, and then passes through a fully connected layer. Perform feature refinement and output. ;in, This represents the fully connected weight matrix in the nonlinear transformation layer. The dimension representing the higher-dimensional latent space. This indicates that the result of the linear transformation is processed by a nonlinear activation function. This represents the refined feature vector obtained after processing by the nonlinear transformation layer.

[0118] S3013, The residual connection-feature fusion layer uses the following formula to convert the initial projection vector... With nonlinear transformation output Perform element-wise addition:

[0119] In the formula, This represents the fused feature vector after processing by the residual connection-feature fusion layer. Represents the initial projection vector. This represents the refined feature vector obtained after processing by the nonlinear transformation layer.

[0120] S3014, Through the linear transformation matrix of the output projection layer The fused features Dimensionality reduction back to the original feature dimension Generate deep semantic feature vectors ;in, express, Represents the fused feature vector. This represents the linear transformation weight matrix of the output projection layer. The dimension representing the higher-dimensional latent space. This represents the dimension of the original semantic feature vector.

[0121] S302. Calculate the semantic similarity between the semantic feature vectors of adjacent subtitle text units to obtain a semantic similarity sequence, and perform minimum value detection on the semantic similarity sequence according to the preset mutation detection threshold to obtain the semantic mutation point.

[0122] Among them, the semantic mutation point can be the moment position in the subtitle text stream where adjacent text units undergo significant and discontinuous changes in semantic content.

[0123] For example, a smart terminal can use a semantic similarity calculation algorithm to calculate the similarity score between each pair of adjacent semantic feature vectors in an ordered semantic feature vector sequence in turn, so as to obtain a semantic similarity sequence arranged in chronological order.

[0124] The expression for the similarity score between adjacent semantic feature vectors is:

[0125]

[0126] In the formula, This represents the similarity score between adjacent semantic feature vectors. Indicates the first The semantic feature vector corresponding to each subtitle text unit The dimension of the semantic feature vector. Represents semantic feature vector and dot product, and Let represent the Euclidean norms of the semantic feature vectors, respectively.

[0127] Secondly, the smart terminal can call the preset mutation detection threshold to scan and analyze the semantic similarity sequence. By detecting local minima in the semantic similarity sequence and filtering them in combination with the preset mutation detection threshold, it is determined to be a valid semantic mutation point only when the similarity score of a local minima is lower than the mutation detection threshold.

[0128] S303. The optimal presentation time point corresponding to the semantic mutation point is taken as the time axis segmentation node, and the time span range of each time axis segmentation node is obtained by combining the generation completion time corresponding to the semantic mutation point.

[0129] Among them, the timeline segmentation node can be the key segmentation position determined on the timeline by the smart terminal based on the semantic mutation points detected in the subtitle text stream during the process of constructing the semantic timeline.

[0130] The time span of each time axis segmentation node can be the time period covered by the smart terminal from the completion time of the generation of the subtitle text unit corresponding to that node to the end of its optimal presentation time.

[0131] For example, after obtaining the semantic mutation point and its corresponding optimal presentation time and generation completion time, the smart terminal can mark the optimal presentation time point as a time axis segmentation node for each semantic mutation point. Secondly, the smart terminal can compare the generation completion time corresponding to the semantic mutation point with the optimal presentation time point, calculate the time difference between the two, and use the time difference as the time span interval represented by the segmentation node.

[0132] S304. Linearly concatenate the time axis segmentation nodes and time span intervals in chronological order to obtain the initial time series framework, and fill the semantic feature vectors into the corresponding time span intervals in the initial time series framework to generate the semantic time axis.

[0133] The initial temporal framework can be an intermediate data structure generated by the smart terminal during the construction of the semantic timeline. It is a skeleton-like timeline with clear time segments and sequential relationships, but it has not yet been filled with specific semantic content, providing a basic container for subsequent semantic information integration.

[0134] For example, the smart terminal can collect the time axis segmentation node-time span interval pairing data generated for all semantic mutation points in step S303. Next, the smart terminal can sort the time axis segmentation node-time span interval pairing data according to the optimal presentation time of each segmentation node in ascending order, and connect them sequentially using an ordered data structure such as a linked list or array to generate a time-progressive, segmented linear sequence, i.e., the initial time series framework.

[0135] Furthermore, the smart terminal can traverse each time span interval in the framework, and select all semantic feature vectors falling within that time range from the sequence of subtitle semantic feature vectors generated in step S301 and arranged in chronological order, based on the start and end times corresponding to the interval. Finally, the smart terminal associates and fills all semantic feature vectors as the semantic content carried by that interval into the corresponding interval positions in the framework, thereby transforming the framework with only a temporal structure into a complete and usable semantic timeline object that integrates time segmentation and specific semantic information.

[0136] In this embodiment, the smart terminal constructs a high-precision semantic timeline through localized deep semantic analysis and temporal modeling, providing a structured benchmark that combines semantic coherence and temporal accuracy for the distribution and presentation of multimodal content.

[0137] In one embodiment, the reinforcement learning model includes a temporal policy decision layer, a dynamic compensation calculation layer, and a temporal synchronization correction layer. Based on the semantic anchor set, semantic timeline, and path network delay parameters, the model, combined with a pre-trained reinforcement learning model, adjusts the playback timing of the original multimodal data to obtain perceptually synchronized semantic anchors. This may include the following steps:

[0138] S401. Extract the semantic feature vector and temporal dynamic features corresponding to the semantic anchor point set and semantic time axis, and concatenate the semantic feature vector, temporal dynamic features and path network delay parameters in multiple dimensions to construct a joint state feature vector; wherein, the joint state feature vector is used to characterize the current network transmission environment and content playback status.

[0139] For example, the smart terminal can extract structured features from the locally stored set of semantic anchors and the semantic timeline, respectively: for the set of semantic anchors, the smart terminal can read the semantic code associated with each anchor and its timestamp to form a set of semantic feature vectors; for the semantic timeline, the smart terminal can analyze its temporal structure, extract dynamic features, and form a set of temporal dynamic features. Secondly, the smart terminal can load the path network delay parameters obtained in step S102, and through feature standardization and dimensional alignment operations, unify the above three sets of features into numerical vectors of the same scale, and perform multi-dimensional splicing and fusion according to a preset splicing order to generate a high-dimensional joint state feature vector.

[0140] Optionally, dynamic features may include, but are not limited to, time interval distribution, semantic paragraph duration, and contextual coherence metrics.

[0141] S402. The joint state feature vector is obtained by nonlinearly mapping the joint state feature vector through the deep feature extraction network of the time-series policy decision layer.

[0142] The deep feature extraction network in the temporal policy decision layer can be a neural network module in a reinforcement learning model used for deep abstraction and representation learning of joint state features. The deep feature extraction network can consist of multiple fully connected layers, with nonlinear activation functions and normalization operations between them. The function of the deep feature extraction network is to extract higher-level, more discriminative abstract features from a representation that integrates semantics, temporal sequence, and network state through multi-level nonlinear transformations.

[0143] High-dimensional latent feature vectors can be feature vectors output by deep feature extraction networks with dimensions significantly higher than the input.

[0144] For example, the smart terminal can load a pre-trained reinforcement learning model locally and invoke the temporal policy decision layer of the reinforcement learning model after constructing a joint state feature vector. Next, the smart terminal can input the joint state feature vector into the deep feature extraction network of the temporal policy decision layer. The deep feature extraction network performs nonlinear transformations and feature recombination on the input joint state feature vector through stacked fully connected layers and activation functions, gradually filtering out redundant information and strengthening key related features. Finally, the smart terminal captures and caches the output of the last layer of the deep feature extraction network as a high-dimensional latent feature vector.

[0145] S403. Through a hierarchical reinforcement learning network in the temporal strategy decision layer, a global temporal adjustment vector is obtained by making a global trend decision on the high-dimensional latent feature vector. A local adjustment step size is obtained by calculating the local step size. The global temporal adjustment vector and the local adjustment step size are fused to output a temporal optimization control strategy for the current distribution period. Among them, the high-dimensional latent feature vector is used to characterize the deep correlation features of the original multimodal data in the semantic, temporal and network state dimensions; the global temporal adjustment vector is used to characterize the optimization direction and overall offset trend of the playback timing under the current network transmission environment; and the local adjustment step size is used to quantify the specific delay compensation magnitude of each semantic anchor point.

[0146] The hierarchical reinforcement learning network includes a high-level policy network, an objective decomposition module, and a low-level policy network. The method for constructing the hierarchical reinforcement learning network includes:

[0147] First, the high-level policy network receives the high-dimensional latent feature vector as input, performs non-linear mapping with several fully connected layers and activation functions, and outputs a global temporal adjustment vector with fixed dimensions. Second, the target decomposition module is a learnable linear projection layer that maps the global temporal adjustment vector to a set of sub-target vectors with the same number of semantic anchors. Furthermore, the low-level policy network is a neural network with shared parameters that can receive the local state-target joint vector and output the local adjustment step size corresponding to the anchor. Still further, the entire hierarchical reinforcement learning network is trained end-to-end through reinforcement learning, using temporal synchronization quality and playback smoothness as reward signals to optimize the parameters of each layer.

[0148] For example, the smart terminal can input the high-dimensional latent feature vector into the high-level policy network; the high-level policy network consists of multiple fully connected layers, and its output is a global timing adjustment vector; the global timing adjustment vector is used to characterize the overall optimization direction and trend of the playback timing under the current network transmission environment;

[0149] Furthermore, the hierarchical reinforcement learning network can decompose the global temporal adjustment vector into sub-target vectors corresponding to each semantic anchor point through the target decomposition module; the sub-target vectors are used to guide the local temporal adjustment of each anchor point.

[0150] Furthermore, the hierarchical reinforcement learning network concatenates the sub-target vectors with the state features of the corresponding semantic anchors to generate a local state-target joint vector, which is then input into the low-level policy network. The low-level policy network is a policy function based on a multilayer perceptron, and its output is the local adjustment step size of each semantic anchor, which is used to quantify the specific time delay compensation magnitude of each semantic anchor.

[0151] Finally, the hierarchical reinforcement learning network can integrate the global temporal adjustment vector and the local adjustment step size to generate a temporal optimization control strategy for the current distribution period; the temporal optimization control strategy includes the overall temporal offset trend and the independent adjustment amount of each semantic anchor point.

[0152] S404. The timing optimization control strategy is analyzed by the policy network of the dynamic compensation calculation layer. The policy feature vector corresponding to each semantic anchor point is extracted, the policy feature vector is mapped to a scalar value, and the scalar value is used as the basic adjustment coefficient of each semantic anchor point.

[0153] The policy network can include an input adaptation layer, a policy parsing attention layer, a feature encoding layer, a coefficient mapping layer, and an output association layer. Each layer can be connected in series through a fully connected link to support end-to-end training and inference. The input is a temporal optimization control policy, and the output is the basic adjustment coefficients corresponding to each semantic anchor point. The output dimension is dynamically matched with the size of the semantic anchor point set.

[0154] The policy feature vector can be a decision feature representation of a specific semantic anchor point parsed from the time-series optimization control policy by the intelligent terminal. The policy feature vector encodes the reinforcement learning model based on the current joint state.

[0155] The basic adjustment coefficient for each semantic anchor can be a quantified scalar value generated by the strategy feature vector mapping, used to characterize the baseline magnitude and direction of the compensation adjustment for the initial playback timestamp of the semantic anchor.

[0156] For example, a policy network can perform parsing and mapping tasks with a data flow as follows:

[0157] S4041. The input adaptation layer can obtain the timing optimization control strategy from the timing strategy decision layer, and transform the input timing optimization control strategy into a fixed-dimensional intermediate representation through a learnable linear projection to obtain the adapted timing optimization control strategy.

[0158] S4042. The policy parsing attention layer can adopt a multi-head self-attention mechanism after obtaining the adapted temporal optimization control policy. Based on the position of each semantic anchor in the original sequence, it calculates the attention weight of each anchor to the entire policy representation, thereby outputting the initial policy features of each anchor.

[0159] S4043. The feature encoding layer can be composed of several fully connected layers and nonlinear activation functions stacked together. The feature encoding layer can refine, denoise and enhance features through multi-layer nonlinear transformation after obtaining the initial policy features of each anchor point, and finally output a high-dimensional, dense policy feature vector.

[0160] S4044. The coefficient mapping layer can be composed of a lightweight multilayer perceptron. The coefficient mapping layer can take the policy feature vector of each semantic anchor point as input and compress and map the policy feature vector of each semantic anchor point into a one-dimensional real value scalar.

[0161] S4045. The output association layer is not a parameter layer in the traditional sense, but a structured output module. The output association layer is used to associate and store the scalar sequence output by the coefficient mapping layer with the corresponding anchor in the semantic anchor set based on the unique identifier of the semantic anchor.

[0162] S405. Based on the semantic feature vectors corresponding to each semantic anchor point, the importance of each semantic anchor point is quantitatively evaluated to obtain the semantic importance weight of each semantic anchor point.

[0163] For example, the smart terminal can load the semantic feature vectors corresponding to each semantic anchor point and analyze them through preset evaluation rules. Specifically, the smart terminal can calculate the norm of each semantic feature vector to initially reflect its information richness, or calculate its distance to the average vector of the entire set of semantic anchor points to measure its semantic uniqueness. Secondly, the smart terminal can assign higher baseline importance values ​​to semantic anchor points that are associated with specific visual objects or occupy structural positions in the semantic timeline, based on preset evaluation rules. Finally, by combining the above quantitative analysis and rule application, the smart terminal can calculate a normalized scalar value for each semantic anchor point as the corresponding semantic importance weight.

[0164] S406. Substitute the basic adjustment coefficient, semantic importance weight, and path network delay parameters into the preset formula to calculate the dynamic delay compensation amount for each semantic anchor point.

[0165] The dynamic latency compensation amount can be the amount of adjustment suggested by the smart terminal for the initial playback timestamp of a specific semantic anchor point, whether to increase or decrease.

[0166] For example, the smart terminal can obtain the basic adjustment coefficients, semantic importance weights, and path network latency parameters for each semantic anchor point, and then substitute these three sets of data into a preset mathematical formula for calculation. Specifically, the smart terminal first extracts the predicted average latency and predicted network latency jitter value of the target distribution path from the path network latency parameters, and obtains the initial playback timestamps of adjacent anchor points from the semantic anchor point set. Secondly, the smart terminal can input variables such as the basic adjustment coefficients, semantic importance weights, the ratio of latency jitter to average latency, and the time interval between adjacent anchor points corresponding to each semantic anchor point into a specific formula including a normalization factor and a saturation function for fusion calculation. Finally, for each semantic anchor point, the smart terminal outputs a specific dynamic latency compensation amount.

[0167] S407. The initial playback timestamp and the corresponding dynamic delay compensation amount are superimposed through the timing synchronization correction layer to obtain the preliminary compensation timestamp.

[0168] The initial compensation timestamp can be an intermediate time point obtained by arithmetically superimposing the initial playback timestamp of the semantic anchor point with the calculated dynamic delay compensation amount during the timing synchronization correction process of the smart terminal, without subsequent smoothness optimization processing.

[0169] For example, the smart terminal can sequentially traverse each semantic anchor in the semantic anchor set, read its initial playback timestamp from the attributes of the semantic anchor, and obtain the corresponding dynamic latency compensation amount from the output of step S406. Next, the smart terminal can perform an arithmetic summation operation on the timestamps, that is, directly add the initial playback timestamp to the dynamic latency compensation amount to calculate the preliminary compensation timestamp of the semantic anchor. Finally, the smart terminal can cache or record this calculation result as a temporary attribute of the anchor, generating an intermediate data set containing the preliminary adjustment results of all semantic anchors.

[0170] S408. Based on the semantic timeline and the temporal correlation between each semantic anchor point determined based on the semantic feature vector, perform contextual temporal smoothness detection on the preliminary compensation timestamp and generate temporal smoothness constraints.

[0171] Among them, the temporal smoothness constraint can be a set of quantization rules or inequalities generated by the smart terminal after detecting the coherence of the initial compensation timestamp sequence. The temporal smoothness constraint can be used to characterize the allowable range and rate of change of timestamp adjustment between adjacent semantic anchors.

[0172] For example, the smart terminal can load a semantic timeline and the semantic feature vectors corresponding to each semantic anchor point, analyze the semantic association strength between anchor points and their paragraph affiliation on the timeline, thereby constructing a graph structure reflecting their inherent temporal dependencies. Secondly, the smart terminal can analyze the preliminary compensated timestamp sequence, calculate the time interval after compensation for adjacent anchor points, and compare it with the original time interval and semantic association strength to detect any abrupt changes in time intervals or temporal inversions that violate semantic logic. Subsequently, based on the detection results, the smart terminal generates specific constraints for anchor pairs or anchor sequences that pose a risk. Finally, the smart terminal formalizes the constraints into mathematical inequalities or sets of rules.

[0173] S409. Based on the temporal smoothness constraint, the initial compensation timestamp is fine-tuned and corrected using an iterative optimization algorithm to obtain the final compensation timestamp.

[0174] The final compensation timestamp can be the target playback time point determined by the smart terminal for each semantic anchor point after completing the fine-tuning correction based on temporal smoothness constraints, which has eliminated temporal jumps and meets the requirements of contextual coherence.

[0175] For example, the smart terminal can load the initial compensation timestamp set and the temporal smoothness constraints generated in step S408, and construct it as a constrained nonlinear optimization problem with temporal smoothness as the objective and the constraints as the limitations. Secondly, the smart terminal can employ an iterative optimization algorithm suitable for handling such constrained optimization problems, using the initial compensation timestamps as the initial solution, and continuously adjusting the timestamps of each semantic anchor point during the iteration process to minimize the variance of the compensation time interval between adjacent semantic anchor points, and ensuring that all timestamp adjustments satisfy the temporal smoothness constraints. Finally, when the optimization process converges, the smart terminal determines the set of timestamps output by the algorithm that satisfies all constraints as the final compensation timestamps for each semantic anchor point.

[0176] S410. Update the timestamp attribute of the corresponding semantic anchor point according to the final compensation timestamp to obtain the corrected semantic anchor point, and use the corrected semantic anchor point as the semantic anchor point for perception synchronization.

[0177] For example, the smart terminal can sequentially traverse the set of semantic anchors and match the identifier of each semantic anchor with the final compensation timestamp calculated in step S409. Next, the smart terminal can call its data update interface to replace the original initial playback timestamp attribute value of the semantic anchor with the corresponding final compensation timestamp, while retaining all other attributes of the anchor unchanged, thereby generating a new anchor data unit with updated timestamp attributes.

[0178] Furthermore, after performing this update operation on all anchor points, the smart terminal can reorganize all newly generated data units according to the original structure, constructing a completely new structured set with finalized timestamp information—that is, the corrected semantic anchor point set. Finally, the smart terminal outputs the semantic anchor points that are synchronized with perception as the semantic anchor points for perception synchronization.

[0179] In this embodiment, the smart terminal uses a pre-trained hierarchical reinforcement learning model to achieve online, adaptive, and dynamic optimization of the playback sequence of multimodal content, effectively coordinating semantic coherence and network fluctuations.

[0180] In one embodiment, the basic adjustment coefficient, semantic importance weight, and path network delay parameters are substituted into a preset formula for calculation to obtain the dynamic delay compensation amount for each semantic anchor point, including:

[0181] The expression for dynamic delay compensation is:

[0182]

[0183] in, The first semantic anchor in the set Index of semantic anchor points Indicates that for the first The dynamic latency compensation amount calculated for each semantic anchor point. The policy network of the reinforcement learning model is represented as the first... The basic adjustment coefficients for the output of each semantic anchor point Indicates the first The semantic importance weight of each semantic anchor point Indicates the first term used by the target terminal Predicting network latency jitter for each distribution path. Indicates the target terminal using the first The predicted average latency of each distribution path, and They represent the first The and the first The initial playback timestamp of each semantic anchor point Represents a timescale reference constant. This represents the fidelity scaling factor, used to control the sensitivity of the compensation amount to the original interval between anchor points. This represents the hyperbolic tangent function, used as a saturation function.

[0184] In this embodiment, the intelligent terminal introduces a mathematical formula that integrates reinforcement learning decision-making, content semantic importance, and network latency characteristics to achieve quantitative calculation of dynamic latency compensation for each semantic anchor point, thereby ensuring the accuracy, adaptability, and robustness of timing adjustment at the mathematical level.

[0185] In one embodiment, the method further includes:

[0186] S501. Obtain the local model update data uploaded by each target terminal, and apply random noise perturbation that meets the differential privacy requirements to the local model update data to obtain noisy model update data.

[0187] The local model update data can be the model parameter gradient information calculated based on local experience data collected in the actual distribution scenario during the process of each target terminal running the pre-trained reinforcement learning model locally to make playback timing decisions.

[0188] Optionally, the real-time data may include, but is not limited to, actual network status, user interaction feedback, and playback performance metrics. Model parameter update information may include, but is not limited to, gradient vectors, weight changes, or the gradient of the loss function.

[0189] Noisy model update data can be data obtained by a smart terminal after receiving local model update data and then subjecting the local model update data to random noise perturbation processing in accordance with differential privacy mechanism to meet privacy protection requirements.

[0190] For example, the smart terminal can perform format verification and normalization on the local model update data uploaded by various target terminals. Secondly, based on preset differential privacy budget parameters, the smart terminal calls the corresponding noise generation algorithm to generate random noise values ​​conforming to a distribution for each dimension of the update data, and then superimposes the random noise values ​​with the original update data to obtain noisy model update data.

[0191] S502. Perform secure aggregation calculations on the noisy model update data to generate a global model update gradient.

[0192] Among them, the global model update gradient can be the comprehensive parameter adjustment direction generated by the smart terminal after performing secure aggregation calculation on all received noisy model update data, which is used to uniformly update the pre-trained reinforcement learning model.

[0193] For example, the smart terminal can acquire noisy model update data from various target terminals and perform format standardization and validity verification on the noisy model update data. Subsequently, the smart terminal can assign aggregation weights to the update data of each target terminal based on the amount of data contributed or a preset reputation weight. Next, without decrypting or restoring any original update information from any terminal, the smart terminal directly performs a weighted average operation on all noisy model update data. The result of the above weighted average calculation constitutes a global model update gradient that integrates multi-terminal experience and satisfies differential privacy requirements. The global model update gradient is used to characterize the overall direction and magnitude of optimizing the parameters of the pre-trained reinforcement learning model based on the current multi-terminal distributed experience.

[0194] Optionally, the weighted average operation follows a federated average algorithm framework. The aggregation weights can be dynamically calculated based on the amount of experience data reported by each terminal to ensure the universality and stability of the update direction.

[0195] S503. Iteratively update the pre-trained reinforcement learning model based on the global model update gradient to update the reinforcement learning model.

[0196] For example, the smart terminal can load the currently pre-trained reinforcement learning model from secure storage and parse the global model update gradient generated in step S502. Next, the smart terminal can use stochastic gradient descent or a variant thereof, taking the global model update gradient as input, to iteratively update the parameters of the reinforcement learning model. Specifically, based on a preset learning rate, the current model parameters are subtracted from the product of the global model update gradient and the learning rate to obtain the new model parameters. This process strictly follows the rules of the secure aggregation framework, using only the aggregated gradient and not involving any of the terminal's original data. After the update is complete, the smart terminal can verify the performance of the updated model based on the retained validation dataset. If the verification passes, the smart terminal can then overwrite the original model with the new parameters.

[0197] In this embodiment, the intelligent terminal introduces differential privacy mechanism and secure aggregation computation during the model update process, thereby enabling the collaborative evolution and continuous optimization of the reinforcement learning model by leveraging the distributed experience of multiple terminals while strictly protecting user privacy.

[0198] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0199] Based on the same inventive concept, this application also provides an intelligent collaborative distribution system for broadcast television multimodal content as described above. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the intelligent collaborative distribution system for broadcast television multimodal content provided below can be found in the limitations of the intelligent collaborative distribution method for broadcast television multimodal content described above, and will not be repeated here.

[0200] In one exemplary embodiment, such as Figure 2 As shown, a broadcast television multimodal content intelligent collaborative distribution system 600 is provided, comprising:

[0201] Alignment analysis module 601 is used to acquire the original multimodal data of broadcast television programs and perform cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline; wherein, the original multimodal data includes video stream, audio stream and native subtitle stream;

[0202] The delay parameter acquisition module 602 is used to acquire historical transmission data of heterogeneous distribution paths, and perform time series analysis on the historical transmission data to obtain the path network delay parameters of each heterogeneous distribution path in the future distribution period.

[0203] The timing synchronization module 603 is used to adjust the playback timing of the original multimodal data based on the semantic anchor point set, semantic timing axis and path network delay parameters, combined with a pre-trained reinforcement learning model, to obtain the semantic anchor points for perceptual synchronization.

[0204] The adaptation task generation module 604 is used to obtain real-time resource status data input by the target terminal, and perform semantic granularity division and adaptation requirement parsing on the original multimodal data in sequence according to the semantic anchor point and semantic time axis of perception synchronization, generate content adaptation tasks corresponding to the original multimodal data, and decouple the content adaptation tasks into local executable tasks and remote collaborative tasks according to the resource status data.

[0205] The instruction generation module 605 is used to perform collaborative scheduling optimization analysis on local executable tasks and remote collaborative tasks, generate corresponding task scheduling instructions, and send the task scheduling instructions to the target terminal; wherein, the task scheduling instructions are used to instruct the target terminal to execute local executable tasks and initiate collaborative processing requests for remote collaborative tasks to edge servers or cloud servers.

[0206] In one embodiment, the alignment analysis module includes:

[0207] The multimodal feature extraction unit is used to extract features from the video stream, audio stream, and native subtitle stream to obtain video spatiotemporal feature sequences, audio acoustic feature sequences, and subtitle semantic feature sequences.

[0208] The cross-modal attention interaction unit is used to perform cross-modal semantic interaction calculations on video spatiotemporal feature sequences, audio acoustic feature sequences, and subtitle semantic feature sequences based on a cross-modal multi-head attention mechanism, generating audio-video semantic association weight matrices and video-subtitle semantic association weight matrices. The video semantic association weight matrix represents the semantic association strength between the video spatiotemporal feature sequences and the audio acoustic feature sequences; the video-subtitle semantic association weight matrix represents the semantic association strength between the video spatiotemporal feature sequences and the subtitle semantic feature sequences.

[0209] The association strength aggregation and synchronization point detection unit is used to perform time-dimensional association strength aggregation calculation on the audio-video semantic association weight matrix and the video subtitle semantic association weight matrix respectively, to obtain the association strength sequence corresponding to each time point, and to select the time points that simultaneously meet the peak conditions in the association strength sequence as synchronization time points according to the preset peak threshold.

[0210] The semantic anchor construction unit is used to construct a triplet by combining the synchronization time point with the corresponding video frame index and subtitle text index, and to use the structured set composed of each triplet as the semantic anchor set.

[0211] The speech recognition and timing correction unit is used to perform real-time speech recognition on the audio acoustic feature sequence and audio stream to obtain the generation completion time of the subtitle text unit in the original subtitle stream. Combined with the time correspondence in the semantic anchor set, the generation completion time is offset and corrected based on semantic coherence to obtain the optimal presentation time of each subtitle text unit.

[0212] The semantic timeline generation unit is used to construct an ordered data structure based on the optimal presentation time and generation completion time of each subtitle text unit, and output it as a semantic timeline.

[0213] In one embodiment, the semantic timeline generation unit includes:

[0214] The semantic feature vector construction sub-unit is used to perform vectorized encoding of semantic features to obtain the initial semantic feature vector, and then the initial semantic feature vector is mapped to the preset high-dimensional semantic space to obtain the semantic feature vector corresponding to each subtitle text unit;

[0215] The mutation point generation sub-unit is used to calculate the semantic similarity between the semantic feature vectors of adjacent subtitle text units, obtain a semantic similarity sequence, and perform minimum value detection on the semantic similarity sequence according to the preset mutation detection threshold to obtain semantic mutation points;

[0216] The time span interval generation sub-unit is used to take the best presentation time point corresponding to the semantic mutation point as the time axis segmentation node, and combine it with the generation completion time corresponding to the semantic mutation point to obtain the time span interval of each time axis segmentation node.

[0217] The semantic timeline construction subunit is used to linearly concatenate the timeline segmentation nodes and time span intervals in chronological order to obtain the initial timeline framework, and then fill the corresponding time span intervals in the initial timeline framework with semantic feature vectors to generate the semantic timeline.

[0218] In one embodiment, the timing synchronization module includes:

[0219] The joint state feature vector construction unit is used to extract the semantic feature vector and temporal dynamic features corresponding to the semantic anchor point set and semantic time axis, and to concatenate the semantic feature vector, temporal dynamic features and path network delay parameters in multiple dimensions to construct the joint state feature vector; wherein, the joint state feature vector is used to characterize the current network transmission environment and content playback status;

[0220] The high-dimensional latent feature vector generation unit is used to perform nonlinear mapping on the joint state feature vector through the deep feature extraction network of the temporal policy decision layer to obtain the high-dimensional latent feature vector of the joint state feature vector.

[0221] The optimization strategy generation unit is used to obtain a global temporal adjustment vector by performing global trend decision on the high-dimensional latent feature vector through a hierarchical reinforcement learning network of the temporal strategy decision layer, and to obtain a local adjustment step size by calculating the local step size. The global temporal adjustment vector and the local adjustment step size are fused to output a temporal optimization control strategy for the current distribution period. Among them, the high-dimensional latent feature vector is used to represent the deep correlation features of the original multimodal data in the semantic, temporal and network state dimensions; the global temporal adjustment vector is used to represent the optimization direction and overall offset trend of the playback timing under the current network transmission environment; and the local adjustment step size is used to quantify the specific delay compensation magnitude of each semantic anchor point.

[0222] The policy parsing and coefficient mapping unit is used to perform feature parsing on the time-series optimization control policy through the policy network of the dynamic compensation calculation layer, extract the policy feature vector corresponding to each semantic anchor point, map the policy feature vector to a scalar value, and use the scalar value as the basic adjustment coefficient of each semantic anchor point.

[0223] The semantic importance assessment unit is used to quantitatively assess the importance of each semantic anchor point based on the semantic feature vector corresponding to each semantic anchor point, and obtain the semantic importance weight of each semantic anchor point.

[0224] The dynamic delay compensation calculation unit is used to substitute the basic adjustment coefficient, semantic importance weight and path network delay parameters into the preset formula to calculate the dynamic delay compensation amount of each semantic anchor point.

[0225] The preliminary compensation timestamp generation unit is used to superimpose the initial playback timestamp and the corresponding dynamic delay compensation amount through the timing synchronization correction layer to obtain the preliminary compensation timestamp;

[0226] The temporal smoothness constraint generation unit is used to perform contextual temporal smoothness detection on the preliminary compensation timestamp based on the semantic temporal axis and the temporal correlation between each semantic anchor point determined based on the semantic feature vector, and generate temporal smoothness constraint conditions.

[0227] The timing fine-tuning optimization unit is used to fine-tune and correct the initial compensation timestamp based on the timing smoothness constraint and using an iterative optimization algorithm to obtain the final compensation timestamp.

[0228] The perception synchronization anchor point generation unit is used to update the timestamp attribute of the corresponding semantic anchor point according to the final compensation timestamp, obtain the corrected semantic anchor point, and use the corrected semantic anchor point as the semantic anchor point for perception synchronization.

[0229] In one embodiment, the dynamic delay compensation calculation unit includes:

[0230] The expression for dynamic delay compensation is:

[0231]

[0232] in, The first semantic anchor in the set Index of semantic anchor points Indicates that for the first The dynamic latency compensation amount calculated for each semantic anchor point. The policy network of the reinforcement learning model is represented as the first... The basic adjustment coefficients for the output of each semantic anchor point Indicates the first The semantic importance weight of each semantic anchor point Indicates the first term used by the target terminal Predicting network latency jitter for each distribution path. Indicates the target terminal using the first The predicted average latency of each distribution path, and They represent the first The and the first The initial playback timestamp of each semantic anchor point Represents a timescale reference constant. This represents the fidelity scaling factor, used to control the sensitivity of the compensation amount to the original interval between anchor points. This represents the hyperbolic tangent function, used as a saturation function.

[0233] In one embodiment, the system further includes:

[0234] The differential privacy perturbation module is used to obtain local model update data uploaded by each target terminal and apply random noise perturbation that meets the differential privacy requirements to the local model update data to obtain noisy model update data.

[0235] The secure aggregation gradient generation module is used to perform secure aggregation calculations on noisy model update data to generate global model update gradients.

[0236] The model federated iterative update module is used to iteratively update the pre-trained reinforcement learning model based on the global model update gradient, so as to update the reinforcement learning model.

[0237] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described intelligent collaborative distribution method for multimodal broadcast television content.

[0238] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0239] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0240] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for intelligent collaborative distribution of multimodal broadcast television content, characterized in that, The method includes: S101. Obtain the original multimodal data of broadcast television programs, and perform cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline; wherein, the original multimodal data includes video stream, audio stream and native subtitle stream; S102. Obtain historical transmission data of heterogeneous distribution paths, and perform time series analysis on the historical transmission data to obtain the path network delay parameters of each heterogeneous distribution path in the future distribution period. S103. Based on the set of semantic anchor points, the semantic time axis, and the path network delay parameters, the playback timing of the original multimodal data is adjusted using a pre-trained reinforcement learning model to obtain semantic anchor points for perceptual synchronization. S104. Obtain real-time resource status data input from the target terminal. Based on the semantic anchor point and semantic time axis of the perception synchronization, perform semantic granularity division and adaptation requirement parsing on the original multimodal data in sequence, generate content adaptation tasks corresponding to the original multimodal data, and decouple the content adaptation tasks into locally executable tasks and remote collaborative tasks based on the resource status data. S105. Perform collaborative scheduling optimization analysis on the locally executable task and the remote collaborative task, generate corresponding task scheduling instructions, and send the task scheduling instructions to the target terminal; wherein, the task scheduling instructions are used to instruct the target terminal to execute the locally executable task and initiate a collaborative processing request for the remote collaborative task to the edge server or cloud server.

2. The method according to claim 1, characterized in that, The process involves performing cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic temporal axis, including: S201. Perform feature extraction on the video stream, the audio stream, and the native subtitle stream to obtain a video spatiotemporal feature sequence, an audio acoustic feature sequence, and a subtitle semantic feature sequence; S202. Based on a cross-modal multi-head attention mechanism, perform cross-modal semantic interaction calculations on the video spatiotemporal feature sequence, the audio acoustic feature sequence, and the subtitle semantic feature sequence to generate an audio-video semantic association weight matrix and a video subtitle semantic association weight matrix; wherein, the video semantic association weight matrix is ​​used to characterize the semantic association strength between the video spatiotemporal feature sequence and the audio acoustic feature sequence; the video subtitle semantic association weight matrix is ​​used to characterize the semantic association strength between the video spatiotemporal feature sequence and the subtitle semantic feature sequence; S203. Perform time-dimensional aggregation calculations on the audio-visual semantic association weight matrix and the video subtitle semantic association weight matrix respectively to obtain the association strength sequence corresponding to each time point, and select the time points that simultaneously meet the peak conditions in the association strength sequence as synchronization time points according to the preset peak threshold. S204. Construct a triplet by combining the synchronization time point with the corresponding video frame index and subtitle text index, and use the structured set composed of each triplet as the semantic anchor set. S205. Perform real-time speech recognition on the audio acoustic feature sequence and the audio stream to obtain the generation completion time of the subtitle text unit in the native subtitle stream, and combine the time correspondence in the semantic anchor set to perform offset correction on the generation completion time based on semantic coherence to obtain the optimal presentation time point of each subtitle text unit. S206. Based on the optimal presentation time and the generation completion time of each subtitle text unit, construct an ordered data structure in chronological order and output it as the semantic timeline.

3. The method according to claim 2, characterized in that, The optimal presentation time and generation completion time of each subtitle text unit are constructed into an ordered data structure in chronological order, and the output is the semantic timeline, including: S301. The semantic features are vectorized and encoded to obtain an initial semantic feature vector, and the initial semantic feature vector is mapped to a preset high-dimensional semantic space to obtain the semantic feature vector corresponding to each subtitle text unit. S302. Calculate the semantic similarity between the semantic feature vectors of adjacent subtitle text units to obtain a semantic similarity sequence, and perform minimum value detection on the semantic similarity sequence according to a preset mutation detection threshold to obtain semantic mutation points; S303. The optimal presentation time point corresponding to the semantic mutation point is taken as the time axis segmentation node, and the time span range of each time axis segmentation node is obtained by combining the generation completion time corresponding to the semantic mutation point. S304. The time axis segmentation nodes and the time span intervals are linearly concatenated in chronological order to obtain an initial time series framework, and the semantic feature vectors are filled into the corresponding time span intervals in the initial time series framework to generate the semantic time axis.

4. The method according to claim 1, characterized in that, The reinforcement learning model includes a temporal policy decision layer, a dynamic compensation calculation layer, and a temporal synchronization correction layer. Based on the semantic anchor point set, the semantic timeline, and the path network delay parameters, the pre-trained reinforcement learning model adjusts the playback timing of the original multimodal data to obtain perceptually synchronized semantic anchor points, including: S401. Extract the semantic feature vector and temporal dynamic features corresponding to the semantic anchor point set and the semantic time axis, and concatenate the semantic feature vector, the temporal dynamic features and the path network delay parameters in a multi-dimensional manner to construct a joint state feature vector; wherein, the joint state feature vector is used to characterize the current network transmission environment and content playback status; S402. The joint state feature vector is nonlinearly mapped through the deep feature extraction network of the time-series policy decision layer to obtain the high-dimensional latent feature vector of the joint state feature vector. S403. Through the hierarchical reinforcement learning network of the temporal strategy decision layer, a global trend decision is made on the high-dimensional latent feature vector to obtain a global temporal adjustment vector, and a local adjustment step size is calculated through local step size calculation. The global temporal adjustment vector and the local adjustment step size are fused to output a temporal optimization control strategy for the current distribution period. The high-dimensional latent feature vector is used to characterize the deep correlation features of the original multimodal data in semantic, temporal, and network state dimensions; the global temporal adjustment vector is used to characterize the optimization direction and overall offset trend of the playback timing under the current network transmission environment; and the local adjustment step size is used to quantify the specific latency compensation magnitude of each semantic anchor point. S404. The timing optimization control strategy is analyzed by the strategy network of the dynamic compensation calculation layer, the strategy feature vector corresponding to each semantic anchor point is extracted, the strategy feature vector is mapped to a scalar value, and the scalar value is used as the basic adjustment coefficient of each semantic anchor point. S405. Based on the semantic feature vectors corresponding to each semantic anchor point, the importance of each semantic anchor point is quantitatively evaluated to obtain the semantic importance weight of each semantic anchor point. S406. Substitute the basic adjustment coefficient, the semantic importance weight, and the path network delay parameter into the preset formula to calculate the dynamic delay compensation amount for each semantic anchor point. S407. The initial playback timestamp and the corresponding dynamic delay compensation amount are superimposed through the timing synchronization correction layer to obtain the preliminary compensation timestamp; S408. Based on the semantic timeline and the temporal correlation between each semantic anchor point determined based on the semantic feature vector, perform contextual temporal smoothness detection on the preliminary compensation timestamp and generate temporal smoothness constraints. S409. Based on the time series smoothness constraint, the preliminary compensation timestamp is fine-tuned and corrected using an iterative optimization algorithm to obtain the final compensation timestamp. S410. Update the timestamp attribute of the corresponding semantic anchor point according to the final compensation timestamp to obtain the corrected semantic anchor point, and use the corrected semantic anchor point as the semantic anchor point for the perception synchronization.

5. The method according to claim 4, characterized in that, The step of substituting the basic adjustment coefficient, the semantic importance weight, and the path network delay parameter into a preset formula to calculate the dynamic delay compensation amount for each semantic anchor point includes: The expression for the dynamic delay compensation amount is: in, The set of semantic anchors represents the first... Index of semantic anchor points Indicates that for the first The dynamic latency compensation amount calculated for each semantic anchor point. The policy network of the reinforcement learning model is the first... The basic adjustment coefficients for the output of each semantic anchor point Indicates the first The semantic importance weight of each semantic anchor point The target terminal uses the first Predicting network latency jitter for each distribution path. Indicates the target terminal using the first The predicted average latency of each distribution path, and They represent the first The and the first The initial playback timestamp of each semantic anchor point Represents a timescale reference constant. This represents the fidelity scaling factor, used to control the sensitivity of the compensation amount to the original interval between anchor points. This represents the hyperbolic tangent function, used as a saturation function.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: S501. Obtain the local model update data uploaded by each of the target terminals, and apply random noise perturbation that meets the differential privacy requirements to the local model update data to obtain noisy model update data. S502. Perform secure aggregation calculation on the noisy model update data to generate a global model update gradient; S503. The pre-trained reinforcement learning model is iteratively updated according to the global model update gradient to update the reinforcement learning model.

7. A smart collaborative distribution system for multimodal broadcast television content, characterized in that, The system includes: The alignment analysis module is used to acquire the original multimodal data of broadcast television programs and perform cross-modal semantic alignment analysis on the original multimodal data to obtain a set of semantic anchor points and a semantic timeline; wherein, the original multimodal data includes video stream, audio stream and native subtitle stream; The delay parameter acquisition module is used to acquire historical transmission data of heterogeneous distribution paths, and perform time series analysis on the historical transmission data to obtain the path network delay parameters of each heterogeneous distribution path in the future distribution period. The timing synchronization module is used to adjust the playback timing of the original multimodal data based on the semantic anchor point set, the semantic timing axis and the path network delay parameters, combined with a pre-trained reinforcement learning model, to obtain the semantic anchor points for perceptual synchronization. The adaptation task generation module is used to obtain real-time resource status data input by the target terminal, and according to the semantic anchor point and semantic time axis of the perception synchronization, perform semantic granularity division and adaptation requirement parsing on the original multimodal data in sequence, generate content adaptation tasks corresponding to the original multimodal data, and decouple the content adaptation tasks into locally executable tasks and remote collaborative tasks according to the resource status data. The instruction generation module is used to perform collaborative scheduling optimization analysis on the locally executable task and the remote collaborative task, generate corresponding task scheduling instructions, and send the task scheduling instructions to the target terminal; wherein, the task scheduling instructions are used to instruct the target terminal to execute the locally executable task and initiate a collaborative processing request for the remote collaborative task to the edge server or cloud server.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images