Network audio and video playing control method and system based on large model

Abstract

The invention discloses a network audio and video playing control method and system based on a large model, and relates to the technical field of audio and video playing control, and the method comprises the steps: collecting three types of data, namely network states, audio and video content features and user behaviors, and carrying out the preprocessing; large model training data are generated based on the space-time slicing technology, a three-level progressive training system is constructed for large model training, and large model parameters are optimized through reinforcement learning; inputting the preprocessed real-time collected data into the large model, predicting a future optimal playing parameter, and combining smooth transition, content perception audio adjustment, bidirectional timestamp synchronization and Kalman filtering feedback to realize initial dynamic adjustment of audio and video playing so as to form a basic playing parameter; and by defining a network stability factor and a content complexity factor, dynamically adjusting a fusion weight of the large model and a code rate adaptive algorithm, and fusing results of the large model and the code rate adaptive algorithm to calculate a final playing parameter. The method and the device are applied to various network audio and video playing scenes.

Description

Technical Field

[0001] This invention relates to the field of audio and video playback control technology, specifically a network audio and video playback control method and system based on a large model. Background Technology

[0002] With the widespread application of online audio and video in scenarios such as online education, telemedicine, and interactive entertainment, users have placed extremely high demands on the real-time performance and smoothness of audio and video playback.

[0003] Traditional network audio and video playback control methods largely rely on adjusting playback parameters based on arrival time deviation calculations. While this can guarantee playback quality to some extent, it has significant limitations in complex network environments (such as weak networks and network jitter) and under the challenge of diverse audio and video content (high bitrate video, multi-channel audio). When network congestion or packet loss occurs, traditional methods cannot quickly and accurately adjust parameters such as audio and video bitrate and playback frame rate, easily leading to audio-visual desynchronization, stuttering, or even playback interruption. Furthermore, they struggle to achieve optimal playback when handling audio and video with different audio sampling rates and encoding formats, failing to meet users' demands for a high-quality audio and video playback experience.

[0004] Large-scale models, with their powerful data processing and analysis capabilities, have achieved remarkable results in fields such as natural language processing and image recognition. Introducing them into the field of network audio and video playback control is expected to break through traditional technical bottlenecks and achieve more intelligent and efficient playback control. Summary of the Invention

[0005] This invention addresses the needs and shortcomings of current technological development by providing a network audio and video playback control method and system based on a large model, effectively solving the problems of traditional playback control methods and bringing users a better video playback experience.

[0006] Firstly, the present invention provides a network audio and video playback control method based on a large model, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0007] A network audio and video playback control method based on a large model includes the following steps:

[0008] S1. Collect network status data, audio and video content feature data, and user behavior data, and perform layered cleaning and normalization preprocessing on the collected data.

[0009] S2. Generate large model training data based on spatiotemporal slicing technology, construct a three-level progressive training system including feature extraction layer, fusion decision layer and reinforcement learning optimization layer for large model training, optimize parameters through reinforcement learning and improve the performance of large model by adopting anti-overfitting and convergence optimization techniques.

[0010] S3. Input the preprocessed real-time acquisition data into the trained large model. The large model predicts the optimal playback parameters in the future. Combined with smooth transition, content-aware audio adjustment, bidirectional timestamp synchronization and Kalman filter feedback, the initial dynamic adjustment of audio and video playback is realized, forming basic playback parameters.

[0011] S4. By defining network stability factors and content complexity factors, the fusion weights of the large model and the bitrate adaptive algorithm are dynamically adjusted, and the results of the two are fused and calculated as the final playback parameters.

[0012] Optionally, step S1 specifically includes:

[0013] S1.1 Data Acquisition: Network status data is collected through a combination of active detection and passive monitoring, including round-trip latency, network throughput, and packet loss rate; audio and video content feature data is extracted during the decoding stage using FFmpeg, including video frame rate, resolution, bitrate fluctuations, and audio encoding format and sampling rate; user behavior data is captured in real time by embedding tracking code in the player, and device performance parameters are collected in conjunction with the device information acquisition interface.

[0014] S1.2 Data Preprocessing: The collected raw data is first filtered for outliers to remove invalid data that exceeds the preset range; then, for time series network data, a sliding window algorithm is used for smoothing to eliminate interference from instantaneous network noise; finally, through Min-Max standardization, all types of data are uniformly mapped to the [0, 1] interval to ensure that the data quality meets the input requirements of large models.

[0015] Further optionally, step S2 specifically includes:

[0016] S2.1. Use spatiotemporal slicing technology to construct training data for a large model. Divide the continuous data stream into 5-second time windows. Each window contains network state sequences, audio and video feature sequences and corresponding playback parameter labels. Finally, generate a structured training dataset and divide it into training set and validation set according to a preset ratio.

[0017] S2.2 Construct a three-level progressive training system comprising a feature extraction layer, a fusion decision layer, and a reinforcement learning optimization layer. The feature extraction layer uses a pre-trained Transformer model to process pre-processed temporal network data to capture long-term dependencies, while simultaneously using an LSTM network to extract temporal patterns from the pre-processed audio and video features. The fusion decision layer inputs the multimodal features output from the feature extraction layer into the MLP, automatically assigns feature weights using an attention mechanism, learns the optimal playback parameter mapping relationship for different scenarios based on the training set, and evaluates the generalization ability of the large model using a validation set. The reinforcement learning optimization layer, based on the DDPG algorithm, constructs a reward function using playback smoothness, latency, and audio-visual synchronization rate, iteratively optimizes the parameters of the feature extraction layer and the fusion decision layer based on the training set, and monitors the optimization effect using a validation set, achieving online optimization of the large model parameters.

[0018] S2.3 During the training of the large model, the parameters are learned by using the training set as input, the performance of the large model is evaluated by the validation set, batch normalization and Dropout techniques are used to prevent overfitting, and the learning rate decay strategy of cosine annealing is used to improve convergence stability. When the evaluation index of the validation set reaches the preset threshold, the training and optimization of the large model is completed.

[0019] Further optionally, step S3 specifically includes:

[0020] S3.1 Prediction stage: The preprocessed real-time collected data is input into the trained large model, and the large model outputs the optimal combination of playback parameters for the next 5 seconds;

[0021] S3.2 Parameter Adjustment Stage: A smooth transition algorithm is adopted to avoid visual jitter caused by parameter abrupt changes. The specific operation is as follows: When it is necessary to reduce the bit rate, the video frame rate is gradually reduced by a preset step size, and then the resolution is adjusted in stages. When the network status monitoring meets the preset conditions, it is determined that the network has recovered. At this time, the following operations are performed: First, the step size is adjusted in reverse according to the original resolution and gradually increased to the original set value, and then the step size is adjusted in reverse according to the original frame rate and increased to the initial level. At the same time, a content-aware volume adjustment algorithm is designed for audio playback. The human voice and background music are separated by audio feature recognition. When the bandwidth is lower than the preset threshold, the background music volume is reduced first, and the bandwidth occupation is reduced by dynamically adjusting the sampling rate.

[0022] S3.3 Monitoring and Feedback: Audio and video delays are calculated synchronously using bidirectional timestamps, and delay trends are predicted using a Kalman filter to adjust audio and video progress in real time, ensuring smooth and synchronized playback.

[0023] Further optionally, step S4 specifically includes:

[0024] S4.1 Constructing the network stability factor SN and content complexity factor CC:

[0025] SN = 1 / (1+e) -k(BW-θBW) ,

[0026] CC=ω1*(E video / E video,max )+ω2*(C audio / C audio,max ),

[0027] In the formula, BW represents the average bandwidth, θBW represents the bandwidth standard deviation, and k is a preset adjustment parameter; E video E represents the entropy difference between video frames. video,max C represents the theoretical maximum value of the inter-frame difference entropy in a video; audio Describing the audio spectral complexity, C audio,max This represents the theoretical maximum value of the audio spectral complexity; ω1 and ω2 are weighting coefficients, and ω1 + ω2 = 1;

[0028] S4.2. Based on the network stability factor SN and the content complexity factor CC, the fusion weight α between the large model and the bitrate adaptive algorithm is calculated using formula a:

[0029] α=(SN*CC) / {SN*CC+(1-SN)*(1-CC)};

[0030] S4.3. Based on the fusion weight α, the prediction results of the large model are weighted and fused with the prediction results based on the bitrate adaptive algorithm to generate the final playback parameters P. final :

[0031] P final =α*P model +(1-α)*P traditional ,

[0032] In the formula, α represents the fusion weight, and P model P represents the prediction result of the large model. traditional This indicates the prediction result based on the adaptive bitrate algorithm.

[0033] Secondly, this invention provides a network audio and video playback control system based on a large model, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0034] A network audio and video playback control system based on a large model, comprising:

[0035] The data acquisition and preprocessing module is used to collect network status data, audio and video content feature data, and user behavior data, and to perform layered cleaning and normalization preprocessing operations on the collected data.

[0036] The large model processing module is used to generate large model training data based on spatiotemporal slicing technology, construct a three-level progressive training system including feature extraction layer, fusion decision layer and reinforcement learning optimization layer for large model training, optimize parameters through reinforcement learning and improve the performance of large model by adopting anti-overfitting and convergence optimization techniques.

[0037] The playback parameter generation module is used to input the real-time data output by the data acquisition and preprocessing module into the trained large model. The large model predicts the optimal playback parameters in the future. Combined with smooth transition, content-aware audio adjustment, bidirectional timestamp synchronization and Kalman filter feedback, it realizes the initial dynamic adjustment of audio and video playback and forms the basic playback parameters.

[0038] The playback parameter optimization module is used to dynamically adjust the fusion weights of the large model and the bitrate-adaptive algorithm by defining network stability factors and content complexity factors, and then fuse the results of the two to calculate the final playback parameters.

[0039] Optionally, the data acquisition and preprocessing modules involved specifically include:

[0040] The data acquisition unit is used to collect network status data, audio and video content feature data, and user behavior data. Among them, network status data is collected through a combination of active detection and passive monitoring, including round-trip latency, network throughput, and packet loss rate; audio and video content feature data is extracted using FFmpeg tools during the decoding stage, including video frame rate, resolution, bitrate fluctuation, and audio encoding format and sampling rate; user behavior data is collected in real time by embedding tracking code in the player to capture user playback operations, and combined with device information acquisition interface to collect device performance parameters.

[0041] The data preprocessing unit first filters out outliers from the collected raw data, removing invalid data that exceeds the preset range. Then, for time-series network data, a sliding window algorithm is used for smoothing to eliminate interference from instantaneous network noise. Finally, through Min-Max standardization, all types of data are uniformly mapped to the [0, 1] interval to ensure that the data quality meets the input requirements of large models.

[0042] Optionally, the large model processing modules involved specifically include:

[0043] The training data construction unit is used to construct training data for large models using spatiotemporal slicing technology. It divides the continuous data stream into 5-second time windows, and each window contains network state sequences, audio and video feature sequences and corresponding playback parameter labels, ultimately generating a structured training dataset.

[0044] Training data partitioning unit, used to divide the training dataset into training set and validation set according to a preset ratio;

[0045] The large model building unit is used to construct a three-level progressive training system comprising a feature extraction layer, a fusion decision layer, and a reinforcement learning optimization layer. The feature extraction layer uses a pre-trained Transformer model to process pre-processed temporal network data to capture long-term dependencies, while simultaneously using an LSTM network to extract temporal patterns from the pre-processed audio and video features. The fusion decision layer inputs the multimodal features output from the feature extraction layer into an MLP, automatically assigns feature weights using an attention mechanism, learns the optimal playback parameter mapping relationship for different scenarios based on the training set, and evaluates the generalization ability of the large model using a validation set. The reinforcement learning optimization layer, based on the DDPG algorithm, constructs a reward function using playback smoothness, latency, and audio-visual synchronization rate, iteratively optimizes the parameters of the feature extraction layer and the fusion decision layer based on the training set, and monitors the optimization effect using the validation set, achieving online optimization of the large model parameters.

[0046] The large model training unit is used to train large models. During training, the training set is used as input to learn parameters, and the model performance is evaluated through the validation set. Batch normalization and Dropout techniques are used during training to prevent overfitting, and a learning rate decay strategy of cosine annealing is used to improve convergence stability. When the evaluation metric of the validation set reaches the preset threshold, the training and optimization of the large model is completed.

[0047] Optionally, the playback parameter generation module may specifically include:

[0048] The model calling unit is used to input the real-time data output by the data acquisition and preprocessing module into the trained large model, and the large model outputs the optimal combination of playback parameters for the next 5 seconds.

[0049] The parameter adjustment unit employs a smooth transition algorithm to avoid visual jitter caused by parameter mutations. Specifically, when the bitrate needs to be reduced, the video frame rate is gradually reduced by a preset step size, and then the resolution is adjusted in stages. When network status monitoring meets preset conditions and network recovery is determined, the step size is gradually increased back to the original set value by adjusting the original resolution step size, and then back to the initial level by adjusting the original frame rate step size. Simultaneously, a content-aware volume adjustment algorithm is designed for audio playback. This algorithm separates human voices from background music through audio feature recognition. When the bandwidth is below a preset threshold, the background music volume is reduced first, and bandwidth usage is reduced by dynamically adjusting the sampling rate.

[0050] The monitoring and feedback unit is used to calculate audio-visual delay synchronously with bidirectional timestamps, predict delay trends through a Kalman filter, and adjust audio and video progress in real time to ensure smooth and synchronized playback.

[0051] Alternatively, the playback parameter optimization module may perform the following operations:

[0052] (1) Construct the network stability factor SN and the content complexity factor CC:

[0053] SN = 1 / (1+e) -k(BW-θBW) ,

[0054] CC=ω1*(E video / E video,max )+ω2*(C audio / C audio,max ),

[0055] In the formula, BW represents the average bandwidth, θBW represents the bandwidth standard deviation, and k is a preset adjustment parameter; E video E represents the entropy difference between video frames. video,max C represents the theoretical maximum value of the inter-frame difference entropy in a video; audio Describing the audio spectral complexity, C audio,max This represents the theoretical maximum value of the audio spectral complexity; ω1 and ω2 are weighting coefficients, and ω1 + ω2 = 1;

[0056] (2) Based on the network stability factor SN and the content complexity factor CC, the fusion weight α between the large model and the bitrate adaptive algorithm is calculated using formula a:

[0057] α=(SN*CC) / {SN*CC+(1-SN)*(1-CC)};

[0058] (3) Based on the fusion weight α, the prediction results of the large model are weighted and fused with the prediction results based on the bitrate adaptive algorithm to generate the final playback parameters P. final :

[0059] P final =α*P model +(1-α)*P traditional ,

[0060] In the formula, α represents the fusion weight, and P model P represents the prediction result of the large model. traditional This indicates the prediction result based on the adaptive bitrate algorithm.

[0061] The network audio and video playback control method and system based on a large model of the present invention have the following advantages compared with the prior art:

[0062] 1. This invention utilizes a large model to perform in-depth analysis and prediction of multi-dimensional data such as network status, audio and video content characteristics, and user behavior, thereby optimizing the dynamic adjustment strategy of audio and video playback parameters and achieving low-latency, high-smoothness network audio and video playback effects.

[0063] 2. This invention utilizes the powerful data processing and analysis capabilities of large models to optimize the display latency calculation and adjustment strategy during video playback, thereby achieving a perfect balance between low latency and high smoothness. Under different network environments and video content conditions, it can dynamically and adaptively adjust video playback parameters, effectively solving the problems existing in traditional playback control methods and bringing users a better video playback experience. Attached Figure Description

[0064] Appendix Figure 1 This is a flowchart of the method according to Embodiment 1 of the present invention;

[0065] Appendix Figure 2 This is a module connection block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0066] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0067] Example 1:

[0068] Combined with appendix Figure 1 This embodiment proposes a network audio and video playback control method based on a large model, which includes the following steps:

[0069] S1. Collect network status data, audio and video content feature data, and user behavior data, and perform layered cleaning and normalization preprocessing on the collected data. This process specifically includes:

[0070] S1.1 Data Acquisition: ① Network status data is collected through a combination of active probing and passive monitoring. Active probing involves sending test data packets to the server to calculate metrics such as round-trip time (RTT) and packet loss rate. Passive monitoring uses statistical information from the TCP / IP protocol stack to obtain parameters such as network throughput and connection status in real time. ② Audio and video content feature data is extracted during the decoding stage using FFmpeg, including video frame rate, resolution, bitrate fluctuations, and audio encoding format (such as AAC, MP3) and sampling rate. ③ User behavior data is captured in real time by embedding tracking code in the player to capture user playback operations (such as play / pause time, fast forward / rewind amplitude), and device performance parameters are collected in conjunction with device information acquisition interfaces (such as Android's Build class and iOS's UIDevice class).

[0071] S1.2 Data Preprocessing: The collected raw data is first filtered for outliers to remove invalid data that exceeds the preset range (such as invalid samples with a packet loss rate of more than 80%). Then, for time series network data, a sliding window algorithm is used for smoothing to eliminate interference caused by instantaneous network noise. Finally, through Min-Max standardization, all types of data are uniformly mapped to the [0, 1] interval to ensure that the data quality meets the input requirements of large models.

[0072] S2. Generate large-scale model training data based on spatiotemporal slicing technology, construct a three-level progressive training system including a feature extraction layer, a fusion decision layer, and a reinforcement learning optimization layer for large-scale model training, optimize parameters through reinforcement learning, and improve the performance of the large-scale model by employing anti-overfitting and convergence optimization techniques; this process specifically includes:

[0073] S2.1. Use spatiotemporal slicing technology to construct training data for a large model. Divide the continuous data stream into 5-second time windows. Each window contains network state sequences, audio and video feature sequences and corresponding playback parameter labels. Finally, generate a structured training dataset and divide it into training set and validation set according to a preset ratio.

[0074] S2.2 Construct a three-level progressive training system comprising a feature extraction layer, a fusion decision layer, and a reinforcement learning optimization layer. The feature extraction layer uses a pre-trained Transformer model to process pre-processed temporal network data to capture long-term dependencies, while simultaneously using an LSTM network to extract temporal patterns from the pre-processed audio and video features. The fusion decision layer inputs the multimodal features output from the feature extraction layer into an MLP (Multilayer Perceptron), automatically assigns feature weights using an attention mechanism, learns the optimal playback parameter mapping relationship for different scenarios based on the training set, and evaluates the generalization ability of the large model using a validation set. The reinforcement learning optimization layer, based on the DDPG (Deep Deterministic Policy Gradient) algorithm, constructs a reward function using playback smoothness, latency, and audio-visual synchronization rate, iteratively optimizes the parameters of the feature extraction layer and the fusion decision layer based on the training set, and monitors the optimization effect using a validation set, achieving online optimization of the large model parameters.

[0075] S2.3 During the training of the large model, the parameters are learned by taking the training set as input, and the performance of the large model is evaluated by taking the validation set. Batch normalization and dropout techniques are used to prevent overfitting, and the learning rate decay strategy of cosine annealing is used to improve convergence stability. When the evaluation index of the validation set reaches the preset threshold, the training and optimization of the large model is completed.

[0076] S3. Input the preprocessed real-time acquired data into the trained large model. The large model predicts the optimal playback parameters for the future. Combined with smooth transitions, content-aware audio adjustment, and bidirectional timestamp synchronization with Kalman filter feedback, preliminary dynamic adjustments to audio and video playback are achieved, forming basic playback parameters. Specifically, this includes:

[0077] S3.1 Prediction stage: The preprocessed real-time collected data is input into the trained large model, and the large model outputs the optimal combination of playback parameters for the next 5 seconds;

[0078] S3.2 Parameter Adjustment Stage: A smooth transition algorithm is adopted to avoid visual jitter caused by parameter abrupt changes. The specific operation is as follows: When it is necessary to reduce the bit rate, the video frame rate is gradually reduced by a preset step size, and then the resolution is adjusted in stages. When the network status monitoring meets the preset conditions, it is determined that the network has recovered. At this time, the following operations are performed: First, the step size is adjusted in reverse according to the original resolution and gradually increased to the original set value, and then the step size is adjusted in reverse according to the original frame rate and increased to the initial level. At the same time, a content-aware volume adjustment algorithm is designed for audio playback. The human voice and background music are separated by audio feature recognition. When the bandwidth is lower than the preset threshold, the background music volume is reduced first, and the bandwidth occupation is reduced by dynamically adjusting the sampling rate.

[0079] S3.3 Monitoring and Feedback: Audio and video delays are calculated synchronously using bidirectional timestamps, and delay trends are predicted using a Kalman filter to adjust audio and video progress in real time, ensuring smooth and synchronized playback.

[0080] S4. By defining network stability and content complexity factors, the fusion weights of the large model and the bitrate-adaptive algorithm are dynamically adjusted, and the results of both are fused to calculate the final playback parameters. Specifically, this includes:

[0081] S4.1 Constructing the network stability factor SN and content complexity factor CC:

[0082] SN = 1 / (1+e) -k(BW-θBW) ,

[0083] CC=ω1*(E video / E video,max )+ω2*(C audio / C audio,max ),

[0084] In the formula, BW represents the average bandwidth, θBW represents the bandwidth standard deviation, and k is a preset adjustment parameter; E video E represents the entropy difference between video frames. video,max C represents the theoretical maximum value of the inter-frame difference entropy in a video; audio Describing the audio spectral complexity, C audio,maxThis represents the theoretical maximum value of the audio spectral complexity; ω1 and ω2 are weighting coefficients, and ω1 + ω2 = 1;

[0085] S4.2. Based on the network stability factor SN and the content complexity factor CC, the fusion weight α between the large model and the bitrate adaptive algorithm is calculated using formula a:

[0086] α=(SN*CC) / {SN*CC+(1-SN)*(1-CC)},

[0087] When the fusion weight α approaches 1, it depends on the prediction results of the large model; when the fusion weight α approaches 0, it depends on the prediction results based on the bitrate adaptive algorithm.

[0088] S4.3. Based on the fusion weight α, the prediction results of the large model are weighted and fused with the prediction results based on the bitrate adaptive algorithm to generate the final playback parameters P. final :

[0089] P final =α*P model +(1-α)*P traditional ,

[0090] In the formula, α represents the fusion weight, and P model P represents the prediction result of the large model. traditional This indicates the prediction result based on the adaptive bitrate algorithm.

[0091] The network audio and video playback control method based on this embodiment is compared with traditional video playback control methods (such as those based on bitrate adaptive algorithms) in a comparative experiment.

[0092] 1. Under the same network environment, hardware environment, and video dataset, use the control method of this embodiment and the bitrate adaptive algorithm to play videos, and record the video playback latency and smoothness indicators (such as the number of stutters and the stutter time).

[0093] 1.1 Hardware Environment: Build different hardware testing platforms, including high-performance servers, ordinary personal computers, and mobile devices (such as mobile phones and tablets), to simulate the performance conditions of different user devices.

[0094] 1.2 Network Environment: Using network simulation tools such as NS-3 and Mininet, various network environments were constructed, including stable high-speed networks, low-speed networks with large fluctuations, and networks with packet loss and latency jitter. Simultaneously, real-world network environments were connected for testing, such as home broadband, 4G / 5G mobile networks, and public WiFi.

[0095] 1.3 Video Datasets: Prepare diverse video datasets covering video files of different resolutions (e.g., standard definition, high definition, 4K), bitrates, frame rates, and content types (e.g., action movies, documentaries, animations).

[0096] II. Conduct the experiment.

[0097] 2.1 Parameter Adjustment Experiment: The key parameters in the control method of this embodiment, such as the hyperparameters of the large model and the fusion weights, are adjusted. The performance of the control method described in this embodiment under different parameter settings is observed to determine the optimal parameter combination.

[0098] 2.2 Robustness Experiment: The robustness of the control method described in this embodiment is tested under conditions of drastic changes in the network environment or abnormalities in the video content. For example, a sudden network interruption or bandwidth drop is simulated during video playback, or the video is switched to a high bitrate, high frame rate video segment. The robustness of the control method described in this embodiment is observed to ensure it can quickly adapt to changes and maintain the stability of video playback.

[0099] III. Analysis of the above experimental results.

[0100] 3.1 Performance Indicator Comparison: Statistical analysis was performed on the performance data obtained from the experiments, including latency and smoothness metrics. Comparative charts were created to visually demonstrate the performance differences between the control method described in this embodiment and the bitrate-adaptive algorithm under different experimental conditions. Data analysis revealed that, under various network environments and video content conditions, the average latency of the control method described in this embodiment is significantly lower than that of the bitrate-adaptive algorithm. Simultaneously, the number of stutters and the duration of stutters are also significantly reduced, effectively improving the smoothness of video playback.

[0101] 3.2 Parameter Impact Analysis: This section analyzes the impact of different parameter settings on algorithm performance and determines the optimal range of values ​​for key parameters. For example, experiments show that when the learning rate of the large model is set within a certain range, the convergence speed and prediction accuracy of the control method described in this embodiment are optimal. The value of the fusion weights is closely related to the stability of the network environment and the complexity of the video content; under different circumstances, the weights need to be adjusted reasonably to obtain the best playback control effect.

[0102] 3.3 Robustness Evaluation: Based on the robustness experiment results, the adaptability of the control method described in this embodiment to changes in network and video content is evaluated. Experiments show that the control method described in this embodiment can quickly adjust playback parameters when the network environment changes drastically, effectively avoiding problems such as screen stuttering and excessive latency; it can also maintain a relatively stable playback effect when video content becomes abnormal, demonstrating strong robustness.

[0103] In summary, the control method described in this embodiment achieves intelligent and dynamic control of network audio and video playback. Compared with traditional methods (such as those based on bitrate adaptive algorithms), the average latency is significantly reduced under various network environments and audio / video content conditions, and the number and duration of audio / video stutters are significantly reduced, effectively ensuring audio-visual synchronization and achieving a high-quality playback effect with low latency and high smoothness.

[0104] Example 2:

[0105] Combined with appendix Figure 2 This embodiment proposes a network audio and video playback control system based on a large model, which includes:

[0106] The data acquisition and preprocessing module is used to collect network status data, audio and video content feature data, and user behavior data, and to perform layered cleaning and normalization preprocessing operations on the collected data.

[0107] The large model processing module is used to generate large model training data based on spatiotemporal slicing technology, construct a three-level progressive training system including feature extraction layer, fusion decision layer and reinforcement learning optimization layer for large model training, optimize parameters through reinforcement learning and improve the performance of large model by adopting anti-overfitting and convergence optimization techniques.

[0108] The playback parameter generation module is used to input the real-time data output by the data acquisition and preprocessing module into the trained large model. The large model predicts the optimal playback parameters in the future. Combined with smooth transition, content-aware audio adjustment, bidirectional timestamp synchronization and Kalman filter feedback, it realizes the initial dynamic adjustment of audio and video playback and forms the basic playback parameters.

[0109] The playback parameter optimization module is used to dynamically adjust the fusion weights of the large model and the bitrate-adaptive algorithm by defining network stability factors and content complexity factors, and then fuse the results of the two to calculate the final playback parameters.

[0110] In this embodiment, the data acquisition and preprocessing module specifically includes:

[0111] The data acquisition unit is used to collect network status data, audio and video content feature data, and user behavior data. ① Network status data is collected through a combination of active probing and passive monitoring. Active probing involves sending test data packets to the server to calculate metrics such as round-trip time (RTT) and packet loss rate. Passive monitoring uses statistical information from the TCP / IP protocol stack to obtain parameters such as network throughput and connection status in real time. ② Audio and video content feature data is extracted during the decoding stage using FFmpeg, including video frame rate, resolution, bitrate fluctuations, and audio encoding formats (such as AAC and MP3) and sampling rates. ③ User behavior data is collected in real time by embedding tracking code in the player to capture user playback operations (such as play / pause time, fast forward / rewind amplitude) and by combining this with device information acquisition interfaces (such as Android's Build class and iOS's UIDevice class) to collect device performance parameters.

[0112] The data preprocessing unit first filters out outliers from the collected raw data, removing invalid data that exceeds the preset range (such as invalid samples with a packet loss rate exceeding 80%). Then, for time-series network data, a sliding window algorithm is used for smoothing to eliminate interference from instantaneous network noise. Finally, through Min-Max standardization, all types of data are uniformly mapped to the [0, 1] interval to ensure that the data quality meets the input requirements of large models.

[0113] In this embodiment, the large model processing module specifically includes:

[0114] The training data construction unit is used to construct training data for large models using spatiotemporal slicing technology. It divides the continuous data stream into 5-second time windows, and each window contains network state sequences, audio and video feature sequences and corresponding playback parameter labels, ultimately generating a structured training dataset.

[0115] Training data partitioning unit, used to divide the training dataset into training set and validation set according to a preset ratio;

[0116] The large model building unit is used to construct a three-level progressive training system comprising a feature extraction layer, a fusion decision layer, and a reinforcement learning optimization layer. The feature extraction layer uses a pre-trained Transformer model to process pre-processed temporal network data to capture long-term dependencies, while simultaneously using an LSTM network to extract temporal patterns from the pre-processed audio and video features. The fusion decision layer inputs the multimodal features output from the feature extraction layer into an MLP, automatically assigns feature weights using an attention mechanism, learns the optimal playback parameter mapping relationship for different scenarios based on the training set, and evaluates the generalization ability of the large model using a validation set. The reinforcement learning optimization layer, based on the DDPG algorithm, constructs a reward function using playback smoothness, latency, and audio-visual synchronization rate, iteratively optimizes the parameters of the feature extraction layer and the fusion decision layer based on the training set, and monitors the optimization effect using the validation set, achieving online optimization of the large model parameters.

[0117] The large model training unit is used to train large models. During training, the training set is used as input to learn parameters, and the model performance is evaluated through the validation set. Batch normalization and Dropout techniques are used during training to prevent overfitting, and a learning rate decay strategy of cosine annealing is used to improve convergence stability. When the evaluation metric of the validation set reaches the preset threshold, the training and optimization of the large model is completed.

[0118] In this embodiment, the playback parameter generation module specifically includes:

[0119] The model calling unit is used to input the real-time data output by the data acquisition and preprocessing module into the trained large model, and the large model outputs the optimal combination of playback parameters for the next 5 seconds.

[0120] The parameter adjustment unit employs a smooth transition algorithm to avoid visual jitter caused by parameter mutations. Specifically, when the bitrate needs to be reduced, the video frame rate is gradually reduced by a preset step size, and then the resolution is adjusted in stages. When network status monitoring meets preset conditions and network recovery is determined, the step size is gradually increased back to the original set value by adjusting the original resolution step size, and then back to the initial level by adjusting the original frame rate step size. Simultaneously, a content-aware volume adjustment algorithm is designed for audio playback. This algorithm separates human voices from background music through audio feature recognition. When the bandwidth is below a preset threshold, the background music volume is reduced first, and bandwidth usage is reduced by dynamically adjusting the sampling rate.

[0121] The monitoring and feedback unit is used to calculate audio-visual delay synchronously with bidirectional timestamps, predict delay trends through a Kalman filter, and adjust audio and video progress in real time to ensure smooth and synchronized playback.

[0122] In this embodiment, the playback parameter optimization module performs the following operations:

[0123] (1) Construct the network stability factor SN and the content complexity factor CC:

[0124] SN = 1 / (1+e) -k(BW-θBW) ,

[0125] CC=ω1*(E video / E video,max )+ω2*(C audio / C audio,max ),

[0126] In the formula, BW represents the average bandwidth, θBW represents the bandwidth standard deviation, and k is a preset adjustment parameter; E video E represents the entropy difference between video frames. video,max C represents the theoretical maximum value of the inter-frame difference entropy in a video; audio Describing the audio spectral complexity, C audio,max This represents the theoretical maximum value of the audio spectral complexity; ω1 and ω2 are weighting coefficients, and ω1 + ω2 = 1;

[0127] (2) Based on the network stability factor SN and the content complexity factor CC, the fusion weight α between the large model and the bitrate adaptive algorithm is calculated using formula a:

[0128] α=(SN*CC) / {SN*CC+(1-SN)*(1-CC)},

[0129] When the fusion weight α approaches 1, it depends on the prediction results of the large model; when the fusion weight α approaches 0, it depends on the prediction results based on the bitrate adaptive algorithm.

[0130] (3) Based on the fusion weight α, the prediction results of the large model are weighted and fused with the prediction results based on the bitrate adaptive algorithm to generate the final playback parameters P. final :

[0131] P final =α*P model +(1-α)*P traditional ,

[0132] In the formula, α represents the fusion weight, and P model P represents the prediction result of the large model. traditional This indicates the prediction result based on the adaptive bitrate algorithm.

[0133] In summary, the network audio and video playback control method and system based on a large model of the present invention optimizes the dynamic adjustment strategy of audio and video playback parameters by deeply analyzing and predicting multi-dimensional data such as network status, audio and video content characteristics and user behavior through a large model. This achieves low latency and high smoothness in network audio and video playback, and can be widely applied to various network audio and video playback scenarios, greatly improving the user's audio and video playback experience.

[0134] The above specific examples illustrate the principles and implementation methods of the present invention in detail. These embodiments are merely for the purpose of helping to understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A network audio and video playback control method based on a large model, characterized in that, Includes the following steps: S1. Collect network status data, audio and video content feature data, and user behavior data, and perform layered cleaning and normalization preprocessing on the collected data. S2. Generate large model training data based on spatiotemporal slicing technology, construct a three-level progressive training system including feature extraction layer, fusion decision layer and reinforcement learning optimization layer for large model training, optimize parameters through reinforcement learning and improve the performance of large model by adopting anti-overfitting and convergence optimization techniques. S3. Input the preprocessed real-time acquisition data into the trained large model. The large model predicts the optimal playback parameters in the future. Combined with smooth transition, content-aware audio adjustment, bidirectional timestamp synchronization and Kalman filter feedback, the initial dynamic adjustment of audio and video playback is realized, forming basic playback parameters. S4. By defining network stability factors and content complexity factors, the fusion weights of the large model and the bitrate adaptive algorithm are dynamically adjusted, and the results of the two are fused and calculated as the final playback parameters.

2. The network audio and video playback control method based on a large model according to claim 1, characterized in that, Step S1 specifically includes: S1.1 Data Acquisition: Network status data is collected through a combination of active detection and passive monitoring, including round-trip latency, network throughput, and packet loss rate; audio and video content feature data is extracted during the decoding stage using FFmpeg, including video frame rate, resolution, bitrate fluctuations, and audio encoding format and sampling rate; user behavior data is captured in real time by embedding tracking code in the player, and device performance parameters are collected in conjunction with the device information acquisition interface. S1.2 Data Preprocessing: The collected raw data is first filtered for outliers to remove invalid data that exceeds the preset range; then, for time series network data, a sliding window algorithm is used for smoothing to eliminate interference from instantaneous network noise; finally, through Min-Max standardization, all types of data are uniformly mapped to the [0, 1] interval to ensure that the data quality meets the input requirements of large models.

3. The network audio and video playback control method based on a large model according to claim 2, characterized in that, Step S2 specifically includes: S2.

1. Use spatiotemporal slicing technology to construct training data for a large model. Divide the continuous data stream into 5-second time windows. Each window contains network state sequences, audio and video feature sequences and corresponding playback parameter labels. Finally, generate a structured training dataset and divide it into training set and validation set according to a preset ratio. S2.2 Construct a three-level progressive training system comprising a feature extraction layer, a fusion decision layer, and a reinforcement learning optimization layer. The feature extraction layer uses a pre-trained Transformer model to process pre-processed temporal network data to capture long-term dependencies, while simultaneously using an LSTM network to extract temporal patterns from the pre-processed audio and video features. The fusion decision layer inputs the multimodal features output from the feature extraction layer into the MLP, automatically assigns feature weights using an attention mechanism, learns the optimal playback parameter mapping relationship for different scenarios based on the training set, and evaluates the generalization ability of the large model using a validation set. The reinforcement learning optimization layer, based on the DDPG algorithm, constructs a reward function using playback smoothness, latency, and audio-visual synchronization rate, iteratively optimizes the parameters of the feature extraction layer and the fusion decision layer based on the training set, and monitors the optimization effect using a validation set, achieving online optimization of the large model parameters. S2.3 During the training of the large model, the parameters are learned by using the training set as input, the performance of the large model is evaluated by the validation set, batch normalization and Dropout techniques are used to prevent overfitting, and the learning rate decay strategy of cosine annealing is used to improve convergence stability. When the evaluation index of the validation set reaches the preset threshold, the training and optimization of the large model is completed.

4. The network audio and video playback control method based on a large model according to claim 3, characterized in that, Step S3 specifically includes: S3.1 Prediction stage: The preprocessed real-time collected data is input into the trained large model, and the large model outputs the optimal combination of playback parameters for the next 5 seconds; S3.2 Parameter Adjustment Stage: A smooth transition algorithm is adopted to avoid visual jitter caused by parameter abrupt changes. The specific operation is as follows: When it is necessary to reduce the bit rate, the video frame rate is gradually reduced by a preset step size, and then the resolution is adjusted in stages. When the network status monitoring meets the preset conditions, it is determined that the network has recovered. At this time, the following operations are performed: First, the step size is adjusted in reverse according to the original resolution and gradually increased to the original set value, and then the step size is adjusted in reverse according to the original frame rate and increased to the initial level. At the same time, a content-aware volume adjustment algorithm is designed for audio playback. The human voice and background music are separated by audio feature recognition. When the bandwidth is lower than the preset threshold, the background music volume is reduced first, and the bandwidth occupation is reduced by dynamically adjusting the sampling rate. S3.3 Monitoring and Feedback: Audio and video delays are calculated synchronously using bidirectional timestamps, and delay trends are predicted using a Kalman filter to adjust audio and video progress in real time, ensuring smooth and synchronized playback.

5. The network audio and video playback control method based on a large model according to claim 4, characterized in that, Step S4 specifically includes: S4.1 Constructing the network stability factor SN and content complexity factor CC: SN<1 / (1+e) -k(BW-θBW) , CC=ω1*(E video / E video,max )+ω2*(C audio / C audio,max ), In the formula, BW represents the average bandwidth, θBW represents the bandwidth standard deviation, and k is a preset adjustment parameter; E video E represents the entropy difference between video frames. video,max C represents the theoretical maximum value of the inter-frame difference entropy in a video; audio Describing the audio spectral complexity, C audio,max This represents the theoretical maximum value of the audio spectral complexity; ω1 and ω2 are weighting coefficients, and ω1 + ω2 = 1; S4.

2. Based on the network stability factor SN and the content complexity factor CC, the fusion weight α between the large model and the bitrate adaptive algorithm is calculated using formula a: α=(SN*CC) / {SN*CC+(1-SN)*(1-CC)}; S4.

3. Based on the fusion weight α, the prediction results of the large model are weighted and fused with the prediction results based on the bitrate adaptive algorithm to generate the final playback parameters P. final : P final =α*P model +(1-α)*P traditional , In the formula, α represents the fusion weight, and P model P represents the prediction result of the large model. traditional This indicates the prediction result based on the adaptive bitrate algorithm.

6. A network audio and video playback control system based on a large model, characterized in that, It includes: The data acquisition and preprocessing module is used to collect network status data, audio and video content feature data, and user behavior data, and to perform layered cleaning and normalization preprocessing operations on the collected data. The large model processing module is used to generate large model training data based on spatiotemporal slicing technology, construct a three-level progressive training system including feature extraction layer, fusion decision layer and reinforcement learning optimization layer for large model training, optimize parameters through reinforcement learning and improve the performance of large model by adopting anti-overfitting and convergence optimization techniques. The playback parameter generation module is used to input the real-time data output by the data acquisition and preprocessing module into the trained large model. The large model predicts the optimal playback parameters in the future. Combined with smooth transition, content-aware audio adjustment, bidirectional timestamp synchronization and Kalman filter feedback, it realizes the initial dynamic adjustment of audio and video playback and forms the basic playback parameters. The playback parameter optimization module is used to dynamically adjust the fusion weights of the large model and the bitrate-adaptive algorithm by defining network stability factors and content complexity factors, and then fuse the results of the two to calculate the final playback parameters.

7. A network audio and video playback control system based on a large model according to claim 6, characterized in that, The data acquisition and preprocessing module specifically includes: The data acquisition unit is used to collect network status data, audio and video content feature data, and user behavior data. Among them, network status data is collected through a combination of active detection and passive monitoring, including round-trip latency, network throughput, and packet loss rate; audio and video content feature data is extracted using FFmpeg tools during the decoding stage, including video frame rate, resolution, bitrate fluctuation, and audio encoding format and sampling rate; user behavior data is collected in real time by embedding tracking code in the player to capture user playback operations, and combined with device information acquisition interface to collect device performance parameters. The data preprocessing unit first filters out outliers from the collected raw data, removing invalid data that exceeds the preset range. Then, for time-series network data, a sliding window algorithm is used for smoothing to eliminate interference from instantaneous network noise. Finally, through Min-Max standardization, all types of data are uniformly mapped to the [0, 1] interval to ensure that the data quality meets the input requirements of large models.

8. The network audio and video playback control system based on a large model according to claim 7, characterized in that, The large model processing module specifically includes: The training data construction unit is used to construct training data for large models using spatiotemporal slicing technology. It divides the continuous data stream into 5-second time windows, and each window contains network state sequences, audio and video feature sequences and corresponding playback parameter labels, ultimately generating a structured training dataset. Training data partitioning unit, used to divide the training dataset into training set and validation set according to a preset ratio; The large model building unit is used to construct a three-level progressive training system comprising a feature extraction layer, a fusion decision layer, and a reinforcement learning optimization layer. The feature extraction layer uses a pre-trained Transformer model to process pre-processed temporal network data to capture long-term dependencies, while simultaneously using an LSTM network to extract temporal patterns from the pre-processed audio and video features. The fusion decision layer inputs the multimodal features output from the feature extraction layer into an MLP, automatically assigns feature weights using an attention mechanism, learns the optimal playback parameter mapping relationship for different scenarios based on the training set, and evaluates the generalization ability of the large model using a validation set. The reinforcement learning optimization layer, based on the DDPG algorithm, constructs a reward function using playback smoothness, latency, and audio-visual synchronization rate, iteratively optimizes the parameters of the feature extraction layer and the fusion decision layer based on the training set, and monitors the optimization effect using the validation set, achieving online optimization of the large model parameters. The large model training unit is used to train large models. During training, the training set is used as input to learn parameters, and the model performance is evaluated through the validation set. Batch normalization and Dropout techniques are used during training to prevent overfitting, and a learning rate decay strategy of cosine annealing is used to improve convergence stability. When the evaluation metric of the validation set reaches the preset threshold, the training and optimization of the large model is completed.

9. A network audio and video playback control system based on a large model according to claim 8, characterized in that, The playback parameter generation module specifically includes: The model calling unit is used to input the real-time data output by the data acquisition and preprocessing module into the trained large model, and the large model outputs the optimal combination of playback parameters for the next 5 seconds. The parameter adjustment unit employs a smooth transition algorithm to avoid visual jitter caused by parameter mutations. Specifically, when the bitrate needs to be reduced, the video frame rate is gradually reduced by a preset step size, and then the resolution is adjusted in stages. When network status monitoring meets preset conditions and network recovery is determined, the step size is gradually increased back to the original set value by adjusting the original resolution step size, and then back to the initial level by adjusting the original frame rate step size. Simultaneously, a content-aware volume adjustment algorithm is designed for audio playback. This algorithm separates human voices from background music through audio feature recognition. When the bandwidth is below a preset threshold, the background music volume is reduced first, and bandwidth usage is reduced by dynamically adjusting the sampling rate. The monitoring and feedback unit is used to calculate audio-visual delay synchronously with bidirectional timestamps, predict delay trends through a Kalman filter, and adjust audio and video progress in real time to ensure smooth and synchronized playback.

10. A network audio and video playback control system based on a large model according to claim 9, characterized in that, The playback parameter optimization module specifically performs the following operations: (1) Construct the network stability factor SN and the content complexity factor CC: SN<1 / (1+e) -k(BW-θBW) , CC=ω1*(E video / E video,max )+ω2*(C audio / C audio,max ), In the formula, BW represents the average bandwidth, θBW represents the bandwidth standard deviation, and k is a preset adjustment parameter; E video E represents the entropy difference between video frames. video,max C represents the theoretical maximum value of the inter-frame difference entropy in a video; audio Describing the audio spectral complexity, C audio,max This represents the theoretical maximum value of the audio spectral complexity; ω1 and ω2 are weighting coefficients, and ω1 + ω2 = 1; (2) Based on the network stability factor SN and the content complexity factor CC, the fusion weight α between the large model and the bitrate adaptive algorithm is calculated using formula a: α=(SN*CC) / {SN*CC+(1-SN)*(1-CC)}; (3) Based on the fusion weight α, the prediction results of the large model are weighted and fused with the prediction results based on the bitrate adaptive algorithm to generate the final playback parameters P. final : P final =α*P model +(1-α)*P traditional , In the formula, α represents the fusion weight, and P model P represents the prediction result of the large model. traditional This indicates the prediction result based on the adaptive bitrate algorithm.

Images