Satellite-ground fusion network video code rate adaptive transmission method based on inverse reinforcement learning

Abstract

The invention discloses a satellite-ground convergence network video code rate adaptive transmission method based on inverse reinforcement learning, and relates to the field of satellite-ground convergence networks. According to the method, a simulation environment is constructed based on real satellite-ground fusion heterogeneous network performance index data, expert sample data are obtained through labeling of multiple experts, and therefore a reward function is solved through inverse reinforcement learning; an optimal strategy is solved according to the obtained reward function training model, and the purpose of simulating an expert strategy is achieved; meanwhile, a progressive action verification module is designed to ensure the trend adaptability, preference consistency and transmission stability of code rate adjustment under multiple constraints of the satellite-ground fusion network, and a new way is provided for solving the problem of satellite-ground fusion network video code rate self-adaption.

Description

Technical Field

[0001] This invention relates to the field of satellite-ground fusion networks, and in particular to a video bitrate adaptive transmission method for satellite-ground fusion networks based on inverse reinforcement learning. Background Technology

[0002] With the rapid development of low-Earth orbit satellite internet, the convergence of space-ground networks—that is, the integration of GEO / MEO / LEO and even terrestrial networks to achieve complementary advantages between systems—is an inevitable trend in the development of private network communications. However, compared with terrestrial public mobile communication networks, space-ground converged networks have limited signal transmission bandwidth, and the quality differences of various communication links are more obvious, making it difficult to guarantee the quality of audio and video communication services.

[0003] Video bitrate adaptive transmission technology is fundamental to reducing stuttering and improving user experience in heterogeneous / unstable scenarios of satellite-terrestrial converged networks. Traditional heuristic bitrate adaptive algorithms adjust the bitrate based on rules such as bandwidth and buffer size, which are too conservative and lack flexibility. In recent years, learning-based bitrate adaptive methods have received increasing attention from the industry, including two categories: network state awareness based on deep learning and bitrate adaptive control strategy generation based on reinforcement learning.

[0004] Regarding network state awareness based on deep learning, invention application number 2021100977647 discloses a deep learning-based intelligent video bitrate adjustment and bandwidth allocation method for internet video services. This method uses a trained bitrate allocation neural network to obtain a bitrate allocation scheme, and then determines the bandwidth occupied by each user based on the bitrate allocation scheme and the currently available bandwidth, thus completing the bandwidth allocation. Invention application number 2020105978336 discloses a server and bitrate determination method for adaptive bitrate transmission for internet video services. This method uses a deep learning network to perform network state prediction and performs bitrate adaptation based on the prediction results. As a type of time-series prediction task, deep learning-based network state awareness methods generally have relatively mature solutions.

[0005] Regarding the generation of bitrate adaptive control strategies, invention application number 2024112900175 discloses a real-time video transmission method based on QoE and resource consumption awareness for live streaming scenarios. This method uses reinforcement learning to learn the optimal encoding ladder by analyzing user QoE feedback and resource consumption under system conditions. Invention application number 2023111240614 discloses an end-to-end bitrate adaptive control method based on multi-agent reinforcement learning for video-on-demand applications. This method utilizes multiple agents to make decisions on encoding parameters such as resolution to directly optimize the user's QoE. However, existing reinforcement learning-based bitrate adaptive control strategy generation methods still have significant limitations in complex scenarios of satellite-terrestrial fusion networks: reward function design is difficult, making it hard to balance multi-objective optimization needs; verification mechanisms and adaptation logic are insufficient, failing to effectively match the dynamic characteristics and multi-dimensional constraints of the scenario. Furthermore, both methods currently mainly target internet video transmission scenarios such as live streaming and video-on-demand; research on video bitrate adaptive transmission technology in satellite-terrestrial network scenarios is urgently needed. Summary of the Invention

[0006] In view of this, this invention proposes a video bitrate adaptive transmission method based on inverse reinforcement learning for satellite-terrestrial fusion networks. This method aims to solve problems such as adaptive video transmission bitrate in satellite-terrestrial fusion network scenarios and the difficulty in setting reward functions for traditional reinforcement learning methods.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A satellite-to-ground fusion network video bitrate adaptive transmission method based on inverse reinforcement learning includes the following steps:

[0009] Step 1: Collect multi-scenario link index data of the space-ground fusion network, and after preprocessing and expansion, construct and verify a simulation environment consistent with the real network to ensure data reliability and simulation authenticity;

[0010] Step 2: Generate candidate decision trajectories through expert annotation, pre-trained model assistance, and quality screening. Combine stuttering and video quality feature clustering to obtain three types of expert preference sample sets: conservative, radical, and adaptive.

[0011] Step 3: Define the state / action space, design the basic reward function, solve for the specific parameters by combining expert trajectories, introduce adversarial rewards and iteratively fuse them to finally obtain the convergent optimal reward function;

[0012] Step 4: Construct a bitrate adaptive model based on the reward function, design a progressive action verification module of "network trend - expert preference - historical lag", and iteratively train by jointly expanding the sample and verification log data to ensure the effectiveness and stability of the strategy;

[0013] Step 5: Lightweight processing of the bitrate adaptive model and deployment, iterative optimization of the model based on real-world interactive data.

[0014] Furthermore, the specific method of step 1 is as follows:

[0015] Step 1.1: Construct a satellite-ground integrated heterogeneous network, including a satellite system and a terrestrial private network communication system, interconnected by gateways; parse standard protocol data and passively extract core link quality indicators, including bandwidth, latency, and packet loss; collect network performance data over a 24-hour period, including different weather conditions such as sunny, rainy, cloudy, foggy, and snowy days, selecting different areas such as densely populated urban areas, general areas, and suburbs; based on the collected network environment dataset, perform data preprocessing, including outlier handling, data alignment, and reasonable scrambling strategies to expand the dataset and generate an enhanced network environment dataset;

[0016] Step 1.2: Based on the enhanced network environment dataset, according to the real network indicator distribution in different weather and regions, set the network damage parameters using the network damage instrument damage configuration table to ensure cross-domain consistency between the simulation environment and the real satellite-ground fusion network data; and quantitatively verify the similarity between the network indicators collected in the simulation environment and the real data collected in Step 1 based on the mean square error index to avoid simulation distortion leading to poor model generalization.

[0017] Furthermore, the specific method for step 2 is as follows:

[0018] Step 2.1: Real-time video communication is conducted based on a satellite-ground fusion simulation network. Experts annotate the times when bitrate adjustments are needed and the adjustment strategies, i.e., the selected bitrate levels, generating candidate expert annotation decision trajectories. Specifically, the network status for the next N seconds is provided to the experts, where N is 10 or 30, thereby improving annotation accuracy. Some candidate decision trajectories are generated based on a public network pre-trained bitrate adaptive model. Offline automatic video quality evaluation and scoring are performed on the video screen recording files corresponding to the above two types of decision trajectories. The trajectory set with high scores is selected and confirmed by experts as the final expert annotation sample set.

[0019] Step 2.2: Generate a sample set of M expert annotations. Cluster them according to the overall evaluation of stuttering, video quality, and number of switching to obtain three preference expert annotation sample sets: "conservative", "aggressive", and "adaptive".

[0020] Furthermore, step 3 is performed as follows:

[0021] Step 3.1: First, define the bitrate adaptive state space as S, including bandwidth / latency / packet loss within the historical window, bandwidth / latency / packet loss within the predicted future window, and stuttering / bitrate level status within the historical window. The state at time t is represented as... ,and The bitrate-adaptive action space is defined as A, containing selectable bitrates (360P, 480P, 720P, 1080P), and the action at time t is represented as... ,and ;

[0022] Step 3.2: Design a basic reward formula based on multiple dimensions including video quality, switching frequency, network adaptation, and stuttering. The formula is as follows:

[0023]

[0024] in It's a video quality item. It is a bitrate switching penalty item. It is a network status adaptation item. It's a lag penalty item. , , , These are the weighting coefficients;

[0025]

[0026] It is an action The corresponding actual bitrate value, It is the lowest bitrate value in the action space. It is the highest bitrate value in the action space;

[0027] This is the bitrate switching penalty, and the specific calculation formula is as follows:

[0028]

[0029] It is the first The bitrate value selected at each moment, from Obtain from historical bitrates. It is an action The corresponding actual bitrate value, It is the lowest bitrate value in the action space. It is the highest bitrate value in the action space;

[0030] This is a network state adaptation item, and the specific calculation formula is as follows:

[0031]

[0032] The first term is the penalty for bitrate exceeding the predicted bandwidth, where It is the first The minimum bandwidth predicted in the future at any time, from The first item is obtained from the predicted future window bandwidth; the second item is the packet loss exceeding penalty item, in which... It is the first The historical average packet loss rate at any given time, from Retrieved from packet loss in the historical window; It is the packet loss rate threshold; The first item is the packet loss penalty coefficient; the second item is the latency exceeding the limit penalty item, in which... It is the first The historical average time delay of a moment, from Obtained from the historical window latency; It is the delay threshold; It is the delay penalty coefficient;

[0033] This is a lag penalty, and the specific calculation formula is as follows:

[0034]

[0035] It is the first The actual duration of the pause at any given moment, from the next state Retrieved during the lag in the history window; It is the lag tolerance threshold. It's a mild lag penalty factor. This is a severe lag penalty coefficient, which must meet the following requirements. ;

[0036] Step 3.3: Define Experts A dedicated set of parameters:

[0037]

[0038] in, Representing three types of experts, the overall trajectory set is denoted as The set of single-class expert trajectories is denoted as ;

[0039] From expert trajectory set Extract all the triples of "current state, selected action, next state" to form a sample set. Total number of samples ,Right now

[0040]

[0041] Assuming optimal expert decision-making, gradient ascent iterative calculation is employed to find the parameter values ​​that maximize the cumulative reward of expert actions.

[0042]

[0043] Substituting the solved parameters into the basic structure yields the results for experts. Initial reward function:

[0044]

[0045] Step 3.4: The adversarial reward function is set to distinguish between expert trajectories and interference trajectories, making the policy's decision-making pattern more consistent with that of the expert; a discriminator model is trained, with the input state being "state ,action Next state The algorithm generates a triplet, outputting the probability that the triplet originates from an expert trajectory. Then, it extracts the adversarial reward from the discriminator. If the triplet originates from an expert trajectory, the adversarial reward is positive; if it originates from an interference policy, the adversarial reward is negative. The formula is:

[0046]

[0047] in, The probability output by the discriminator. These are the parameters of the discriminator;

[0048] Step 3.5: Combine expert rewards and adversarial rewards to form the final reward function:

[0049]

[0050] Based on the current reward function, the action that maximizes the cumulative reward is selected when the policy interacts with the environment. The trajectory generated based on this will be closer to the expert trajectory than the previous policy. The new policy is then denoted as π_new.

[0051] Repeat step 3.3 to extract triplet samples from the trajectory of the new strategy. Combined with expert samples The parameters are then solved again using maximum likelihood estimation. ;

[0052] Repeat step 3.4 to update the base reward with the new parameters. Retrain the discriminator and adjust the weights. Receive the new round of adjusted rewards ;

[0053] During the iteration process, when the expert parameter set The change in the value is extremely small, indicating that the parameters have stabilized, meaning the reward function has met the convergence condition. At this point, the reward function is in its final form:

[0054]

[0055] in These are the converged expert parameters. It is the optimal fusion weight.

[0056] Furthermore, step 4 is specifically implemented as follows:

[0057] Step 4.1: Define the core space of the scene: In this step, the bitrate adaptive model is constructed through reinforcement learning. Its action space and state space are consistent with those of the inverse reinforcement learning in Step 3. See Step 3.1 for process definition.

[0058] Step 4.2: The action verification module performs verifications in the following order: "Network trend consistency verification → Expert preference adaptation verification → Historical stuttering correlation verification". If the verification fails, the historical bitrate is retained; if it passes, the original action is output. At the same time, a structured verification log is output for subsequent optimization of expert-annotated sample sets and iteration of reinforcement learning models.

[0059] Input to the verification module includes: the initial bitrate level output by the reinforcement learning model, historical window data of bandwidth / latency / packet loss / bitrate / stuttering over the past 5 periods, bandwidth, latency, packet loss rate indicators and confidence prediction window data for the next 3 periods, and conservative / aggressive / adaptive expert preference labels output by the previous clustering module.

[0060] Network trend consistency verification: This verification ensures that the direction of bitrate adjustment is consistent with the current trend of network status changes; this section calculates the network status trend based on historical window data and prediction window data, and the specific calculation formula is as follows:

[0061]

[0062] Based on the above formula, the subsequent network status is determined by setting a threshold. When the bandwidth increases, the latency decreases, and the packet loss rate decreases, if the current action switching bitrate is greater than or equal to the historical bitrate, the bitrate is switched according to the current action. When the bandwidth decreases, the latency increases, and the packet loss rate increases, if the current action switching bitrate is greater than or equal to the historical bitrate, the bitrate is switched according to the current action.

[0063] Expert preference adaptation verification: The current decision trajectory is classified using the clustering model in step 4 to obtain expert preferences. The action is adjusted based on the expert preferences in three dimensions: bandwidth threshold, adjustment range, and confidence level to prevent inconsistencies between the action and the expert preferences. The minimum bandwidth, bandwidth threshold, confidence threshold, and adjustment range limit of the three types of sample data (conservative, aggressive, and adaptive) are statistically determined according to the quantile method. If the threshold is not met, bitrate switching based on the current action is prohibited.

[0064] Historical stuttering correlation verification: The historical stuttering correlation verification stage checks whether stuttering has occurred in the last 3 periods. If a historical stuttering record is detected, rule verification is triggered. When the action is executed to switch the bitrate upward, the bandwidth must be greater than or equal to the minimum bandwidth of the target bitrate × the redundancy coefficient. The conservative / adaptive preference is a redundancy coefficient of 1.5, and the aggressive preference is 1.2. If the condition is not met, the rate increase is prohibited. When the action is to downgrade, continuous rate downgrading is not allowed to avoid excessive degradation.

[0065] Step 4.3: Iterative optimization based on the reward function: The reward function in this step is obtained from step 3, see step 3.5 for details. The reinforcement learning model is trained based on the expanded expert sample dataset and the data generated by the verification module. When the average cumulative reward over multiple rounds is greater than or equal to the set threshold and the standard deviation is less than or equal to the set threshold, the iterative optimization is completed.

[0066] Furthermore, step 5 is specifically implemented as follows:

[0067] The bitrate adaptive model generated in step 4 is distilled and quantized to adapt to hardware resources with limited computing power. After the reinforcement learning model is deployed to the real environment after action verification, the real environment will provide feedback on "action-state-reward" data to support the model's iterative optimization.

[0068] Due to the adoption of the above technical solution, the beneficial effects of this invention compared with the prior art are as follows:

[0069] 1. This invention proposes a method for obtaining video bitrate decision samples in a space-ground fusion network. This method is based on real data combined with a network impairment instrument to ensure cross-domain data consistency. It also ensures the consistency of simulation data and real data distribution through mean square error, thereby solving the problem of difficulty in simulating heterogeneous space-ground networks.

[0070] 2. This invention proposes a method for generating reward functions for video bitrate adaptive models based on inverse reinforcement learning. This method designs a composite reward function that integrates differentiated reward terms that combine expert decision-making preferences with adversarial reward terms generated by the discriminator, and solves the problem of difficulty in setting reward functions and inability to accurately balance the relationship between factors such as stuttering, video quality, and number of switching in reinforcement learning-based bitrate adaptive methods.

[0071] 3. This invention proposes a progressive action verification module for a satellite-ground fusion network video bitrate adaptive model. Through multi-dimensional input fusion and progressive verification logic design, it improves the stability and image quality adaptability of satellite-ground fusion network video transmission while avoiding the risk of bitrate adjustment mismatch. Attached Figure Description

[0072] Figure 1This is an overall flowchart of the satellite-ground fusion network video bitrate adaptive transmission method based on inverse reinforcement learning in this embodiment of the invention.

[0073] Figure 2 for Figure 1 A schematic diagram of the overall architecture of the satellite-ground fusion simulation network.

[0074] Figure 3 This is a schematic diagram of the process of obtaining the reward function based on inverse reinforcement learning in an embodiment of the present invention.

[0075] Figure 4 This is a schematic diagram of the action verification module in an embodiment of the present invention. Detailed Implementation

[0076] The invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0077] A video bitrate adaptive transmission method based on inverse reinforcement learning for satellite-terrestrial fusion networks is proposed. A simulation environment is constructed using performance index data from real satellite-terrestrial fusion heterogeneous networks. Expert sample data is obtained through annotation by multiple experts, and the reward function is solved using inverse reinforcement learning. The model is then trained based on the obtained reward function to solve for the optimal strategy, achieving the goal of mimicking expert strategies. Simultaneously, a progressive action verification module is designed to ensure the trend adaptability, preference consistency, and transmission stability of bitrate adjustment under multiple constraints in the satellite-terrestrial fusion network, providing a new approach to the video bitrate adaptive problem in satellite-terrestrial fusion networks. Figure 1 As shown, it includes the following steps:

[0078] Step 1: Collect multi-scenario link index data of the space-ground fusion network, and after preprocessing and expansion, construct and verify a simulation environment consistent with the real network to ensure data reliability and simulation authenticity;

[0079] Step 2: Generate candidate decision trajectories through expert annotation, pre-trained model assistance, and quality screening. Combine stuttering and video quality feature clustering to obtain three types of expert preference sample sets: conservative, radical, and adaptive.

[0080] Step 3: Define the state / action space, design the basic reward function, solve for the specific parameters by combining expert trajectories, introduce adversarial rewards and iteratively fuse them to finally obtain the convergent optimal reward function;

[0081] Step 4: Construct a bitrate adaptive model based on the reward function, design a progressive action verification module of "network trend - expert preference - historical lag", and iteratively train by jointly expanding the sample and verification log data to ensure the effectiveness and stability of the strategy;

[0082] Step 5: Lightweight processing of the bitrate adaptive model and deployment, iterative optimization of the model based on real-world interactive data.

[0083] Furthermore, the specific method of step 1 is as follows:

[0084] Step 1.1: Construct a satellite-ground integrated heterogeneous network, including a satellite system and a terrestrial private network communication system, interconnected by gateways; parse standard protocol data and passively extract core link quality indicators, including bandwidth, latency, and packet loss; collect network performance data over a 24-hour period, including different weather conditions such as sunny, rainy, cloudy, foggy, and snowy days, selecting different areas such as densely populated urban areas, general areas, and suburbs; based on the collected network environment dataset, perform data preprocessing, including outlier handling, data alignment, and reasonable scrambling strategies to expand the dataset and generate an enhanced network environment dataset;

[0085] Step 1.2: Based on the enhanced network environment dataset, and according to the distribution of real network indicators in different weather conditions and regions, such as... Figure 2 As shown, network damage parameters are set using the network damage configuration table of the network damage instrument to ensure cross-domain consistency between the simulation environment and the real satellite-ground fusion network data; and the similarity between the network indicators collected in the simulation environment and the distribution of the real data collected in step 1 is quantitatively verified based on the mean square error index to avoid simulation distortion leading to poor model generalization.

[0086] Furthermore, the specific method for step 2 is as follows:

[0087] Step 2.1: Real-time video communication is conducted based on a satellite-ground fusion simulation network. Experts annotate the times when bitrate adjustments are needed and the adjustment strategies, i.e., the selected bitrate levels, generating candidate expert-annotated decision trajectories. Specifically, experts are informed of the network status for the next N seconds, where N is 10 or 30, to improve annotation accuracy. Some candidate decision trajectories are generated based on a public network pre-trained bitrate adaptive model, specifically using the Pensieve model. Offline automatic video quality evaluation and scoring are performed on the video screen recording files corresponding to the above two types of decision trajectories. The trajectory set with the highest score is selected and confirmed by experts as the final expert annotation sample set.

[0088] Step 2.2: Generate a sample set of M expert annotations. Cluster them according to the overall evaluation of stuttering, video quality, and number of switching to obtain three preference expert annotation sample sets: "conservative", "aggressive", and "adaptive".

[0089] Specifically, a conservative approach is preferred: video quality should meet basic requirements, switching frequency should be controlled, the core objective is to minimize buffering, and the video quality should be downgraded to avoid buffering risks when there are slight network fluctuations.

[0090] Aggressive preference: With the core goal of improving video quality, it can tolerate a high switching frequency, allow a small number of short-term stutters, and upgrade to pursue high image quality as soon as the network is slightly optimized.

[0091] Adaptive Preferences: Dynamically balance video quality and switching frequency, adjust weights according to network trends, improve image quality and moderately relax switching frequency when the network is stable, and reduce switching frequency and control stuttering when the network fluctuates.

[0092] Furthermore, such as Figure 3 As shown, the specific method for step 3 is as follows:

[0093] Step 3.1: First, define the bitrate adaptive state space as S, including bandwidth / latency / packet loss within the historical window, bandwidth / latency / packet loss within the predicted future window, and stuttering / bitrate level status within the historical window. The state at time t is represented as... ,and The bitrate-adaptive action space is defined as A, containing selectable bitrates (360P, 480P, 720P, 1080P), and the action at time t is represented as... ,and ;

[0094] Step 3.2: Design a basic reward formula based on multiple dimensions including video quality, switching frequency, network adaptation, and stuttering. The formula is as follows:

[0095]

[0096] in It's a video quality item. It is a bitrate switching penalty item. It is a network status adaptation item. It's a lag penalty item. , , , These are the weighting coefficients;

[0097]

[0098] It is an action The corresponding actual bitrate value, It is the lowest bitrate value in the action space. It is the highest bitrate value in the action space;

[0099] This is the bitrate switching penalty, and the specific calculation formula is as follows:

[0100]

[0101] It is the first The bitrate value selected at each moment, from Obtain from historical bitrates. It is an action The corresponding actual bitrate value, It is the lowest bitrate value in the action space. It is the highest bitrate value in the action space;

[0102] This is a network state adaptation item, and the specific calculation formula is as follows:

[0103]

[0104] The first term is the penalty for bitrate exceeding the predicted bandwidth, where It is the first The minimum bandwidth predicted in the future at any time, from The first item is obtained from the predicted future window bandwidth; the second item is the packet loss exceeding penalty item, in which... It is the first The historical average packet loss rate at any given time, from Retrieved from packet loss in the historical window; It is the packet loss rate threshold; The first item is the packet loss penalty coefficient; the second item is the latency exceeding the limit penalty item, in which... It is the first The historical average time delay of a moment, from Obtained from the historical window latency; It is the delay threshold; It is the delay penalty coefficient;

[0105] This is a lag penalty, and the specific calculation formula is as follows:

[0106]

[0107] It is the first The actual duration of the pause at any given moment, from the next state Retrieved during the lag in the history window; It is the lag tolerance threshold. It's a mild lag penalty factor. This is a severe lag penalty coefficient, which must meet the following requirements. ;

[0108] Step 3.3: Define Experts A dedicated set of parameters:

[0109]

[0110] in, Representing three types of experts, the overall trajectory set is denoted as The set of single-class expert trajectories is denoted as ;

[0111] From expert trajectory set Extract all the triples of "current state, selected action, next state" to form a sample set. Total number of samples ,Right now

[0112]

[0113] Assuming optimal expert decision-making, gradient ascent iterative calculation is employed to find the parameter values ​​that maximize the cumulative reward of expert actions.

[0114]

[0115] Substituting the solved parameters into the basic structure yields the results for experts. Initial reward function:

[0116]

[0117] Step 3.4: The adversarial reward function is set to distinguish between expert trajectories and interference trajectories, making the policy's decision-making pattern tend to align with the expert's; a discriminator model is trained, specifically, the discriminator model uses a perceptron model, with the input state being "state ,action Next state The algorithm generates a triplet, outputting the probability that the triplet originates from an expert trajectory. Then, it extracts the adversarial reward from the discriminator. If the triplet originates from an expert trajectory, the adversarial reward is positive; if it originates from an interference policy, the adversarial reward is negative. The formula is:

[0118]

[0119] in, The probability output by the discriminator. These are the parameters of the discriminator;

[0120] Step 3.5: Combine expert rewards and adversarial rewards to form the final reward function:

[0121]

[0122] Based on the current reward function, the action that maximizes the cumulative reward is selected when the policy interacts with the environment. The trajectory generated based on this will be closer to the expert trajectory than the previous policy. The new policy is then denoted as π_new.

[0123] Repeat step 3.3 to extract triplet samples from the trajectory of the new strategy. Combined with expert samples The parameters are then solved again using maximum likelihood estimation. ;

[0124] Repeat step 3.4 to update the base reward with the new parameters. Retrain the discriminator and adjust the weights. Receive the new round of adjusted rewards ;

[0125] During the iteration process, when the expert parameter set The change in the value is extremely small, indicating that the parameters have stabilized, meaning the reward function has met the convergence condition. At this point, the reward function is in its final form:

[0126]

[0127] in These are the converged expert parameters. It is the optimal fusion weight.

[0128] Furthermore, step 4 is specifically implemented as follows:

[0129] Step 4.1: Define the core space of the scene: In this step, the bitrate adaptive model is constructed through reinforcement learning. Specifically, the bitrate adaptive model adopts the Deep Q-Network (DQN) model, and its action space and state space are consistent with those of the inverse reinforcement learning in step 3. See the process definition in step 3.1 for details.

[0130] Step 4.2: As Figure 4 As shown, the action verification module performs verification in the following order: "Network trend consistency verification → expert preference adaptation verification → historical stuttering correlation verification". If the verification fails, the historical bitrate is maintained; if it passes, the original action is output. At the same time, a structured verification log is output for subsequent optimization of expert-annotated sample sets and iteration of reinforcement learning models.

[0131] The verification module input includes: the initial bitrate level output by the reinforcement learning model, historical window data of bandwidth / latency / packet loss / bitrate / stuttering over the past 5 periods, bandwidth, latency, and packet loss rate data and confidence prediction window data for the next 3 periods, and conservative / aggressive / adaptive expert preference labels output by the previous clustering module; the confidence value represents the reliability of the bandwidth / latency / packet loss values ​​predicted in the future window.

[0132] Network trend consistency verification: This verification ensures that the direction of bitrate adjustment is consistent with the current trend of network status changes; this section calculates the network status trend based on historical window data and prediction window data, and the specific calculation formula is as follows:

[0133]

[0134] Based on the above formula, the subsequent network status is determined by setting a threshold. When the bandwidth increases, the latency decreases, and the packet loss rate decreases, if the current action switching bitrate is greater than or equal to the historical bitrate, the bitrate is switched according to the current action. When the bandwidth decreases, the latency increases, and the packet loss rate increases, if the current action switching bitrate is greater than or equal to the historical bitrate, the bitrate is switched according to the current action.

[0135] Expert preference adaptation verification: The current decision trajectory is classified using the clustering model in step 4 to obtain expert preferences. The action is adjusted based on the expert preferences in three dimensions: bandwidth threshold, adjustment range, and confidence level to prevent inconsistencies between the action and the expert preferences. The minimum bandwidth, bandwidth threshold, confidence threshold, and adjustment range limit of the three types of sample data (conservative, aggressive, and adaptive) are statistically determined according to the quantile method. If the threshold is not met, bitrate switching based on the current action is prohibited.

[0136] The adjustment range refers to the step size of the bitrate switching, such as the range (360P, 480P, 720P, 1080P). The step size for 360P→720P is 2, and the step size for 360P→480P is 1.

[0137] Historical stuttering correlation verification: The historical stuttering correlation verification stage checks whether stuttering has occurred in the last 3 periods. If a historical stuttering record is detected, rule verification is triggered. When the action is executed to switch the bitrate upward, the bandwidth must be greater than or equal to the minimum bandwidth of the target bitrate × the redundancy coefficient. The conservative / adaptive preference is a redundancy coefficient of 1.5, and the aggressive preference is 1.2. If the condition is not met, the rate increase is prohibited. When the action is to downgrade, continuous rate downgrading is not allowed to avoid excessive degradation.

[0138] Step 4.3: Iterative optimization based on the reward function: The reward function in this step is obtained from step 3, see step 3.5 for details. The reinforcement learning model is trained based on the expanded expert sample dataset and the data generated by the verification module. When the average cumulative reward over multiple rounds is greater than or equal to the set threshold and the standard deviation is less than or equal to the set threshold, the iterative optimization is completed.

[0139] Furthermore, step 5 is specifically implemented as follows:

[0140] The bitrate adaptive model generated in step 4 is distilled and quantized to adapt to hardware resources with limited computing power. After the reinforcement learning model is deployed to the real environment after action verification, the real environment will provide feedback on "action-state-reward" data to support the model's iterative optimization.

[0141] In summary, the core protection of this invention is a video bitrate adaptive transmission method based on inverse reinforcement learning for satellite-ground fusion networks. It aims to solve problems such as video transmission bitrate adaptation in satellite-ground fusion network scenarios and the difficulty in setting reward functions for traditional reinforcement learning methods. The key points are as follows:

[0142] (1) A method for obtaining expert samples for video bitrate decision-making in satellite-ground fusion networks is proposed. The method is characterized by: collecting real satellite-ground fusion network link index data in multiple scenarios at all times and expanding it through data augmentation; configuring network impairment parameters based on the index data to construct a simulation environment, and verifying its distribution consistency with real data through mean square error quantification; combining the network state in the next N seconds to assist expert annotation, integrating the public network pre-trained model to generate decision trajectories, and constructing a high-quality expert annotation sample set through quality evaluation; and finally clustering the expert decision sample sets with three types of preferences: conservative, aggressive, and adaptive.

[0143] (2) A method for generating reward function of video bitrate adaptive model based on inverse reinforcement learning is proposed. The method is characterized by: constructing a state space S containing historical and predicted network performance indicators, stuttering and bitrate levels and a multi-level bitrate action space A; based on the expert decision sample set, and with the help of the inverse reinforcement learning framework, obtaining a composite reward function composed of differentiated reward items that integrate expert decision preferences and adversarial reward items generated by the discriminator.

[0144] (3) A progressive action verification module for a satellite-ground fusion network video bitrate adaptive model is proposed. The module is characterized by constructing a multi-dimensional input system including initial bitrate, historical network state data, network prediction data and confidence interval for the next N seconds, and expert preference labels. The module performs verification according to the progressive logic of "network trend consistency verification → expert preference adaptation verification → historical stuttering correlation verification", so as to achieve accurate adaptation of actions with network state and expert decision preferences.

[0145] Those skilled in the art will recognize that the described embodiments are intended to help readers understand the principles of the invention and should be understood as not limiting the scope of protection of the invention to the described embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A method for video rate adaptive transmission of a satellite-ground integrated network based on inverse reinforcement learning, characterized in that, The method comprises the following steps: Step 1: Collect the multi-scene link index data of the satellite-ground fusion network, pre-process and expand the data, build and verify the simulation environment consistent with the real network, and ensure the reliability of the data and the authenticity of the simulation; Step 2: Generate candidate decision trajectories through expert annotation, pre-trained model assistance and quality screening, and obtain conservative / aggressive / adaptive three types of expert preference sample sets by clustering based on the features of lag and video quality; Step 3: Define the state / action space, design the basic reward function, combine the expert trajectories to solve the exclusive parameters, introduce the adversarial reward and iteratively fuse, and finally obtain the converged optimal reward function; Step 4: Build a rate adaptation model based on the reward function, design a "network trend-expert preference-historical lag" progressive action verification module, iteratively train the model by expanding the sample and verifying the log data, and ensure the effectiveness and stability of the strategy; Step 5: Lighten the rate adaptation model and deploy it, and iteratively optimize the model based on real environment interaction data.

2. The inverse reinforcement learning based satellite-ground integrated network video code rate adaptive transmission method according to claim 1, characterized in that, The specific way of step 1 is: Step 1.1: Build a satellite-ground fusion heterogeneous network, including a satellite system and a ground special network communication system, and interconnect them through a gateway; analyze the standard protocol data, passively extract the link quality core indicators, including bandwidth, delay and packet loss; the network performance data collection period covers 24 hours, including different weathers such as sunny, rainy, cloudy, foggy and snowy days, and different regions such as dense urban areas, general areas and suburban areas; Based on the collected network environment data set, perform data preprocessing, including outlier processing, data alignment and reasonable scrambling strategy to expand the data set, and generate an enhanced network environment data set; Step 1.2: Based on the enhanced network environment data set, according to the real network index distribution of different weathers and regions, use the network damage instrument damage configuration table to set the network damage parameters, ensure the consistency of the simulation environment and the real satellite-ground fusion network data across domains; and based on the mean square error index, the similarity of the collected network indicators in the simulation environment and the distribution of the real data collected in step 1 is quantitatively verified to avoid simulation distortion leading to poor model generalization.

3. The inverse reinforcement learning based satellite-ground integrated network video code rate adaptive transmission method according to claim 1, characterized in that, The specific way of step 2 is: Step 2.1: Real-time video communication is carried out based on the satellite-ground fusion simulation network, and the expert is informed of the network state in the next N seconds, N being 10 or 30, so as to improve the annotation accuracy, and part of the candidate decision trajectories are generated based on the pre-trained rate adaptation model of the public network; the video recording files corresponding to the above two types of decision trajectories are automatically evaluated and scored offline, and the trajectories with high scores are selected by the expert as the final expert annotation sample set; Step 2.2: Generate M expert annotation sample sets, cluster them according to the overall evaluation lag, video quality and switching frequency features, and obtain "conservative", "aggressive" and "adaptive" three types of preference expert annotation sample sets.

4. The inverse reinforcement learning based satellite-ground fusion network video code rate adaptive transmission method according to claim 1, characterized in that, The specific way of step 3 is: Step 3.1: Firstly, the state space of rate adaptation is defined as S, including bandwidth / delay / packet loss in the history window, predicted bandwidth / delay / packet loss in the future window, and the state of the history window of the stall / rate level, the state at the t-th moment is represented as , and ; the action space of rate adaptation is defined as A, including the selectable rate (360P, 480P, 720P, 1080P), the action at the t-th moment is represented as , and ; Step 3.2: Design a basic reward formula from the dimensions of video quality, switching frequency, network adaptation and lag, and the formula is as follows: ； wherein is a video quality term, is a code rate switching penalty term, is a network status adaptation term, is a stall penalty term, , , , are weight coefficients; ； is the action corresponding actual code rate value, is the lowest code rate value in the action space, is the highest code rate value in the action space; is the code rate switching penalty term, and the specific formula is as follows: ； is the selected code rate value at the moment, is the selected code rate value at the moment, is the selected code rate value at the moment, is the selected code rate value at the moment, is the selected code rate value at the moment, is the selected code rate value at the moment, is the selected code rate value at the moment, is the network state adaptation item, and the specific calculation formula is as follows: ； The first term is a penalty term for the code rate exceeding the predicted bandwidth, where is the predicted future minimum bandwidth at time , which is obtained from the predicted future window bandwidth of ; the second term is a penalty term for the packet loss exceeding the threshold, where is the historical average packet loss rate at time , which is obtained from the historical window packet loss of ; is the packet loss rate threshold; is the packet loss penalty coefficient. The third term is a latency exceeding penalty term, wherein is the historical average latency at the moment, obtained from the historical window latency of ; and is a latency threshold value; is a latency penalty coefficient. is the Cartan penalty term, which is calculated as follows: ； is the actual frame freezing duration at the moment, which is obtained from the historical window freezing of the next state . ; is the freezing tolerance threshold, is the mild freezing penalty coefficient, is the severe freezing penalty coefficient, which satisfies ; Step 3.3: Define the expert's set of parameters : the expert's set of parameters ； wherein, represent three types of experts, the overall trajectory set is denoted as , and the single-class expert trajectory set is denoted as ; From the set of expert trajectories all "current state, selected action, next state" triples are extracted, forming a sample set , the total number of samples i.e. ； With the optimal expert decision as the precondition, gradient ascent iteration calculation is adopted to solve the parameter value that maximizes the cumulative reward of the expert action ; ； Substitute the solved parameters into the base structure to obtain the initial reward function for the expert : ； Step 3.4: Set the setting of the adversarial reward function to distinguish the expert trajectory from the interference trajectory, so that the decision mode of the strategy is consistent with the expert; train a discriminator model, input the state as the "state , action , next state " triplet, output the probability that the triplet comes from the expert trajectory, then extract the adversarial reward from the discriminator, if the triplet comes from the expert trajectory, the adversarial reward is positive, if it comes from the interference strategy, the adversarial reward is negative, the formula is: ； wherein, is the probability output by the discriminator, is a parameter of the discriminator; Step 3.5: Combine the expert reward and the adversarial reward to form the final reward function: ； Based on the current reward function, select the action that maximizes the cumulative reward when interacting with the environment, and generate a trajectory based on this. The new policy is recorded as π_new. Repeat step 3.3 to extract triple samples from the new policy's trajectories , combined with expert samples , re-solve the parameters by maximum likelihood estimation ; Repeat step 3.4, update base reward with new parameters , retrain discriminator, adjust weights Get new round of adjusted rewards ; In the iteration process, when the variation of the expert parameter set is very small, it indicates that the parameters have been stabilized, i.e. the reward function has met the convergence condition, at which time the reward function is the final form: ； wherein is the converged expert parameter, is the optimal fusion weight.

5. The inverse reinforcement learning based satellite-ground fusion network video code rate adaptive transmission method according to claim 1, characterized in that, The specific way of step 4 is: Step 4.1: Define the scene core space: In this step, the rate adaptation model is constructed through reinforcement learning, and its action space and state space are consistent with the inverse reinforcement learning in step 3.

1. See step 3.1 process definition for details. Step 4.2: The action verification module performs verification in the order of "network trend consistency verification → expert preference adaptation verification → historical stall correlation verification". If the verification fails, the historical code rate is maintained, and if the verification passes, the original action is output, and a structured verification log is output for subsequent expert sample set optimization and reinforcement learning model iteration: Verification module input: initial code rate output by reinforcement learning model, historical window data of bandwidth / delay / packet loss / code rate / stall in the past 5 periods, future 3 period bandwidth / delay / packet loss rate index data and confidence prediction window data, conservative / aggressive / adaptive expert preference labels output by the previous clustering module; Network trend consistency verification: This verification ensures that the direction of code rate adjustment is consistent with the trend of the current network state. Based on the historical window data and the prediction window data, the network state trend is calculated, and the specific calculation formula is: ； Based on the above formula, by setting a threshold, the subsequent network state is determined. When the bandwidth increases, the delay decreases, and the packet loss rate decreases, if the current action switching code rate ≥ historical code rate, the code rate is switched according to the current action; when the bandwidth decreases, the delay increases, and the packet loss rate increases, if the current action switching code rate ≥ historical code rate, the code rate is not switched according to the current action; Expert preference adaptation verification: Use the clustering model in step 4 to classify the current decision trajectory to obtain the expert preference. Adjust the action from three dimensions of bandwidth threshold, adjustment amplitude, and confidence to prevent the action from being inconsistent with the expert preference. According to the quantile method, the minimum bandwidth, bandwidth threshold, confidence threshold, and adjustment amplitude limit of the three types of sample data are calculated. If the threshold is not met, the code rate is not switched according to the current action; Historical stall correlation verification: The historical stall correlation verification stage checks whether there has been a stall in the last 3 periods. If a historical stall record is detected, the rule verification is triggered. When the action is performed and the code rate is switched up, the bandwidth must be greater than or equal to the target code rate minimum bandwidth x redundancy coefficient, where the conservative / adaptive preference redundancy coefficient is 1.5, and the aggressive preference is 1.

2. If it is not met, the upgrade is prohibited. When the action is downgraded, it is not allowed to be downgraded continuously to avoid excessive degradation. Step 4.3: Iterative optimization based on reward function: The reward function of this step is obtained from step 3, which is detailed in step 3.

5. The reinforcement learning model is trained based on the extended expert sample dataset and the data generated by the verification module. When the average value of the cumulative reward of multiple rounds is greater than or equal to the set threshold and the standard deviation is less than or equal to the set threshold, the iterative optimization is completed.

6. The inverse reinforcement learning based satellite-ground fusion network video code rate adaptive transmission method according to claim 1, characterized in that, The specific way of step 5 is: The code rate adaptive model generated in step 4 is distilled and quantized to adapt to hardware resources with limited computing power. After the reinforcement learning model is deployed to the real environment after action verification, the real environment will feedback "action-state-reward" data to support the iterative optimization of the model.

Images