A layered reinforcement learning-based adaptive video encoding rate control and bandwidth allocation method, system, device and medium

Abstract

The application provides a layered reinforcement learning-based adaptive video encoding rate control and bandwidth allocation method, system, device and medium, relates to the technical field of video transmission, and obtains high-layer observation information corresponding to a high-layer time interval composed of K continuous low-layer time intervals through an edge base station, sets a video encoding rate in the high-layer time interval for each video collection terminal by using a high-layer decision network, dynamically allocates a bandwidth proportion in the corresponding low-layer time interval for each video collection terminal by using a low-layer decision network according to low-layer observation information corresponding to each low-layer time interval, enables each video collection terminal to generate a to-be-transmitted video according to the set video encoding rate, and transmits the video in the to-be-transmitted video queue to the edge base station based on the allocated bandwidth proportion, so that the queue stability can be maintained while the video transmission quality is ensured, and the calculation complexity of online decision is significantly reduced.

Description

Technical Field

[0001] This application relates to the field of video transmission technology, and in particular to an adaptive video coding rate control and bandwidth allocation method, system, device and medium based on hierarchical reinforcement learning. Background Technology

[0002] In wide-area video network scenarios, mid-to-high-position cameras are typically deployed in remote areas such as rural areas, mountainous regions, and coastal defense areas. Due to the lagging infrastructure development in these areas, the deployment cost of wired networks such as fiber optics is extremely high and the construction is difficult. Therefore, wireless communication methods such as 4G / 5G are mostly used for video transmission between cameras and edge base stations.

[0003] However, wireless channels are highly susceptible to weather changes, geographical environment, and electromagnetic interference, exhibiting significant time-varying characteristics. Furthermore, they currently lack online adaptive capabilities to dynamic network environments. In actual deployments, network states such as channel gain and available bandwidth fluctuate rapidly. If real-time joint optimization of transmission video coding rate and bandwidth allocation is not possible, it can lead to video stuttering at best, and queue overflow and data loss at worst, seriously threatening the reliability of monitoring tasks. Jointly optimizing bitrate control and bandwidth allocation is crucial for improving video transmission quality and reducing latency, but these two aspects inherently differ in their decision-making time scales. Therefore, this application proposes an adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide an adaptive video coding rate control and bandwidth allocation method, system, device and medium based on hierarchical reinforcement learning, so as to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect, this application provides an adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning, the method comprising: The edge base station acquires low-level observation information corresponding to the low-level time interval, and acquires high-level observation information corresponding to the high-level time interval composed of the K consecutive low-level time intervals; K is an integer greater than 1. Based on the high-level observation information, the edge base station sets the video encoding rate for each video acquisition terminal within the high-level time interval every K low-level time intervals through the high-level decision network. Each video acquisition terminal generates a video to be transmitted and adds it to its respective video queue to be transmitted based on its video encoding rate within the high-level time interval. Based on the low-level observation information, the edge base station allocates the bandwidth ratio of each video acquisition terminal to each video acquisition terminal in each low-level time interval through the low-level decision network. Each video acquisition terminal transmits the video in its respective video queue to the edge base station based on the total bandwidth of the edge base station within the low-layer time interval and the bandwidth ratio of each video acquisition terminal within the low-layer time interval.

[0006] Optionally, each video acquisition terminal transmits the video in its respective video queue to the edge base station according to its bandwidth within the lower-layer time interval, including: Each video acquisition terminal determines its uplink transmission rate within the lower-layer time interval based on its bandwidth and channel gain within that lower-layer time interval. Each video acquisition terminal determines its own video transmission volume within the low-layer time interval based on the duration of the low-layer time interval and its own uplink transmission rate within the low-layer time interval. Each video acquisition terminal transmits the video in its respective video queue to the edge base station according to the video transmission volume within the lower-level time interval.

[0007] Optionally, each video acquisition terminal generates a video to be transmitted and adds it to its respective video queue to be transmitted based on its video encoding rate within the high-level time interval, including: Each video acquisition terminal determines its video encoding rate in the high-level time interval as its video encoding rate in the corresponding K consecutive low-level time intervals. Each video acquisition terminal encodes the video data acquired in its respective low-level time interval according to its own video encoding rate in the low-level time interval, generates a video to be transmitted, and adds it to its own video queue to be transmitted. The method further includes: Each video acquisition terminal determines the amount of video generated within the low-level time interval based on the duration of the low-level time interval and its respective video encoding rate within the low-level time interval. Each video acquisition terminal determines the length of its video queue to be transmitted in the next low-level time interval based on the length of its video queue to be transmitted, the amount of video transmitted, and the amount of video generated in the current low-level time interval.

[0008] Optionally, The low-level observation information includes: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the low-level time interval, and the video coding rate of each video acquisition terminal in the high-level time interval. The video encoding rate of a video acquisition terminal within K low-level time intervals of the same high-level time interval is the video encoding rate set by the high-level decision network for the video acquisition terminal within that high-level time interval. The high-level observation information includes: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the high-level time interval, and the video coding rate of each video acquisition terminal in the previous high-level time interval. The length of the video queue to be transmitted and the channel gain of a video acquisition terminal in a higher-level time interval are equal to the length of the video queue to be transmitted and the channel gain of the video acquisition terminal in the first lower-level time interval of that higher-level time interval.

[0009] Optionally, the method further includes: The training method aims to minimize the variation in the length of the video queues to be transmitted from each video acquisition terminal and maximize the video quality of the videos transmitted from each video acquisition terminal to the edge base station. This is achieved by alternately training the high-level and low-level training networks. The alternate training process includes: With the model parameters of the high-level network to be trained frozen, the low-level network obtains low-level training samples through interaction with the environment, and uses the low-level training samples to perform low-level reinforcement learning training on the low-level network to be trained, thus obtaining the low-level network to be trained with updated parameters. With the model parameters of the lower-level network to be trained frozen after the parameter update, the higher-level network obtains higher-level training samples through interaction with the environment, and uses the higher-level training samples to perform higher-level reinforcement learning training on the higher-level network to be trained, thus obtaining the higher-level network to be trained after parameter update. Repeat the above process to update the high-level decision network and the low-level decision network.

[0010] Optionally, the observation space of the low-level network to be trained is: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the low-level time interval of the sample, and the video coding rate of each video acquisition terminal in the high-level time interval of the sample to which the low-level time interval belongs. The action space of the low-level network to be trained is: the bandwidth allocation weight of each video acquisition terminal in the low-level time interval of the sample. The reward function value of the low-level network to be trained represents the change between the length of the video queue to be transmitted by each video acquisition terminal in the current low-level time interval of the sample and the length of the video queue to be transmitted in the next low-level time interval of the sample.

[0011] Optionally, the observation space of the high-level network to be trained is: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the sample high-level time interval, and the video coding rate of each video acquisition terminal in the sample high-level time interval. The action space of the high-level network to be trained is: the video encoding rate of each video acquisition terminal within the sample high-level time interval. The reward function value of the high-level network to be trained includes: the video quality utility value of each video acquisition terminal in the high-level time interval of the sample, and the sum of the reward function values ​​corresponding to the K low-level time intervals of the sample in the high-level time interval. There is a corresponding relationship between the video quality utility value of each video acquisition terminal in the sample high-level time interval and the video encoding rate of each video acquisition terminal in the sample high-level time interval. The corresponding relationship is that the video quality utility value increases monotonically with the video encoding rate and the marginal effect decreases.

[0012] A second aspect of this application provides an adaptive video coding rate control and bandwidth allocation system based on hierarchical reinforcement learning, the system comprising: The acquisition module is used to acquire low-level observation information corresponding to low-level time intervals through edge base stations, and to acquire high-level observation information corresponding to high-level time intervals composed of K consecutive low-level time intervals; where K is an integer greater than 1. The setting module is used to set the video encoding rate of each video acquisition terminal in the high-level time interval by means of the edge base station according to the high-level observation information, every K low-level time interval, through the high-level decision network. The generation module is used to generate videos to be transmitted and add them to their respective video queues based on the video encoding rate of each video acquisition terminal within the high-level time interval. The allocation module is used to allocate the bandwidth ratio of each video acquisition terminal in each low-level time interval to the edge base station based on the low-level observation information and through the low-level decision network in each low-level time interval. The transmission module is used to transmit the video in each video queue to the edge base station according to the total bandwidth of the edge base station in the low-layer time interval and the bandwidth ratio of each video acquisition terminal in the low-layer time interval.

[0013] A third aspect of this application provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning as described in the first aspect of this application.

[0014] A fourth aspect of this application provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning as described in the first aspect of this application.

[0015] The beneficial effects of this application are: This application provides an adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning. The method includes: an edge base station acquiring low-level observation information corresponding to low-level time intervals, and acquiring high-level observation information corresponding to high-level time intervals composed of K consecutive low-level time intervals; where K is an integer greater than 1; the edge base station, based on the high-level observation information, sets the video coding rate for each video acquisition terminal within the high-level time interval through a high-level decision network every K low-level time intervals; each video acquisition terminal generates a video to be transmitted and adds it to its respective video queue to be transmitted based on its video coding rate within the high-level time interval; the edge base station, based on the low-level observation information, allocates a bandwidth ratio for each video acquisition terminal within the low-level time interval through a low-level decision network at each low-level time interval; and each video acquisition terminal transmits the video in its respective video queue to the edge base station based on the total bandwidth of the edge base station within the low-level time interval and the bandwidth ratio of each video acquisition terminal within the low-level time interval.

[0016] This application acquires high-level observation information corresponding to high-level time intervals consisting of K consecutive low-level time intervals through an edge base station. It then uses a high-level decision network to set the video encoding rate for each video acquisition terminal every K low-level time intervals. Simultaneously, based on the low-level observation information corresponding to each low-level time interval, the low-level decision network dynamically allocates a bandwidth ratio for each video acquisition terminal within each low-level time interval. This enables each video acquisition terminal to generate video to be transmitted according to the set video encoding rate and transmit the video in the video queue to the edge base station based on the allocated bandwidth ratio. This effectively addresses the inherent differences in decision-making scales between rate control and bandwidth allocation, alleviating problems such as video stuttering, large transmission delays, video queue overflow, and data loss caused by channel fluctuations and bandwidth limitations. It maintains queue stability while ensuring video transmission quality and significantly reduces the computational complexity of online decision-making. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the steps of an adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning provided in an embodiment of this application. Figure 2 This is a flowchart illustrating the steps of a method for transmitting video in a video queue to an edge base station, as provided in an embodiment of this application. Figure 3 This is a graph showing the changes in the rewards of the low-level decision network, the rewards of the high-level decision network, the average video coding rate, and the average queue length during the training process, as provided in an embodiment of this application. Figure 4 This is a graph showing the changes in video encoding rate, video quality, and queue length of different cameras during the training process, as provided in an embodiment of this application. Figure 5 This is a graph showing the variation of average video quality, average queue length, and high-level decision network reward value under different methods with different trade-off parameters V, as provided in the embodiments of this application. Figure 6 This is a graph showing the variation of average video quality, average queue length, and high-level decision network reward for different methods under different total system bandwidth resources, as provided in the embodiments of this application. Figure 7This application provides a graph showing the changes in average video quality, average queue length, and high-level decision network reward value for different methods under different numbers of cameras. Figure 8 This is a schematic diagram of an adaptive video coding rate control and bandwidth allocation system based on hierarchical reinforcement learning provided in this application; Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0020] In a first aspect, this application provides an adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning, such as... Figure 1 As shown, the method includes: S101, the edge base station obtains the low-level observation information corresponding to the low-level time interval, and obtains the high-level observation information corresponding to the high-level time interval composed of K consecutive low-level time intervals; K is an integer greater than 1.

[0021] In this step, low-level observation information corresponding to low-level time intervals is obtained through edge base stations, and high-level observation information corresponding to high-level time intervals consisting of K consecutive low-level time intervals is obtained; where K is an integer greater than 1.

[0022] In this application, the low-layer time interval can be understood as the decision time interval for bandwidth allocation, and the high-layer time interval can be understood as the decision time interval for video coding rate. A high-layer time interval consists of K consecutive low-layer time intervals. The edge base station can acquire the low-layer observation information corresponding to each low-layer time interval within the high-layer time interval, and acquire the high-layer observation information corresponding to the high-layer time interval.

[0023] S102, the edge base station, based on the high-level observation information, sets the video encoding rate for each video acquisition terminal within the high-level time interval every K low-level time intervals through the high-level decision network.

[0024] In this step, the edge base station inputs the high-level observation information into the high-level decision network. Every K low-level time intervals, the high-level decision network outputs the video encoding rate corresponding to each video acquisition terminal within that high-level time interval. That is, within the same high-level time interval, each video acquisition terminal can perform video encoding according to the video encoding rate set by the edge base station.

[0025] S103, each video acquisition terminal generates a video to be transmitted and adds it to its respective video queue to be transmitted according to its video encoding rate within the high-level time interval.

[0026] In this step, after acquiring video data, each video acquisition terminal can encode the video data according to its corresponding video encoding rate to obtain the video to be transmitted, and add the video to be transmitted to its local video queue. In this application, the video queue is used to buffer video data that has not yet been transmitted to the edge base station.

[0027] S104, the edge base station, based on the low-level observation information, allocates the bandwidth ratio of each video acquisition terminal to each video acquisition terminal in each low-level time interval through the low-level decision network.

[0028] In this step, for the lower-level time interval in the higher-level time interval, the edge base station can input the lower-level observation information into the lower-level decision network, and the lower-level decision network can output the bandwidth ratio of each video acquisition terminal in each lower-level time interval.

[0029] S105, each video acquisition terminal transmits the video in its respective video queue to the edge base station according to the total bandwidth of the edge base station in the low-layer time interval and the bandwidth ratio of each video acquisition terminal in the low-layer time interval.

[0030] In this step, each video acquisition terminal determines its available transmission bandwidth within the lower-layer time interval based on the total bandwidth of the edge base station within the lower-layer time interval and its corresponding bandwidth ratio. Based on this determined transmission bandwidth, it transmits the video in the video queue to the edge base station. Therefore, the edge base station can control the video encoding rate within the higher-layer time interval and allocate bandwidth ratios within each lower-layer time interval.

[0031] This application acquires high-level observation information corresponding to high-level time intervals consisting of K consecutive low-level time intervals through an edge base station. It then uses a high-level decision network to set the video encoding rate for each video acquisition terminal every K low-level time intervals. Simultaneously, based on the low-level observation information corresponding to each low-level time interval, the low-level decision network dynamically allocates a bandwidth ratio for each video acquisition terminal within each low-level time interval. This enables each video acquisition terminal to generate video to be transmitted according to the set video encoding rate and transmit the video in the video queue to the edge base station based on the allocated bandwidth ratio. This effectively addresses the inherent differences in decision-making scales between rate control and bandwidth allocation, mitigating problems such as video stuttering, large transmission delays, video queue overflow, and data loss caused by channel fluctuations and bandwidth limitations. Ultimately, this improves video transmission quality, bandwidth utilization efficiency, and the reliability of monitoring tasks.

[0032] In one embodiment, each video acquisition terminal transmits the video in its respective video queue to the edge base station according to its bandwidth within the lower-layer time interval, including as follows: Figure 2 The steps shown are as follows: S201, each video acquisition terminal determines its uplink transmission rate within the low-level time interval based on its bandwidth and channel gain within the low-level time interval.

[0033] In this step, assuming any video acquisition terminal j, the bandwidth allocated by the edge base station to video acquisition terminal j within the lower-layer time interval is denoted as . , where n represents the slot number of the current low-layer time interval, and the channel gain between the video acquisition terminal j and the edge base station within the current low-layer time interval is denoted as . .

[0034] Then, the bandwidth allocated by the edge base station to the video acquisition terminal j within the lower-layer time interval is denoted as... The channel gain between the video acquisition terminal j and the edge base station during the lower-layer time interval is denoted as . Determine the uplink transmission rate of video acquisition terminal j within the lower layer time interval. The uplink transmission rate can be expressed by the following formula: (1) in, Indicates the transmit power of a given terminal; This represents the power spectral density of additive white Gaussian noise.

[0035] S202, each video acquisition terminal determines its own video transmission amount within the low-layer time interval based on the duration of the low-layer time interval and its own uplink transmission rate within the low-layer time interval.

[0036] In this step, the uplink transmission rate is determined. Then, further based on the duration of the lower-level time interval Determine the amount of video data that video acquisition terminal j can theoretically transmit within this low-level time interval, i.e. Considering that at the start of the lower-level time interval, the amount of existing video data in the video queue to be transmitted of video acquisition terminal j is... Therefore, the actual amount of video transmitted by video acquisition terminal j within the lower time interval is... This can be expressed by the following formula: (2) S203, each video acquisition terminal transmits the video in its respective video queue to the edge base station according to the video transmission volume in the lower layer time interval.

[0037] In this step, each video acquisition terminal can retrieve the corresponding amount of video data from its respective video queue to be transmitted according to the video transmission volume determined by itself in the lower layer time interval, and send it to the edge base station through the wireless channel.

[0038] In one embodiment, each video acquisition terminal generates a video to be transmitted and adds it to its respective video queue to be transmitted based on its respective video encoding rate within the high-level time interval, including: Each video acquisition terminal determines its video encoding rate in the high-level time interval as its video encoding rate in the corresponding K consecutive low-level time intervals. Each video acquisition terminal encodes the video data acquired in its respective low-level time interval according to its own video encoding rate in the low-level time interval, generates a video to be transmitted, and adds it to its own video queue to be transmitted. The method further includes: Each video acquisition terminal determines the amount of video generated within the low-level time interval based on the duration of the low-level time interval and its respective video encoding rate within the low-level time interval. Each video acquisition terminal determines the length of its video queue to be transmitted in the next low-level time interval based on the length of its video queue to be transmitted, the amount of video transmitted, and the amount of video generated in the current low-level time interval.

[0039] In this embodiment, for any video acquisition terminal j, the video encoding rate of the video acquisition terminal j in its corresponding higher-level time interval can be determined as the video encoding rate of the video acquisition terminal j in the lower-level time interval. .

[0040] The video capture terminal j operates according to its video encoding rate within its lower time interval. The video data acquired within the lower time interval is encoded to generate a video to be transmitted, and the generated video to be transmitted is added to the video queue to be transmitted of the video acquisition terminal j.

[0041] Furthermore, assume the duration of the lower-level time interval is... The video acquisition terminal j is based on the video encoding rate. Duration of the time interval with the lower layer Determine the amount of video generated within the lower-level time interval, which can be expressed as: .

[0042] Furthermore, assume that the length of the video queue to be transmitted at the beginning of the lower-level time interval for video acquisition terminal j is... Based on the above embodiments, the video transmission volume of video acquisition terminal j within the lower-level time interval is calculated as follows: And the video acquisition terminal j according to the video encoding rate Duration of the time interval with the lower layer The calculated video generation volume within the lower-level time interval Determine the length of the video queue to be transmitted for the video acquisition terminal j in the next lower-level time interval.

[0043] In this embodiment, on the video acquisition terminal side, each video acquisition terminal j maintains an independent queue of video data to be transmitted. During the lower time interval, the video acquisition terminal j... The encoded video stream is generated at a rate and injected into the end of the queue, while simultaneously... The uplink transmission rate sends the data at the front of the queue to the edge base station. Considering the causality of data processing (i.e., new data generated in the current time interval cannot be sent immediately), the length of the video queue to be transmitted by the video acquisition terminal j in the next lower-level time interval is shown in the following formula: (3) in, This represents the length of the video queue to be transmitted by video acquisition terminal j in the (n+1)th lower-level time interval.

[0044] In one embodiment, the low-level observation information includes: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the low-level time interval, and the video coding rate of each video acquisition terminal in the high-level time interval. The video encoding rate of a video acquisition terminal within K low-level time intervals of the same high-level time interval is the video encoding rate set by the high-level decision network for the video acquisition terminal within that high-level time interval. The high-level observation information includes: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the high-level time interval, and the video coding rate of each video acquisition terminal in the previous high-level time interval. The length of the video queue to be transmitted and the channel gain of a video acquisition terminal in a higher-level time interval are equal to the length of the video queue to be transmitted and the channel gain of the video acquisition terminal in the first lower-level time interval of that higher-level time interval.

[0045] In this embodiment, the low-layer observation information includes the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the low-layer time interval, as well as the video coding rate of each video acquisition terminal in the current high-layer time interval. In this embodiment, within the K low-level time intervals contained in the same high-level time interval, for any video acquisition terminal, the video encoding rate of the video acquisition terminal is the video encoding rate set by the high-level decision network for the video acquisition terminal within the high-level time interval. That is, after the high-level decision network sets the video encoding rate for each video acquisition terminal at the beginning of a high-level time interval, the video encoding rate can remain effective within the K low-level time intervals corresponding to the high-level time interval. When the low-level decision network allocates bandwidth ratios within each low-level time interval, it can use the video encoding rate as part of the low-level observation information.

[0046] In this embodiment, the higher-level observation information includes the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the current higher-level time interval, as well as the video coding rate of each video acquisition terminal in the previous higher-level time interval. For any video acquisition terminal, the length of the video queue to be transmitted and the channel gain of that video acquisition terminal in the current higher-level time interval are equal to the length of the video queue to be transmitted and the channel gain of that video acquisition terminal in the first lower-level time interval of the current higher-level time interval.

[0047] In one embodiment, the high-level training network and the low-level training network are trained alternately with the goal of minimizing the variation in the length of the video queue to be transmitted by each video acquisition terminal and maximizing the video quality of the video transmitted by each video acquisition terminal to the edge base station. The alternating training process includes: With the model parameters of the high-level network to be trained frozen, the low-level network obtains low-level training samples through interaction with the environment, and uses the low-level training samples to perform low-level reinforcement learning training on the low-level network to be trained, thus obtaining the low-level network to be trained with updated parameters. With the model parameters of the lower-level network to be trained frozen after the parameter update, the higher-level network obtains high-level training samples through interaction with the environment, and uses the high-level training samples to perform high-level reinforcement learning training on the higher-level network to be trained, thus obtaining the higher-level network to be trained after parameter update. Repeat the above process to update the high-level decision network and the low-level decision network.

[0048] In this embodiment, the high-level decision network and the low-level decision network are obtained by alternating training of the high-level network to be trained and the low-level network to be trained with the goal of minimizing the change in the length of the video queue to be transmitted by each video acquisition terminal and maximizing the video quality of the video transmitted by each video acquisition terminal to the edge base station.

[0049] During the alternating training process, the model parameters of the high-level network to be trained can be frozen first, so that the high-level network to be trained does not update its parameters. The low-level network obtains low-level training samples through interaction with the environment, and uses the low-level training samples to perform low-level reinforcement learning training on the low-level network to be trained, resulting in the low-level network to be trained with updated parameters. Then, the model parameters of the low-level network to be trained with updated parameters are frozen, so that the low-level network to be trained does not update its parameters. The high-level network obtains high-level training samples through interaction with the environment, and uses the high-level training samples to perform high-level reinforcement learning training on the high-level network to be trained, resulting in the high-level network to be trained with updated parameters. In this embodiment, the environment represents the current wireless communication environment. Repeat the alternating update process of low-level training and high-level training to update both the high-level and low-level decision networks.

[0050] In this embodiment, when performing low-level reinforcement learning training on the low-level network to be trained, the low-level network to be trained can first collect N... low After testing with low-level training samples, parameters are updated; when training high-level reinforcement learning on the network to be trained, the high-level network to be trained can collect N samples. highAfter collecting one high-level training sample, the parameters are updated. Since the decision-making time scales of the high-level and low-level training networks are different, for every high-level training sample collected by the high-level training network, the low-level training network can collect K corresponding low-level training samples. Through this alternating training method, the high-level and low-level training networks can update their parameters in relatively stable training environments.

[0051] In this embodiment, since the high-level and low-level training networks correspond to different decision time scales, the two decision processes of coding rate control and bandwidth allocation are processed in layers. This allows the high-level training network to focus on learning video coding rate control strategies at longer time scales, while the low-level training network focuses on learning bandwidth allocation strategies at shorter time scales, thereby achieving coordinated optimization of video coding rate and bandwidth allocation.

[0052] In one embodiment, the observation space of the low-level network to be trained is: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the sample low-level time interval, and the video coding rate of each video acquisition terminal in the sample high-level time interval to which the sample low-level time interval belongs. The action space of the low-level network to be trained is: the bandwidth allocation weight of each video acquisition terminal in the low-level time interval of the sample. The reward function value of the low-level network to be trained represents the change between the length of the video queue to be transmitted by each video acquisition terminal in the current low-level time interval of the sample and the length of the video queue to be transmitted in the next low-level time interval of the sample.

[0053] In this application, the low-level network to be trained is used to learn the bandwidth allocation strategy of each video acquisition terminal in the low-level time interval of the samples.

[0054] In this embodiment, the observation space of the low-level network to be trained includes the length of the video queue to be transmitted, the channel gain, and the video coding rate of each video acquisition terminal in the high-level time interval to which the low-level time interval belongs.

[0055] Among them, the observation space of the low-level network to be trained This can be expressed by the following formula: (4) in, This indicates the target coding rate set by the high-level decision-making network.

[0056] In this application, K low-level time intervals constitute a high-level decision cycle. The target coding rate refers to the target coding rate set by the high-level decision network at the high-level decision moment corresponding to this cycle. This target coding rate will be used by each low-level time interval in the next high-level time interval. That is, the target coding rate output by the high-level decision network is not only used for a single low-level time interval, but is continuously effective within the corresponding K low-level time intervals; the low-level decision network needs to use this target coding rate when allocating bandwidth in each of these K low-level time intervals.

[0057] The action space of the low-level network to be trained consists of the bandwidth weights allocated to each video acquisition terminal within the sample low-level time interval.

[0058] Among them, the action space of the lower-level network to be trained This can be expressed by the following formula: (5) Among them, satisfying W is the total system bandwidth, and, .

[0059] The reward function value of the low-level network under training is used to characterize the change between the length of the video queue to be transmitted at each video acquisition terminal in the nth sample low-level time interval and the length of the video queue to be transmitted in the (n+1)th sample low-level time interval. This reward function is used to guide the low-level reinforcement learning algorithm to learn a reasonable bandwidth allocation strategy to reduce queue backlog.

[0060] Specifically, if the length of the video queue to be transmitted by each video acquisition terminal in the (n+1)th sample low-level time interval decreases relative to the nth sample low-level time interval, it indicates that the bandwidth allocation weights of the low-level network to be trained are beneficial to alleviating queue backlog; if the length of the video queue to be transmitted by each video acquisition terminal in the (n+1)th sample low-level time interval increases relative to the nth sample low-level time interval, it indicates that the current bandwidth allocation result may lead to continued queue backlog.

[0061] The reward function of the lower-level network to be trained can be expressed by the following formula: (6) In one embodiment, the observation space of the high-level network to be trained is: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the sample high-level time interval, and the video coding rate of each video acquisition terminal in the sample high-level time interval. The action space of the high-level network to be trained is: the video encoding rate of each video acquisition terminal within the sample high-level time interval. The reward function value of the high-level network to be trained includes: the video quality utility value of each video acquisition terminal in the high-level time interval of the sample, and the sum of the reward function values ​​corresponding to the K low-level time intervals of the sample in the high-level time interval. There is a corresponding relationship between the video quality utility value of each video acquisition terminal in the sample high-level time interval and the video encoding rate of each video acquisition terminal in the sample high-level time interval. The corresponding relationship is that the video quality utility value increases monotonically with the video encoding rate and the marginal effect decreases.

[0062] In this application, the high-level training network is used to learn the video encoding rate setting strategy of each video acquisition terminal within the sample high-level time interval.

[0063] In this embodiment, the observation space of the high-level network to be trained includes: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the sample high-level time interval, and the video coding rate of each video acquisition terminal in the previous sample high-level time interval.

[0064] The observation space of the high-level network to be trained can be represented by the following formula: (7) in, , where k represents the k-th high-level time interval.

[0065] The action space of the high-level network to be trained is the video encoding rate of each video acquisition terminal within the sample high-level time interval.

[0066] The action space of the high-level network to be trained can be represented by the following formula: (8) Among them, satisfying .

[0067] The reward function value of the high-level network to be trained includes the video quality utility value of each video acquisition terminal within the sample high-level time interval, and the cumulative value of the reward function values ​​corresponding to K sample low-level time intervals within that sample high-level time interval. There is a correlation between the video quality utility value and the video coding rate; the video quality utility value monotonically increases with the video coding rate, but its marginal effect decreases.

[0068] The reward function of the high-level network to be trained can be expressed by the following formula (9): (9) in, Indicates the trade-off parameters, VThe larger the value, the more emphasis is placed on video quality. This represents the video quality weighting coefficient corresponding to the j-th video acquisition terminal.

[0069] In this application, both the high-level and low-level training networks employ two fully connected layers with 64 hidden neurons and Tanh activation function. The output of the low-level decision network uses Softmax to generate the bandwidth allocation ratio, which is then multiplied by the total bandwidth to obtain the specific bandwidth value. The high-level decision network directly outputs the encoding rate.

[0070] In this application, the high-level decision network operates with a large time step, dynamically adjusting the target coding rate of each video acquisition terminal based on global channel quality, queue length, and current video bitrate, and is responsible for bitrate decision-making on a slow time scale. The low-level decision network operates in each time slot, performing millisecond-level fine-grained bandwidth allocation based on instantaneous channel gain, queue status, and the bitrate target issued by the high-level network. Through the decoupling and coordination between the long-term bitrate adjustment of the high-level decision network and the instantaneous bandwidth allocation of the low-level decision network, the problem of the inherent difference in the decision-making time scale between joint optimization bitrate control and bandwidth allocation is effectively solved.

[0071] In this application, the performance of the proposed algorithm in video transmission is verified through experiments. First, the parameter settings of the simulation environment and the training parameters of the reinforcement learning algorithm during training are given, along with a convergence analysis of the reinforcement learning algorithm. Second, the performance of the proposed algorithm is compared with that of other algorithms under different conditions to verify the effectiveness and robustness of the proposed algorithm.

[0072] Among them, the camera data M is set to 5, the total system bandwidth W is set to 8MHz, and the video encoding rate range is... and Set to 1Mbps and 8Mbps respectively, with the time interval... Set to 0.01s, K is set to 20, and the channel gain is adjusted for different cameras. The parameters of different cameras satisfy a uniform distribution within the interval [0.0001, 0.1]. The values ​​were set to 1.0, 1.5, 2.0, 2.5, and 3.0 respectively, with V set to 0.5. To ensure stable training of the reinforcement learning algorithm, the low-level decision network... The value is set to 600 (to match the environment's K, which is a multiple of 20), the update rounds are 10, the mini-batch size is 128, and the learning rate is 3×10. 4 Entropy regularization coefficient Set to 0.01, value loss coefficient Set to 0.5. High-level decision-making network. The settings are set to 256, update rounds to 10, mini-batch size to 64, and learning rate to 3×10. 5 Entropy regularization coefficient Set to 0.001, value loss coefficient Set to 0.5. Discount factor for high-level and low-level decision networks. Set to 0.99, GAE parameter Set to 0.95 and the pruning parameter to 0.2. It should be noted that during the alternating training phase of the following simulation experiment results, the target bitrate of the upper layer will not be adjusted during the update of the lower layer PPO parameter. Instead, the bitrate of the upper layer will be kept fixed (or restored to the initial bitrate configuration after the environment is reset). During the testing phase, the upper layer bitrate control and lower layer bandwidth allocation will be executed according to the complete hierarchical decision-making process.

[0073] Figure 3 This graph shows the changes in rewards for the low-level decision network, high-level decision network, average video coding rate, and average queue length during the training process of this application. The horizontal axis represents the index of the alternating training phases. Each phase consists of one high-level PPO training cycle and one low-level PPO training cycle, with a test performed after each phase. The graph shows that as training progresses, the rewards for both the low-level and high-level decision networks gradually increase and then stabilize, indicating that both networks learn optimal strategies through interaction with the environment. The average video coding rate and average queue length gradually increase and then stabilize, demonstrating that the high-level decision network learns to increase the system's video coding rate to obtain higher rewards, while the low-level decision network adjusts its strategy according to the changes in the high-level decision network's strategy, maintaining a lower average queue length. Figure 4 The figure illustrates the changes in video encoding rate, video quality, and queue length for different cameras during training. As training progresses, the video encoding rate and video quality of different cameras gradually increase and remain stable, while the queue length also gradually increases and remains stable with the increase in video encoding rate. This demonstrates that the high-level decision network, through interaction with the environment, has learned that providing higher encoding rates to cameras yields higher high-level rewards. However, due to… V The existence of this necessitates a balance between video quality and system queue length in the high-level decision-making network. Therefore, video encoding rate and quality ultimately stabilize within a certain range of fluctuations. Comparing the video encoding rate, video quality, and queue length of different cameras reveals that… Larger cameras will achieve higher video encoding rates and video quality, but their queue lengths will also fluctuate within a wider range. This is because... This represents the sensitivity of video quality to video encoding rate. The larger the value, the more sensitive the video quality is to the video encoding rate, and therefore the higher the video encoding rate will be assigned.

[0074] To verify the effectiveness of the hierarchical reinforcement learning method proposed in this application, the following four baseline algorithms are selected for comparison with the proposed RL algorithm: RL (the method presented in this paper): The high-level decision network outputs the video coding rate of different cameras based on environmental observations (queue length of different cameras, channel gain, and previous video coding rate); the low-level decision network outputs the bandwidth allocation ratio based on environmental observations, which is then normalized by Softmax and multiplied by the total bandwidth.

[0075] JRCA: Joint Rate Control and Bandwidth Allocation Algorithm, also known as the joint algorithm. A high-level decision network solves for the optimal video coding rate using a closed-form solution; a low-level decision network iteratively solves for bandwidth allocation using a binary search. This method relies on an accurate system model and has high computational complexity.

[0076] Random: A randomized algorithm. The high-level decision network samples uniformly and randomly within the video coding rate range [1, 8]; the low-level decision network generates random weights and normalizes them.

[0077] Average: An average allocation algorithm. In the high-level decision network, all cameras have a fixed video encoding rate of 4.5; in the low-level decision network, all cameras have bandwidth allocated equally.

[0078] Greedy: A greedy algorithm. The video encoding rate of the high-level decision network is inversely proportional to the queue length: clip(8.0-queue / 10, 1, 8), meaning that the longer the queue, the lower the video encoding rate is set, and vice versa; the bandwidth allocation ratio of the lower layers is directly proportional to the queue length.

[0079] Evaluation metrics include system utility (average video quality, average queue length (reflecting system latency and stability) and high-level rewards, and will also compare the computational latency of JRCA and RL methods.

[0080] Figure 5 Table 1 shows the different trade-off parameters. V Average video quality, average queue length, and high-level decision network reward value under different methods. VUnder these conditions, both the RL algorithm proposed in this chapter and the JRCA method derived mathematically can maintain a balance between video quality and average queue length. The overall average video quality of the RL algorithm is slightly lower than that of the JRCA algorithm, but its overall performance can reach over 97% of JRCA, as evidenced by the high-level decision network rewards. Since the JRCA algorithm represents a mathematical upper bound, this effectively proves that the RL algorithm can learn optimal bitrate control and resource allocation strategies through interaction with the environment. While the Random and Average methods achieve high video quality, they also have extremely high average queue lengths, indicating that these two bitrate control and resource allocation methods cannot achieve stable average queue lengths. The Greedy method balances video quality and average queue length more effectively than the Random and Average methods, but it has lower overall average video quality and a significantly increased average queue length compared to the RL and JRCA algorithms. V As the value increases, the average video quality of both the RL and JRCA methods generally shows a trend of first increasing and then stabilizing, while the average queue length decreases with increasing value. V The value gradually increases as it increases. This is because at larger values... V At higher bitrates, the system tends to achieve higher average video quality. To achieve this, the RL and JRCA methods set higher bitrates for the camera, which in turn increases the average queue length.

[0081] Table 1 shows the high-level decision network rewards for each method under different V value settings.

[0082]

[0083] Table 1 Figure 6 Table 2 shows the average video quality, average queue length, and high-level decision network reward for different methods under different total system bandwidth resources. It can be seen that while the Random and Average methods for bitrate control and bandwidth allocation can achieve higher video quality, they also have extremely high average queue lengths, indicating that these two methods cannot balance system video quality to obtain a stable average queue length. Although the heuristic greedy strategy can better balance system video quality and average queue length to some extent, it still has a high average queue length, therefore its high-level decision network reward value is lower than that of the RL-based method and the JRCA method. The RL-based method achieves better average video quality, lower average queue length, and higher high-level decision network reward than the JRCA method under different total system bandwidth resources.

[0084] Table 2 shows the high-level decision network rewards for each method under different total bandwidths.

[0085]

[0086] Table 2 Figure 7 Table 3 shows the changes in average video quality, average queue length, and high-level decision network reward for different methods under different numbers of cameras. It can be seen that the Random and Average methods for bitrate control and bandwidth allocation cannot achieve a balance between average video quality and average queue length because these two methods cannot adjust the bitrate control and bandwidth allocation strategies according to dynamic changes in the environment. The heuristic greedy method can effectively improve the efficiency of bitrate control and resource allocation compared to the above two methods, obtaining a higher high-level decision network reward; however, it has a higher average queue length at lower average video quality, and its performance is still lower than that of the RL-based method and the JRCA method. The RL-based method can effectively balance average video quality and average queue length, achieving performance similar to JRCA, indicating that the RL-based method can learn a near-optimal strategy through interactive learning with the environment.

[0087] Table 3 shows the high-level decision network rewards for each method under different numbers of cameras.

[0088]

[0089] Table 3 The above are different V Comparative experiments under different system bandwidth resources and different numbers of cameras all demonstrate that the RL-based method can learn performance approximating or exceeding that of JRCA through interaction with the environment. Based on the above experiments, this application also compares the CPU computation time of the traditional iterative algorithm JRCA and the RL algorithm when performing 20,000 decision steps, as shown in Table 4.

[0090] Table 4 compares the computation time of the algorithm for different numbers of cameras.

[0091]

[0092] Table 4 As shown in Table 4, the computation time of JRCA increases significantly with the increase in camera size. In contrast, the computation time of the reinforcement learning model remains stable at around 4.8 seconds, since the inference phase only involves a few layers of forward propagation in the neural network. Overall, RL-based methods can improve decision-making speed by 50-80 times.

[0093] To verify the effectiveness of the proposed algorithm, this application compares it with four baseline algorithms: JRCA, Random, Average, and Greedy. Experimental results show that during training, the rewards for low-level decision networks, high-level decision networks, the system's average bitrate, and the average queue length all gradually increase and tend to stabilize. The bitrate and video quality of different cameras are correlated with their content sensitivity coefficients. The positive correlation indicates that the algorithm can adaptively learn reasonable resource scheduling strategies. Under different trade-off parameters V, the overall performance of the proposed RL algorithm can reach more than 97% of the JRCA (mathematical upper bound), and significantly outperforms the Greedy, Random, and Average methods. Under different total system bandwidths and different numbers of cameras, the proposed RL algorithm can also effectively balance average video quality and average queue length, obtaining high-level decision network rewards similar to JRCA, and its performance is consistently better than other baseline algorithms. In terms of computational time, since the reinforcement learning model inference only involves a small number of neural network forward propagations, the decision time of the proposed RL algorithm is stable at about 4.8 seconds (20,000 steps), which is 50-80 times better than the JRCA algorithm that requires iterative solutions, fully meeting the real-time scheduling requirements of practical systems.

[0094] This application employs a hierarchical architecture where a higher layer determines the bitrate and a lower layer allocates bandwidth. This effectively addresses the inherent time-scale differences between bitrate control and bandwidth allocation: the higher-layer decision network operates with large time steps (every K time intervals), dynamically adjusting the encoding rate of each camera based on channel quality, queue length, and historical bitrate information; the lower-layer decision network operates in each time interval, efficiently allocating bandwidth based on instantaneous channel gain, queue status, and the bitrate target set by the higher layer. This decouples long-term queue stability from the goal of maximizing video quality and transforms it into a hierarchical reinforcement learning model with single-step reward functions for both the higher and lower layers, effectively guiding the alternating training and convergence of the high- and low-layer decision networks.

[0095] Based on the same inventive concept, a second aspect of the embodiments of this application provides an adaptive video coding rate control and bandwidth allocation system based on hierarchical reinforcement learning, such as... Figure 8 As shown, the system includes: The acquisition module 301 is used to acquire low-level observation information corresponding to low-level time intervals through edge base stations, and to acquire high-level observation information corresponding to high-level time intervals composed of K consecutive low-level time intervals; K is an integer greater than 1. The setting module 302 is used to set the video encoding rate of each video acquisition terminal in the high-level time interval by means of the edge base station according to the high-level observation information, every K low-level time interval, through the high-level decision network. The generation module 303 is used to generate videos to be transmitted and add them to their respective video queues based on the video encoding rate of each video acquisition terminal within the high-level time interval. The allocation module 304 is used to allocate the bandwidth ratio of each video acquisition terminal in each low-level time interval to the edge base station based on the low-level observation information and through the low-level decision network in each low-level time interval. The transmission module 305 is used to transmit the video in each video queue to the edge base station according to the total bandwidth of the edge base station in the low-layer time interval and the bandwidth ratio of each video acquisition terminal in the low-layer time interval.

[0096] Based on the same inventive concept, a third aspect of the embodiments of this application provides a method as follows: Figure 9 The electronic device 100 shown includes a processor 120, a memory 110, and a program or instructions stored in the memory 110 and executable on the processor 120. When the program or instructions are executed by the processor 120, they implement the steps of the adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning as described in the first aspect of this application.

[0097] Based on the same inventive concept, in a fourth aspect of this application, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning as described in the first aspect of this application.

[0098] Each embodiment in this specification focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0104] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0105] The above provides a detailed description of the adaptive video coding rate control and bandwidth allocation method, system, device, and medium based on hierarchical reinforcement learning. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning, characterized in that, The method includes: Edge base stations acquire low-level observation information corresponding to low-level time intervals, and acquire high-level observation information corresponding to high-level time intervals composed of K consecutive low-level time intervals; K is an integer greater than 1. Based on the high-level observation information, the edge base station sets the video encoding rate for each video acquisition terminal within the high-level time interval every K low-level time intervals through the high-level decision network. Each video acquisition terminal generates a video to be transmitted and adds it to its respective video queue to be transmitted based on its video encoding rate within the high-level time interval. Based on the low-level observation information, the edge base station allocates the bandwidth ratio of each video acquisition terminal to each video acquisition terminal in each low-level time interval through the low-level decision network. Each video acquisition terminal transmits the video in its respective video queue to the edge base station based on the total bandwidth of the edge base station within the low-layer time interval and the bandwidth ratio of each video acquisition terminal within the low-layer time interval.

2. The adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning according to claim 1, characterized in that, Each video acquisition terminal transmits the video in its respective video queue to the edge base station according to its bandwidth within the lower-layer time interval, including: Each video acquisition terminal determines its uplink transmission rate within the lower-layer time interval based on its bandwidth and channel gain within that lower-layer time interval. Each video acquisition terminal determines its own video transmission volume within the low-layer time interval based on the duration of the low-layer time interval and its own uplink transmission rate within the low-layer time interval. Each video acquisition terminal transmits the video in its respective video queue to the edge base station according to the video transmission volume within the lower-level time interval.

3. The adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning according to claim 2, characterized in that, Each video acquisition terminal generates a video to be transmitted and adds it to its respective video queue to be transmitted based on its video encoding rate within the high-level time interval, including: Each video acquisition terminal determines its video encoding rate in the high-level time interval as its video encoding rate in the corresponding K consecutive low-level time intervals. Each video acquisition terminal encodes the video data acquired in its respective low-level time interval according to its own video encoding rate in the low-level time interval, generates a video to be transmitted, and adds it to its own video queue to be transmitted. The method further includes: Each video acquisition terminal determines the amount of video generated within the low-level time interval based on the duration of the low-level time interval and its respective video encoding rate within the low-level time interval. Each video acquisition terminal determines the length of its video queue to be transmitted in the next low-level time interval based on the length of its video queue to be transmitted, the amount of video transmitted, and the amount of video generated in the current low-level time interval.

4. The adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The low-level observation information includes: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the low-level time interval, and the video coding rate of each video acquisition terminal in the high-level time interval. The video encoding rate of a video acquisition terminal within K low-level time intervals of the same high-level time interval is the video encoding rate set by the high-level decision network for the video acquisition terminal within that high-level time interval. The high-level observation information includes: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the high-level time interval, and the video coding rate of each video acquisition terminal in the previous high-level time interval. The length of the video queue to be transmitted and the channel gain of a video acquisition terminal in a higher-level time interval are equal to the length of the video queue to be transmitted and the channel gain of the video acquisition terminal in the first lower-level time interval of that higher-level time interval.

5. The adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The method further includes: The training method aims to minimize the variation in the length of the video queues to be transmitted from each video acquisition terminal and maximize the video quality of the videos transmitted from each video acquisition terminal to the edge base station. This is achieved by alternately training the high-level and low-level training networks. The alternate training process includes: With the model parameters of the high-level network to be trained frozen, the low-level network obtains low-level training samples through interaction with the environment, and uses the low-level training samples to perform low-level reinforcement learning training on the low-level network to be trained, thus obtaining the low-level network to be trained with updated parameters. With the model parameters of the lower-level network to be trained frozen after the parameter update, the higher-level network obtains higher-level training samples through interaction with the environment, and uses the higher-level training samples to perform higher-level reinforcement learning training on the higher-level network to be trained, thus obtaining the higher-level network to be trained after parameter update. Repeat the above process to update the high-level decision network and the low-level decision network.

6. The adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning according to claim 5, characterized in that, The observation space of the low-level network to be trained is: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the low-level time interval of the sample, and the video coding rate of each video acquisition terminal in the high-level time interval of the sample to which the low-level time interval belongs. The action space of the low-level network to be trained is: the bandwidth allocation weight of each video acquisition terminal in the low-level time interval of the sample. The reward function value of the low-level network to be trained represents the change between the length of the video queue to be transmitted for each video acquisition terminal in the current low-level time interval of the sample and the length of the video queue to be transmitted in the next low-level time interval of the sample.

7. The adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning according to claim 5, characterized in that, The observation space of the high-level network to be trained is: the length of the video queue to be transmitted and the channel gain of each video acquisition terminal in the sample high-level time interval, and the video coding rate of each video acquisition terminal in the sample high-level time interval. The action space of the high-level network to be trained is: the video encoding rate of each video acquisition terminal within the sample high-level time interval. The reward function value of the high-level network to be trained includes: the video quality utility value of each video acquisition terminal in the high-level time interval of the sample, and the sum of the reward function values ​​corresponding to the K low-level time intervals of the sample in the high-level time interval. There is a corresponding relationship between the video quality utility value of each video acquisition terminal in the sample high-level time interval and the video encoding rate of each video acquisition terminal in the sample high-level time interval. The corresponding relationship is that the video quality utility value increases monotonically with the video encoding rate and the marginal effect decreases.

8. An adaptive video coding rate control and bandwidth allocation system based on hierarchical reinforcement learning, characterized in that, The system includes: The acquisition module is used to acquire low-level observation information corresponding to low-level time intervals through edge base stations, and to acquire high-level observation information corresponding to high-level time intervals composed of K consecutive low-level time intervals; K is an integer greater than 1. The setting module is used to set the video encoding rate of each video acquisition terminal in the high-level time interval by means of the edge base station based on the high-level observation information, every K low-level time interval, through the high-level decision network. The generation module is used to generate videos to be transmitted and add them to their respective video queues based on the video encoding rate of each video acquisition terminal within the high-level time interval. The allocation module is used to allocate bandwidth proportions for each video acquisition terminal in each low-level time interval through the low-level decision network based on the low-level observation information from the edge base station. The transmission module is used to transmit the video in each video queue to the edge base station according to the total bandwidth of the edge base station in the low-layer time interval and the bandwidth ratio of each video acquisition terminal in the low-layer time interval.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the adaptive video coding rate control and bandwidth allocation method based on hierarchical reinforcement learning as described in any one of claims 1-7.

Images