Model training method, method for determining video stream scheduling strategy, and electronic device

Abstract

The application discloses a model training method, a method for determining a video stream scheduling strategy and an electronic device. The model training method comprises the following steps: inputting at least one state vector sequence into a buffer estimation model to obtain a first probability and a first number of buffer emptying corresponding to each state vector sequence; the state vector sequence represents a state vector corresponding to simultaneous transmission of data packets of multiple video streams by a base station at different time points, and each state vector represents multiple network state parameters; based on the calibration probability, the calibration number, the first probability and the first number corresponding to each state vector sequence in the at least one state vector sequence, a loss value of the buffer estimation model is calculated; and the weight parameters of the buffer estimation model are updated according to the calculated loss value. The buffer estimation model trained by using the above model training method can accurately predict the probability and number of buffer emptying in the scene of simultaneous pushing of multiple videos.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a model training method, a method for determining video stream scheduling strategies, and an electronic device. Background Technology

[0002] In recent years, numerous measurement studies have provided new design space for improving the Quality of Experience (QoE) of video streaming services in cellular networks. On the one hand, the periodicity and variability of video streaming service patterns can be leveraged to design time-related resource allocation strategies based on historical information; on the other hand, network operators can set different priorities for short-view and long-view services, thereby enhancing the QoE of video streaming services without incurring complex algorithm design or significant costs.

[0003] To orchestrate QoE-oriented scheduling strategies, a theoretical model is needed to capture the probability of buffer starvation (or buffer emptying) in video streaming services. Assuming geographical homogeneity of user behavior, observations are mapped to users located in the same wireless cell, which helps to detect the QoE metric for each video stream from a micro perspective. This QoE metric includes the probability of buffer starvation. Video streams are distinguished as short or long streams based on a super-exponential distribution of viewing time. Based on the QoE metric values, a weighted proportional fairness algorithm and a weighted round-robin algorithm are used to schedule short and long streams at the base station (BS).

[0004] However, only a single video stream is considered to have a fixed video length (in seconds) or size (in bits), while the video length or size of multiple video streams is dynamically changing. Therefore, in related technologies, the video stream scheduling method on the base station side is not suitable for scenarios where multiple videos are pushed simultaneously. Summary of the Invention

[0005] In view of this, embodiments of this application provide a model training method, a method for determining video stream scheduling strategies, and an electronic device to solve the technical problem that the video stream scheduling method on the base station side in related technologies is not applicable to scenarios where multiple videos are pushed simultaneously.

[0006] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0007] This application provides a model training method, including:

[0008] At least one state vector sequence is input into the buffer estimation model to obtain the first probability and the first number of the first buffer becoming empty for each state vector sequence; wherein, the state vector sequence represents the state vector corresponding to the base station transmitting data packets of multiple video streams at different times, each state vector represents multiple network state parameters, and the first buffer represents the video buffer of the video stream server.

[0009] Based on the calibration probability, calibration count, first probability, and first count corresponding to each state vector sequence in the at least one state vector sequence, the loss value of the buffer estimation model is calculated.

[0010] The weight parameters of the buffer estimation model are updated based on the calculated loss value.

[0011] In the above scheme, the buffer estimation model includes a recurrent neural network, a bidirectional gated recurrent neural network, and a multi-task prediction network; when inputting at least one state vector sequence into the buffer estimation model to obtain the first probability and the first number of the first buffer becoming empty corresponding to each state vector sequence, the method includes:

[0012] The state vector sequence is input into the recurrent neural network to obtain the real number corresponding to each state vector in the state vector sequence, and attention weights are applied to the real number corresponding to each state vector in the state vector sequence to obtain the first feature vector; wherein, the real number is obtained by feature compression of the state vector; the real number represents the global feature of the state vector;

[0013] The first feature vector is input into the bidirectional gated recurrent neural network to obtain the second feature vector; the second feature vector represents the correlation between state vectors in the state vector sequence.

[0014] The second feature vector is input into the multi-task prediction network to obtain the first probability and the first number of the first buffer becoming empty corresponding to the state vector sequence.

[0015] In the above scheme, calculating the loss value of the buffer estimation model based on the calibration probability, calibration count, first probability, and first count corresponding to each state vector sequence in the at least one state vector sequence includes:

[0016] The first loss value is calculated based on the calibration number and the first time number corresponding to each state vector sequence in the at least one state vector sequence;

[0017] The second loss value is calculated based on the calibration probability and the first probability corresponding to each state vector sequence in the at least one state vector sequence.

[0018] The loss value of the buffer estimation model is calculated based on the product of the first coefficient and the first loss value, and the product of the second coefficient and the second loss value.

[0019] In the above scheme, the first loss value represents the mean squared error; the second loss value represents the cross-entropy.

[0020] This application also provides a method for determining a video stream scheduling strategy, including:

[0021] The first state vector sequence is input into the first model to obtain the second probability and the second number of times the second buffer corresponding to the first state vector sequence becomes empty; the first state vector sequence represents the state vector corresponding to the base station when it receives data packets in the data packet queues corresponding to multiple video streams, each state vector represents multiple network state parameters, and the second buffer represents the video buffer of the video stream server.

[0022] A reward function is constructed based on channel traffic, the total number of data packets in the data packet queue, the second probability, the second count, and the historical state vectors corresponding to different flow states of the marked video stream.

[0023] Multiple historical state vectors and their corresponding scheduling policies are input into the reinforcement learning model, and the total reward for executing each scheduling policy is calculated using the reward function.

[0024] The value gradient is calculated based on the third probability and the first value; wherein the third probability represents the probability of selecting the corresponding target scheduling strategy for the target historical state vector corresponding to the maximum total reward; and the first value represents the value of executing the target scheduling strategy.

[0025] The weight parameters of the reinforcement learning model are updated based on the value gradient;

[0026] When the reinforcement learning model reaches the set convergence condition, the first state vector sequence is input into the reinforcement learning model after updating the weight parameters to obtain the scheduling policy that maximizes the total reward for each state vector in the first state vector sequence; wherein...

[0027] The first model is a buffer estimation model trained by any of the model training methods described above.

[0028] In the above scheme, the reward function includes a first reward function and a second reward function; the construction of the reward function based on channel traffic, the total number of data packets in the data packet queue, the second probability, the second count, and the historical state vectors corresponding to different stream states of the marked video stream includes:

[0029] A first reward function is constructed based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second count.

[0030] A second reward function is constructed based on the historical state vectors corresponding to different flow states of the labeled video stream.

[0031] In the above scheme, constructing the first reward function based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second count includes:

[0032] Based on the total number of data packets corresponding to the data packet queue, the second probability, and the second number of times, a fourth probability of the second buffer becoming empty is determined;

[0033] Based on the channel traffic and the total number of data packets corresponding to the data packet queue, a first time interval is calculated; the first time interval represents the time interval between two consecutive times the second buffer becomes empty;

[0034] A first reward function is constructed based on the fourth probability and the first time interval.

[0035] In the above scheme, the stream state includes the stream state before the video stream arrives, the stream state when the video stream transmission is started, and the stream state when the video stream is transmitted.

[0036] The second reward function is constructed based on the historical state vectors corresponding to different flow states of the labeled video stream, including:

[0037] The first historical state vector corresponding to the first stream state of the first video stream to be marked and the first scheduling strategy are input into the first neural network to predict the first feature vector corresponding to the second historical state vector;

[0038] The second historical state vector corresponding to the second stream state of the first video stream to be labeled is input into the second neural network to obtain the second feature vector corresponding to the second historical state vector.

[0039] Based on the first and second feature vectors, a second reward function is constructed; where,

[0040] When the first stream state is the stream state before the video stream arrives, the second stream state is the stream state when the video stream transmission is started; when the first stream state is the stream state when the video stream transmission is started, the second stream state is the stream state when the video stream is transmitted.

[0041] In the above scheme, when constructing the second reward function based on the first feature vector and the second feature vector, the method further includes:

[0042] The first feature vector is adjusted based on environmental noise.

[0043] In the above scheme, the second neural network is trained in the following way:

[0044] Input at least one first historical state vector corresponding to the first stream state and the second historical state vector corresponding to the second stream state of the labeled second video stream into the second neural network to obtain a third feature vector corresponding to each first historical state vector and a fourth feature vector corresponding to each second historical state vector.

[0045] Based on the third feature vector and the corresponding fourth feature vector, the scheduling strategy used in the corresponding first-flow state is predicted.

[0046] Based on the predicted scheduling strategy and the actual scheduling strategy used in the corresponding first-flow state, the loss value of the second neural network is calculated.

[0047] The weight parameters of the second neural network are adjusted based on the calculated loss value.

[0048] This invention also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0049] When the processor runs the computer program, it performs one of the following:

[0050] The steps of any of the above model training methods;

[0051] The steps of any of the above video stream scheduling methods.

[0052] This invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, performs one of the following:

[0053] The steps of any of the above model training methods;

[0054] The steps of any of the above video stream scheduling methods.

[0055] In this embodiment, a buffer estimation model is trained using at least one state vector sequence. During training, the loss value of the buffer estimation model is calculated based on the calibration probability, calibration count, first probability, and first count corresponding to each state vector sequence. The weight parameters of the buffer estimation model are then updated based on the calculated loss value. Each state vector sequence represents a state vector corresponding to the base station simultaneously transmitting data packets of multiple video streams at different times, and each state vector represents multiple network state parameters. Since the buffer estimation model is trained using state vector sequences in scenarios with simultaneous streaming of multiple videos, the trained model can accurately predict the probability and number of times the buffer becomes empty in such scenarios. This improves the accuracy of the predicted probability and number of times the buffer becomes empty, allowing for accurate determination of the corresponding scheduling strategy to schedule multiple video streams based on these probabilities and counts. This improves the precision of video stream scheduling, reduces the occurrence of buffer emptying in scenarios with simultaneous streaming of multiple videos, and improves the QoE of the video stream service. Attached Figure Description

[0056] Figure 1 A schematic diagram illustrating the implementation process of the model training method provided in this application embodiment;

[0057] Figure 2 A schematic diagram illustrating the implementation process of the buffer estimation model for processing state vector sequences provided in this application embodiment;

[0058] Figure 3 A schematic diagram illustrating the implementation flow of the method for determining a video stream scheduling strategy provided in an embodiment of this application;

[0059] Figure 4 A schematic diagram illustrating the implementation flow of the method for constructing a reward function provided in this application embodiment;

[0060] Figure 5 A schematic diagram illustrating the implementation flow of the method for constructing the first reward function provided in an embodiment of this application;

[0061] Figure 6 A schematic diagram illustrating the implementation flow of the method for constructing the second reward function provided in this application embodiment;

[0062] Figure 7 A schematic diagram illustrating a method for determining a video stream scheduling strategy provided in an application embodiment of this application;

[0063] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0064] Figure 9A schematic diagram of the structure of an electronic device provided in another embodiment of this application;

[0065] Figure 10 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0066] With the rapid development of mobile video services, the latency and stability of network video streaming have received increasing attention. In particular, Video on Demand (VoD) dominates streaming media traffic on mobile networks. Compared to voice communication, VoD has less stringent requirements for end-to-end latency, but it uses a set of Key Performance Indicators (KPIs) to evaluate the QoE of video streaming services. Among these KPIs, buffer inadequacy is the most prominent.

[0067] During video playback, if the video streaming service's buffer becomes empty (also known as buffer starvation), the video player will enter a buffering state, at which point the user will see a frozen image. Studies show that when 10% of the time is wasted waiting for video buffering, nearly 80% of users choose to terminate video playback before buffering finishes. Therefore, ensuring smooth video playback is crucial. Considering the importance of whether the video streaming service's buffer becomes empty, numerous studies have simulated the delicate trade-off between initial startup latency and buffer starvation, and applied buffer probability assessment models to channel resource scheduling.

[0068] As described in the background section, a video stream scheduling method for the base station side is provided in the related art.

[0069] However, although the video stream originates from the video content provider, the transmission process is controlled by the network operator. In scenarios where multiple videos are streamed simultaneously, the multiple video streams traversing the same channel of the base station can cause the video stream service buffer to become more congested, increasing the probability of the buffer becoming empty. Furthermore, only a single video stream is considered to have a fixed video length (in seconds) or size (in bits). In scenarios where multiple videos are streamed simultaneously, the video length or size of the multiple video streams changes dynamically. In other words, the probability of the buffer becoming empty determined in related technologies does not match the probability of the buffer becoming empty in scenarios where multiple videos are streamed simultaneously, and the methods for determining short or long streams are not applicable to scenarios where multiple videos are streamed simultaneously. Therefore, the video stream scheduling methods on the base station side in related technologies are not suitable for scenarios where multiple videos are streamed simultaneously.

[0070] Based on this, embodiments of this application provide a method for determining video stream scheduling strategies. A trained buffer estimation model can be used to obtain the probability and frequency of buffer emptying in scenarios with multiple video streams, improving the accuracy of the determined probability and frequency. A trained reinforcement learning model can be used to obtain the scheduling strategy with the highest total reward in scenarios with multiple video streams, thus enabling the use of the optimal scheduling strategy to schedule video streams, improving the accuracy of video stream scheduling, and reducing the occurrence of buffer emptying in scenarios with multiple videos streaming simultaneously, thereby improving the QoE of the video stream service.

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] Figure 1 This is a schematic diagram illustrating the implementation flow of the model training method provided in this application embodiment, wherein the execution entity of the flow is an electronic device such as a terminal device, server, or base station. Figure 1 The model training methods shown include:

[0073] Step 101: Input at least one state vector sequence into the buffer estimation model to obtain the first probability and the first number of the first buffer becoming empty for each state vector sequence; wherein, the state vector sequence represents the state vector corresponding to the base station transmitting data packets of multiple video streams at different times, each state vector represents multiple network state parameters, and the first buffer represents the video buffer of the video stream server.

[0074] Here, the electronic device determines at least one state vector sequence, inputs the determined at least one state vector sequence into the buffer estimation model, and processes the at least one state vector sequence using the buffer estimation model to obtain the first probability and the first number of the first buffer becoming empty corresponding to each state vector sequence output by the buffer estimation model.

[0075] Here, the state vector sequence determined by the electronic device serves as the training sample, which can be obtained from a sample library. The sample library can exist in the electronic device's local database or a remote database. The state vector sequence consists of state vectors corresponding to different times, and the state vector sequences in the sample library represent the state vectors corresponding to the base station simultaneously transmitting data packets of multiple video streams at different times. It should be noted that the state vector sequences in the sample library are all historical data.

[0076] Each state vector represents multiple network state parameters. Each state vector in each sequence represents network state parameters of the same type, and these network state parameters include at least one of the following categories:

[0077] The first network state parameter represents the flow arrival information;

[0078] The second network state parameter represents user behavior information;

[0079] The third network state parameter characterizes the traffic load.

[0080] The first network state parameters include at least one of the following: packet arrival rate, packet service rate, packet arrival probability, packet departure probability, set startup delay, and traffic intensity. The set startup delay refers to the period after which transmission begins. Both the packet arrival probability and packet departure probability are obtained by the base station. It should be noted that the startup delay is reconfigured when the buffer becomes empty or insufficient.

[0081] The second network state parameter includes the duration of the service slot.

[0082] The third network status parameter includes at least one of the following: total number of data packets, file size per data packet, average file size, minimum file size, and a start threshold per packet. The start threshold indicates that transmission will only begin after a set number of data packets have been acquired.

[0083] In some embodiments, the buffer estimation model includes a recurrent neural network (RNN), a bidirectional gated recurrent neural network, and a multi-task prediction network. In practical applications, the buffer estimation model is formed by cascading a recurrent neural network, a bidirectional gated recurrent neural network, and a multi-task prediction network. Figure 2 As shown, when inputting at least one state vector sequence into the buffer estimation model to obtain the first probability and the first number of the first buffer becoming empty for each state vector sequence, the method includes:

[0084] Step 201: Input the state vector sequence into the recurrent neural network to obtain the real number corresponding to each state vector in the state vector sequence, and apply attention weights to the real number corresponding to each state vector in the state vector sequence to obtain the first feature vector; wherein, the real number is obtained by feature compression of the state vector; the real number represents the global feature of the state vector.

[0085] Here, the electronic device inputs the state vector sequence into a recurrent neural network. The recurrent neural network performs feature compression on each state vector in the state vector sequence to obtain the real number corresponding to each state vector. An attention mechanism is then used to apply attention weights to the real numbers corresponding to each state vector in the state vector sequence to obtain the first feature vector corresponding to the state vector sequence. The first feature vector represents the deep-level features of the state vectors in the state vector sequence.

[0086] When a recurrent neural network compresses features of a state vector, it compresses the features of the state vector along the channels of the recurrent neural network and the dimensions of the state vector to obtain the corresponding real number. This real number has a global receptive field and represents the global features of the corresponding state vector.

[0087] Attention weights are generated by neurons in a recurrent neural network; attention weights are the weight parameters to be trained.

[0088] Step 202: Input the first feature vector into the bidirectional gated recurrent neural network to obtain the second feature vector; the second feature vector represents the correlation between state vectors in the state vector sequence.

[0089] Here, the electronic device inputs the first feature vector corresponding to the state vector sequence into a bidirectional gated recurrent neural network for feature extraction, and obtains the second feature vector corresponding to the state vector sequence. The second feature vector represents the correlation between state vectors in the state vector sequence.

[0090] The bidirectional gated recurrent neural network includes at least one layer of bidirectional gated recurrent unit (BiGRU).

[0091] In practical applications, a bidirectional gated recurrent neural network (BRN) consists of three cascaded layers of bidirectional gated recurrent units. Each BRN includes a reset gate and an update gate. The reset gate determines how new input information is combined with previous memories, thus deciding how much past information needs to be forgotten; the update gate helps the BRN decide how much past information should be passed to the future. The reset and update gates determine which information is ultimately output by the gated recurrent unit.

[0092] Since, compared to unidirectional gated recurrent neural networks, bidirectional gated recurrent neural networks can pass information about the state vectors in the state vector sequence from front to back and from back to front, the second feature vector output by the bidirectional gated recurrent neural network can fully reflect the correlation between the state vectors in the state vector sequence.

[0093] Step 203: Input the second feature vector into the multi-task prediction network to obtain the first probability and the first number of the first buffer becoming empty corresponding to the state vector sequence.

[0094] Here, the electronic device inputs the second feature vector corresponding to the state vector sequence into the multi-task prediction network for processing, and obtains the first probability and the first number of the first buffer becoming empty corresponding to the state vector sequence.

[0095] It should be noted that, in this embodiment, the electronic device uses a buffer estimation model to process each state vector sequence according to steps 201 to 203.

[0096] In this embodiment, the deep features of the state vectors in the state vector sequence are extracted by the recurrent neural network in the buffer estimation model. The bidirectional gated recurrent neural network in the buffer estimation model can comprehensively extract the correlation features between the state vectors in the state vector sequence. As a result, the first probability and the first number predicted based on the second feature vector can be more accurate.

[0097] Step 102: Calculate the loss value of the buffer estimation model based on the calibration probability, calibration number, first probability and first number corresponding to each state vector sequence in the at least one state vector sequence.

[0098] Here, the electronic device calculates the loss value corresponding to each state vector sequence based on the calibration probability, calibration number, first probability, and first number corresponding to each state vector sequence in the current batch; based on the loss value corresponding to each state vector sequence, it calculates the average loss value corresponding to the state vector sequence in the current batch, and obtains the loss value of the buffer estimation model.

[0099] To improve the accuracy of the calculated loss value, in one embodiment, the calculation of the loss value of the buffer estimation model based on the calibration probability, calibration count, first probability, and first count corresponding to each state vector sequence in the at least one state vector sequence includes:

[0100] The first loss value is calculated based on the calibration number and the first time number corresponding to each state vector sequence in the at least one state vector sequence;

[0101] The second loss value is calculated based on the calibration probability and the first probability corresponding to each state vector sequence in the at least one state vector sequence.

[0102] The loss value of the buffer estimation model is calculated based on the product of the first coefficient and the first loss value, and the product of the second coefficient and the second loss value.

[0103] Here, the electronic device calculates a first loss value based on the number of calibrations and the first count for each state vector sequence in the current batch; it calculates a second loss value based on the calibration probability and the first probability for each state vector sequence in the current batch; it calculates a first product between the first coefficient and the first loss value, a second product between the second coefficient and the second loss value, and finally, it calculates the sum of the first and second products to obtain the loss value of the buffer estimation model. Here, the first and second coefficients are both weight parameters to be trained.

[0104] To improve the accuracy of the calculated loss value, in some embodiments, the first loss value represents the mean squared error; the second loss value represents the cross-entropy.

[0105] In practical applications, the loss value of the buffer estimation model is calculated using the defined loss function Loss = α × MeanSquareError + β × CrossEntropy. Here, α represents the first coefficient, MeanSquareError represents the mean squared error of the loss value between the number of calibrations and the first calibration, β represents the second coefficient, and CrossEntropy represents the cross-entropy between the calibration probability and the first probability.

[0106] Specifically, the electronic device calculates the average loss value corresponding to the state vector sequence of the current batch based on the loss value 1 between the calibration number corresponding to each state vector sequence of the current batch and the first number; based on the calculated average loss value and the loss value 1 corresponding to each state vector sequence, it calculates the mean square error corresponding to the state vector sequence of the current batch to obtain the first loss value.

[0107] The electronic device calculates the cross-entropy of each state vector sequence based on the calibration probability and the first probability corresponding to each state vector sequence in the current batch, and obtains the loss value 2 corresponding to each state vector; based on the loss value 2 corresponding to each state vector sequence in the current batch, it calculates the corresponding average loss value, and obtains the second loss value.

[0108] Step 103: Update the weight parameters of the buffer estimation model based on the calculated loss value.

[0109] Here, the electronic device updates the weight parameters of the buffer estimation model based on the loss value of the buffer estimation model, in order to improve the accuracy of the first probability and the first number output by the buffer estimation model.

[0110] In this process, the electronic device backpropagates the loss value of the buffer estimation model within the buffer estimation model. During the backpropagation of the loss value to each layer of the buffer estimation model, the gradient of the loss function is calculated based on the loss value, and the weight parameters backpropagated to the current layer are updated along the descent direction of the gradient.

[0111] The electronic device will use the updated weight parameters as the weight parameters for the trained buffer estimation model.

[0112] Here, an update stopping condition can be set. When the update stopping condition is met, the weight parameters obtained from the last update are determined as the weight parameters used by the trained buffer estimation model. The update stopping condition is such as the set training epoch. One training epoch is the process of training the buffer estimation model once based on at least one sequence of state vectors. Of course, the update stopping condition is not limited to this; for example, it can also be the set average accuracy (mAP, mean Average Precision), etc.

[0113] It should be noted that in practical applications, model training is achieved by minimizing the loss function.

[0114] Backpropagation is the opposite of forward propagation. Forward propagation refers to the feedforward processing of the model, while backpropagation proceeds in the opposite direction. Backpropagation involves updating the weight parameters of each layer of the model based on the model's output. For example, in a buffer estimation model comprising a recurrent neural network (RNN), a bidirectional gated RNN, and a multi-task prediction network, forward propagation processes the data in the order of RNN - bidirectional gated RNN - multi-task prediction network, while backpropagation updates the weight parameters of each layer sequentially in the order of multi-task prediction network - bidirectional gated RNN - RNN.

[0115] In this embodiment, a buffer estimation model is trained using at least one state vector sequence. During the training process, the loss value of the buffer estimation model is calculated based on the calibration probability, calibration count, first probability, and first count corresponding to each state vector sequence. The weight parameters of the buffer estimation model are then updated based on the calculated loss value. Each state vector sequence represents the state vector corresponding to the base station simultaneously transmitting data packets of multiple video streams at different times, and each state vector represents multiple network state parameters. Since the buffer estimation model is trained using state vector sequences under scenarios of simultaneous streaming of multiple videos, the trained buffer estimation model can accurately predict the probability and number of times the buffer becomes empty in scenarios of simultaneous streaming of multiple videos. This improves the accuracy of the predicted probability and number of times the buffer becomes empty, allowing for accurate determination of the corresponding scheduling strategy to schedule multiple video streams based on the probability and number of times the buffer becomes empty. This improves the accuracy of video stream scheduling, thereby reducing the occurrence of buffer emptying in scenarios of simultaneous streaming of multiple videos and improving the QoE of video stream services.

[0116] Once the buffer estimation model is trained, it can be put into use. For example, in scenarios where multiple videos are streamed simultaneously, electronic devices can use the buffer estimation model trained through the above embodiments to process the state vector sequence corresponding to the queue of data packets to be transmitted. It should be noted that the electronic device in the embodiments for training the buffer estimation model may be different from the electronic device for putting the buffer estimation model into use.

[0117] Figure 3 This is a schematic diagram illustrating the implementation flow of the method for determining a video stream scheduling strategy provided in an embodiment of this application. The execution entity of the process is an electronic device such as a base station used to transmit data packets for multiple video streams. The following description uses the base station as the execution entity of the process. Figure 3 The methods shown for determining video stream scheduling strategies include:

[0118] Step 301: Input the first state vector sequence into the first model to obtain the second probability and the second number of times the second buffer becomes empty corresponding to the first state vector sequence; wherein, the first state vector sequence represents the state vector corresponding to the base station when receiving data packets in the data packet queues corresponding to multiple video streams, each state vector represents multiple network state parameters, and the second buffer represents the video buffer of the video stream server; the first model is a buffer estimation model trained using the model training method described in any of the above embodiments.

[0119] Here, in the scenario of multiple videos being pushed simultaneously, the base station records the state vector corresponding to the data packets received at different times during the process of receiving data packets corresponding to multiple video streams, and obtains the first state vector sequence corresponding to the data packet queue to be transmitted; the first state vector sequence is input into the first model to obtain the second probability and the second number of times the second buffer corresponding to the first state vector sequence becomes empty.

[0120] The first model processes the first state vector sequence to obtain the corresponding second probability and second number. This process is similar to the process of obtaining the first probability and first number by the buffer estimation model in step 101. Please refer to the relevant description in step 101. It will not be repeated here.

[0121] Step 302: Construct a reward function based on channel traffic, the total number of data packets in the data packet queue, the second probability, the second number of times, and the historical state vectors corresponding to the different flow states of the marked video stream.

[0122] The stream state includes the stream state before the video stream arrives, the stream state when transmission of the video stream is initiated, and the stream state during transmission. The historical state vector refers to the state vector corresponding to the different stream states of the marked video stream before the data packet queue to be transmitted was received.

[0123] In practical applications, the process of scheduling video streams by the base station before receiving the queue of data packets to be transmitted is described as a Markov process. Three Markov processes are used to characterize the scheduling of the video streams when they are in different stream states. From these three constructed Markov processes, the base station determines the historical state vectors corresponding to the different stream states of the labeled video streams.

[0124] The three Markov processes include:

[0125] MC1 represents the process of scheduling a video stream when it is in a stream state before the arrival of the tagged video stream;

[0126] MC2 represents the process of scheduling the video stream corresponding to the stream state when the tagged video stream is started for transmission;

[0127] MC3 represents the process of scheduling the video stream corresponding to the stream state during video stream transmission.

[0128] CM1 is constructed for two purposes: one is to calculate the probability of a stream request being rejected due to the permission control policy, and the other is to obtain the static distribution of the network state at any time when the tagged video stream is added. MC2 is constructed to analyze the network state distribution at the end of the initial prefetch phase. MC3 is constructed to analyze the impact of the probability of the second buffer becoming empty on the state transitions of the tagged video stream.

[0129] A Markov process can be represented as (S; A; P(s'|s,a); π(a|s)). Wherein,

[0130] S represents the set of state vectors; A represents the set of actions, where actions refer to scheduling policies configured under different flow states. Scheduling policies include packet prefetching policies and start delays. Each action will cause a change in the state vector; the packet prefetching policy refers to prefetching a set number of packets before transmitting them; the start delay is reconfigured when the buffer becomes empty or insufficient; P(s'|s,a) represents the probability that state vector s will transform into the next state vector s' when action a is performed on state vector s; the policy function π(a|s) represents the probability of selecting action a for state vector s.

[0131] In some embodiments, the reward function includes a first reward function and a second reward function; such as Figure 4As shown, the reward function is constructed based on channel traffic, the total number of data packets in the data packet queue, the second probability, the second count, and the historical state vectors corresponding to different flow states of the marked video stream, including:

[0132] Step 401: Construct a first reward function based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second count;

[0133] Step 402: Construct a second reward function based on the historical state vectors corresponding to different flow states of the labeled video stream.

[0134] In practical applications, R i The expression for the reward function is: R = R e +R i Among them, R e Characterizes the first reward function; R i Characterizes the second reward function.

[0135] The first reward function represents the extrinsic reward, and the second reward function represents the exploration reward.

[0136] In practical applications, after the base station starts the video stream transmission service, it acquires the channel traffic, determines the total number of data packets in the data packet queue to be transmitted, and obtains the total number of data packets corresponding to the data packet queue. Based on the channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second number of times, a first reward function is constructed.

[0137] Based on the historical state vectors corresponding to different flow states of the labeled video stream, the base station determines the feature information of the historical state vectors corresponding to different flow states of the labeled video stream, and constructs a second reward function based on the feature information corresponding to each pair of adjacent flow states.

[0138] In practical applications, the base station can determine the historical state vector corresponding to different stream states of the tagged video stream from MC1, MC2 and MC3.

[0139] In this embodiment, the constructed reward function includes a first reward function and a second reward function, thereby accurately calculating the total reward corresponding to the execution scheduling strategy through the constructed reward function.

[0140] like Figure 5 As shown, in some embodiments, constructing the first reward function based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second number of times includes:

[0141] Step 501: Based on the total number of data packets corresponding to the data packet queue, the second probability, and the second number of times, determine the fourth probability that the second buffer becomes empty.

[0142] In practical applications, base stations are based on formulas The fourth probability of the second buffer becoming empty is determined. Here, x1 represents the total number of packets corresponding to the packet queue; k represents the second probability; p represents the second probability; P s The fourth probability represents the second buffer becoming empty.

[0143] Since the second probability obtained in step 301 represents the probability of the second buffer becoming empty when the data packet queue to be transmitted is transmitted as a whole, rather than the probability of the second buffer becoming empty at any moment during the transmission of the data packet queue, in this embodiment, a fourth probability of the second buffer becoming empty is determined based on the total number of data packets corresponding to the data packet queue, the second probability, and the second count. The fourth probability represents the probability of the second buffer becoming empty at any moment during the transmission of the data packet queue. This improves the accuracy of the reward value calculated by the first reward function.

[0144] Step 502: Calculate the first time interval based on the channel traffic and the total number of data packets corresponding to the data packet queue; the first time interval represents the time interval between two consecutive times the second buffer becomes empty.

[0145] Here, the base station is based on the formula. The first time interval is calculated. Here, ρ represents the channel traffic, λ represents a positive number, and E(T) represents the first time interval.

[0146] Step 503: Construct a first reward function based on the fourth probability and the first time interval.

[0147] Here, the base station, having obtained the fourth probability and the first time interval, constructs the first reward function based on the fourth probability and the first time interval.

[0148] In practical applications, the expression for the first reward function is R. e =-(P s +γ×(g(E(T)))). Where g(E(T)) represents a strictly monotonically increasing convex function with E(T) as the variable; γ represents the reward decay factor, which is greater than or equal to zero and less than or equal to 1.

[0149] like Figure 6 As shown, in some embodiments, the stream state includes the stream state before the video stream arrives, the stream state when the video stream transmission is started, and the stream state when the video stream is transmitted; the construction of the second reward function based on the historical state vectors corresponding to different stream states of the marked video stream includes:

[0150] Step 601: Input the first historical state vector corresponding to the first stream state of the first video stream to the first neural network and the first scheduling strategy to predict the first feature vector corresponding to the second historical state vector.

[0151] Here, the base station inputs the first historical state vector corresponding to the first stream state of the first video stream being marked and the corresponding first scheduling strategy into the first neural network. The first neural network extracts the feature information of the first historical state vector and the feature information of the first scheduling strategy. Based on the feature information of the first historical state vector and the feature information of the first scheduling strategy, the base station predicts the first feature vector of the second historical state vector corresponding to the first stream state when the marked first video stream changes from the first stream state to the second stream state after the first scheduling strategy is applied to the first historical state vector.

[0152] In some embodiments, the first neural network is trained in the following manner:

[0153] Input the first historical state vector corresponding to at least one labeled second video stream in the first stream state and the corresponding scheduling strategy into the first neural network to obtain the predicted feature vector corresponding to the second historical state vector corresponding to each labeled second video stream in the second stream state.

[0154] The loss value of the first neural network is calculated based on the actual feature vector and the corresponding predicted feature vector of the second historical state vector corresponding to the second stream state of each labeled second video stream.

[0155] The weight parameters of the first neural network are adjusted based on the calculated loss value.

[0156] Here, the first neural network is trained based on the historical state vectors and scheduling strategies corresponding to the labeled second video stream in different stream states. The labeled first video stream is different from the labeled second video stream. The historical state vectors and scheduling strategies can be determined from MC1, MC2, and MC3.

[0157] During the training of the first neural network, at least one first historical state vector corresponding to the first stream state of the labeled second video stream and the corresponding scheduling strategy are determined from MC1, MC2, and MC3, and a second historical state vector corresponding to the second stream state of the labeled second video stream is determined. The first historical state vector corresponding to the first stream state of the labeled second video stream and the corresponding scheduling strategy are input into the first neural network. The first neural network extracts feature information of the first historical state vector and feature information of the corresponding scheduling strategy. Based on the feature information of each first historical state vector and the feature information of the corresponding scheduling strategy, a predicted feature vector of the second historical state vector corresponding to the transition of the labeled second video stream from the first stream state to the second stream state is predicted after applying the corresponding scheduling strategy to the first historical state vector.

[0158] Using the defined L2 norm expression, the loss value of the first neural network is calculated based on the actual feature vector and the corresponding predicted feature vector of the second historical state vector corresponding to the second stream state of each labeled second video stream. The weight parameters of the first neural network are adjusted based on the loss value of the first neural network until the first neural network converges, at which point training stops.

[0159] In practical applications, the predicted feature vector is represented as: in, The vector representing the first historical state corresponding to the first-flow state, 'a' representing the scheduling strategy corresponding to the first-flow state, and θ... F The weight parameters characterize the first neural network.

[0160] Step 602: Input the second historical state vector corresponding to the second stream state of the first video stream being labeled into the second neural network to obtain the second feature vector corresponding to the second historical state vector; wherein,

[0161] When the first stream state is the stream state before the video stream arrives, the second stream state is the stream state when the video stream transmission is started; when the first stream state is the stream state when the video stream transmission is started, the second stream state is the stream state when the video stream is transmitted.

[0162] Here, the base station inputs the second historical state vector corresponding to the first video stream being marked as being in the second stream state into the second neural network, and uses the second neural network to extract the feature information of the second historical state vector to obtain the second feature vector actually corresponding to the second historical state vector.

[0163] It should be noted that the first neural network and the second neural network are two different neural networks. The first flow state and the second flow state are two adjacent flow states.

[0164] In practical applications, both the first and second neural networks are convolutional neural networks. The first historical state vector and the first scheduling strategy corresponding to the labeled first video stream can be determined from the constructed CM1 or CM2, and the second historical state vector corresponding to the labeled first video stream can be determined from CM2 or CM3.

[0165] In some embodiments, the second neural network is trained in the following manner:

[0166] Input at least one first historical state vector corresponding to the first stream state and the second historical state vector corresponding to the second stream state of the labeled second video stream into the second neural network to obtain a third feature vector corresponding to each first historical state vector and a fourth feature vector corresponding to each second historical state vector.

[0167] Based on the third feature vector and the corresponding fourth feature vector, the scheduling strategy used in the corresponding first-flow state is predicted.

[0168] Based on the predicted scheduling strategy and the actual scheduling strategy used in the corresponding first-flow state, the loss value of the second neural network is calculated.

[0169] The weight parameters of the second neural network are adjusted based on the calculated loss value.

[0170] Here, from MC1, MC2, and MC3, at least one first historical state vector and a second historical state vector corresponding to the labeled second video stream are determined. It should be noted that the training samples used to train the second neural network can be different from those used to train the first neural network.

[0171] At least one first historical state vector and its corresponding second historical state vector are input into a second neural network to obtain a third feature vector corresponding to each first historical state vector and a fourth feature vector corresponding to each second historical state vector. Based on the third feature vector and its corresponding fourth feature vector, the scheduling strategy to be used in the corresponding first flow state is predicted. In practical applications, the predicted scheduling strategy can be expressed as: a p =DNN(v,v';θ) I Where s represents the first flow state; v' represents the second flow state; θ I The weight parameters characterize the second neural network.

[0172] Based on the predicted scheduling strategy and the actual scheduling strategy used in the corresponding first-stream state, the loss value corresponding to each labeled second video stream is calculated. Based on the calculated loss value, the average loss value is calculated to obtain the loss value of the second neural network. The weight parameters of the second neural network are adjusted based on the calculated loss value until the second neural network converges, at which point training stops.

[0173] In practical applications, the loss value of the second neural network is calculated using the set L1 norm expression, based on the predicted scheduling strategy and the actual scheduling strategy used in the corresponding first-flow state.

[0174] It should be noted that both the predicted scheduling strategy and the actual scheduling strategy used are represented by vectors.

[0175] In this embodiment, the performance of the second neural network is improved by minimizing the error between the predicted scheduling policy and the actual scheduling policy.

[0176] Step 603: Construct a second reward function based on the first feature vector and the second feature vector.

[0177] Here, the base station constructs an expression for the L2 norm based on the determined first feature vector and the determined second feature vector, and constructs a second reward function based on the constructed expression for the L2 norm.

[0178] In practical applications, the expression for the second reward function is: Wherein, η represents a constant; Characterizing the L2 norm, Characterizing the second eigenvector, The first eigenvector is represented.

[0179] Considering the presence of environmental noise in practical applications, in some embodiments, when constructing the second reward function based on the first feature vector and the second feature vector, the method further includes:

[0180] The first feature vector is adjusted based on environmental noise.

[0181] In the presence of environmental noise, the first feature vector predicted by the first neural network may have errors. Adjusting the first feature vector based on the environmental noise can compensate for the errors caused by the environmental noise and improve the accuracy of the first feature vector.

[0182] Step 303: Input multiple historical state vectors and corresponding scheduling policies into the reinforcement learning model, and use the reward function to calculate the total reward corresponding to each scheduling policy.

[0183] Here, after constructing the reward function, multiple historical state vectors and their corresponding scheduling policies are input into the reinforcement learning model, and the total reward corresponding to the execution of the scheduling policy for each historical state vector is calculated using the constructed reward function.

[0184] Step 304: Calculate the value gradient based on the third probability and the first value; wherein the third probability represents the probability of selecting the corresponding target scheduling strategy for the target historical state vector corresponding to the maximum total reward; and the first value represents the value of executing the target scheduling strategy.

[0185] Here, based on the total reward corresponding to the scheduling policy corresponding to each historical state vector, the maximum total reward is determined, the scheduling policy corresponding to the maximum total reward is determined as the target scheduling policy, and the historical state vector corresponding to the target scheduling policy is determined as the target historical state vector; from P(s'|s,a) in MC1, MC2 or MC3, the probability of selecting the corresponding target scheduling policy for the target historical state vector corresponding to the maximum total reward is determined, and the third probability is obtained.

[0186] The value of the execution target scheduling strategy is calculated using the defined advantage function, yielding the first value. The advantage function can be represented as A. π (s,a); A π (s,a) represents the value of choosing action a for state vector s, where action refers to scheduling strategy.

[0187] Given the first value and the third probability, the value gradient is calculated using the defined gradient function based on the first value and the third probability.

[0188] In practical applications, the gradient function is set as follows:

[0189] in, The gradient of the cumulative long-term reward with respect to the parameter θ is represented; θ and γ are both weight parameters of the reinforcement learning model; γ represents the reward decay factor, which is greater than or equal to zero and less than or equal to 1; π(s,a) is the policy function, representing the probability of choosing action a for state vector s.

[0190] It should be noted that the first value corresponds to A. π The value of (s,a) corresponds to the value of π(s,a), and the third probability corresponds to the value of π(s,a).

[0191] In practical applications, the A3C algorithm, or Asynchronous Advantageactor-Critic, is used to train the reinforcement learning model. During training with the A3C algorithm, the optimal scheduling policy is learned, thereby estimating the gradient of the expected cumulative long-term reward.

[0192] Step 305: Update the weight parameters of the reinforcement learning model based on the value gradient.

[0193] Here, the weight parameters of the reinforcement learning model are updated based on the calculated value gradient, and steps 303 to 305 are executed until the reinforcement learning model reaches the set convergence condition, at which point the training of the reinforcement learning model is stopped.

[0194] In practical applications, a convergence flag is set to indicate that the value gradient is less than or equal to a set gradient threshold.

[0195] The trained reinforcement learning model can output the total reward and the corresponding scheduling policy given the input state vector.

[0196] Step 306: When the reinforcement learning model reaches the set convergence condition, the first state vector sequence is input into the reinforcement learning model after updating the weight parameters to obtain the scheduling policy with the maximum total reward for each state vector in the first state vector sequence.

[0197] Here, once the reinforcement learning model reaches the set convergence condition, the first state vector sequence is input into the reinforcement learning model after updating the weight parameters. The reinforcement learning model predicts the scheduling strategy that maximizes the total reward for each state vector in the first state vector sequence. Based on the scheduling strategy that maximizes the total reward for each state vector in the first state vector sequence, the base station schedules the queue of data packets to be transmitted. The scheduling strategy that maximizes the total reward is the optimal scheduling strategy.

[0198] The temporal sequence of the reinforcement learning model predicting the scheduling policy that maximizes the total reward for each state vector is as follows:

[0199] The reinforcement learning model determines multiple scheduling policies corresponding to each state vector in the first state vector sequence, and uses the constructed reward function to calculate the total reward of each scheduling policy among the multiple scheduling policies corresponding to each state vector. Based on the total reward of each scheduling policy among the multiple scheduling policies corresponding to each state vector, the scheduling policy corresponding to the maximum total reward is determined, thus obtaining the scheduling policy with the maximum total reward for each state vector.

[0200] In this embodiment, the first state vector sequence corresponding to the data packet queue to be transmitted is input into the trained buffer estimation model to obtain the second probability and the second number of times the second buffer becomes empty corresponding to the first state vector sequence; a reward function is constructed based on the channel traffic, the total number of data packets in the data packet queue, the second probability, the second number, and the historical state vectors corresponding to the labeled video stream in different flow states; multiple historical state vectors and corresponding scheduling strategies are input into the reinforcement learning model, and the total reward corresponding to each scheduling strategy is calculated using the reward function; the value gradient is calculated based on the third probability and the first value; the weight parameters of the reinforcement learning model are updated based on the value gradient; when the reinforcement learning model reaches the set convergence condition, the first state vector sequence is input into the reinforcement learning model after updating the weight parameters to obtain the scheduling strategy with the largest total reward corresponding to each state vector in the first state vector sequence, and the video stream is scheduled according to the scheduling strategy corresponding to each state vector using the determined scheduling strategy with the largest total reward. The above scheme can obtain the probability and number of times the buffer becomes empty in the scenario of multiple video streaming by using the trained buffer estimation model, thereby improving the accuracy of determining the probability and number of times the buffer becomes empty. The trained reinforcement learning model can obtain the optimal scheduling strategy corresponding to each state vector in the first state vector sequence in the scenario of multiple video streaming, so that the optimal scheduling strategy can be used to schedule the video stream, thereby improving the accuracy of scheduling the video stream and reducing the occurrence of buffer emptying in the scenario of multiple videos streaming at the same time, thus improving the QoE of the video stream service.

[0201] Each state vector represents network state parameters related to user behavior, such as the duration of a service slot. Thus, the trained reinforcement learning model can determine the optimal scheduling strategy for video streams of different durations.

[0202] Figure 7 This is a schematic diagram illustrating a method for determining a video stream scheduling strategy provided in an application embodiment of this application. Figure 7 As shown, the methods for determining the video stream scheduling strategy include:

[0203] Step 701: Input the first state vector sequence into the trained buffer estimation model to obtain the second probability and the second number of times the second buffer becomes empty corresponding to the first state vector sequence.

[0204] Here, step 701 is the same as step 301, and the implementation process is requested to refer to the relevant description in step 301, which will not be repeated here.

[0205] Step 702: Construct a first reward function based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second number of times.

[0206] Here, step 702 is the same as step 401, and the implementation process is requested to refer to the relevant description in step 401, which will not be repeated here.

[0207] Step 703: Input the first historical state vector corresponding to the first stream state of the first video stream to the first neural network and the first scheduling strategy to predict the first feature vector corresponding to the second historical state vector.

[0208] Here, step 703 is the same as step 601, and the implementation process is requested to refer to the relevant description in step 601, which will not be repeated here.

[0209] Step 704: Input the second historical state vector corresponding to the second stream state of the first video stream being labeled into the second neural network to obtain the second feature vector corresponding to the second historical state vector; wherein,

[0210] When the first stream state is the stream state before the video stream arrives, the second stream state is the stream state when the video stream transmission is started; when the first stream state is the stream state when the video stream transmission is started, the second stream state is the stream state when the video stream is transmitted.

[0211] Here, step 704 is the same as step 602, and the implementation process is requested to refer to the relevant description in step 602, which will not be repeated here.

[0212] Step 705: Construct a second reward function based on the first feature vector and the second feature vector.

[0213] Here, step 705 is the same as step 603, and the implementation process is requested to refer to the relevant description in step 603, which will not be repeated here.

[0214] In practical applications, after determining the first feature vector, the first feature vector is adjusted based on environmental noise, and a second reward function is constructed based on the adjusted first and second feature vectors.

[0215] Step 706: Construct the overall reward function based on the first reward function and the second reward function.

[0216] In practical applications, the overall reward function is expressed as: R = R e +R i .

[0217] Step 707: Input multiple historical state vectors and corresponding scheduling policies into the reinforcement learning model, and use the overall reward function to calculate the total reward corresponding to each scheduling policy.

[0218] Here, step 707 is the same as step 303, and the implementation process is requested to refer to the relevant description in step 303, which will not be repeated here.

[0219] Step 708: Calculate the value gradient based on the third probability and the first value; wherein the third probability represents the probability of selecting the corresponding target scheduling strategy for the target historical state vector corresponding to the maximum total reward; and the first value represents the value of executing the target scheduling strategy.

[0220] Here, step 708 is the same as step 304, and the implementation process is requested to refer to the relevant description in step 304, which will not be repeated here.

[0221] Step 709: Update the weight parameters of the reinforcement learning model based on the value gradient.

[0222] Here, step 709 is the same as step 305, and the implementation process is requested to refer to the relevant description in step 305, which will not be repeated here.

[0223] Step 710: When the reinforcement learning model reaches the set convergence condition, the first state vector sequence is input into the reinforcement learning model after updating the weight parameters to obtain the scheduling policy with the maximum total reward corresponding to each state vector in the first state vector sequence.

[0224] Here, step 710 is the same as step 306, and the implementation process is requested to refer to the relevant description in step 306, which will not be repeated here.

[0225] To implement the model training method of this application embodiment, this application embodiment also provides an electronic device, which can correspondingly implement all the above-described model training method embodiments, such as... Figure 8 As shown, the electronic device includes:

[0226] Training unit 81 is used to train a buffer estimation model by inputting at least one state vector sequence into it, and to obtain the first probability and the first number of the first buffer becoming empty for each state vector sequence; wherein, the state vector sequence represents the state vector corresponding to the base station transmitting data packets of multiple video streams at different times, each state vector represents multiple network state parameters, and the first buffer represents the video buffer of the video stream server.

[0227] The calculation unit 82 is used to calculate the loss value of the buffer estimation model based on the calibration probability, calibration number, first probability and first number corresponding to each state vector sequence in the at least one state vector sequence;

[0228] The update unit 83 is used to update the weight parameters of the buffer estimation model based on the calculated loss value.

[0229] In some embodiments, the buffer estimation model includes a recurrent neural network, a bidirectional gated recurrent neural network, and a multi-task prediction network; the training unit 81 is specifically used for:

[0230] The state vector sequence is input into the recurrent neural network to obtain the real number corresponding to each state vector in the state vector sequence, and attention weights are applied to the real number corresponding to each state vector in the state vector sequence to obtain the first feature vector; wherein, the real number is obtained by feature compression of the state vector; the real number represents the global feature of the state vector;

[0231] The first feature vector is input into the bidirectional gated recurrent neural network to obtain the second feature vector; the second feature vector represents the correlation between state vectors in the state vector sequence.

[0232] The second feature vector is input into the multi-task prediction network to obtain the first probability and the first number of the first buffer becoming empty corresponding to the state vector sequence.

[0233] In some embodiments, the calculation unit 82 is specifically used for:

[0234] The first loss value is calculated based on the calibration number and the first time number corresponding to each state vector sequence in the at least one state vector sequence;

[0235] The second loss value is calculated based on the calibration probability and the first probability corresponding to each state vector sequence in the at least one state vector sequence.

[0236] The loss value of the buffer estimation model is calculated based on the product of the first coefficient and the first loss value, and the product of the second coefficient and the second loss value.

[0237] In some embodiments, the first loss value represents the mean squared error; the second loss value represents the cross-entropy.

[0238] In practical applications, the training unit 81, the calculation unit 82, and the update unit 83 can be implemented by processors in electronic devices, such as central processing units (CPUs), digital signal processors (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs). Of course, the processor needs to run programs stored in memory to implement the functions of each of the above program modules.

[0239] It should be noted that the above embodiments of the electronic device, when training the buffer estimation model, are only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the electronic device and the buffer estimation model method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0240] To implement the method for determining a video stream scheduling strategy according to the embodiments of this application, this application also provides an electronic device that can correspondingly implement all the embodiments of the method for determining a video stream scheduling strategy described above, such as... Figure 9 As shown, the electronic device includes:

[0241] Prediction unit 91 is used to input the first state vector sequence into the first model to obtain the second probability and the second number of times the second buffer corresponding to the first state vector sequence becomes empty; the first state vector sequence represents the state vector corresponding to the base station when it receives data packets in the data packet queues corresponding to multiple video streams, each state vector represents multiple network state parameters, and the second buffer represents the video buffer of the video stream server.

[0242] Construction unit 92 is used to construct a reward function based on channel traffic, the total number of data packets in the data packet queue, the second probability, the second number of times, and the historical state vectors corresponding to different flow states of the marked video stream;

[0243] The first computing unit 93 is used to input multiple historical state vectors and corresponding scheduling strategies into the reinforcement learning model, and use the reward function to calculate the total reward corresponding to the execution of each scheduling strategy.

[0244] The second calculation unit 94 is used to calculate the value gradient based on the third probability and the first value; wherein the third probability represents the probability of selecting the corresponding target scheduling strategy for the target historical state vector corresponding to the maximum total reward; and the first value represents the value of executing the target scheduling strategy.

[0245] Update unit 95 is used to update the weight parameters of the reinforcement learning model based on the value gradient;

[0246] Determining unit 96 is used to, when the reinforcement learning model reaches a set convergence condition, input the first state vector sequence into the reinforcement learning model after updating the weight parameters, to obtain the scheduling policy that maximizes the total reward for each state vector in the first state vector sequence; wherein...

[0247] The first model is a buffer estimation model trained by the model training method described in any of the above embodiments.

[0248] In some embodiments, the reward function includes a first reward function and a second reward function; the construction unit 92 is specifically used for:

[0249] A first reward function is constructed based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second count.

[0250] A second reward function is constructed based on the historical state vectors corresponding to different flow states of the labeled video stream.

[0251] In some embodiments, the building unit 92 is specifically used for:

[0252] Based on the total number of data packets corresponding to the data packet queue, the second probability, and the second number of times, a fourth probability of the second buffer becoming empty is determined;

[0253] Based on the channel traffic and the total number of data packets corresponding to the data packet queue, a first time interval is calculated; the first time interval represents the time interval between two consecutive times the second buffer becomes empty;

[0254] A first reward function is constructed based on the fourth probability and the first time interval.

[0255] In some embodiments, the stream state includes the stream state before the video stream arrives, the stream state when the video stream transmission is started, and the stream state when the video stream is transmitted; the construction unit 92 is specifically used for:

[0256] The first historical state vector corresponding to the first stream state of the first video stream to be marked and the first scheduling strategy are input into the first neural network to predict the first feature vector corresponding to the second historical state vector;

[0257] The second historical state vector corresponding to the second stream state of the first video stream to be labeled is input into the second neural network to obtain the second feature vector corresponding to the second historical state vector.

[0258] Based on the first and second feature vectors, a second reward function is constructed; where,

[0259] When the first stream state is the stream state before the video stream arrives, the second stream state is the stream state when the video stream transmission is started; when the first stream state is the stream state when the video stream transmission is started, the second stream state is the stream state when the video stream is transmitted.

[0260] In some embodiments, the electronic device further includes:

[0261] An adjustment unit is used to adjust the first feature vector based on environmental noise.

[0262] In some embodiments, the electronic device further includes:

[0263] A training unit is used to train the second neural network; specifically, the training unit is used for:

[0264] Input at least one first historical state vector corresponding to the first stream state and the second historical state vector corresponding to the second stream state of the labeled second video stream into the second neural network to obtain a third feature vector corresponding to each first historical state vector and a fourth feature vector corresponding to each second historical state vector.

[0265] Based on the third feature vector and the corresponding fourth feature vector, the scheduling strategy used in the corresponding first-flow state is predicted.

[0266] Based on the predicted scheduling strategy and the actual scheduling strategy used in the corresponding first-flow state, the loss value of the second neural network is calculated.

[0267] The weight parameters of the second neural network are adjusted based on the calculated loss value.

[0268] In practical applications, the prediction unit 91, construction unit 92, first calculation unit 93, second calculation unit 94, update unit 95, determination unit 96, adjustment unit, and training unit can be implemented by a processor in an electronic device, such as a CPU, DSP, MCU, or FPGA. Of course, the processor needs to run programs stored in memory to implement the functions of each of the above program modules.

[0269] It should be noted that the electronic device provided in the above embodiments, when determining the video stream scheduling strategy, is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the electronic device provided in the above embodiments and the method embodiments for determining the video stream scheduling strategy belong to the same concept; the specific implementation process is detailed in the method embodiments and will not be repeated here.

[0270] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of the present invention, the embodiments of the present invention also provide an electronic device. Figure 10 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of the present invention, such as... Figure 10 As shown, the electronic device 10 includes:

[0271] The communication interface 101 is capable of exchanging information with other devices, such as network devices;

[0272] The processor 102 is connected to the communication interface 101 to enable information interaction with other devices. When running a computer program, it executes the model training method provided by one or more of the above-mentioned technical solutions, or the method for determining a video stream scheduling strategy provided by one or more of the above-mentioned technical solutions. The computer program is stored in the memory 103.

[0273] Of course, in practical applications, the various components in electronic device 10 are coupled together through bus system 104. It can be understood that bus system 104 is used to realize the connection and communication between these components. In addition to a data bus, bus system 104 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 10 The general labeled all buses as Bus System 104.

[0274] The memory 103 in this embodiment of the invention is used to store various types of data to support the operation of the electronic device 10. Examples of such data include any computer program used to operate on the electronic device 10.

[0275] It is understood that memory 103 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 103 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0276] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 102. Processor 102 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 102 or by instructions in software form. The processor 102 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 103. Processor 102 reads the program in memory 103 and combines it with its hardware to complete the steps of the aforementioned method.

[0277] Optionally, when the processor 102 executes the program, it implements the corresponding processes implemented by the terminal in the various methods of the embodiments of the present invention. For the sake of brevity, these will not be described in detail here.

[0278] In an exemplary embodiment, the present invention also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 103 storing a computer program, which can be executed by the processor 102 of a terminal to complete the steps of the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0279] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0280] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0281] Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0282] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0283] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, include: At least one state vector sequence is input into the buffer estimation model to obtain the first probability and the first number of the first buffer becoming empty for each state vector sequence; wherein, the state vector sequence represents the state vector corresponding to the base station transmitting data packets of multiple video streams at different times, each state vector represents multiple network state parameters, and the first buffer represents the video buffer of the video stream server. Based on the calibration probability, calibration count, first probability, and first count corresponding to each state vector sequence in the at least one state vector sequence, the loss value of the buffer estimation model is calculated. The weight parameters of the buffer estimation model are updated based on the calculated loss value; The buffer estimation model includes a recurrent neural network, a bidirectional gated recurrent neural network, and a multi-task prediction network. When inputting at least one state vector sequence into the buffer estimation model to obtain the first probability and the first count of the first buffer becoming empty for each state vector sequence, the method includes: The state vector sequence is input into the recurrent neural network to obtain a real number corresponding to each state vector in the state vector sequence, and attention weights are applied to the real number corresponding to each state vector in the state vector sequence to obtain a first feature vector; wherein, the real number is obtained by feature compression of the state vector; the real number represents the global feature of the state vector; the first feature vector is input into the bidirectional gated recurrent neural network to obtain a second feature vector; the second feature vector represents the correlation between state vectors in the state vector sequence; the second feature vector is input into the multi-task prediction network to obtain the first probability and the first number of the first buffer becoming empty corresponding to the state vector sequence.

2. The method according to claim 1, characterized in that, The step of calculating the loss value of the buffer estimation model based on the calibration probability, calibration count, first probability, and first count corresponding to each state vector sequence in the at least one state vector sequence includes: The first loss value is calculated based on the calibration number and the first time number corresponding to each state vector sequence in the at least one state vector sequence; The second loss value is calculated based on the calibration probability and the first probability corresponding to each state vector sequence in the at least one state vector sequence. The loss value of the buffer estimation model is calculated based on the product of the first coefficient and the first loss value, and the product of the second coefficient and the second loss value.

3. The method according to claim 2, characterized in that, The first loss value represents the mean squared error; the second loss value represents the cross-entropy.

4. A method for determining a video stream scheduling strategy, characterized in that, include: The first state vector sequence is input into the first model to obtain the second probability and the second number of times the second buffer corresponding to the first state vector sequence becomes empty. The first state vector sequence represents the state vector of the base station when it receives data packets from the data packet queues corresponding to multiple video streams. Each state vector represents multiple network state parameters. The second buffer represents the video buffer of the video stream server. A reward function is constructed based on channel traffic, the total number of data packets in the data packet queue, the second probability, the second count, and the historical state vectors corresponding to different flow states of the marked video stream. Multiple historical state vectors and their corresponding scheduling policies are input into the reinforcement learning model, and the total reward for executing each scheduling policy is calculated using the reward function. The value gradient is calculated based on the third probability and the first value; wherein the third probability represents the probability of selecting the corresponding target scheduling strategy for the target historical state vector corresponding to the maximum total reward; and the first value represents the value of executing the target scheduling strategy. The weight parameters of the reinforcement learning model are updated based on the value gradient; When the reinforcement learning model reaches the set convergence condition, the first state vector sequence is input into the reinforcement learning model after updating the weight parameters to obtain the scheduling policy that maximizes the total reward for each state vector in the first state vector sequence; wherein... The first model is a buffer estimation model trained by the model training method as described in any one of claims 1 to 3.

5. The method according to claim 4, characterized in that, The reward function includes a first reward function and a second reward function; the reward function is constructed based on channel traffic, the total number of data packets in the data packet queue, the second probability, the second count, and the historical state vectors corresponding to different stream states of the marked video stream, including: A first reward function is constructed based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second count. A second reward function is constructed based on the historical state vectors corresponding to different flow states of the labeled video stream.

6. The method according to claim 5, characterized in that, The first reward function is constructed based on channel traffic, the total number of data packets corresponding to the data packet queue, the second probability, and the second count, including: Based on the total number of data packets corresponding to the data packet queue, the second probability, and the second number of times, a fourth probability of the second buffer becoming empty is determined; Based on the channel traffic and the total number of data packets corresponding to the data packet queue, a first time interval is calculated; the first time interval represents the time interval between two consecutive times the second buffer becomes empty; A first reward function is constructed based on the fourth probability and the first time interval.

7. The method according to claim 5, characterized in that, The stream state includes the stream state before the video stream arrives, the stream state when the video stream transmission is started, and the stream state when the video stream is transmitted. The second reward function is constructed based on the historical state vectors corresponding to different flow states of the labeled video stream, including: The first historical state vector corresponding to the first stream state of the first video stream to be marked and the first scheduling strategy are input into the first neural network to predict the first feature vector corresponding to the second historical state vector; The second historical state vector corresponding to the second stream state of the first video stream to be labeled is input into the second neural network to obtain the second feature vector corresponding to the second historical state vector. Based on the first and second feature vectors, a second reward function is constructed; where, When the first stream state is the stream state before the video stream arrives, the second stream state is the stream state when the video stream transmission is started; when the first stream state is the stream state when the video stream transmission is started, the second stream state is the stream state when the video stream is transmitted.

8. The method according to claim 7, characterized in that, When constructing the second reward function based on the first feature vector and the second feature vector, the method further includes: The first feature vector is adjusted based on environmental noise.

9. The method according to claim 7, characterized in that, The second neural network was trained in the following way: Input at least one first historical state vector corresponding to the first stream state and the second historical state vector corresponding to the second stream state of the labeled second video stream into the second neural network to obtain a third feature vector corresponding to each first historical state vector and a fourth feature vector corresponding to each second historical state vector. Based on the third feature vector and the corresponding fourth feature vector, the scheduling strategy used in the corresponding first-flow state is predicted. Based on the predicted scheduling strategy and the actual scheduling strategy used in the corresponding first-flow state, the loss value of the second neural network is calculated. The weight parameters of the second neural network are adjusted based on the calculated loss value.

10. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor runs the computer program, it performs at least one of the following: The steps of the model training method according to any one of claims 1 to 3; The steps of the method for determining a video stream scheduling strategy as described in any one of claims 4 to 9.

11. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs at least one of the following: The steps of the model training method according to any one of claims 1 to 3; The steps of the method for determining a video stream scheduling strategy as described in any one of claims 4 to 9.

Images