A streaming media recommendation and transmission method, device and computer-readable storage medium

Abstract

An embodiment of the present invention provides a streaming media recommendation and transmission method, device and computer-readable storage medium, the method comprising: obtaining observation data from a base station; establishing a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data; based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, determining and sending to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame, so that the base station can recommend the corresponding file to the user and adjust the transmission power of the file transmission; and continuously training the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on feedback information of file playback.

Description

Technical Field

[0001] The present invention relates to the field of mobile communication technology, and in particular to a streaming media recommendation and transmission method, device and computer-readable storage medium. Background Art

[0002] Currently, low-energy streaming media transmission solutions optimize transmission strategies based on prediction information in a prediction window, and cannot flexibly adjust the strategies. Moreover, the main goal of the solution is to maximize user experience, and the transmission energy consumption of streaming media is not taken into account. In addition, the relevant solutions do not consider the waste of transmission energy caused by users terminating playback prematurely. Summary of the invention

[0003] In view of this, embodiments of the present invention are intended to provide a method, device, and computer-readable storage medium for recommending and transmitting streaming media.

[0004] To achieve the above purpose, the technical solution of the embodiment of the present invention is implemented as follows:

[0005] An embodiment of the present invention provides a streaming media recommendation and transmission method, which is applied to a central unit and includes:

[0006] Obtain observation data from the base station;

[0007] Establishing a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data;

[0008] Based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, determine and send to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame, so that the base station can recommend the corresponding file to the user and adjust the transmission power of the file transmission;

[0009] The recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework are continuously trained based on feedback information of file playback.

[0010] The observation data includes but is not limited to the following information:

[0011] Index of files whose playing time in the user's file request history is greater than a preset threshold;

[0012] The time the user watched the requested file and the total playback time of the file;

[0013] The average channel gain vector of the user in the current frame and the past frames;

[0014] The amount of data to be played remaining in the user's current cache;

[0015] The transmission power of the base station to which the user is connected in the current time slot.

[0016] Wherein, the establishing of a recommendation strategy reinforcement learning framework based on the observation data includes:

[0017] The state in the recommendation strategy reinforcement learning framework is set to the file index whose playing time ratio in the user's file request history in the observation data is greater than a preset threshold; the action is set to the file index to be recommended to the user at the current time step in the observation data; the reward function is set to the playing time ratio of the file in the observation data;

[0018] Based on the states, actions and rewards in the recommendation strategy reinforcement learning framework, a deep reinforcement learning algorithm is used to learn the recommendation strategy.

[0019] Wherein, establishing a transmission strategy reinforcement learning framework based on the observation data includes:

[0020] The state in the transmission strategy reinforcement learning framework is set to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames; the action is set to the average data rate that the base station in the observed data needs to provide in the current frame, and the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information is given; the reward function is set to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality;

[0021] Based on the states, actions and rewards in the transmission strategy reinforcement learning framework, the transmission strategy is learned using a deep reinforcement learning algorithm.

[0022] When the reinforcement learning algorithm is a deep Q network DQN algorithm, based on the observation data and the recommendation strategy reinforcement learning framework, determining and sending to the base station a file index to be recommended to the user includes:

[0023] Using the file indexes in the observed data whose playing time ratio in the file request history of the user is greater than a preset threshold as the input of the DQN for learning the recommendation strategy, and outputting the Q value estimation of the action of each recommendation strategy reinforcement learning framework;

[0024] The ε-greedy method is used to execute the strategy, and the selected action is sent to the base station to which the user is connected.

[0025] When the reinforcement learning algorithm is a deep policy gradient DDPG algorithm, based on the observation data and the transmission strategy reinforcement learning framework, the average data rate required to be provided by the base station in the current frame is determined and sent to the base station, including:

[0026] The amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames are used as inputs of the actor network in the DDPG algorithm, and the output is the action of the transmission strategy reinforcement learning framework; the input of the critic network in the DDPG algorithm is the state and action of the transmission strategy reinforcement learning framework, and the output is the corresponding Q value;

[0027] DDPG encourages exploration of the action space by adding noise terms to the output when executing actions, and selects the actions to be executed and sends them to the base station to which the user is connected.

[0028] The continuous training of the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on the feedback information of the file playback includes:

[0029] When the current frame ends, the reward function and the next state of the transmission strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch of samples are randomly selected to update the network parameters in the transmission strategy reinforcement learning framework;

[0030] When it is determined that the current file playback is terminated, the reward function and the next state of the recommendation strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch sample is randomly selected to update the network parameters in the recommendation strategy reinforcement learning framework.

[0031] The embodiment of the present invention further provides a streaming media recommendation and transmission method, which is applied to a base station and includes:

[0032] Receiving a file index to be recommended to a user and an average data rate required to be provided by the base station in a current frame sent by a central unit; the file index and the average data rate are determined by the central unit based on observed data, a recommendation strategy reinforcement learning framework, and a transmission strategy reinforcement learning framework;

[0033] The corresponding file is recommended to the user based on the file index to be recommended to the user, and the transmission power of the file transmission is adjusted based on the average data rate required to be provided by the base station in the current frame.

[0034] The adjusting the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame includes:

[0035] Based on the average data rate that the base station needs to provide in the current frame, and according to the functional relationship between the transmit power of the base station in each time slot and the average data rate and the channel information, determine the transmit power of each time slot in the current frame;

[0036] The transmission power of the file transmission is adjusted based on the determined result.

[0037] The embodiment of the present invention further provides a streaming media recommendation and transmission device, which is applied to a central unit and includes:

[0038] A data observation module, used to obtain observation data from a base station;

[0039] A framework building module, used to build a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data;

[0040] A processing module, configured to determine and send to a base station: a file index to be recommended to a user and an average data rate required to be provided by the base station in a current frame based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, so that the base station can recommend corresponding files to the user and adjust the transmission power of the file transmission;

[0041] The model training module is used to continuously train the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on feedback information of file playback.

[0042] The embodiment of the present invention further provides a streaming media recommendation and transmission device, which is applied to a base station and includes:

[0043] A receiving module, used for receiving a file index to be recommended to a user and an average data rate required to be provided by the base station in a current frame sent by the central unit; the file index and the average data rate are determined by the central unit based on the observed data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework;

[0044] A recommendation module, used for recommending corresponding files to users based on the file index to be recommended to users;

[0045] The adjustment module is used to adjust the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame.

[0046] The embodiment of the present invention further provides a streaming media recommendation and transmission device, the device comprising: a processor and a memory for storing a computer program that can be run on the processor,

[0047] Wherein, the processor is used to execute the steps of the above method when running the computer program.

[0048] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0049] The streaming media recommendation and transmission method, device and computer-readable storage medium provided in the embodiments of the present invention obtain observation data from a base station; establish a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data; determine and send to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, so as to recommend the corresponding file to the user and adjust the transmission power of the file transmission; and continuously train the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on the feedback information of the file playback. The embodiments of the present invention establish a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on a model-free, which can flexibly adjust the strategy online according to the changes in the streaming media transmission environment, reduce the transmission energy waste caused by the probability of the user terminating the playback early, and reduce the base station energy consumption caused by wireless streaming media transmission while ensuring the user service quality.

[0050] The embodiment of the present invention does not require road testing to establish a signal map, and does not require converting prediction information into knowledge required for decision-making, thereby reducing system implementation and calculation complexity and being easy to implement; and does not require GPS data, thereby avoiding user privacy issues.

[0051] Moreover, the embodiment of the present invention uses a central unit with strong computing power and wide coverage for centralized training, selects the action in the transmission strategy reinforcement learning framework as the average data rate required by the base station in each frame (making decisions in frames), and sets the corresponding relationship between the transmission power, the average data rate and the instantaneous channel information. After receiving the average data rate requirement within the frame given by the central unit, the base station can perform distributed transmission power adjustment according to the real-time channel information in each time slot. Compared with the central unit making decisions in each time slot, the signaling overhead between the central unit and the base station can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the flow of the streaming media recommendation and transmission method according to an embodiment of the present invention Figure 1 ;

[0053] Figure 2 Schematic diagram of the flow of the streaming media recommendation and transmission method according to an embodiment of the present invention Figure 2 ;

[0054] Figure 3 Schematic diagram of the structure of the streaming media recommendation and transmission device according to an embodiment of the present invention Figure 1 ;

[0055] Figure 4 Schematic diagram of the structure of the streaming media recommendation and transmission device according to an embodiment of the present invention Figure 2 ;

[0056] Figure 5 Schematic diagram of the flow of the streaming media recommendation and transmission method according to an embodiment of the present invention Figure 3 . DETAILED DESCRIPTION

[0057] The present invention is described below in conjunction with the accompanying drawings and embodiments.

[0058] The embodiment of the present invention provides a streaming media recommendation and transmission method, such as Figure 1 As shown, the method is applied to a central unit and comprises:

[0059] Step 101: Obtain observation data from a base station;

[0060] Step 102: establishing a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data;

[0061] Step 103: Based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, determine and send to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame, so that the base station can recommend the corresponding file to the user and adjust the transmission power of the file transmission;

[0062] Step 104: Continuously train the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on the feedback information of the file playback.

[0063] The embodiments of the present invention establish a model-free recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework, which can flexibly adjust the strategy online according to the changes in the streaming media transmission environment, reduce the transmission energy waste caused by the probability of users terminating the playback early, and reduce the base station energy consumption caused by wireless streaming media transmission while ensuring the user service quality.

[0064] In the embodiment of the present invention, the observation data includes but is not limited to the following information:

[0065] Index of files whose playing time in the user's file request history is greater than a preset threshold;

[0066] The time the user watched the requested file and the total playback time of the file;

[0067] The average channel gain vector of the user in the current frame and the past frames;

[0068] The amount of data to be played remaining in the user's current cache;

[0069] The transmission power of the base station to which the user is connected in the current time slot.

[0070] In an embodiment of the present invention, the step of establishing a recommendation strategy reinforcement learning framework based on the observation data includes:

[0071] The state in the recommendation strategy reinforcement learning framework is set to the file index whose playing time ratio in the user's file request history in the observation data is greater than a preset threshold; the action is set to the file index to be recommended to the user at the current time step in the observation data; the reward function is set to the playing time ratio of the file in the observation data;

[0072] Based on the states, actions and rewards in the recommendation strategy reinforcement learning framework, a deep reinforcement learning algorithm is used to learn the recommendation strategy.

[0073] In an embodiment of the present invention, a transmission strategy reinforcement learning framework is established based on the observation data, including:

[0074] The state in the transmission strategy reinforcement learning framework is set to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames; the action is set to the average data rate that the base station in the observed data needs to provide in the current frame, and the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information is given; the reward function is set to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality;

[0075] Based on the states, actions and rewards in the transmission strategy reinforcement learning framework, the transmission strategy is learned using a deep reinforcement learning algorithm.

[0076] In one embodiment of the present invention, when the reinforcement learning algorithm is a deep Q network DQN algorithm, based on the observation data and the recommendation strategy reinforcement learning framework, determining and sending to the base station a file index to be recommended to the user includes:

[0077] Using the file indexes in the observed data whose playing time ratio in the file request history of the user is greater than a preset threshold as the input of the DQN for learning the recommendation strategy, and outputting the Q value estimation of the action of each recommendation strategy reinforcement learning framework;

[0078] The ε-greedy method is used to execute the strategy, and the selected action is sent to the base station to which the user is connected.

[0079] In one embodiment of the present invention, when the reinforcement learning algorithm is a deep policy gradient DDPG algorithm, based on the observation data and the transmission strategy reinforcement learning framework, determining and sending to the base station the average data rate required to be provided by the base station in the current frame includes:

[0080] The amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames are used as inputs of the actor network in the DDPG algorithm, and the output is the action of the transmission strategy reinforcement learning framework; the input of the critic network in the DDPG algorithm is the state and action of the transmission strategy reinforcement learning framework, and the output is the corresponding Q value;

[0081] DDPG encourages exploration of the action space by adding noise terms to the output when executing actions, and selects the actions to be executed and sends them to the base station to which the user is connected.

[0082] In an embodiment of the present invention, the continuous training of the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on the feedback information of the file playback includes:

[0083] When the current frame ends, the reward function and the next state of the transmission strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch of samples are randomly selected to update the network parameters in the transmission strategy reinforcement learning framework;

[0084] When it is determined that the current file playback is terminated, the reward function and the next state of the recommendation strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch sample is randomly selected to update the network parameters in the recommendation strategy reinforcement learning framework.

[0085] The embodiment of the present invention also provides a streaming media recommendation and transmission method, such as Figure 2 As shown, the method is applied to a base station, comprising:

[0086] Step 201: receiving a file index to be recommended to a user and an average data rate required to be provided by a base station in a current frame sent by a central unit; the file index and the average data rate are determined by the central unit based on observed data, a recommendation strategy reinforcement learning framework, and a transmission strategy reinforcement learning framework;

[0087] Step 202: recommending a corresponding file to the user based on the file index to be recommended to the user, and adjusting the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame.

[0088] In the embodiment of the present invention, adjusting the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame includes:

[0089] Based on the average data rate that the base station needs to provide in the current frame, and according to the functional relationship between the transmit power of the base station in each time slot and the average data rate and the channel information, determine the transmit power of each time slot in the current frame;

[0090] The transmission power of the file transmission is adjusted based on the determined result.

[0091] In order to implement the above method embodiment, the embodiment of the present invention also provides a streaming media recommendation and transmission device, such as Figure 3 As shown, the device is applied to a central unit and comprises:

[0092] The data observation module 301 is used to obtain observation data from the base station;

[0093] A framework building module 302, used to build a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data;

[0094] The processing module 303 is used to determine and send to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, so that the base station can recommend the corresponding file to the user and adjust the transmission power of the file transmission;

[0095] The model training module 304 is used to continuously train the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on the feedback information of the file playback.

[0096] In the embodiment of the present invention, the observation data includes but is not limited to the following information:

[0097] Index of files whose playing time in the user's file request history is greater than a preset threshold;

[0098] The time the user watched the requested file and the total playback time of the file;

[0099] The average channel gain vector of the user in the current frame and the past frames;

[0100] The amount of data to be played remaining in the user's current cache;

[0101] The transmission power of the base station to which the user is connected in the current time slot.

[0102] In the embodiment of the present invention, the framework establishment module 302 establishes a recommendation strategy reinforcement learning framework based on the observation data, including:

[0103] The state in the recommendation strategy reinforcement learning framework is set to the file index whose playing time ratio in the user's file request history in the observation data is greater than a preset threshold; the action is set to the file index to be recommended to the user at the current time step in the observation data; the reward function is set to the playing time ratio of the file in the observation data;

[0104] Based on the states, actions and rewards in the recommendation strategy reinforcement learning framework, a deep reinforcement learning algorithm is used to learn the recommendation strategy.

[0105] In the embodiment of the present invention, the framework establishment module 302 establishes a transmission strategy reinforcement learning framework based on the observation data, including:

[0106] The state in the transmission strategy reinforcement learning framework is set to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames; the action is set to the average data rate that the base station in the observed data needs to provide in the current frame, and the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information is given; the reward function is set to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality;

[0107] Based on the states, actions and rewards in the transmission strategy reinforcement learning framework, the transmission strategy is learned using a deep reinforcement learning algorithm.

[0108] In one embodiment of the present invention, when the reinforcement learning algorithm is a deep Q network DQN algorithm, the processing module 303 determines and sends a file index to be recommended to the user to the base station based on the observation data and the recommendation strategy reinforcement learning framework, including:

[0109] Using the file indexes in the observed data whose playing time ratio in the file request history of the user is greater than a preset threshold as the input of the DQN for learning the recommendation strategy, and outputting the Q value estimation of the action of each recommendation strategy reinforcement learning framework;

[0110] The ε-greedy method is used to execute the strategy, and the selected action is sent to the base station to which the user is connected.

[0111] In one embodiment of the present invention, when the reinforcement learning algorithm is a deep policy gradient DDPG algorithm, the processing module 303 determines and sends to the base station an average data rate required to be provided by the base station in the current frame based on the observation data and the transmission strategy reinforcement learning framework, including:

[0112] The amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames are used as inputs of the actor network in the DDPG algorithm, and the output is the action of the transmission strategy reinforcement learning framework; the input of the critic network in the DDPG algorithm is the state and action of the transmission strategy reinforcement learning framework, and the output is the corresponding Q value;

[0113] DDPG encourages exploration of the action space by adding noise terms to the output when executing actions, and selects the actions to be executed and sends them to the base station to which the user is connected.

[0114] In the embodiment of the present invention, the model training module 304 continuously trains the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on the feedback information of the file playback, including:

[0115] When the current frame ends, the reward function and the next state of the transmission strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch of samples are randomly selected to update the network parameters in the transmission strategy reinforcement learning framework;

[0116] When it is determined that the current file playback is terminated, the reward function and the next state of the recommendation strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch sample is randomly selected to update the network parameters in the recommendation strategy reinforcement learning framework.

[0117] The embodiment of the present invention also provides a streaming media recommendation and transmission device, such as Figure 4 As shown, the device is applied to a base station, comprising:

[0118] The receiving module 401 is used to receive the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame sent by the central unit; the file index and the average data rate are determined by the central unit based on the observed data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework;

[0119] A recommendation module 402, configured to recommend corresponding files to the user based on the file index to be recommended to the user;

[0120] The adjustment module 403 is used to adjust the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame.

[0121] In the embodiment of the present invention, the adjusting module 403 adjusts the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame, including:

[0122] Based on the average data rate that the base station needs to provide in the current frame, and according to the functional relationship between the transmit power of the base station in each time slot and the average data rate and the channel information, determine the transmit power of each time slot in the current frame;

[0123] The transmission power of the file transmission is adjusted based on the determined result.

[0124] The embodiment of the present invention further provides a streaming media recommendation and transmission device, the device comprising: a processor and a memory for storing a computer program that can be run on the processor,

[0125] Wherein, when the processor is used to run the computer program, it executes:

[0126] Obtain observation data from the base station;

[0127] Establishing a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data;

[0128] Based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, determine and send to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame, so that the base station can recommend the corresponding file to the user and adjust the transmission power of the file transmission;

[0129] The recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework are continuously trained based on feedback information of file playback.

[0130] The observation data includes but is not limited to the following information:

[0131] Index of files whose playing time in the user's file request history is greater than a preset threshold;

[0132] The time the user watched the requested file and the total playback time of the file;

[0133] The average channel gain vector of the user in the current frame and the past frames;

[0134] The amount of data to be played remaining in the user's current cache;

[0135] The transmission power of the base station to which the user is connected in the current time slot.

[0136] When the recommendation strategy reinforcement learning framework is established based on the observation data, the processor is also used to execute the following when running the computer program:

[0137] The state in the recommendation strategy reinforcement learning framework is set to the file index whose playing time ratio in the user's file request history in the observation data is greater than a preset threshold; the action is set to the file index to be recommended to the user at the current time step in the observation data; the reward function is set to the playing time ratio of the file in the observation data;

[0138] Based on the states, actions and rewards in the recommendation strategy reinforcement learning framework, a deep reinforcement learning algorithm is used to learn the recommendation strategy.

[0139] When establishing a transmission strategy reinforcement learning framework based on the observation data, the processor is further configured to execute, when running the computer program:

[0140] The state in the transmission strategy reinforcement learning framework is set to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames; the action is set to the average data rate that the base station in the observed data needs to provide in the current frame, and the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information is given; the reward function is set to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality;

[0141] Based on the states, actions and rewards in the transmission strategy reinforcement learning framework, the transmission strategy is learned using a deep reinforcement learning algorithm.

[0142] When the reinforcement learning algorithm is a deep Q network DQN algorithm, based on the observation data and the recommendation strategy reinforcement learning framework, the file index to be recommended to the user is determined and sent to the base station, and the processor is also used to run the computer program to execute:

[0143] Using the file indexes in the observed data whose playing time ratio in the file request history of the user is greater than a preset threshold as the input of the DQN for learning the recommendation strategy, and outputting the Q value estimation of the action of each recommendation strategy reinforcement learning framework;

[0144] The ε-greedy method is used to execute the strategy, and the selected action is sent to the base station to which the user is connected.

[0145] The processor is further configured to execute, when running the computer program:

[0146] When the reinforcement learning algorithm is a deep policy gradient DDPG algorithm, based on the observation data and the transmission strategy reinforcement learning framework, determining and sending to the base station the average data rate required to be provided by the base station in the current frame, the processor is also used to execute the following when running the computer program:

[0147] The amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames are used as inputs of the actor network in the DDPG algorithm, and the output is the action of the transmission strategy reinforcement learning framework; the input of the critic network in the DDPG algorithm is the state and action of the transmission strategy reinforcement learning framework, and the output is the corresponding Q value;

[0148] DDPG encourages exploration of the action space by adding noise terms to the output when executing actions, and selects the actions to be executed and sends them to the base station to which the user is connected.

[0149] When the feedback information based on file playback continuously trains the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, the processor is further used to execute the following when running the computer program:

[0150] When the current frame ends, the reward function and the next state of the transmission strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch of samples are randomly selected to update the network parameters in the transmission strategy reinforcement learning framework;

[0151] When it is determined that the current file playback is terminated, the reward function and the next state of the recommendation strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch sample is randomly selected to update the network parameters in the recommendation strategy reinforcement learning framework.

[0152] The embodiment of the present invention further provides a streaming media recommendation and transmission device, the device comprising: a processor and a memory for storing a computer program that can be run on the processor,

[0153] Wherein, when the processor is used to run the computer program, it executes:

[0154] Receiving a file index to be recommended to a user and an average data rate required to be provided by the base station in a current frame sent by a central unit; the file index and the average data rate are determined by the central unit based on observed data, a recommendation strategy reinforcement learning framework, and a transmission strategy reinforcement learning framework;

[0155] The corresponding file is recommended to the user based on the file index to be recommended to the user, and the transmission power of the file transmission is adjusted based on the average data rate required to be provided by the base station in the current frame.

[0156] When the transmission power of the file transmission is adjusted based on the average data rate required to be provided by the base station in the current frame, the processor is further configured to execute, when running the computer program:

[0157] Based on the average data rate that the base station needs to provide in the current frame, and according to the functional relationship between the transmit power of the base station in each time slot and the average data rate and the channel information, determine the transmit power of each time slot in the current frame;

[0158] The transmission power of the file transmission is adjusted based on the determined result.

[0159] It should be noted that: the device provided in the above embodiment only uses the division of the above program modules as an example when performing streaming media recommendation and transmission. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the device provided in the above embodiment and the corresponding method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0160] In an exemplary embodiment, an embodiment of the present invention further provides a computer-readable storage medium, which may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0161] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program performs:

[0162] Obtain observation data from the base station;

[0163] Establishing a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework based on the observation data;

[0164] Based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, determine and send to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame, so that the base station can recommend the corresponding file to the user and adjust the transmission power of the file transmission;

[0165] The recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework are continuously trained based on feedback information of file playback.

[0166] The observation data includes but is not limited to the following information:

[0167] Index of files whose playing time in the user's file request history is greater than a preset threshold;

[0168] The time the user watched the requested file and the total playback time of the file;

[0169] The average channel gain vector of the user in the current frame and the past frames;

[0170] The amount of data to be played remaining in the user's current cache;

[0171] The transmission power of the base station to which the user is connected in the current time slot.

[0172] When the recommendation strategy reinforcement learning framework is established based on the observation data, the computer program is executed by the processor and further executes:

[0173] The state in the recommendation strategy reinforcement learning framework is set to the file index whose playing time ratio in the user's file request history in the observation data is greater than a preset threshold; the action is set to the file index to be recommended to the user at the current time step in the observation data; the reward function is set to the playing time ratio of the file in the observation data;

[0174] Based on the states, actions and rewards in the recommendation strategy reinforcement learning framework, a deep reinforcement learning algorithm is used to learn the recommendation strategy.

[0175] When the transmission strategy reinforcement learning framework is established based on the observation data, the computer program is executed by the processor and further executes:

[0176] The state in the transmission strategy reinforcement learning framework is set to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames; the action is set to the average data rate that the base station in the observed data needs to provide in the current frame, and the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information is given; the reward function is set to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality;

[0177] Based on the states, actions and rewards in the transmission strategy reinforcement learning framework, the transmission strategy is learned using a deep reinforcement learning algorithm.

[0178] When the reinforcement learning algorithm is a deep Q network DQN algorithm, when determining and sending a file index to be recommended to the user to the base station based on the observation data and the recommendation strategy reinforcement learning framework, when the computer program is executed by the processor, the following is further executed:

[0179] Using the file indexes in the observed data whose playing time ratio in the file request history of the user is greater than a preset threshold as the input of the DQN for learning the recommendation strategy, and outputting the Q value estimation of the action of each recommendation strategy reinforcement learning framework;

[0180] The ε-greedy method is used to execute the strategy, and the selected action is sent to the base station to which the user is connected.

[0181] When the computer program is executed by a processor, it also performs:

[0182] When the reinforcement learning algorithm is a deep policy gradient DDPG algorithm, based on the observation data and the transmission strategy reinforcement learning framework, when determining and sending to the base station the average data rate required to be provided by the base station in the current frame, the computer program is executed by the processor and further executes:

[0183] The amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames are used as inputs of the actor network in the DDPG algorithm, and the output is the action of the transmission strategy reinforcement learning framework; the input of the critic network in the DDPG algorithm is the state and action of the transmission strategy reinforcement learning framework, and the output is the corresponding Q value;

[0184] DDPG encourages exploration of the action space by adding noise terms to the output when executing actions, and selects the actions to be executed and sends them to the base station to which the user is connected.

[0185] When the feedback information based on file playback continuously trains the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, the computer program is executed by the processor and further executes:

[0186] When the current frame ends, the reward function and the next state of the transmission strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch of samples are randomly selected to update the network parameters in the transmission strategy reinforcement learning framework;

[0187] When it is determined that the current file playback is terminated, the reward function and the next state of the recommendation strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch sample is randomly selected to update the network parameters in the recommendation strategy reinforcement learning framework.

[0188] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program performs:

[0189] Receiving a file index to be recommended to a user and an average data rate required to be provided by the base station in a current frame sent by a central unit; the file index and the average data rate are determined by the central unit based on observed data, a recommendation strategy reinforcement learning framework, and a transmission strategy reinforcement learning framework;

[0190] The corresponding file is recommended to the user based on the file index to be recommended to the user, and the transmission power of the file transmission is adjusted based on the average data rate required to be provided by the base station in the current frame.

[0191] When the transmission power of the file transmission is adjusted based on the average data rate required to be provided by the base station in the current frame, when the computer program is executed by the processor, the following steps are further performed:

[0192] Based on the average data rate that the base station needs to provide in the current frame, and according to the functional relationship between the transmit power of the base station in each time slot and the average data rate and the channel information, determine the transmit power of each time slot in the current frame;

[0193] The transmission power of the file transmission is adjusted based on the determined result.

[0194] The present invention is described below in conjunction with scenario embodiments.

[0195] The embodiment of the invention of this application is a low-energy wireless streaming media recommendation and transmission strategy based on model-free learning. By establishing a model-free reinforcement learning framework, the strategy can be flexibly adjusted online according to changes in the environment, thereby reducing the waste of transmission energy caused by the probability of users terminating playback early, and minimizing the base station energy consumption caused by wireless streaming media transmission while ensuring the user service quality.

[0196] In a multi-cell system, each cell is equipped with a base station, and each base station is connected to the central unit. Users can move between multiple cells and access the base station with the largest average channel gain. Whenever a user may need streaming media services (for example, opening a streaming media application or just finishing watching a streaming media file), the central unit will control the base station to which the user is connected to recommend a streaming media file to the user. The user may accept the recommended file and request that the file be played. The user may also not accept the file recommended by the base station and request another file to be played. The length of time a user watches a single file is called the length of a time step, and the length of time the user watches the file in the jth time step is T. j , file recommendation is performed in time steps. The total playback time of the file requested by the user at the jth time step is L j , when the user is not interested in the requested file, the file playback may be terminated in advance, then T j ≤L j , and called T j / L j is the playback time ratio.

[0197] Assume that each streaming media file is divided into N c The playback time of each segment contains M frames. The duration of each frame is ΔT (seconds), which is N s The transmission power allocation of the base station is carried out in units of time slots, and the duration of each time slot is τ=ΔT / M (millisecond level).

[0198] set up is the average channel gain vector of the user in the t frame, where α n，t Indicates that the user has the same average channel gain as n The average channel gain between strong base stations; a n,t The n in the table is 1 to N. b Any integer in a. 1,t represents the average channel gain between the user and the base station with the strongest average channel, a n,t represents the average channel gain between the user and the nth strongest base station in the average channel, a Nb,t Indicates that the user is Nth on the average channel bThe average channel gain between strong base stations. The central unit obtains and records observation data from the base station, learns recommendation and transmission strategies, and gives the files that need to be recommended to the user in the current time step and the average data rate provided by the base station to the user in the current frame. Specifically, the observation data of the central unit includes at least but not limited to the following information:

[0199] The J file indexes f whose playback time ratio in the user's recent file request history is greater than the threshold θ (any value between 0-1, such as: 0.8) j-1 , …, f j-J ;

[0200] The time T that the user watches the requested file j And the total playback time of the file L j ;

[0201] The user is in the current frame and the past N t The average channel gain vector of frames (preset number of past frames)

[0202] The amount of data to be played remaining in the user's current cache B t ;

[0203] The transmission power p of the base station to which the user is connected in the current time slot ti .

[0204] The embodiments of the present invention are first described from the perspective of establishing a recommendation strategy reinforcement learning framework and a transmission strategy reinforcement learning framework.

[0205] 1. Reinforcement Learning Framework for Recommendation Strategies

[0206] The purpose of file recommendation is to recommend videos of interest to users, increase the probability of users playing the complete video, and thus reduce the energy waste caused by video transmission. action And rewards

[0207] Considering that the user's viewing history can reflect the user's preferences, the state in the recommendation strategy reinforcement learning framework is set to action Set to the file index that needs to be recommended at the current time step. The reward function is set to the user's play time ratio

[0208] After setting the states, actions, and rewards in the recommendation strategy reinforcement learning framework, we can use existing deep reinforcement learning algorithms to learn the recommendation strategy, such as the DQN algorithm. rec (s, a; θ rec) represents a DQN for learning recommendation strategies, whose input is the state The output for each action is The Q value of the network is estimated and the network parameters are θ rec DQN uses the ε-greedy method to execute the strategy, randomly selecting actions to execute with a probability of ε, and selecting actions with a probability of 1-ε. Execution. The parameters θ of DQN rec Perform online iterative update according to the following formula

[0209]

[0210] in is a set of sample indexes randomly selected from historical observations, δ rec is the learning rate of DQN, is the target DQN, whose structure is similar to Q rec Same, the parameters of target DQN according to To update, rec is the update rate of the target DQN.

[0211] 2. Reinforcement Learning Framework for Transfer Strategy

[0212] The purpose of the transmission strategy is to flexibly adjust the base station transmission power in each time slot to reduce the base station energy consumption caused by streaming media transmission while ensuring that users can play videos without interruption. The following sets the states in the transmission strategy reinforcement learning framework action And rewards

[0213] Since the amount of data that the base station needs to transmit should be adjusted according to the channel conditions and the amount of data remaining to be played in the user buffer, the state in the transmission strategy reinforcement learning framework is set to

[0214] action Set to the average data rate that the base station needs to provide in the tth frame, that is

[0215] When determining the average data rate that the base station accessed by the user needs to provide in frame t (The output action is from the reinforcement learning algorithm of the transmission strategy After obtaining), the transmission power of the base station in the i-th time slot of the t-th frame is given by the following formula:

[0216]

[0217] where σ 2 is the power of the noise, g tiis the small-scale fading gain between the user and the base station it connects to in the i-th time slot of the t-th frame, The following equation can be solved by bisection method:

[0218]

[0219] Where W is the transmission bandwidth. In particular, when the small-scale fading obeys the Rayleigh distribution, we have:

[0220]

[0221] in is the exponential integral.

[0222] The reward function is set as:

[0223]

[0224] in is the transmission energy consumption of the base station in the tth frame, λ is the penalty factor, S t+1 Indicates the size of the segment to be played in the next frame, where B t+1 =max{B t +R t -S t I end (t), 0} is the amount of data to be played in the user cache at the next time step, R t is the amount of data received by the user in the tth frame, the indicative function I end (t) = 1 means that the current segment has been played when the tth frame ends, otherwise I end (t) = 0. Penalty term -λ(S t+1 -B t+1 )×min{B t+1 -S t+1 ,0} only in B t+1 -S t+1 <, that is, it takes effect when the next frame to be played is not transmitted to the user's cache in advance. The role of the penalty item is to ensure the user's service quality, that is, to avoid interruptions in playback.

[0225] After setting the state, action, and reward in the transfer strategy reinforcement learning framework, the existing deep reinforcement learning algorithm can be used to learn the transfer strategy. Considering that both the state and the action belong to continuous space, the deep policy gradient (DDPG) algorithm can be used. The DDPG algorithm is based on the actor-critic architecture. The input is the state Output as Action The parameters are Critics Network The input is the state With action The output is the corresponding Q value. DDPG performs actions by adding a noise term to the output To encourage exploration of the action space, i.e., to perform actions

[0226] Critic Network Parameters Update according to the following formula:

[0227]

[0228] in is a set of sample indexes randomly selected from historical observations, is the learning rate of the critic network, and They are the target actor network and the critic network, and their structures are respectively tr and Q tr Same, and according to Update the parameters, w tr is the update rate.

[0229] Actor Network Parameters Update according to the following formula:

[0230]

[0231] in is the learning rate of the actor network.

[0232] Based on the above two reinforcement learning frameworks, the implementation steps of the method of the embodiment of the present invention are as follows: Figure 5 As shown, including:

[0233] Step 501: Initialize the reinforcement learning framework: set the learning rate δ in the reinforcement learning algorithm rec , and the target network update rate w rec 、w tr , randomly initialize the neural network parameters θ rec ,

[0234] Step 502: The central unit obtains the state based on the observation Randomly select an action with probability ε Select an action with probability 1-ε And sent to the base station accessed by the user;

[0235] Step 503: The base station to which the user is connected takes an action according to the action given by the central unit. Recommend files to users;

[0236] Step 504: The central unit obtains the state based on the observation Select Action And sent to the base station accessed by the user;

[0237] Step 505: The base station accessed by the user is based on And the current channel information, using the dichotomy method to solve equation (3) to obtain And calculate the transmission power of each time slot in the current frame according to formula (2);

[0238] Step 506: If it is determined that the current frame ends, the central unit observes the reward and the next state (the state of the next frame), record the state transition sample Then execute step 507; otherwise, return to step 505;

[0239] Step 507: The central unit randomly selects a small batch of samples from the historical state transition samples Update the critic and actor network parameters according to equations (6) and (7) according to Update the target critic network and actor network parameters and continue to train the reinforcement learning framework;

[0240] Step 508: If it is determined that the user has ended the file playback or the file playback is complete, the central unit observes the reward and the next state (the state of the next time step), record the state transition sample Then execute step 509; otherwise, return to step 504;

[0241] Step 509: The central unit randomly selects a small batch of samples from the historical observation records Update the DQN network parameters according to formula (1) according to Update the target DQN network parameters, return to step 502, and continue training the reinforcement learning framework.

[0242] It should be noted that the reinforcement learning framework provided in the embodiment of the present invention is not limited to the DQN algorithm and DDPG algorithm described above, and other reinforcement learning algorithms, such as dueling DDQN, actor-critic (A2C) and neighbor policy optimization (PPO) algorithms, may also be used.

[0243] The embodiment of the present invention is based on model-free reinforcement learning, and does not require accurate prediction of future information. It can flexibly adjust the recommendation and transmission strategies online according to changes in the environment (network). The embodiment of the present invention does not require road testing to establish a signal map, and does not require the prediction information to be converted into the knowledge required for decision-making, thereby reducing the system implementation and calculation complexity, and is easy to implement; it does not require GPS data, thereby avoiding user privacy issues.

[0244] Moreover, the embodiments of the present invention can significantly reduce the energy consumption of wireless streaming media transmission while ensuring the quality of user service. By using a central unit with strong computing power and large coverage for centralized training, the action in the transmission strategy reinforcement learning framework is selected as the average data rate required by the base station in each frame (making decisions in frames), and by setting the corresponding relationship between the transmission power and the average data rate and the instantaneous channel information, the base station can perform distributed transmission power adjustment according to the real-time channel information in each time slot after receiving the average data rate requirement within the frame given by the central unit. Compared with the central unit making decisions in each time slot, the signaling overhead between the central unit and the base station can be greatly reduced.

[0245] In addition, the embodiment of the present invention recommends files of interest to users based on file index information whose playback time in the user's file request history is greater than a preset threshold through the application of a recommendation strategy reinforcement learning framework, thereby reducing the probability of users terminating file playback prematurely and thus reducing the waste of transmission resources.

[0246] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A streaming media recommendation and transmission method, characterized in that: The method is applied to the central unit and includes: Obtain observation data from the base station; The state in the recommendation strategy reinforcement learning framework is set to the file index whose playing time ratio in the file request history of the user in the observation data is greater than a preset threshold; the action is set to the file index to be recommended to the user at the current time step in the observation data; the reward function is set to the playing time ratio of the file in the observation data; based on the state, action and reward in the recommendation strategy reinforcement learning framework, the recommendation strategy is learned using a deep reinforcement learning algorithm; The state in the transmission strategy reinforcement learning framework is set to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames; the action is set to the average data rate that the base station in the observed data needs to provide in the current frame, and the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information is given; the reward function is set to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality; based on the state, action and reward in the transmission strategy reinforcement learning framework, the transmission strategy is learned using a deep reinforcement learning algorithm; Based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, determine and send to the base station: the file index to be recommended to the user and the average data rate required to be provided by the base station in the current frame, so that the base station can recommend the corresponding file to the user and adjust the transmission power of the file transmission; The recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework are continuously trained based on feedback information of file playback.

2. The method according to claim 1, characterized in that The observation data includes but is not limited to the following information: Index of files whose playing time in the user's file request history is greater than a preset threshold; The time the user watched the requested file and the total playback time of the file; The average channel gain vector of the user in the current frame and the past frames; The amount of data to be played remaining in the user's current cache; The transmission power of the base station to which the user is connected in the current time slot.

3. The method according to claim 1, characterized in that When the reinforcement learning algorithm is a deep Q network DQN algorithm, based on the observation data and the recommendation strategy reinforcement learning framework, determining and sending to the base station a file index to be recommended to the user, includes: Using the file indexes in the observed data whose playing time ratio in the file request history of the user is greater than a preset threshold as the input of the DQN for learning the recommendation strategy, and outputting the Q value estimation of the action of each recommendation strategy reinforcement learning framework; The ε-greedy method is used to execute the strategy, and the selected action is sent to the base station to which the user is connected.

4. The method according to claim 1, characterized in that When the reinforcement learning algorithm is a deep policy gradient DDPG algorithm, based on the observation data and the transmission strategy reinforcement learning framework, the average data rate required to be provided by the base station in the current frame is determined and sent to the base station, including: The amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames are used as inputs of the actor network in the DDPG algorithm, and the output is the action of the transmission strategy reinforcement learning framework; the input of the critic network in the DDPG algorithm is the state and action of the transmission strategy reinforcement learning framework, and the output is the corresponding Q value; DDPG encourages exploration of the action space by adding noise terms to the output when executing actions, and selects the actions to be executed and sends them to the base station to which the user is connected.

5. The method according to claim 1, characterized in that The feedback information based on file playback continuously trains the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, including: When the current frame ends, the reward function and the next state of the transmission strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch of samples are randomly selected to update the network parameters in the transmission strategy reinforcement learning framework; When it is determined that the current file playback is terminated, the reward function and the next state of the recommendation strategy reinforcement learning framework are recorded, and the state transition samples are recorded; a preset batch sample is randomly selected to update the network parameters in the recommendation strategy reinforcement learning framework.

6. A streaming media recommendation and transmission method, characterized in that: The method is applied to a base station, comprising: Receive the file index to be recommended to the user and the average data rate that the base station needs to provide in the current frame sent by the central unit; the file index and the average data rate are determined by the central unit based on the observed data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework; the central unit is used to set the state in the recommendation strategy reinforcement learning framework to the file index whose playback time ratio in the user's file request history in the observed data is greater than a preset threshold; set the action to the file index to be recommended to the user at the current time step in the observed data; set the reward function to the playback time ratio of the files in the observed data; based on the state, action and reward in the recommendation strategy reinforcement learning framework, use The recommendation strategy is learned using a deep reinforcement learning algorithm; the central unit is also used to set the state in the transmission strategy reinforcement learning framework to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frames; the action is set to the average data rate required to be provided by the base station in the current frame in the observed data, and the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information is given; the reward function is set to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality; based on the state, action and reward in the transmission strategy reinforcement learning framework, the transmission strategy is learned using a deep reinforcement learning algorithm; The corresponding file is recommended to the user based on the file index to be recommended to the user, and the transmission power of the file transmission is adjusted based on the average data rate required to be provided by the base station in the current frame.

7. The method according to claim 6, characterized in that The adjusting the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame includes: Based on the average data rate that the base station needs to provide in the current frame, and according to the functional relationship between the transmit power of the base station in each time slot and the average data rate and the channel information, determine the transmit power of each time slot in the current frame; The transmission power of the file transmission is adjusted based on the determined result.

8. A streaming media recommendation and transmission device, characterized in that: The device is applied to the central unit and comprises: A data observation module, used to obtain observation data from a base station; A framework establishment module is used to set the state in the recommendation strategy reinforcement learning framework to the file index whose playback time ratio in the file request history of the user in the observed data is greater than a preset threshold; set the action to the file index to be recommended to the user at the current time step in the observed data; set the reward function to the playback time ratio of the file in the observed data; based on the state, action and reward in the recommendation strategy reinforcement learning framework, use the deep reinforcement learning algorithm to learn the recommendation strategy; set the state in the transmission strategy reinforcement learning framework to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frame; set the action to the average data rate required to be provided by the base station in the current frame in the observed data, and give the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information; set the reward function to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality; based on the state, action and reward in the transmission strategy reinforcement learning framework, use the deep reinforcement learning algorithm to learn the transmission strategy; A processing module, configured to determine and send to a base station: a file index to be recommended to a user and an average data rate required to be provided by the base station in a current frame based on the observation data, the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework, so that the base station can recommend corresponding files to the user and adjust the transmission power of the file transmission; The model training module is used to continuously train the recommendation strategy reinforcement learning framework and the transmission strategy reinforcement learning framework based on feedback information of file playback.

9. A streaming media recommendation and transmission device, characterized in that: The device is applied to a base station, comprising: A receiving module is used to receive a file index to be recommended to a user and an average data rate that a base station needs to provide in a current frame, sent by a central unit; the file index and the average data rate are determined by the central unit based on observed data, a recommendation strategy reinforcement learning framework, and a transmission strategy reinforcement learning framework; the central unit is used to set the state in the recommendation strategy reinforcement learning framework to a file index in the observed data whose playing time ratio in the user's file request history is greater than a preset threshold; set the action to the file index to be recommended to the user in the current time step in the observed data; set the reward function to the playing time ratio of the file in the observed data; and Reward, using deep reinforcement learning algorithm to learn recommendation strategy; the central unit is also used to set the state in the transmission strategy reinforcement learning framework to the amount of data to be played remaining in the user's current cache in the observed data and the average channel gain of the past frame; set the action to the average data rate required to be provided by the base station in the current frame in the observed data, and give the functional relationship between the base station's transmission power in each time slot and the average data rate and channel information; set the reward function to the sum of the negative value of the transmission energy consumption of the current frame in the observed data and the penalty term reflecting the user service quality; based on the state, action and reward in the transmission strategy reinforcement learning framework, use the deep reinforcement learning algorithm to learn the transmission strategy; A recommendation module, used for recommending corresponding files to users based on the file index to be recommended to users; The adjustment module is used to adjust the transmission power of the file transmission based on the average data rate required to be provided by the base station in the current frame.

10. A streaming media recommendation and transmission device, characterized in that: The apparatus comprises: a processor and a memory for storing a computer program capable of running on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 5, or executes the steps of the method described in claim 6 or 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to claim 6 or 7.

Images