Cell cooperative dormancy strategy generation model training method and cell cooperative dormancy method

Abstract

The invention provides a cell cooperative dormancy strategy generation model training method and a cell cooperative dormancy method, and the training method comprises the steps: carrying out the training based on the historical flow data of each cell in a target region corresponding to each unit time and expert demonstration action data used for controlling the cooperative dormancy of each cell, and obtaining an expert demonstration network through training; and migrating the expert demonstration network to a current neural network and a target neural network of a reinforcement learning agent, continuing to learn an expert demonstration result on the basis of the network based on a deep reinforcement learning mode, and continuously updating an optimization strategy to train to obtain a cell cooperative dormancy strategy generation model. According to the invention, the training effectiveness and reliability of the cell cooperative dormancy strategy generation model can be improved, and the problem of decision limitation in imitation learning can be solved, so that the application effectiveness and reliability of the cell cooperative dormancy strategy generated based on the cell cooperative dormancy strategy generation model can be improved; therefore, a dynamic and adaptive cell switching decision is realized.

Description

Technical Field

[0001] The present application relates to the field of wireless communication network technology, and in particular to a cell collaborative sleep strategy generation model training method and a cell collaborative sleep method. Background Art

[0002] With the rapid development of modern communication technology, the communication load has exploded. The emergence of 5G communication technology has enabled the large-scale deployment of 5G cells. According to statistics, cells account for 60% to 80% of the power consumption of communication technology. The large amount of power consumption generated by cells not only increases CO2 emissions, but also increases the operating expenses (OPEX) of operators. In recent years, with the widespread application of artificial intelligence (AI), more and more studies have applied AI algorithms to cell energy saving.

[0003] Traditional single-cell decision-making has problems such as low resource utilization, signal interference, and inability to achieve seamless switching, and cannot adapt to the current complex communication environment. The introduction of multi-cell collaborative decision-making can improve the performance, coverage, capacity and user experience of mobile communication networks, while also helping to reduce network operating costs and energy consumption. Multi-cell collaborative decision-making refers to the cooperation between different cells to make switching decisions to optimize network performance and user experience. Formulating a cell energy-saving plan that coordinates multiple standard cells can optimize the energy consumption and network performance of the entire network.

[0004] In order to solve the problem of multi-cell collaborative energy saving, many studies have been conducted by establishing the optimization problem of cell sleep and resource allocation and using heuristic algorithms to iteratively solve it. For example, a researcher proposed a centralized sleep scheme (CSS) to consider the performance of BS state stability, which is defined as the number of cell on or off state transitions. Furthermore, in order to minimize the system energy consumption and improve the system state stability, a dual-objective optimization problem was established, and then a fast exhaustive algorithm (CSSE) and a low-complexity improved particle swarm algorithm (CSS-PSO) were proposed to solve the dual-objective optimization problem. Secondly, a researcher developed a multi-objective optimization framework to reduce the overall energy consumption of the network through the cell sleep mode (SLM) in an OFDMA-based cellular system. To find the optimal set of active cells for SLM operation, a genetic algorithm was used to obtain the solution of the considered multi-objective optimization problem with a fast convergence rate.

[0005] However, when modeling large-scale cell collaboration, the above heuristic cell collaborative energy-saving scheme may require a lot of computing resources and time to find a feasible solution, and there is no guarantee of finding the optimal solution. In response to this, some researchers have applied deep reinforcement learning to the problem of cell energy saving. This is because, on the one hand, reinforcement learning can dynamically adjust strategies and obtain effective strategies by continuously learning and training a large amount of data. On the other hand, reinforcement learning considers long-term returns, can consider the temporal correlation of cell energy-saving issues over a period of time, and considers the impact of cell sleep on the network in many aspects. However, the centralized cell collaborative sleep algorithm based on reinforcement learning faces the problem of non-stationarity in the cell collaborative sleep scenario. In the communication environment, traffic will show significant fluctuations at different time periods in a day, which will lead to the instability of the dynamic environment of the cell switch. Due to the interference of these factors, the calculation results of the reward function are also prone to large fluctuations. This fluctuation may cause the agent to be unable to identify whether it is caused by action or environment, which further affects the final learning effect.

[0006] To solve this problem, some researchers have proposed an imitation learning method based on behavior cloning (BC). This method learns random or deterministic policy networks by imitating the behavior of experts, thereby avoiding the non-stationarity problem faced by reinforcement learning algorithms in the scenario of coordinated sleep in a cell. However, this method only updates the policy network by learning the behavior of experts, does not require learning rewards, and can only learn the optimal strategy under expert demonstration, and cannot break through the strategy demonstrated by experts.

[0007] That is to say, firstly, in the training process of the existing cell cooperative sleep strategy generation model, most of them use traditional deep reinforcement learning algorithms to establish dynamic cell switch models. Due to the dynamic nature of the communication environment, traditional deep reinforcement learning models may not be able to adapt to highly dynamic environments, resulting in poor model training effects; secondly, existing research introduces imitation learning to solve the non-stationarity problem in deep reinforcement learning. However, by simply imitating the actions demonstrated by experts, strategies can only be learned under expert demonstrations, and strategies demonstrated by experts cannot be surpassed.

[0008] Based on this, there is an urgent need to design a new cell collaborative sleep strategy generation model training method to simultaneously solve the problems of poor reinforcement learning training effect caused by the dynamics and complexity of the environment and the decision-making limitations in the imitation learning model. Summary of the invention

[0009] In view of this, the embodiments of the present application provide a cell collaborative sleep strategy generation model training method and a cell collaborative sleep method to eliminate or improve one or more defects existing in the prior art.

[0010] One aspect of the present application provides a method for training a cell collaborative sleep strategy generation model, including:

[0011] Based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each cell, a preset neural network is trained to train the neural network into an expert demonstration network for predicting the expert demonstration action data corresponding to each cell in the target area;

[0012] The network parameters of the expert demonstration network are used to initialize the current neural network and the target neural network in the reinforcement learning agent, so that the reinforcement learning agent performs joint iterative training on the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area.

[0013] In some embodiments of the present application, the method of training a preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each cell, so as to train the neural network as an expert demonstration network for predicting the expert demonstration action data corresponding to each cell in the target area, includes:

[0014] Obtaining target expert demonstration action vectors corresponding to each of the unit times for controlling the coordinated sleep of each of the cells;

[0015] Convert the target expert demonstration action vectors corresponding to each of the unit times into corresponding target expert demonstration action instructions respectively;

[0016] The target expert demonstration action indication corresponding to each of the unit times is used as a label of the pre-acquired historical traffic data of each of the cells corresponding to each of the unit times, so as to obtain samples corresponding to each of the unit times and an expert demonstration action data set composed of each of the samples;

[0017] Dividing the expert demonstration action dataset into a training set and a validation set;

[0018] Based on the training set, a preset neural network is trained using a stochastic gradient descent algorithm, and the trained neural network is verified using the verification set, so that the verified neural network is determined as the expert demonstration network currently used to predict the expert demonstration action data of the target area within a unit time after the current time, and the expert demonstration network parameters corresponding to the expert demonstration network are stored.

[0019] In some embodiments of the present application, the obtaining of the target expert demonstration action vectors corresponding to each of the unit times for controlling the coordinated sleep of each of the cells includes:

[0020] An expert demonstration algorithm based on heuristic search is used to calculate and obtain first expert demonstration action vectors corresponding to each of the unit times and used to control the coordinated sleep of each of the cells;

[0021] And, using an expert demonstration algorithm based on the ranking of cell values ​​to calculate and obtain second expert demonstration action vectors corresponding to each of the unit times and used to control the coordinated sleep of each of the cells;

[0022] According to the power consumption values ​​of the first expert demonstration motion vector and the second expert demonstration motion vector corresponding to each unit time, one of the power consumption values ​​of the first expert demonstration motion vector and the second expert demonstration motion vector corresponding to each unit time is selected as the target expert demonstration motion vector corresponding to the unit time.

[0023] In some embodiments of the present application, the method of initializing the current neural network and the target neural network in the reinforcement learning agent using the network parameters of the expert demonstration network, so that the reinforcement learning agent performs joint iterative training on the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area, includes:

[0024] Initializing the current neural network and the target neural network in the reinforcement learning agent using the expert demonstration network parameters;

[0025] Initialize an experience replay buffer and a maximum number of iterations corresponding to the reinforcement learning agent, wherein the experience replay buffer is used to store tuple samples, and the tuple samples include: current state, action, reward, and next state;

[0026] The working state of each cell in a unit time is used as the action corresponding to the unit time, the predicted traffic of each cell in the unit time and the sleep state switch action of the previous unit time are used as the current state corresponding to the unit time, and a reward function corresponding to the reward is set; wherein the working state includes: an active state represented by 1 and a sleep state represented by 0;

[0027] Initialize the current neural network and the target neural network of the reinforcement learning agent according to the expert demonstration network parameters;

[0028] Selecting the action according to the current neural network and executing the action, and calculating the corresponding reward based on the reward function, at which time the environment state changes to the next state to obtain the corresponding tuple sample, and storing the tuple sample in the experience replay buffer;

[0029] If the data of the tuple samples in the experience replay buffer meets the preset quantity threshold, a plurality of the tuple samples are randomly selected from the experience replay buffer so that the reinforcement learning agent, in the current iteration round, uses a deep reinforcement learning method to jointly train the current neural network and the target neural network in the reinforcement learning agent based on the randomly selected tuple samples until the number of iterations reaches the maximum number of iterations, then the training is stopped, and the network parameters of the current neural network are used as model parameters corresponding to the cell collaborative sleep strategy generation model for generating a prediction result of the cell collaborative sleep strategy for the target area within a unit time after the current time.

[0030] In some embodiments of the present application, the reward function is composed of a benefit function of power saving, a user's service quality satisfaction function, and an action error function that imitates an expert strategy;

[0031] The value of the power saving benefit function is calculated based on the total power consumption corresponding to a unit time of all cells in the target area that are in an active state, the working state of each cell in the unit time, and the power consumption model;

[0032] The value of the user's service quality satisfaction function is calculated based on the pre-acquired user available service rate, user required traffic and switch variables;

[0033] The value of the action error function imitating the expert strategy is calculated based on the pre-acquired target expert demonstration action vector;

[0034] The user available service rate is calculated based on the bandwidth resources allocated by the active cell to the associated user in unit time and the signal-to-noise ratio from the active cell to the associated user.

[0035] In some embodiments of the present application, before training a preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each cell, the method further includes:

[0036] Collecting historical traffic data, the total number of users and the total number of cells corresponding to each cell in the target area, and preprocessing the historical traffic data to obtain the traffic data of each cell in the target area at each unit time;

[0037] Inputting the flow data of each of the cells at each of the unit times into a preset flow prediction model, so that the flow prediction model outputs a flow prediction result of each of the cells at a unit time after each of the unit times;

[0038] The traffic prediction results of each cell at one unit time after each unit time are summed to obtain the predicted total traffic of the target area at one unit time after each unit time, and based on the predicted total traffic and the total number of users, the user expected traffic and user demand traffic corresponding to one unit time after each unit time are allocated to each user in the target area.

[0039] In some embodiments of the present application, before training a preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each cell, the method further includes:

[0040] Setting a power consumption model, wherein the power consumption model corresponding to each cell in a unit time is composed of a fixed power consumption of the cell in the unit time and a transmission power consumption of the cell in the unit time;

[0041] The cell working state is set, where the cell working state includes: a value of 1 is when the cell is in an active state for a unit time, and a value of 0 is when the cell is in a dormant state for a unit time.

[0042] In some embodiments of the present application, the method of calculating the first expert demonstration action vectors for controlling the coordinated sleep of each of the cells corresponding to each of the unit times by using an expert demonstration algorithm based on heuristic search includes:

[0043] The working state of each cell in the target area is defined as the position of a particle, the total power consumption of all cells in the target area is defined as the fitness of a particle, a particle group is defined according to a plurality of switch combinations for all cells in the target area, and an update process of a particle is defined;

[0044] Based on the position of the particle, the fitness of the particle, the particle group and the update process of the particle, an expert demonstration algorithm based on heuristic search is adopted to search for the optimal solution of the corresponding cell collaborative sleep problem for each of the cells in the target area, so as to obtain the first expert demonstration action vector corresponding to each of the unit times for controlling the collaborative sleep of each of the cells.

[0045] In some embodiments of the present application, the expert demonstration algorithm based on the cell value ranking is used to calculate the second expert demonstration action vector corresponding to each of the unit times and used to control the coordinated sleep of each of the cells, including:

[0046] Determine the total traffic load of the target area at each unit time based on the pre-acquired traffic load of each cell at each unit time;

[0047] Determine the cell value of each cell at each unit time according to the traffic load of each cell at each unit time and the total traffic load;

[0048] Inputting the cell value of each of the cells at each historical unit time before a unit time into a preset time series prediction model, so that the time series prediction model outputs the cell value prediction result of each of the cells at each unit time;

[0049] The corresponding working status of each of the cells in the target area is solved according to the cell value prediction results of each of the cells at each unit time, so as to obtain the second expert demonstration action vector corresponding to each of the unit time for controlling the coordinated sleep of each of the cells.

[0050] Another aspect of the present application provides a cell coordinated sleep method, including:

[0051] Obtain target traffic data of each cell in the target area within a historical unit time;

[0052] Input the target traffic data into a preset cell collaborative sleep strategy generation model so that the cell collaborative sleep strategy generation model outputs the cell collaborative sleep strategy prediction result data of the target area within a unit time after the current time, wherein the cell collaborative sleep strategy generation model is pre-trained based on the cell collaborative sleep strategy generation model training method provided in the first aspect of the present application;

[0053] According to the cell coordinated sleep strategy prediction result data, corresponding sleep state switch actions are performed for each of the cells in the target area.

[0054] A third aspect of the present application provides a cell coordinated sleep strategy generation model training device, including:

[0055] A model training module based on expert demonstration, used to train a preset neural network based on historical traffic data of each cell in the target area corresponding to each unit time and expert demonstration action data for controlling the coordinated sleep of each of the cells, so as to train the neural network into an expert demonstration network for predicting the expert demonstration action data corresponding to each of the cells in the target area;

[0056] The model transfer and reinforcement learning module is used to initialize the current neural network and the target neural network in the reinforcement learning agent using the network parameters of the expert demonstration network, so that the reinforcement learning agent can jointly iteratively train the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area.

[0057] A fourth aspect of the present application provides a cell coordinated sleep device, including:

[0058] A traffic acquisition module, used to acquire target traffic data of each cell in the target area within a historical unit time;

[0059] A cell collaborative sleep strategy prediction module, used for inputting the target traffic data into a preset cell collaborative sleep strategy generation model, so that the cell collaborative sleep strategy generation model outputs the cell collaborative sleep strategy prediction result data of the target area within a unit time after the current time, wherein the cell collaborative sleep strategy generation model is pre-trained based on the cell collaborative sleep strategy generation model training method provided in the first aspect of the present application;

[0060] The cell coordinated sleep control module is used to execute corresponding sleep state switch actions for each of the cells in the target area according to the cell coordinated sleep strategy prediction result data.

[0061] The fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the cell collaborative sleep strategy generation model training method and / or the cell collaborative sleep method are implemented.

[0062] The sixth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cell collaborative sleep strategy generation model training method and / or the cell collaborative sleep method.

[0063] The seventh aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the cell collaborative sleep strategy generation model training method and / or the cell collaborative sleep method.

[0064] The cell collaborative sleep strategy generation model training method provided in the present application trains a preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the collaborative sleep of each of the cells, so as to train the neural network into an expert demonstration network for predicting the expert demonstration action data corresponding to each of the cells in the target area; migrate the expert demonstration network to the current neural network and the target neural network in the reinforcement learning agent, so that the reinforcement learning agent performs joint iterative training on the current neural network of the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network in the reinforcement learning agent into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area; take into account the communication Due to the dynamics and complexity of the environment, the training method provided in this application introduces the idea of ​​transfer learning. First, the network status and expert demonstration actions are input into the neural network for training, and then the trained network model parameters are transferred to the current neural network and the target neural network in the deep reinforcement learning, so that the reinforcement learning agent can continuously update the optimization strategy based on the expert demonstration results, and generate a cell switching strategy that is better than the expert demonstration. It can simultaneously solve the problems of poor reinforcement learning training effect caused by the dynamics and complexity of the environment and the decision-making limitation problem in the imitation learning model, and can effectively improve the training effectiveness and reliability of the cell collaborative sleep strategy generation model, and can solve the decision-making limitation problem in imitation learning, and can effectively improve the application effectiveness and reliability of the cell collaborative sleep strategy generated based on the cell collaborative sleep strategy generation model, so as to realize dynamic and adaptive cell switching decisions.

[0065] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0066] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0068] Figure 1 This is a schematic diagram of the first flow chart of a method for training a cell collaborative sleep strategy generation model in an embodiment of the present application.

[0069] Figure 2 A second flow chart of a method for training a cell collaborative sleep strategy generation model in an embodiment of the present application.

[0070] Figure 3 This is a third flow chart of a method for training a cell collaborative sleep strategy generation model in an embodiment of the present application.

[0071] Figure 4 This is a fourth flow chart of a method for training a cell collaborative sleep strategy generation model in an embodiment of the present application.

[0072] Figure 5 A flowchart of an expert demonstration algorithm based on heuristic search provided for Application Example 3 of this application.

[0073] Figure 6 A flowchart of an expert demonstration algorithm based on cell value ranking provided for application example 4 of the present application.

[0074] Figure 7 It is a flowchart of a cell coordinated sleep method in an embodiment of the present application.

[0075] Figure 8 A schematic diagram of the structure of a cell collaborative sleep strategy generation model training device in an embodiment of the present application.

[0076] Fig. 9 It is a structural diagram of a cell coordinated sleep device in one embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0078] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0079] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0080] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0081] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0082] In order to simultaneously solve the problem of poor reinforcement learning training effect caused by the dynamics and complexity of the environment and the problem of limited decision-making in the imitation learning model, the embodiments of the present application respectively provide a cell collaborative sleep strategy generation model training method, a cell collaborative sleep method, a cell collaborative sleep strategy generation model training device for executing the cell collaborative sleep strategy generation model training method, a cell collaborative sleep device for executing the cell collaborative sleep method, an electronic device, a computer-readable storage medium and a computer program product, which can realize dynamic and adaptive cell switching decisions, improve the problem of poor training effect in traditional reinforcement learning due to the high complexity of the environment, and solve the problem of limited decision-making in imitation learning.

[0083] The details are described in detail through the following examples.

[0084] Based on this, the embodiment of the present application provides a cell collaborative sleep strategy generation model training method that can be implemented by a cell collaborative sleep strategy generation model training device, see Figure 1 The cell collaborative sleep strategy generation model training method specifically includes the following contents:

[0085] Step 100: Based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each of the cells, a preset neural network is trained to train the neural network into an expert demonstration network for predicting the expert demonstration action data corresponding to each of the cells in the target area.

[0086] In one or more embodiments of the present application, each unit time refers to a time period of equal and continuous duration.

[0087] It is understandable that in order to enable the reinforcement learning agent (agent, which can be referred to as the agent) to adapt to the dynamics of the environment, step 100 introduces expert demonstrations so that the agent can continuously optimize the switching strategy according to the expert's actions. Considering that only imitating the expert's actions will result in the agent being unable to produce a strategy that exceeds or is equal to the expert's actions, in order to solve this problem, this embodiment first inputs the expert demonstration actions into the neural network for training before the reinforcement learning algorithm is trained to obtain the expert demonstration parameters, and then the following step 200 can be used to migrate the trained expert demonstration network to the current neural network and the target neural network in the reinforcement learning, specifically referring to: migrating the model parameters (also referred to as expert demonstration parameters) corresponding to the trained expert demonstration network to the current neural network and the target neural network in the reinforcement learning.

[0088] Step 200: Use the network parameters of the expert demonstration network to initialize the current neural network and the target neural network in the reinforcement learning agent, so that the reinforcement learning agent performs joint iterative training on the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area.

[0089] In step 200, the current neural network generates a strategy through training expert demonstration, and through joint training with the target neural network, the final cell collaborative sleep strategy is obtained through continuous iteration. For the reinforcement learning part, this embodiment adopts the Deep Q Learning (DQN) algorithm, and determines the model parameters in the final converged current neural network as the cell collaborative sleep strategy generation model for the final online application.

[0090] In one or more embodiments of the present application, the expert demonstration action data for controlling the coordinated sleep of each of the cells refers to the expert demonstration of the sleep state switch action for changing the corresponding sleep state of each cell in the target area within a unit time; the cell coordinated sleep strategy includes executing the corresponding sleep state switch action for each of the cells in the target area within a unit time after the current time; wherein the sleep state switch action includes: an action of turning on the sleep state of the cell and an action of turning off the sleep state of the cell.

[0091] From the above description, it can be seen that the cell collaborative sleep strategy generation model training method provided in the embodiment of the present application can effectively improve the training effectiveness and reliability of the cell collaborative sleep strategy generation model, and can solve the decision-making limitation problem in imitation learning, thereby effectively improving the application effectiveness and reliability of the cell collaborative sleep strategy generated based on the cell collaborative sleep strategy generation model, so as to realize dynamic and adaptive cell switching decisions.

[0092] In order to further improve the effectiveness and reliability of the training expert demonstration network process, in a cell collaborative sleep strategy generation model training method provided in an embodiment of the present application, see Figure 2 , step 100 in the cell collaborative sleep strategy generation model training method specifically includes the following contents:

[0093] Step 110: Obtain the target expert demonstration action vector corresponding to each of the unit times and used to control the coordinated sleep of each of the cells.

[0094] It can be understood that the target expert demonstration motion vector can also be called the optimal expert demonstration motion vector.

[0095] Step 120: Convert the target expert demonstration action vectors corresponding to each of the unit times into corresponding target expert demonstration action instructions respectively.

[0096] Step 130: Use the target expert demonstration action instructions corresponding to each of the unit times as labels of the pre-acquired historical traffic data of each of the cells corresponding to each of the unit times, so as to obtain samples corresponding to each of the unit times and an expert demonstration action data set composed of each of the samples.

[0097] Step 140: Divide the expert demonstration action dataset into a training set and a validation set.

[0098] Step 150: Based on the training set, a preset neural network is trained using a stochastic gradient descent algorithm, and the trained neural network is verified using the verification set, so as to determine the verified neural network as the expert demonstration network currently used to predict the expert demonstration action data of the target area within a unit time after the current time, and store the expert demonstration network parameters corresponding to the expert demonstration network.

[0099] In the transfer reinforcement learning method based on expert demonstration, in order to obtain expert demonstration actions, this application proposes two centralized expert demonstration algorithms, namely, an expert demonstration algorithm based on heuristic search and an expert demonstration algorithm based on cell value ranking. These two algorithms can shut down cells to the maximum extent, and the second algorithm can greatly reduce the time complexity; correspondingly, in a cell collaborative sleep strategy generation model training method provided in an embodiment of the present application, see Figure 3 , step 110 in the cell collaborative sleep strategy generation model training method specifically includes the following contents:

[0100] Step 111: using an expert demonstration algorithm based on heuristic search to calculate and obtain first expert demonstration action vectors corresponding to each of the unit times for controlling the coordinated sleep of each of the cells.

[0101] And, step 112: using an expert demonstration algorithm based on cell value ranking to calculate a second expert demonstration action vector corresponding to each of the unit times for controlling the coordinated sleep of each of the cells.

[0102] It can be understood that the first expert demonstration action vector and the second expert demonstration action vector are both expert demonstration action vectors, and the difference in their expressions is only used to distinguish between the result data calculated by the expert demonstration algorithm based on heuristic search or the result data calculated by the expert demonstration algorithm based on cell value ranking.

[0103] Step 113: According to the power consumption values ​​of the first expert demonstration motion vector and the second expert demonstration motion vector corresponding to each of the unit times, select one of the power consumption values ​​of the first expert demonstration motion vector and the second expert demonstration motion vector corresponding to each of the unit times as the target expert demonstration motion vector corresponding to the unit time.

[0104] In order to further improve the effectiveness and reliability of the reinforcement learning process, in a cell collaborative sleep strategy generation model training method provided in an embodiment of the present application, see Figure 2 or Figure 3 , step 200 in the cell collaborative sleep strategy generation model training method specifically includes the following contents:

[0105] Step 210: Use the expert demonstration network parameters to initialize the current neural network and the target neural network in the reinforcement learning agent.

[0106] Step 220: Initialize the experience replay buffer and the maximum number of iterations corresponding to the reinforcement learning agent, wherein the experience replay buffer is used to store tuple samples, and the tuple samples include: current state, action, reward and next state.

[0107] Step 230: The working state of each of the cells in a unit time is taken as the action corresponding to the unit time, the predicted traffic of each of the cells in the unit time and the sleep state switch action in the previous unit time are taken as the current state corresponding to the unit time, and a reward function corresponding to the reward is set; wherein the working state includes: an active state represented by 1 and a sleep state represented by 0.

[0108] Step 240: Initialize the current neural network and target neural network of the reinforcement learning agent according to the expert demonstration network parameters.

[0109] Step 250: Select the action according to the current neural network and execute the action, and calculate the corresponding reward based on the reward function. At this time, the environment state changes to the next state to obtain the corresponding tuple sample, and store the tuple sample in the experience replay buffer.

[0110] Step 260: If the data of the tuple samples in the experience replay buffer meets the preset quantity threshold, a plurality of the tuple samples are randomly selected from the experience replay buffer so that the reinforcement learning agent, in the current iteration round, uses deep reinforcement learning to jointly train the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on the randomly selected tuple samples until the number of iterations reaches the maximum number of iterations, then the training is stopped, and the network parameters of the current neural network are used as model parameters corresponding to the cell collaborative sleep strategy generation model for generating a prediction result of the cell collaborative sleep strategy for the target area within a unit time after the current time.

[0111] In order to further improve the application effectiveness and reliability of the reward function, in a cell collaborative sleep strategy generation model training method provided in an embodiment of the present application, the reward function in the cell collaborative sleep strategy generation model training method is composed of a power consumption saving benefit function, a user's service quality satisfaction function, and an action error function that imitates an expert strategy;

[0112] The value of the power saving benefit function is calculated based on the total power consumption corresponding to a unit time of all cells in the target area that are in an active state, the working state of each cell in the unit time, and the power consumption model;

[0113] The value of the user's service quality satisfaction function is calculated based on the pre-acquired user available service rate, user required traffic and switch variables;

[0114] The value of the action error function imitating the expert strategy is calculated based on the pre-acquired target expert demonstration action vector;

[0115] The user available service rate is calculated based on the bandwidth resources allocated by the active cell to the associated user in unit time and the signal-to-noise ratio from the active cell to the associated user.

[0116] In order to further improve the calculation effectiveness and reliability of service quality satisfaction, in a cell collaborative sleep strategy generation model training method provided in an embodiment of the present application, see Figure 4 , the cell collaborative sleep strategy generation model training method further specifically includes the following contents before step 100:

[0117] Step 010: Collect historical traffic data, total number of users and total number of cells corresponding to each cell in the target area, and pre-process the historical traffic data to obtain traffic data of each cell in the target area at each unit time.

[0118] Step 020: Input the traffic data of each of the cells at each of the unit times into a preset traffic prediction model, so that the traffic prediction model correspondingly outputs the traffic prediction results of each of the cells at a unit time after each of the unit times.

[0119] Step 030: Sum the traffic prediction results of each of the cells at one unit time after each of the unit times to obtain the predicted total traffic of the target area at one unit time after each of the unit times, and allocate the user expected traffic and user demand traffic corresponding to one unit time after each of the unit times to each of the users in the target area based on the predicted total traffic and the total number of users.

[0120] In order to further improve the calculation effectiveness and reliability of the benefits of power saving, in a cell cooperative sleep strategy generation model training method provided in an embodiment of the present application, see Figure 4 The cell collaborative sleep strategy generation model training method further includes the following contents before step 100 and after step 030:

[0121] Step 040: Setting a power consumption model, wherein the power consumption model corresponding to each cell in a unit time is composed of the fixed power consumption of the cell in the unit time and the transmission power consumption of the cell in the unit time;

[0122] Step 050: Set the cell working state, where the cell working state includes: a cell being in an active state for a unit time is a value of 1, and a cell being in a dormant state for a unit time is a value of 0.

[0123] In order to further improve the application effectiveness and reliability of the expert demonstration algorithm based on heuristic search, in a cell collaborative sleep strategy generation model training method provided in an embodiment of the present application, step 111 in the cell collaborative sleep strategy generation model training method specifically includes the following contents:

[0124] 1) The working state of each cell in the target area is defined as the position of the particle, the total power consumption of all cells in the target area is defined as the fitness of the particle, a particle group is defined according to multiple switch combinations for all cells in the target area, and an update process of the particle is defined.

[0125] 2) Based on the position of the particle, the fitness of the particle, the particle swarm and the update process of the particle, an expert demonstration algorithm based on heuristic search is used to search for the optimal solution of the corresponding cell collaborative sleep problem for each of the cells in the target area, so as to obtain the first expert demonstration action vector corresponding to each of the unit times for controlling the collaborative sleep of each of the cells.

[0126] In order to further improve the application effectiveness and reliability of the expert demonstration algorithm based on cell value ranking, in a cell collaborative sleep strategy generation model training method provided in an embodiment of the present application, step 112 in the cell collaborative sleep strategy generation model training method specifically includes the following contents:

[0127] 1) Based on the pre-acquired traffic load of each of the cells at each unit time, determine the total traffic load of the target area at each unit time.

[0128] 2) According to the traffic load of each cell at each unit time and the total traffic load, the cell value of each cell at each unit time is determined respectively.

[0129] 3) Inputting the cell value of each of the cells at each historical unit time before a unit time into a preset time series prediction model, so that the time series prediction model outputs the cell value prediction result of each of the cells at each unit time.

[0130] 4) Solving the corresponding working status of each of the cells in the target area according to the cell value prediction results of each of the cells at each unit time, so as to obtain the second expert demonstration action vector corresponding to each of the unit time for controlling the coordinated sleep of each of the cells.

[0131] In order to further illustrate the cell collaborative sleep strategy generation model training method provided in the above embodiment, the present application also provides the following four application examples.

[0132] (I) Application Example 1

[0133] Application Example 1 provides a cell cooperative sleep strategy generation model training method based on expert demonstration transfer reinforcement learning, which is used to explain the aforementioned embodiments of steps 100 to 200, embodiments of steps 111 to 113, embodiments of steps 210 to 260, embodiments of reward functions, embodiments of steps 010 to 030, and embodiments of steps 040 to 050. Specifically, Application Example 1 includes the following contents:

[0134] Step 1-1: Collect cell traffic data. Collect cell historical traffic data (air interface uplink and downlink traffic, Packet Data Convergence Protocol (PDCP) uplink and downlink traffic, etc.) through the network management or data collection system, and perform pre-processing such as cleaning, construction, aggregation, and screening on the collected data, such as removing outliers and filling in missing values.

[0135] Step 1-2: Cell traffic prediction. Based on the cell historical traffic obtained in step 1-1, a machine learning algorithm is used to predict future cell traffic. This is because the cell switching decision is made based on the cell traffic in the future time period. Therefore, it is necessary to perform time series prediction on the cell historical traffic collected in step 1-1. For the time series prediction of cell traffic, traffic prediction models such as the Holt-winters algorithm and the Long Short Term Memory (LSTM) network can be used. Assuming that a target area contains M cells and N users, the cell set and user set can be expressed as and Where m represents the mth cell in the target area, and n represents the nth user in the target area. The traffic of all M cells in the tth unit time can be expressed as in represents the traffic volume of the mth cell at the tth unit time. In order to predict the cell traffic volume of cell m at the tth unit time, the historical traffic sequence of l consecutive unit times before the tth unit time is Input the aforementioned traffic prediction model, and the traffic prediction model outputs the traffic prediction result at the tth unit time, which is defined as This application example can further derive the predicted traffic sequence of cell m within the next T time.

[0136] Step 1-3: User traffic acquisition. By predicting the traffic of all cells in the target area, the predicted traffic of all cells at the tth unit time can be obtained, that is, At the tth unit time, the predicted traffic of all cells in the target area is first summed up to obtain the predicted total traffic in the target area at the tth unit time in the future, that is, Then, given the total traffic in the target area, simulate the traffic of each user. Assuming that the traffic of all users follows Poisson distribution, the specific calculation process is as follows:

[0137] Step 1-31: User expected traffic distribution. From step 1-1, we can get that the total number of users in the entire target area is N, and the traffic of each user follows Poisson distribution. Assume that the expected value of the traffic of each user is Forecast the total flow in the target area The common method of allocating to each user is uniform distribution. The expected traffic of user n can be obtained by formula (1):

[0138]

[0139] If there are specific user preferences, weighted allocation can be made based on this information.

[0140] Step 1-32: Obtain user demand traffic. For each user n, since the Poisson distribution describes the number of events that occur within a fixed time interval, the demand traffic of each user n is is a parameter The Poisson random variable can be obtained by formula (2):

[0141]

[0142] Finally, the demand flow of all users in the target area can be expressed as

[0143] Step 1-4: Fitting the cell power consumption model. In a wireless communication network, the power consumption of a cell is not only proportional to the service load within its coverage area. As a major energy consumer in the communication network, the power consumption of a cell is mainly divided into fixed power consumption and transmission power consumption. Among them, fixed power consumption is related to hardware equipment such as refrigeration equipment and cell circuits, and usually does not change much. Transmission power consumption is related to power amplifiers, radio frequency units, etc., and it is related to the transmission load of the cell. This application example fits a general power consumption model by collecting the power consumption values ​​and traffic loads of the cell at different times. The power consumption model of cell m in the tth unit time is shown in formula (3):

[0144]

[0145] Among them, P fix represents fixed power consumption, is the transmission power consumption of cell m, which is nonlinearly related to the traffic load of cell m. Formula (3) can be obtained by fitting the relationship between cell power consumption and traffic load.

[0146] Step 1-5: Define the working state of the cell and the association state of the user with the cell. Based on the power consumption model of the cell in the active state obtained in step 1-4, the power consumption of the cell under different traffic loads can be obtained. Since the power consumption of the cell is huge, this application example aims to reduce the power consumption of the cell by switching the low-load cell to sleep mode. In order to ensure that each user can be served by the cell in the communication network, a 0-1 switch variable is introduced. To indicate whether user n is associated with cell m at the tth unit time. If user n is associated with cell m at the tth unit time, then on the contrary, It can be expressed as

[0147] Step 1-6: Cell collaborative sleep. In order to reduce cell power consumption and improve network efficiency, this application example adjusts some cells with low traffic load to sleep state to reduce unnecessary power consumption. Based on the cell power consumption model and cell working status defined in the above steps 1-4 to 1-5, this application example uses a deep reinforcement learning model to solve the problem of cell collaborative sleep. In order to enhance the adaptability of the intelligent agent to the environment and improve the training effect, this application example proposes a cell collaborative sleep method based on expert demonstration transfer reinforcement learning, including the following steps 1-61 to step 1-62:

[0148] Step 1-61: First, convert the cell cooperative sleep optimization problem into a Markov decision process (MDP) and define the variables as follows:

[0149] 1) Action: The action determines the working state (active / dormant) of all M cells at the tth unit time, which can be written as in, When cell m is active at the tth unit time is 1, otherwise it is 0.

[0150] 2) State: The system state is determined by the predicted traffic of all M cells in the entire target area at the tth unit time. and the cell switch action of the t-1th unit time, expressed as The predicted traffic of all N users in the entire target area at the tth unit time Obtained from steps 1-3.

[0151] 3) Reward function: In order to evaluate the current switching strategy of the cell, the reward function can be t Take action a t The instantaneous reward obtained is used as the basis for judgment. In order to ensure that the user's QoS is not reduced while reducing the cell power consumption, and to achieve a balance between power consumption and user service quality, the reward function can be R(s t ,a t ) The benefit of power saving c p (s t ,a t ), user's satisfaction with quality of service (QoS) c s (s t ,a t ) and the action error c of imitating the expert strategy exp (s t ,a t ). The reward function can be obtained by formula (4), where λ, μ, and v are the weight parameter values ​​of each part of the reward function:

[0152] R(s t , a t )=λ·c p (s t ,a t )+μ·c s (s t ,a t )+v·c exp (s t ,a t ) (4)

[0153] (3a) The benefit of power saving in the reward function c p (s t ,a t ) is shown in formula (5):

[0154]

[0155] in, It represents the total power consumption when all M cells are in active state in the tth unit time. Formula (5) represents the benefit of the overall power consumption reduction of the target area due to the cell adopting the sleep strategy.

[0156] (3b) The user’s QoS satisfaction c in the reward function s (s t ,a t ) is shown in formula (6):

[0157]

[0158] User QoS satisfaction is determined by the service rate that the user can obtain and the ratio of the user's required service volume. Among them, the service rate that the user can obtain (that is, the user's available service rate) The service rate that user n can obtain by associating to active cell m at the tth unit time; the user's required service volume Obtained from steps 1-32.

[0159] in, The service rate that user n can obtain when associated with active cell m at the tth unit time can be calculated using the Shannon formula, as shown in formula (7):

[0160]

[0161] in, Refers to the bandwidth allocated by active cell m to associated user n in the tth unit time. is the signal-to-noise ratio from the active cell m to the associated user n at the tth unit time, expressed as is the power resource allocated to the associated user n by the active cell m at the tth unit time, is the channel gain from active cell m to associated user n at the tth unit time, N 0 is the Gaussian noise power spectral density.

[0162] (3c) The action error c of imitating the expert strategy in the reward function exp (s t ,a t ), the specific calculation is introduced in step 1-62.

[0163] Step 1-62: Transfer reinforcement learning algorithm based on expert demonstration. Based on the Markov decision process defined in step 1-61, this application example proposes a transfer reinforcement learning algorithm based on expert demonstration to solve the problem of coordinated sleep in cells. Specifically, in order to enable the agent to adapt to the dynamics of the environment, this application example introduces expert demonstrations so that the agent can continuously optimize the switching strategy according to the actions of the experts. Considering that only imitating the expert's actions will result in the agent being unable to produce a strategy that exceeds or is equal to the expert's actions, in order to solve this problem, this application example first inputs the expert demonstration actions into the neural network for training before training the reinforcement learning algorithm to obtain the expert demonstration parameters, and then transfers the expert demonstration parameters trained by the neural network to the current neural network and the target neural network in the reinforcement learning agent. The current neural network is jointly trained with the target neural network, and the final coordinated sleep strategy for the cell is obtained through continuous iteration. For the reinforcement learning part, this application example uses the Deep Q Learning (Deep Q Network, DQN) algorithm. The process of the transfer reinforcement learning algorithm based on expert demonstration is as follows:

[0164] 1) Expert demonstration action acquisition. By using the application example three algorithm, the expert demonstration action vector of the heuristic search is obtained. Using application example 4, we get expert demonstration action vectors based on and based on cell value ranking.

[0165] 2) Construction of the optimal expert demonstration action set. The optimal expert demonstration action set in, is the optimal expert demonstration action vector (i.e., the target expert demonstration action vector) for the tth unit time. It is the expert demonstration action vector generated based on application examples 3 and 4. as well as By calculating and comparing the power consumption values ​​of the two, the lower one is selected and recorded as

[0166] 3) Calculation of action error of imitating expert strategy. The influence of expert demonstration action on current action is that for actions in the same cell at the same time, if the two actions are the same, positive reward is given, and if the two actions are different, negative penalty is given. The reward and penalty process is shown in formula (8):

[0167]

[0168] 4) Neural network model training based on expert demonstration. The optimal expert demonstration action set Train the neural network ED demonstrated by the expert and obtain the trained neural network parameters θ EDThe neural network training process based on expert demonstration is detailed in Application Example 2.

[0169] Step 1-7: DQN training parameter initialization and training, including the following steps 1-71 to 1-74:

[0170] Step 1-71: Initialize the experience replay buffer R exp and the maximum number of iterations E. Among them, R exp Stores the current state a t , Actions t , reward R(s t ,a t )={r 1 ,r 2 ,...,r t ...,r T} and the next state a t+1 And other data.

[0171] Step 1-72: Initialize the neural network. The DQN algorithm usually uses two neural networks, namely the current neural network Q w (s t ,a t ;θ) and the target neural network Q w′ (s t ,a t ; θ′). Here, the trained neural network parameters θ are used. ED Initialize the current neural network and the target neural network.

[0172] Step 1-73: Update the Q value of the neural network. Use the Bellman equation to update the Q value, that is, calculate the mean square error between the target Q value and the current Q value, and update the weight of the neural network by gradient descent, which specifically includes the following steps (a) to (e):

[0173] Step (a), according to the current neural network Q w (s t ,a t ;θ) Select action a using the ε-greedy strategy t ; Execute action a t , get reward R(s t ,a t ), the environment state becomes s t+1 .

[0174] Step (b), the above data is converted into (s t ,s t ,r t ,s t+1 ) in the form of experience playback buffer R exp middle.

[0175] Step (c), when R exp When there are enough data in , randomly collect B data from it, and get {(s i ,a i ,r i ,s i+1 )} i＝1,2,...,B .

[0176] Step (d), the target neural network is used to calculate the collected data, and the calculated value is shown in formula (9):

[0177] y i =r i +γmaxQ w′ (s i+1 , a i ;θ′) (9)

[0178] Step (e) calculates the target loss value and uses the gradient descent algorithm to update the current neural network parameter θ. The loss value is shown in formula (10):

[0179]

[0180] Step 1-74: Update the target neural network parameters θ′. This stage requires updating the target neural network Q w′ (s t ,a t ; θ′) parameter θ′. Determine whether the target neural network parameter θ′ needs to be updated at the current time t. If necessary, update the target neural network Q w′ (s t ,a t ;θ′), θ′=θ.

[0181] Step 1-75: Repeat the above process until the maximum number of iterations E is reached and the model converges.

[0182] (II) Application Example 2

[0183] Application Example 2 provides an expert demonstration network training process, which is used to explain the embodiments of the aforementioned steps 110 to 150. Specifically, Application Example 2 includes the following contents:

[0184] Based on the application example 1, this application example proposes a neural network training method based on expert demonstration to obtain the expert demonstration parameter θ ED Specifically, the expert demonstration network training process is as follows:

[0185] Step 2-1: Optimal expert demonstration action dataset The data set has a total of T samples. Among them, sample d t The attribute is the traffic of M cells in the tth unit time Sample d t The label is the optimal expert demonstration action instruction (i.e. the target expert demonstration action instruction) y t . The set of actions demonstrated by the best expert The action vector at the tth unit time is Each bit of the target area cell (i.e., the switch state 0 or 1) is regarded as a binary number, and the binary number is converted into a decimal number and written as y t ,Right now

[0186] Step 2-2: Divide the data set into a training set and a test set. The training set is used to train the parameters of the expert demonstration network ED. The expert demonstration network ED adjusts its own parameters by learning the sample data in the training set to minimize the loss function. The test set is used to evaluate the performance of the expert demonstration network ED during the training process.

[0187] Step 2-3: Randomly initialize the expert demonstration network ED model parameters θ ED_ini , initialize the maximum learning round E ED .

[0188] Step 2-4: Use the stochastic gradient descent algorithm to train the expert demonstration network ED, which specifically includes steps 2-41 and 2-42:

[0189] Step 2-41: Randomly sample a batch of data from As a sample set. The batch refers to a certain number of data samples randomly selected. The size of the batch data is N d , d t is an empirical sample in the sampling batch data D.

[0190] Step 2-42: Update the Expert Demo Network The parameter θ ED , specifically comprising step 2-42(a) and step 2-42(b):

[0191] Step 2-42(a), expert demonstration network parameters θ ED The loss function is shown in formula (11):

[0192]

[0193] Step 2-42(b), using the stochastic gradient descent algorithm, the expert demonstration network The network parameters θ ED Update, the update process is shown in formula (12):

[0194]

[0195] Among them, l ED is the learning rate of the expert demonstration network, l ED ∈[0,1].

[0196] Step 2-5: Test the loss value and prediction accuracy of the expert demonstration network on the validation set.

[0197] Step 2-6: Repeat the above process until the maximum learning round E is reached ED , the model converges, and the model parameters θ ED Save it.

[0198] (III) Application Example 3

[0199] Application Example 3 provides an expert demonstration algorithm based on heuristic search, which is used to explain the embodiment of the above-mentioned expert demonstration algorithm based on heuristic search to calculate the first expert demonstration action vector corresponding to each of the unit time for controlling the coordinated sleep of each of the cells. Specifically, Application Example 3 includes the following contents:

[0200] As can be seen from Application Example 1, this application uses a transfer reinforcement learning algorithm based on expert demonstration to solve the problem of cell collaborative dormancy. The algorithm enables the agent to make strategies that surpass expert demonstrations by learning neural network training parameters based on expert demonstrations. In order to obtain expert demonstration strategies, this application proposes two expert demonstration algorithms, namely, an expert demonstration algorithm based on heuristic search (Application Example 3) and an expert demonstration algorithm based on cell value ranking (Application Example 4).

[0201] This application example explains in detail the expert demonstration algorithm based on heuristic search. The heuristic search refers to a heuristic search (PSO-GA) based on the combination of particle swarm optimization (PSO) and genetic algorithm (GA). In the traditional particle swarm optimization algorithm, the particle position and velocity update process is continuous, but the cell collaborative sleep problem is a binary variable solution process for integer programming. Therefore, based on the traditional particle swarm optimization algorithm, this application example draws on the idea of ​​genetic algorithm to make improvements to the particle update process.

[0202] The flowchart of the expert demonstration algorithm based on heuristic search is as follows: Figure 5 shown.

[0203] Step 3-1: Define particle p k In the traditional particle swarm algorithm, particle p k Has three attributes in is the particle p k The position of represents a feasible solution to the optimization problem. is the particle p k The fitness value of , indicates the quality of the feasible solution. In the heuristic search expert demonstration algorithm, for the problem of cell cooperative sleep, this application example uses particle p k The properties of are redefined as follows:

[0204] Step 3-11: Redefine particle p k Location Indicates the working status of M cells in the overall target area. From the application example 1, we can see that there are M cells in the overall target area, and the cell set is expressed as Particle p k The position can be expressed as Among them, x k,m Refers to the working status of cell m. k,m is a 0-1 integer variable. If cell m is put into dormant state, then x k,m = 0, otherwise, x k,m =1.

[0205] Step 3-12: Redefine particle p k The fitness of is: the overall energy consumption of all cells in the target area. k Middle position The working status of M cells in the indicated target area, the total power consumption of the target area can be expressed as:

[0206]

[0207] Step 3-2: Define the particle swarm P. The algorithm initializes the solution of K cell cooperative sleep problem, that is, K particles, which is equivalent to K switch combinations of all M cells. The particle swarm P composed of K particles to search for the optimal solution of the cell cooperative sleep problem = {p 1 ,p 2 ,...,p K}.

[0208] Step 3-3: Define the particle p in the heuristic search expert demonstration algorithm k The heuristic search expert demonstration algorithm proposed in this application example is based on the traditional particle swarm optimization algorithm and modifies the particle update process. This application example introduces the idea of ​​genetic algorithm to The variable x indicating the working status of a cell m k,mIt is regarded as the gene position in the genetic algorithm, and the mutation operation and crossover operation are introduced into the particle position update process. The specific description is as follows:

[0209] 1) Selection: The selection operation refers to the particle p k By taking the best solution from history and particle swarm historical optimal solution The gene position is randomly selected to generate the particle p in the next iteration k Here, ε 1 and ε 2 Indicates the best solution from history and the group's historical optimal solution The number of loci selected in .

[0210] 2) Crossover: The crossover operation combines the historical optimal solution and the group's historical optimal solution Generate particle p for the next iteration k The coordinate position of .

[0211] 3) Mutation: In order to avoid the local optimal problem, this application example applies mutation operation to update the particle p k Location The remaining gene positions of particle p k Location The gene values ​​of the remaining gene positions are randomly changed, and λ is represented by particle p k The number of mutated gene loci in .

[0212] Step 3-4: Search for the optimal solution to the cell collaborative sleep problem based on the heuristic search expert demonstration algorithm. Taking the solution of the cell collaborative sleep problem for the tth unit time as an example, the specific algorithm flow is as follows:

[0213] Step 3-41: Initialize variables. Initialize particle p k Location (i.e. switching action), initialize the crossover factor ε 1 , ε 2 , and the maximum variation factor λ max , initialize the mutation factor λ = λ ini , initialize the number of iterations N r .

[0214] Step 3-42: User association. According to particle p k Initial position The indicated cell working state, all users in the target area access the active cell closest to them. The user cell association process is detailed in step 62 in application example 1.

[0215] Step 3-43: Determine the validity of the initial particles. Determine particle p k Initial position Whether the determined active cell can meet the user's service rate, that is, whether it meets The service rate of the active cell m' to user n The definition of is detailed in formula (7) in Application Example 1. If there is a user whose service rate cannot be satisfied, the position of the particle cannot be used as a solution to the cell cooperative sleep problem.

[0216] Step 3-44: Initialize the optimal particle solution. According to particle p k Initial position The working status of all M cells indicated and the cell power consumption formula (3) are used to calculate the initial fitness value of the particle. And the current location Set to particle p k The optimal solution of

[0217] Step 3-45: Initialize the optimal solution of the particle swarm. For the particle swarm P, initialize the optimal solution of the particle swarm

[0218] Step 3-46: Particle position update. In the rth iteration, From the particle p k The optimal solution and particle swarm optimal solution Select ε 1 and ε 2 gene positions, and the remaining λ gene positions mutate randomly to update the particle p k Location

[0219] Step 3-47: User reselects a cell. According to particle p k Location (i.e., switching action) The user reselects the adjacent active cell to connect and updates the traffic load of cell m

[0220] Step 3-48: Determine the validity of the particle. Based on the updated traffic load of active cells Determine particle p k The coordinate position of Whether the indicated active cell is overloaded and whether it can meet the user's service rate. If the active cell is overloaded or the user's service rate cannot be met, the position of the particle cannot be used as a solution to the cell coordinated sleep problem.

[0221] Step 3-49: Update the particle fitness value. Using the traffic load of cell m updated in step 3-47 The power consumption of cell m is calculated using formula (3). Based on the calculated power consumption of all M cells, the particle p is calculated according to formula (13): k Current location The fitness value of

[0222] Step 3-410: Update particle p according to formula (14) k The optimal solution in the rth iteration. According to the particle p k Current location The fitness value of And the historical optimal solution of the first r-1 rounds of iterations The fitness value of Update particle p k The historical optimal solution up to the rth iteration Particle p k The historical optimal solution up to round r The update expression is:

[0223]

[0224] Step 3-411: Update the optimal solution of the particle swarm P in the rth iteration process The update expression of the optimal solution of the particle swarm P that searches for the optimal solution of the cell cooperative sleep problem in the rth round of iteration is:

[0225]

[0226] Step 3-412: Determine whether the heuristic search algorithm falls into a local optimal solution during the rth iteration. Not updated and the mutation factor λ does not exceed λ max , then the variation factor λ is increased. This is because if the particle swarm optimal solution No improvement indicates that the algorithm may have fallen into a local optimum at this time, so it is necessary to increase the value of the mutation factor to increase randomness, thereby increasing the trend of expanding the global search.

[0227] Step 3-413: Repeat steps 3-46 to 3-412 until the maximum number of iterations is reached, i.e., r = N. r . r Particle swarm optimal solution of round iteration As the working state of all M cells in the overall target area at the t-th unit time, and as the expert demonstration action vector at the t-th unit time Right now

[0228] (IV) Application Example 4

[0229] Application Example 4 provides an expert demonstration algorithm based on cell value ranking, which is used to explain the embodiment of using the expert demonstration algorithm based on cell value ranking to calculate the second expert demonstration action vector corresponding to each of the unit time for controlling the coordinated sleep of each of the cells. Specifically, see Figure 6 , Application Example 4 contains the following contents:

[0230] This application example 4 proposes an expert demonstration algorithm based on cell value ranking. The algorithm determines whether the cell is active or dormant based on the value of the cell. Specifically, the algorithm closes low-value cells as much as possible and opens high-value cells. The specific process is as follows:

[0231] Step 4-1: Definition of cell value. In order to better evaluate the energy saving potential of a cell, this application example introduces the concept of cell value. In this application example, cell value refers to the cell traffic load. The lower the cell value, the greater the possibility that it will be shut down.

[0232] Step 4-2: Calculate the cell value. According to the cell traffic load acquisition process described in Application Example 1, the traffic load of cell m at the tth unit time can be obtained: Therefore, based on the target area defined in Application Example 1, the total traffic load of the entire target area is expressed as Next, the cell value of cell m at the tth unit time can be expressed as The calculation process is shown in formula (16):

[0233]

[0234] Step 4-3: Prediction of cell value. Since the cell switching decision is based on the cell value in the future time period, it is necessary to predict the historical cell value obtained in step 4-2. Based on the historical value data of the cell, a machine learning algorithm is used to predict the value of a certain time period in the future. For the time series prediction of cell value, traffic prediction models such as the triple exponential smoothing algorithm (Holt-winters) and the long short-term memory network (Long Short Term Memory, LSTM) can be used. In order to predict the cell value of cell m at the tth unit time, the historical value sequence of cell m for l consecutive unit times before the tth unit time is Input the time series prediction model, and after continuous iteration, obtain the prediction output of the prediction model at the tth unit time, which is defined as the cell value prediction result

[0235] Step 4-4: Solve the sleep status of all M cells in the target area based on the cell value prediction results. Use the expert demonstration algorithm based on cell value ranking to obtain the working status of all M cells in the target area at the tth unit time in, is the working state of the mth cell in the tth unit time, When cell m is active at the tth unit time is 1, otherwise it is 0. Taking the solution of the cell coordinated sleep problem of the t-th unit time as an example, the specific algorithm flow is as follows:

[0236] Step 4-41: When iteration round l=0, assume that all M cells are in active state.

[0237] Step 4-42: In the first iteration, based on the M cell value prediction results obtained in step 4-3 Sort the M cells in descending order.

[0238] Step 4-43: In the first iteration, the cell m with the lowest value is closed, and the users connected to the cell m perform cell reselection, and the user connection status of the cell in the first iteration is updated. The cell reselection process of the user is detailed in step 62 of application example 1.

[0239] Step 4-44: According to user connection Statistics are collected on the service flow data of all M cells after cell reselection.

[0240] Step 4-45: Determine whether the cell is overloaded. For the active cell set For each cell m′ in the cell, the latest user connection status obtained after cell reselection And the traffic demand of all N users ρ t , calculate the active cell set The value of each active cell m′ in the calculation formula of the value of the active cell m′ is:

[0241]

[0242] The active cell set obtained according to formula (17) The traffic load of each active cell m′ in the active cell set is used to determine the active cell set Whether each active cell m′ is overloaded.

[0243] Step 4-46: Determine whether the user's required rate is met. For each user n in the user set, determine whether the active cell m' it currently accesses can provide it with a satisfactory service rate. Whether it satisfies The definition of the service rate from the active cell m' to the user n is detailed in formula (6) in application example 1. If the service rate of each user n meets the user traffic demand And if the power consumption obtained according to formula (1) in application example 1 is reduced compared with that before shutdown, the cell m can be put into a dormant state, otherwise it remains in an active state.

[0244] Step 4-47: The algorithm re-sets the active cells The middle cells m′ are sorted in descending order of value, and steps 4-42 to 4-46 are repeated until the active cell set is The cell is overloaded, or the user rate in the user set cannot be met. The working status of the M cells obtained above is used as the expert demonstration action vector of the M cells in the tth unit time

[0245] Based on the above-mentioned cell collaborative sleep strategy generation model training method embodiment, the present application also provides a cell collaborative sleep method embodiment, see Figure 7 The cell coordinated sleep method specifically includes the following contents:

[0246] Step 300: Obtain target traffic data of each cell in the target area within a historical unit time.

[0247] Step 400: Input the target traffic data into a preset cell collaborative sleep strategy generation model so that the cell collaborative sleep strategy generation model outputs the cell collaborative sleep strategy prediction result data of the target area within a unit time after the current time, wherein the cell collaborative sleep strategy generation model is pre-trained based on the cell collaborative sleep strategy generation model training method.

[0248] Step 500: According to the cell coordinated sleep strategy prediction result data, a corresponding sleep state switch action is performed for each of the cells in the target area.

[0249] Among them, the implementation example of the cell collaborative sleep strategy generation model training method provided in step 400 can specifically adopt the processing flow of the implementation example of the cell collaborative sleep strategy generation model training method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the implementation example of the cell collaborative sleep strategy generation model training method in the above-mentioned embodiment.

[0250] From the above description, it can be seen that the cell collaborative sleep method provided in the embodiment of the present application can effectively improve the application effectiveness and reliability of the cell collaborative sleep strategy generated based on the cell collaborative sleep strategy generation model, so as to realize dynamic and adaptive cell switching decisions.

[0251] The present application also provides a cell collaborative sleep strategy generation model training device for executing all or part of the cell collaborative sleep strategy generation model training method, see Figure 8 The cell collaborative sleep strategy generation model training device specifically includes the following contents:

[0252] The model training module 10 based on expert demonstration is used to train a preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each of the cells, so as to train the neural network into an expert demonstration network for predicting the expert demonstration action data corresponding to each of the cells in the target area.

[0253] The model transfer and reinforcement learning module 20 is used to initialize the current neural network and the target neural network in the reinforcement learning agent using the network parameters of the expert demonstration network, so that the reinforcement learning agent can jointly iteratively train the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area.

[0254] The embodiment of the cell collaborative sleep strategy generation model training device provided in the present application can be specifically used to execute the processing flow of the embodiment of the cell collaborative sleep strategy generation model training method in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned cell collaborative sleep strategy generation model training method embodiment.

[0255] The part of the cell collaborative sleep strategy generation model training device that performs cell collaborative sleep strategy generation model training can be executed in the server, and can be completed in the client device. The specific selection can be based on the processing capability of the client device, the limitations of the user's usage scenario, etc. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the cell collaborative sleep strategy generation model training.

[0256] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0257] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0258] From the above description, it can be seen that the cell collaborative sleep strategy generation model training device provided in the embodiment of the present application can effectively improve the training effectiveness and reliability of the cell collaborative sleep strategy generation model, and can solve the decision-making limitation problem in imitation learning, thereby effectively improving the application effectiveness and reliability of the cell collaborative sleep strategy generated based on the cell collaborative sleep strategy generation model, so as to realize dynamic and adaptive cell switching decisions.

[0259] The present application also provides a cell coordinated sleep device for executing all or part of the cell coordinated sleep method, see Fig. 9The cell collaborative sleep device specifically includes the following contents:

[0260] The traffic acquisition module 30 is used to acquire target traffic data of each cell in the target area within a historical unit time.

[0261] The cell collaborative sleep strategy prediction module 40 is used to input the target traffic data into a preset cell collaborative sleep strategy generation model so that the cell collaborative sleep strategy generation model outputs the cell collaborative sleep strategy prediction result data of the target area within a unit time after the current time, wherein the cell collaborative sleep strategy generation model is pre-trained based on the cell collaborative sleep strategy generation model training method.

[0262] The cell coordinated sleep control module 50 is used to execute corresponding sleep state switch actions for each of the cells in the target area according to the cell coordinated sleep strategy prediction result data.

[0263] The embodiment of the cell collaborative sleep device provided in the present application can be specifically used to execute the processing flow of the embodiment of the cell collaborative sleep method in the above embodiment. Its functions are not repeated here, and reference can be made to the detailed description of the above cell collaborative sleep method embodiment.

[0264] The part of the cell collaborative sleep device that performs the cell collaborative sleep can be executed in the client device or in the server. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the cell collaborative sleep.

[0265] From the above description, it can be seen that the cell collaborative sleep device provided in the embodiment of the present application can effectively improve the application effectiveness and reliability of the cell collaborative sleep strategy generated based on the cell collaborative sleep strategy generation model, so as to realize dynamic and adaptive cell switching decisions.

[0266] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the cell collaborative sleep strategy generation model training method and / or the cell collaborative sleep method mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as by bus connection. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0267] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0268] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the cell collaborative sleep strategy generation model training method and / or the cell collaborative sleep method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the cell collaborative sleep strategy generation model training method and / or the cell collaborative sleep method in the above method embodiment.

[0269] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0270] The one or more modules are stored in the memory, and when executed by the processor, the cell collaborative sleep strategy generation model training method in the embodiment is executed.

[0271] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0272] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0273] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0274] The embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned cell collaborative sleep strategy generation model training method and / or cell collaborative sleep method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0275] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the aforementioned cell collaborative sleep strategy generation model training method and / or cell collaborative sleep method.

[0276] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0277] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0278] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0279] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cell collaborative sleep strategy generation model training method, characterized in that: include: Based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each of the cells, a preset neural network is trained to train the neural network into an expert demonstration network for predicting the expert demonstration action data corresponding to each of the cells in the target area; The network parameters of the expert demonstration network are used to initialize the current neural network and the target neural network in the reinforcement learning agent, so that the reinforcement learning agent performs joint iterative training on the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area.

2. The cell collaborative sleep strategy generation model training method according to claim 1, characterized in that: The method of training a preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each cell, so as to train the neural network into an expert demonstration network for predicting the expert demonstration action data corresponding to each cell in the target area, comprises: Obtaining target expert demonstration action vectors corresponding to each of the unit times for controlling the coordinated sleep of each of the cells; Convert the target expert demonstration action vectors corresponding to each of the unit times into corresponding target expert demonstration action instructions respectively; The target expert demonstration action instructions corresponding to each of the unit times are respectively used as labels of the pre-acquired historical traffic data of each of the cells corresponding to each of the unit times, so as to obtain samples corresponding to each of the unit times and expert demonstration action data sets composed of each of the samples; Dividing the expert demonstration action dataset into a training set and a validation set; Based on the training set, a preset neural network is trained using a stochastic gradient descent algorithm, and the trained neural network is verified using the verification set, so that the verified neural network is determined as the expert demonstration network currently used to predict the expert demonstration action data of the target area within a unit time after the current time, and the expert demonstration network parameters corresponding to the expert demonstration network are stored.

3. The cell collaborative sleep strategy generation model training method according to claim 2 is characterized in that: The obtaining of target expert demonstration action vectors corresponding to each of the unit times and used to control the coordinated sleep of each of the cells includes: An expert demonstration algorithm based on heuristic search is used to calculate and obtain first expert demonstration action vectors corresponding to each of the unit times and used to control the coordinated sleep of each of the cells; And, using an expert demonstration algorithm based on the ranking of cell values ​​to calculate and obtain second expert demonstration action vectors corresponding to each of the unit times and used to control the coordinated sleep of each of the cells; According to the power consumption values ​​of the first expert demonstration motion vector and the second expert demonstration motion vector corresponding to each unit time, one of the power consumption values ​​of the first expert demonstration motion vector and the second expert demonstration motion vector corresponding to each unit time is selected as the target expert demonstration motion vector corresponding to the unit time.

4. The cell collaborative sleep strategy generation model training method according to claim 2, characterized in that: The method of using the network parameters of the expert demonstration network to initialize the current neural network and the target neural network in the reinforcement learning agent, so that the reinforcement learning agent performs joint iterative training on the current neural network in the reinforcement learning agent and the target neural network in the reinforcement learning agent based on deep reinforcement learning, so as to train the current neural network into a cell collaborative sleep strategy generation model for predicting the cell collaborative sleep strategy corresponding to each of the cells in the target area, includes: Initializing the current neural network and the target neural network in the reinforcement learning agent using the expert demonstration network parameters; Initialize an experience replay buffer and a maximum number of iterations corresponding to the reinforcement learning agent, wherein the experience replay buffer is used to store tuple samples, and the tuple samples include: current state, action, reward, and next state; The working state of each cell in a unit time is used as the action corresponding to the unit time, the predicted traffic of each cell in the unit time and the sleep state switch action of the previous unit time are used as the current state corresponding to the unit time, and a reward function corresponding to the reward is set; wherein the working state includes: an active state represented by 1 and a sleep state represented by 0; Initialize the current neural network and the target neural network of the reinforcement learning agent according to the expert demonstration network parameters; Selecting the action according to the current neural network and executing the action, and calculating the corresponding reward based on the reward function, at which time the environment state changes to the next state to obtain the corresponding tuple sample, and storing the tuple sample in the experience replay buffer; If the data of the tuple samples in the experience replay buffer meets the preset quantity threshold, a plurality of the tuple samples are randomly selected from the experience replay buffer so that the reinforcement learning agent, in the current iteration round, uses a deep reinforcement learning method to jointly train the current neural network and the target neural network in the reinforcement learning agent based on the randomly selected tuple samples until the number of iterations reaches the maximum number of iterations, then the training is stopped, and the network parameters of the current neural network are used as model parameters corresponding to the cell collaborative sleep strategy generation model for generating a prediction result of the cell collaborative sleep strategy for the target area within a unit time after the current time.

5. The cell collaborative sleep strategy generation model training method according to claim 4 is characterized in that: The reward function is composed of a power saving benefit function, a user's service quality satisfaction function, and an action error function that imitates an expert strategy; The value of the power saving benefit function is calculated based on the total power consumption corresponding to a unit time of all cells in the target area that are in an active state, the working state of each cell in the unit time, and the power consumption model; The value of the user's service quality satisfaction function is calculated based on the pre-acquired user available service rate, user required traffic and switch variables; The value of the action error function imitating the expert strategy is calculated based on the pre-acquired target expert demonstration action vector; The user available service rate is calculated based on the bandwidth resources allocated by the active cell to the associated user in unit time and the signal-to-noise ratio from the active cell to the associated user.

6. The method for training a cell collaborative sleep strategy generation model according to claim 5, characterized in that: Before training the preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each cell, the method further includes: Collecting historical traffic data, the total number of users and the total number of cells corresponding to each cell in the target area, and preprocessing the historical traffic data to obtain the traffic data of each cell in the target area at each unit time; Inputting the flow data of each of the cells at each of the unit times into a preset flow prediction model, so that the flow prediction model outputs a flow prediction result of each of the cells at a unit time after each of the unit times; The traffic prediction results of each cell at one unit time after each unit time are summed to obtain the predicted total traffic of the target area at one unit time after each unit time, and based on the predicted total traffic and the total number of users, the user expected traffic and user demand traffic corresponding to one unit time after each unit time are allocated to each user in the target area.

7. The method for training a cell collaborative sleep strategy generation model according to claim 6, characterized in that: Before training the preset neural network based on the historical traffic data of each cell in the target area corresponding to each unit time and the expert demonstration action data for controlling the coordinated sleep of each cell, the method further includes: Setting a power consumption model, wherein the power consumption model corresponding to each cell in a unit time is composed of a fixed power consumption of the cell in the unit time and a transmission power consumption of the cell in the unit time; The cell working state is set, where the cell working state includes: a value of 1 is when the cell is in an active state for a unit time, and a value of 0 is when the cell is in a dormant state for a unit time.

8. The cell collaborative sleep strategy generation model training method according to claim 3, characterized in that: The method of using an expert demonstration algorithm based on heuristic search to calculate and obtain first expert demonstration action vectors corresponding to each of the unit times and used to control the coordinated sleep of each of the cells includes: The working state of each cell in the target area is defined as the position of a particle, the total power consumption of all cells in the target area is defined as the fitness of a particle, a particle group is defined according to a plurality of switch combinations for all cells in the target area, and an update process of a particle is defined; Based on the position of the particle, the fitness of the particle, the particle group and the update process of the particle, an expert demonstration algorithm based on heuristic search is adopted to search for the optimal solution of the corresponding cell collaborative sleep problem for each of the cells in the target area, so as to obtain the first expert demonstration action vector corresponding to each of the unit times for controlling the collaborative sleep of each of the cells.

9. The cell collaborative sleep strategy generation model training method according to claim 3, characterized in that: The method of using an expert demonstration algorithm based on the ranking of cell values ​​to calculate the second expert demonstration action vectors corresponding to each of the unit times and used to control the coordinated sleep of each of the cells includes: Determine the total traffic load of the target area at each unit time based on the pre-acquired traffic load of each cell at each unit time; Determine the cell value of each cell at each unit time according to the traffic load of each cell at each unit time and the total traffic load; Inputting the cell value of each of the cells at each historical unit time before a unit time into a preset time series prediction model, so that the time series prediction model outputs the cell value prediction result of each of the cells at each unit time; The corresponding working status of each of the cells in the target area is solved according to the cell value prediction results of each of the cells at each unit time, so as to obtain the second expert demonstration action vector corresponding to each of the unit time for controlling the coordinated sleep of each of the cells.

10. A cell coordinated sleep method, characterized in that: include: Obtain target traffic data of each cell in the target area within a historical unit time; Input the target traffic data into a preset cell collaborative sleep strategy generation model, so that the cell collaborative sleep strategy generation model outputs the cell collaborative sleep strategy prediction result data of the target area within a unit time after the current time, wherein the cell collaborative sleep strategy generation model is pre-trained based on the cell collaborative sleep strategy generation model training method according to any one of claims 1 to 9; According to the cell coordinated sleep strategy prediction result data, corresponding sleep state switch actions are performed for each of the cells in the target area.

Images