Network optimization method, device, electronic device and computer-readable storage medium
By combining imitating neural networks and deep reinforcement learning models to generate network optimization strategies, the global optimization problem in mobile communication network optimization is solved, dynamic optimization and rapid convergence are achieved, and the problem of optimization failure is avoided.
Patent Information
- Application Number
- CN202210383749.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-12
AI Technical Summary
The existing mobile communication network optimization methods cannot achieve global optimization under all possible circumstances, especially when an unknown state is encountered, resulting in the optimization failure.
Using a method of fusion imitating neural networks and deep reinforcement learning models, the network optimization strategy is generated by obtaining network parameter data and optimization indicators, and the network parameters are optimized. The model is trained using tuning empirical data that imitates neural networks, and switch to depth when needed. Reinforcement learning models to converge quickly.
It realizes perceived decisions for dynamic optimization in mobile communication networks, makes full use of existing empirical data, avoids the adverse effects of the exploration process on the network, and improves optimization effect and efficiency.
Smart Images

Figure CN114828045B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of mobile communication technology. Specifically, the present application relates to a network optimization method, device, electronic device and computer storage medium. Background Art
[0002] Various expert methods have been developed in the field of mobile communications and have been used to optimize wireless networks with various objectives to solve various performance problems. These expert methods include methods for radio resource management (RRM) and self-organizing networks (SON). Network optimization is a global optimization process. When adjusting parameters of a cell, it is likely to affect the network status of its neighboring cells. Therefore, network optimization should not only be local (or cell) optimal, but also global (or neighboring multiple cells) optimal.
[0003] At present, the existing solutions for mobile communication network optimization mainly include: (1) Network self-optimization through manually formulated rule tables, by querying various network states and making corresponding adjustments according to the rule tables formulated by experts; however, the rule tables of this solution cannot cover all possible situations, and the optimization effects of the rule tables applied to different devices vary. (2) Network optimization based on iterative optimization models, by traversing various parameters of base station antennas, testing and analyzing the collected network data information, with the goal of maximizing the network performance in the surrounding area, iteratively optimizing different combinations of collected sample data and antenna parameters, and finally obtaining the best parameter combination to achieve the optimization goal; however, the optimization model of this solution cannot cover all possible situations. If an unknown state is encountered, optimization processing cannot be performed, resulting in optimization failure. Summary of the invention
[0004] The embodiments of the present application provide a network optimization method, device, electronic device and computer storage medium, which can integrate the imitation neural network and deep reinforcement learning model to perform network optimization, and provide a solution for the perception decision of dynamic optimization of mobile communication networks. The specific technical solution is as follows:
[0005] According to one aspect of an embodiment of the present application, a network optimization method is provided, the method comprising:
[0006] Obtain the current network parameter data of the target optimization cell;
[0007] Based on the network parameter data and the network optimization index of the target optimization cell, according to the preset judgment strategy, determine to generate a network optimization strategy by imitating a neural network or a deep reinforcement learning model, wherein the imitating neural network is generated by training a predetermined neural network based on a network parameter tuning experience data set of the target optimization cell obtained in advance, and the deep reinforcement learning model is generated based on the imitating neural network;
[0008] According to the network optimization strategy, the network parameters of the target cell are optimized.
[0009] In a possible implementation, a predetermined neural network is trained based on a pre-acquired network parameter tuning experience data set of a target optimized cell to generate an imitation neural network, including:
[0010] A training sample data set is selected from the network parameter tuning experience data set. Each training sample in the training sample data set includes state data and action data. The state data is the cell state S of the target optimized cell. i , the action data is the target optimized cell from the cell state S i Transfer to the next cell state S i+1 The action performed i , i is a natural number;
[0011] The state data of each training sample in the training sample data set is used as the input data of the predetermined neural network, and the predetermined neural network is trained through an imitation learning algorithm until the error between the output data of the predetermined network and the action data corresponding to the state data of each training sample meets the predetermined condition.
[0012] In a possible implementation, each network parameter tuning experience data in the network parameter tuning experience data set is a five-tuple data, and the five-tuple data includes a cell identifier of a target optimized cell, a current cell state of the target optimized cell, a next cell state of the current cell state of the target optimized cell, an action performed by the target optimized cell to transfer from the current state to the next state, and a reward value obtained after performing the action;
[0013] Among them, the training sample data set is selected from the network parameter tuning experience data set, including:
[0014] A training sample data set is selected from a network parameter tuning experience data set according to a current cell state of a target optimized cell, a next cell state of the current cell state of the target optimized cell, and an action performed to transfer the target optimized cell from the current cell state to the next cell state.
[0015] In a possible implementation, the deep reinforcement learning model includes a first neural network and a second neural network; wherein generating the deep reinforcement learning model based on imitating the neural network includes:
[0016] Constructing a first neural network and a second neural network according to the structure of the imitative neural network, and copying the parameters of the imitative neural network to the first neural network and the second neural network;
[0017] Based on the current cell state S of the target optimized cell, the target parameter value θ of the first neural network is determined by performing a loop iteration on the first neural network, and during the loop iteration, the parameter value θ' of the second neural network is updated by copying the target parameter value θ to the second neural network at predetermined intervals.
[0018] In a possible implementation, based on the acquired current cell state S of the target optimization cell, determining the target parameter of the first neural network by iterating the first neural network in a loop includes:
[0019] Determine, by means of a first neural network, an action a performed to transfer the target optimized cell from a current cell state S to a next cell state S', and determine a Q estimation value according to the action a;
[0020] Determine the actual value of Q through a second neural network;
[0021] Determine the loss function of the first neural network according to the difference between the actual value of Q and the estimated value of Q;
[0022] The corresponding parameter value when the loss function is minimized is solved, and the parameter value is determined as the target parameter value θ.
[0023] In a possible implementation, based on the network parameter data and the network optimization index of the target optimized cell, according to a preset decision strategy, it is determined to generate a network optimization strategy by imitating a neural network or a deep reinforcement learning model, including:
[0024] Based on the network parameter data and the network optimization indicators of the target optimization cell, it is directly determined to generate a network optimization strategy by imitating the neural network. When the network optimization strategy generated by imitating the neural network cannot meet the requirements, it is determined to generate a network optimization strategy by a deep reinforcement learning model.
[0025] In a possible implementation manner, after optimizing the network parameters of the target optimized cell according to the network optimization strategy, the method further includes:
[0026] The optimized network parameters of the target optimized cell are obtained, and the optimized network parameters are updated to the network parameter tuning experience data set of the target optimized cell.
[0027] According to another aspect of an embodiment of the present application, a network optimization device is provided, the device comprising:
[0028] An acquisition module is used to obtain the current network parameter data of the target optimization cell;
[0029] A processing module, used for determining, based on the network parameter data and the network optimization index of the target optimization cell, to generate a network optimization strategy by imitating a neural network or a deep reinforcement learning model according to a preset decision strategy, wherein the imitating neural network is generated by training a predetermined neural network based on a network parameter tuning experience data set of the target optimization cell acquired in advance, and the deep reinforcement learning model is generated based on the imitating neural network;
[0030] The optimization module is used to optimize the network parameters of the target optimization cell according to the network optimization strategy.
[0031] In a possible implementation, the processing model is also used to train a predetermined neural network based on a pre-acquired network parameter tuning experience data set of a target optimized cell to generate an imitation neural network, wherein the processing model is specifically used to:
[0032] A training sample data set is selected from the network parameter tuning experience data set. Each training sample in the training sample data set includes state data and action data. The state data is the cell state S of the target optimized cell. i , the action data is the target optimized cell from the cell state S i Transfer to the next cell state S i+1 The action performed i , i is a natural number;
[0033] The state data of each training sample in the training sample data set is used as the input data of the predetermined neural network, and the predetermined neural network is trained through an imitation learning algorithm until the error between the output data of the predetermined network and the action data corresponding to the state data of each training sample meets the predetermined condition.
[0034] In a possible implementation, each network parameter tuning experience data in the network parameter tuning experience data set is a five-tuple data, and the five-tuple data includes a cell identifier of a target optimized cell, a current cell state of the target optimized cell, a next cell state of the current cell state of the target optimized cell, an action performed by the target optimized cell to transfer from the current state to the next state, and a reward value obtained after performing the action;
[0035] Wherein, when the processing module selects the training sample data set from the network parameter tuning experience data set, it is used to:
[0036] A training sample data set is selected from a network parameter tuning experience data set according to a current cell state of a target optimized cell, a next cell state of the current cell state of the target optimized cell, and an action performed to transfer the target optimized cell from the current cell state to the next cell state.
[0037] In a possible implementation, the deep reinforcement learning model includes a first neural network and a second neural network; wherein the processing module, when generating the deep reinforcement learning model based on the imitation neural network, is used to:
[0038] Constructing a first neural network and a second neural network according to the structure of the imitative neural network, and copying the parameters of the imitative neural network to the first neural network and the second neural network;
[0039] Based on the current cell state S of the target optimized cell, the target parameter value θ of the first neural network is determined by performing a loop iteration on the first neural network, and during the loop iteration, the parameter value θ' of the second neural network is updated by copying the target parameter value θ to the second neural network at predetermined intervals.
[0040] In a possible implementation, when the processing module determines the target parameters of the first neural network by iterating the first neural network based on the acquired current cell state S of the target optimization cell, it is configured to:
[0041] Determine, by means of a first neural network, an action a performed to transfer the target optimized cell from a current cell state S to a next cell state S', and determine a Q estimation value according to the action a;
[0042] Determine the actual value of Q through a second neural network;
[0043] Determine the loss function of the first neural network according to the difference between the actual value of Q and the estimated value of Q;
[0044] The corresponding parameter value when the loss function is minimized is solved, and the parameter value is determined as the target parameter value θ.
[0045] In a possible implementation, the processing module is further configured to:
[0046] Based on the network parameter data and the network optimization indicators of the target optimization cell, it is directly determined to generate a network optimization strategy by imitating the neural network. When the network optimization strategy generated by imitating the neural network cannot meet the requirements, it is determined to generate a network optimization strategy by a deep reinforcement learning model.
[0047] In a possible implementation, the device further includes an updating module, where the updating module is configured to:
[0048] The optimized network parameters of the target optimized cell are obtained, and the optimized network parameters are updated to the network parameter tuning experience data set of the target optimized cell.
[0049] According to another aspect of an embodiment of the present application, an electronic device is provided. The electronic device includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above-mentioned network optimization method.
[0050] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, and when the computer program is executed by a processor, the steps of the above-mentioned network optimization method are implemented.
[0051] According to one aspect of an embodiment of the present application, a computer program product is provided, and when the computer program is executed by a processor, the steps of the above-mentioned network optimization method are implemented.
[0052] The network optimization method provided in the embodiment of the present application can not only optimize and adjust network parameters directly based on the imitation neural network, thereby making full use of the existing network parameter tuning experience, but also optimize and adjust network parameters through the deep reinforcement learning model constructed on the basis of the imitation empirical neural network, thereby making full use of the advantage of the deep reinforcement learning model that can converge quickly and effectively avoid the adverse effects of the exploration process on the network; by integrating the imitation neural network and the deep reinforcement learning model, deep learning and reinforcement learning are combined to achieve the complementary advantages of the two, thereby providing a solution for the perception decision-making of dynamic optimization of mobile communication networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in describing the embodiments of the present application.
[0054] Figure 1 A schematic diagram of a mobile communication network provided in an embodiment of the present application;
[0055] Figure 2 A schematic diagram of a cellular network provided in an embodiment of the present application;
[0056] Figure 3 A schematic diagram of the reinforcement learning process provided in the embodiment of the present application;
[0057] Figure 4 A flowchart of a network optimization method provided in an embodiment of the present application;
[0058] Figure 5 A schematic diagram of the basic principle of network optimization provided in an embodiment of the present application;
[0059] Figure 6 A schematic diagram of a network dynamic optimization device that integrates a simulated neural network and a deep reinforcement learning model provided in an embodiment of the present application;
[0060] Figure 7 A schematic diagram of a network optimization process provided in an embodiment of the present application;
[0061] Figure 8 A schematic diagram of the implementation process of the deep reinforcement learning model provided in the embodiment of the present application;
[0062] Fig. 9 A schematic diagram of the structure of a network optimization device provided in an embodiment of the present application;
[0063] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.
[0065] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" indicates that it is implemented as "A", or is implemented as "A", or is implemented as "A and B".
[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0067] First, the relevant technologies and terms involved in this application are introduced and explained:
[0068] Figure 1FIG. 1 shows a mobile communication network including a base station with a coverage area (or cell), a plurality of mobile devices, and a backhaul network. Figure 1 As shown, the base station establishes an uplink connection and a downlink connection with the mobile device, and these connections are used to carry bidirectional transmission data between the mobile device and the base station. The data carried by the uplink connection and the downlink connection may include data transmitted between the mobile devices, and also include data transmitted to or from the remote end through the backhaul network.
[0069] The term "base station" refers to a device or any component or collection of components used to provide wireless access to a network, such as an enhanced base station, a macro cell, a femto cell, a Wi-Fi access point, etc. A base station may provide wireless access according to one or more wireless communication protocols, such as Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), High Speed Packet Access (HSPA), etc.
[0070] The term "mobile device" refers to a component or a collection of components that can establish a wireless connection with a base station, such as User Equipment (UE), Mobile Station (STA), etc.
[0071] A cellular network is a typical mobile communication network. A cellular network may include base stations that form multiple interconnected cells. Through the arrangement of base stations, a cellular network may provide wireless communication coverage over a larger geographical area and enable wireless communication devices to communicate with other wireless communication devices at any geographical location in the network. Figure 2 A cellular network of multiple base stations providing cell coverage is shown, each cell represents a geographical area, and the areas covered by the cells may not overlap with each other or may partially overlap. Each cell has a hexagonal shape, and each base station may be located at the center of the cell or at a corner of the cell hexagon and cover three adjacent hexagonal cells (or three-sector cells). The cellular network may have a specific layout or topology, which includes the relative distances between base stations and the angular directions of their antennas relative to each other. Figure 2 The cellular network shown is only a simple example and may vary in detail in different application implementation scenarios.
[0072] Network optimization technology can be applied to cellular networks to adjust one or more parameters or performance indicators of the cellular network to improve the performance of the cellular network. For example, cellular network optimization can include optimization for coverage, capacity, interference reduction, etc. or adjustment and optimization for other key performance indicators (KPIs). Cellular network parameters can include antenna electronic tilt, antenna azimuth, antenna mechanical tilt, and transmit power.
[0073] In a cellular network, adjacent cells can affect each other, so that changes in the settings of a base station associated with one cell may affect the network performance, such as coverage and capacity, of adjacent cells. Changing the parameter settings of one base station to improve the coverage and capacity of its cell may cause interference to its adjacent cells and may reduce the coverage and capacity of these adjacent cells, as well as the coverage and capacity of the entire network. Increasing the number of cells in a cellular network can lead to an exponential increase in the number of interactions, relationships, and potential interference between adjacent cells. Therefore, a cellular network is not only affected by each individual cell, but also by the relationships between cells. The optimization of a cellular network needs to take these influencing factors into account.
[0074] Imitation learning is a method of learning from expert examples, and a method that enables intelligent agents (robots) to make intelligent decisions like human experts. Using these expert examples to teach intelligent agents to make intelligent decisions is the main problem that imitation learning solves. In addition, a virtual world can also be constructed through the imitation learning method, allowing intelligent agents to freely try and learn in it. Taking network optimization as an example, the expert examples here refer to the state-action sequence collected by the optimal or suboptimal strategy, parsing the existing network parameter tuning experience data, the state is the cell parameter, and the action is the base station parameter associated with this cell. The network parameter tuning data is input into the neural network. After the neural network is trained, a certain cell parameter is input to output the empirically optimal base station adjustment parameters.
[0075] Figure 3 The figure shows a reinforcement learning (RL) process, in which the agent interacts dynamically with the environment. When the agent perceives the state information of the environment, it determines the next action based on the possible reward of its action and further observes the reaction of the environment, repeating the cycle until it converges to a steady-state target.
[0076] Reinforcement learning consists of five elements consisting of two subjects and three interaction data. The two subjects are the intelligent agent and the environment, and the three interaction data are state, action, and reward.
[0077] Agent: A decision maker that interacts with the environment by performing actions, obtaining state values, and receiving rewards.
[0078] Environment: Anything outside the agent.
[0079] State: All data that can affect the agent's next action is considered the agent's state.
[0080] Action: The medium through which the agent interacts with the environment. It must have some control over the environment, i.e. it can change the future rewards.
[0081] Reward: A scalar feedback that measures how well the agent did in state .
[0082] Deep reinforcement learning is a new algorithm that combines deep learning and reinforcement learning to achieve end-to-end learning from perception to action. Simply put, it perceives input information like humans, and then directly outputs actions through deep neural networks. Deep reinforcement learning has the potential to enable intelligent agents to truly learn one or more skills completely autonomously.
[0083] The dynamic optimization of mobile communication networks is a perception decision-making problem. Deep learning has strong perception capabilities, but lacks certain decision-making capabilities. While reinforcement learning has decision-making capabilities, it is relatively helpless in dealing with complex perception problems in state space and action space. The network optimization method that integrates imitation learning and deep reinforcement learning proposed in the embodiment of the present application combines deep learning and reinforcement learning, complementing each other's advantages, and provides a solution for perception decision-making in dynamic optimization of mobile communication networks.
[0084] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.
[0085] Figure 4 A flowchart of a network optimization method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes: step S410, obtaining the current network parameter data of the target optimized cell; step S420, based on the network parameter data and the network optimization index of the target optimized cell, according to a preset judgment strategy, determining to generate a network optimization strategy by imitating a neural network or a deep reinforcement learning model, wherein the imitating neural network is generated by training a predetermined neural network based on a network parameter tuning experience data set of the target optimized cell acquired in advance, and the deep reinforcement learning model is generated based on the imitating neural network; step S430, according to the network optimization strategy, optimizing the network parameters of the target optimized cell.
[0086] In an embodiment of the present application, by interacting with the target optimized cell, the current network parameter data of the target optimized cell can be periodically collected and analyzed at predetermined time intervals. The network parameter data includes but is not limited to cell status data, network environment data, etc. The cell status data (or recorded as cell status) includes but is not limited to cell traffic volume, cell resource utilization, cell reference signal reception quality, cell access success rate, etc.; the network environment data (or recorded as network environment) includes but is not limited to base station performance indicator data and configuration data (such as azimuth, transmission data), etc.
[0087] After obtaining the current number of network parameters of the target optimized cell, based on the network parameter data and the network optimization index of the target optimized cell, according to the preset decision strategy, it can be determined to generate the network optimization strategy of the target optimized cell by imitating the neural network, or to generate the network optimization strategy of the target optimized cell by the deep reinforcement learning model, so as to optimize the network parameters of the target optimized cell. In practical applications, the current number of network parameters of the target optimized cell can be used as input data and input into the imitating neural network or the deep reinforcement learning model, and the output data of the imitating neural network or the deep reinforcement learning model is the network optimization strategy of the target optimized cell (which can also be recorded as a network optimization action).
[0088] The network parameter tuning experience dataset is used to refer to expert experience data, historical data, and other data that can be used to help select actions when training a predetermined neural network. The network parameter tuning experience dataset can be used to train a predetermined neural network to optimize a wireless network. The use of the network parameter tuning experience dataset can help accelerate the training of the predetermined neural network.
[0089] In practical applications, by interacting with the target optimized cell set, the network tuning parameter data source can be collected offline at regular intervals, and the collected network tuning parameter data source can be parsed to generate a network parameter tuning experience data set; then, based on this network parameter tuning experience data set, a predetermined neural network is trained to generate an imitation neural network for optimizing and adjusting the network parameters of the target optimized cell.
[0090] After the imitation neural network is generated, a deep reinforcement learning model can be constructed based on the imitation neural network. After a limited number of interactions with the real network of the target optimization cell, the deep reinforcement learning model can quickly converge to a stable state, and then optimize and adjust the network parameters of the target optimization cell.
[0091] The network optimization method provided in the embodiment of the present application can not only optimize and adjust network parameters directly based on the imitation neural network, thereby making full use of the existing network parameter tuning experience, but also optimize and adjust network parameters through the deep reinforcement learning model constructed on the basis of the imitation empirical neural network, thereby making full use of the advantage of the deep reinforcement learning model that can converge quickly and effectively avoid the adverse effects of the exploration process on the network; by integrating the imitation neural network and the deep reinforcement learning model, deep learning and reinforcement learning are combined to achieve the complementary advantages of the two, thereby providing a solution for the perception decision-making of dynamic optimization of mobile communication networks.
[0092] The basic principle of the embodiment of the present application is as follows Figure 5 Before introducing the embodiments of the present application in detail, the rewards for states and actions that may be involved in the simulated neural network and deep reinforcement learning model of the embodiments of the present application are first specifically defined in the network optimization scenario, as shown below:
[0093] Status, including statistics of base station performance indicators and information obtained from measurement reports provided by UE, such as cell traffic, cell resource utilization, cell reference signal reception quality RSRQ, cell access success rate, etc. According to the specific network optimization goals, a suitable subset of cell indicators can be selected as the cell status space.
[0094] action, so that the cell changes from the current state S t Transition to next state S t+1 Actions performed. Cell actions include parameters used to adjust base stations associated with the cell, such as cell transmit power, antenna azimuth, antenna downtilt, cell access threshold, cell handover threshold, etc. A suitable subset of cell parameters can be selected as the cell action space according to specific network optimization goals.
[0095] Reward, the cost function is designed to calculate the reward through the network optimization goal. When there is more than one optimization goal, they can be considered simultaneously when the reward function is involved, and the importance of the goal can be reflected by setting the weight. For example, to optimize the network for coverage and interference, the cost function can be designed as: Fc = ((w*N1) + (1-w)*N2) / N, where w is the weight coefficient, the value range is [0,1], N1 is the number of MRs received from UEs whose RSRP of the cell is greater than or equal to the RSRP threshold, N2 is the number of MRs received from UEs whose RSRQ of the cell is greater than or equal to the RSRQ threshold, and N = N1 + N2. The (w*N1) part is used to reflect the weight of coverage in the cost function, and the (1-w)*N2 part is used to reflect the weight of interference in the cost function.
[0096] In a possible implementation method of an embodiment of the present application, in the process of determining to generate a network optimization strategy by imitating a neural network or a deep reinforcement learning model based on network parameter data and network optimization indicators of a target optimized cell according to a preset judgment strategy, the following processing can be performed: based on the network parameter data and the network optimization indicators of the target optimized cell, directly determine to generate a network optimization strategy by imitating a neural network, and when the network optimization strategy generated by imitating the neural network cannot meet the requirements, determine to generate a network optimization strategy by a deep reinforcement learning model.
[0097] In one example, the network optimization strategy can be generated by intelligently selecting an imitation neural network or a deep reinforcement learning model based on the changes in the network optimization indicators of the target optimized cell. In practical applications, based on the strategy principles of experience priority and a small amount of exploration, priority can be given to the application of imitation neural networks to make full use of the existing network parameter tuning experience data set; in this case, the current number of network parameters of the target optimized cell can be used as input data and input into the imitation neural network, and the output of the imitation neural network is the network optimization strategy of the target optimized cell (or recorded as a network optimization action). When the network optimization effect generated by the imitation neural network cannot meet the optimization requirements, the deep reinforcement learning model is triggered to generate the network optimization strategy for the next cycle; in this case, the current number of network parameters of the target optimized cell can be used as input data and input into the deep reinforcement learning model, and the output of the deep reinforcement learning model is the network optimization strategy of the target optimized cell (or recorded as a network optimization action).
[0098] In a possible implementation method of an embodiment of the present application, after optimizing the network parameters of the target optimized cell according to the network optimization strategy, the optimized network parameters of the target optimized cell can also be obtained, and the optimized network parameters can be updated to the network parameter tuning experience data set of the target optimized cell.
[0099] In practical applications, in the process of optimizing the target optimized cell, the optimized network parameters of the target optimized cell can be periodically obtained at predetermined time intervals, and the optimized network parameters can be updated to the network parameter tuning experience data set of the target optimized cell. For example, the target optimized cell is interacted with, and the optimized communication base station performance indicator data and configuration data and other network parameters are periodically collected and parsed, and used as incremental updates of the network parameter tuning experience data set, that is, by continuously updating the optimized network parameters of the target optimized cell to the network parameter tuning experience data set, the existing network parameter tuning experience data is incrementally updated, thereby incrementally updating the memory storage of the imitation neural network and deep reinforcement learning model, and further accelerating the iterative process of interactive learning between the imitation neural network and deep reinforcement learning model and the cell network environment.
[0100] In other words, as the network parameter tuning experience data source continues to be parsed, the network parameter tuning experience data set will be incrementally updated. Based on the generated incremental update data set, the storage memory of the imitation neural network and deep reinforcement learning model can be incrementally updated. On the one hand, it can enable the newly created intelligent agent (such as the deep intensity learning model, etc.) to obtain the full amount of network parameter tuning experience. On the other hand, it can enable the existing intelligent agent to learn the incrementally updated network parameter tuning experience, which can further accelerate the intelligent agent learning process and reduce the number of iterations of interactive learning with the environment.
[0101] In a possible implementation of the embodiment of the present application, a predetermined neural network is trained based on a network parameter tuning experience data set of a target optimized cell acquired in advance. In the process of generating an imitation neural network, the following processing can be performed: first, a training sample data set is screened out from the network parameter tuning experience data set. Each training sample in the training sample data set includes state data and action data. The state data is the cell state S of the target optimized cell. i , the action data is the target optimized cell from the cell state S i Transfer to the next cell state S i+1 The action performed i , i is a natural number; then, the state data of each training sample in the training sample data set is used as the input data of the predetermined neural network, and the predetermined neural network is trained by the imitation learning algorithm until the error between the output data of the predetermined network and the action data corresponding to the state data of each training sample meets the predetermined condition.
[0102] Among them, each network parameter tuning experience data in the network parameter tuning experience data set is a quintuple data, which includes the cell identifier of the target optimized cell, the current cell state of the target optimized cell, the next cell state of the current cell state of the target optimized cell, the action performed by the target optimized cell to transfer from the current state to the next state, and the reward value obtained after performing the action.
[0103] In an example, the five-tuple data may be a specific mathematical identifier as shown below:
[0104] (I c , S t , S t+1 , a t , r t )
[0105] Among them, I c Represents the cell identifier, S t Indicates the current state of the cell, S t+1 Indicates the next state of the cell, a t Indicates that the cell changes from the current state St Transition to next state S t+1 The action performed, r t Indicates the action taken t Rewards received afterwards.
[0106] Specifically, the process of filtering out a training sample data set from the network parameter tuning experience data set can be: filtering out a training sample data set from the network parameter tuning experience data set according to the current cell state of the target optimized cell, the next cell state of the current cell state of the target optimized cell, and the action performed to transfer the target optimized cell from the current cell state to the next cell state.
[0107] The above process of generating an imitation neural network can be divided into two main stages. The first stage is to construct a learning data set (i.e., a training sample data set) for imitating the neural network; the second stage is to model the imitation neural network, i.e., to generate a network optimization strategy by training a predetermined neural network to imitate the learning training sample data set. Among them:
[0108] Before constructing a training sample data set that simulates a neural network, the network parameter tuning experience data source of the cell can be collected offline in advance. By parsing and extracting the network parameter tuning experience data source, a network parameter tuning experience data set can be obtained.
[0109] In the process of building a learning data set that simulates a neural network, since a network parameter tuning experience data set has been obtained, and the existing network parameter tuning experience data set often gives a current cell state S with poor performance. i Transfer to the next cell state S where the performance meets the standard i+1 (i.e. the current cell status S i The next cell state), the action to be performed a i , i is a natural number (i.e., i is 0 or a positive integer); therefore, a training sample data set A of a simulated neural network from the cell state to the empirical optimal action can be extracted from the network parameter tuning empirical data set = {(S i , a i )}. Among them, (S i , a i ) represents a training sample data, S i is the input X of the simulated neural network. The Si can be regarded as the state data included in the training sample data. i It imitates the output y of the neural network, and the ai can be regarded as the action data included in the training sample data.
[0110] Next, the state data (e.g., Si) of each training sample in the training sample data set (e.g., {(Si, ai)}) is used as the input data of the predetermined neural network, and the predetermined neural network is trained by the imitation learning algorithm until the error between the output data (e.g., ai) of the predetermined network and the action data corresponding to the state data of each training sample meets the predetermined condition, thereby obtaining a trained imitation neural network. For example, in the process of imitating neural network modeling, the predetermined neural network can be used to learn the network tuning experience data set, wherein the input of the predetermined neural network is the cell state of the problem cell (i.e., the target optimization cell), and the output is the optimal or suboptimal experience action (i.e., the error between the output data of the predetermined network and the action data corresponding to the state data of each training sample meets the predetermined condition), and executing this optimal or suboptimal experience action can make the problem cell network performance meet the standard.
[0111] The method of the embodiment of the present application creates an imitation neural network by applying an imitation learning algorithm, learns the network parameter tuning experience data set, and generates a network optimization strategy from the cell state to the optimal experience action. This imitation neural network will serve as the basis for the subsequent construction of a deep reinforcement learning model.
[0112] In a possible implementation of an embodiment of the present application, a deep reinforcement learning model includes a first neural network and a second neural network; wherein, in the process of generating a deep reinforcement learning model based on an imitation neural network, the following processing can be performed: first, the first neural network and the second neural network are constructed according to the structure of the imitation neural network, and the parameters of the imitation neural network are copied to the first neural network and the second neural network; then, based on the current cell state S of the target optimized cell obtained, the target parameter value θ of the first neural network is determined by iterating the first neural network, and during the iterative process, the parameter value θ' of the second neural network is updated by copying the target parameter value θ to the second neural network at predetermined intervals.
[0113] In a possible implementation, in the process of determining the target parameters of the first neural network by looping and iterating the first neural network based on the current cell state S of the target optimized cell, the following processing may be performed: first, the action a performed to make the target optimized cell transfer from the current cell state S to the next cell state S' is determined by the first neural network, and the Q estimation value is determined according to the action a; then, the Q actual value is determined by the second neural network;
[0114] According to the difference between the actual value of Q and the estimated value of Q, the loss function of the first neural network is determined; then, the parameter value corresponding to the minimization of the loss function is solved, and the parameter value is determined as the target parameter value θ.
[0115] In practical applications, DQN (Deep Q Network) is used in the embodiments of the present application to generate a deep reinforcement learning model. DQN (Deep Q-network) combines Q-Learning with deep learning, uses a deep neural network to represent the value function to predict the Q value, and continuously updates the neural network to learn the optimal action path. Among them, the Q learning algorithm is a value-based reinforcement learning algorithm, Q is the abbreviation of quality, and the Q function Q (state, action) represents the quality of executing an action action under the state state, that is, what is the Q value that can be obtained.
[0116] DQN contains two neural networks, eval_net (i.e. the first neural network mentioned above) and target_net (i.e. the second neural network mentioned above), with the same structure but different parameters. At the beginning, the parameters of the eval_net neural network and the target_net neural network are the same, and the one with relatively fixed parameters is the target_net, which is used to obtain the Q target value or actual value (Q_target); while the other network that is constantly updated is used to obtain the Q evaluation value or estimated value (Q_eval), but after every N actions, the parameters of the eval_net neural network will be synchronized to the target_net neural network, so the target_net is called relatively fixed. The reason for this is that the difference between Q_target and Q_eval is the loss function. The purpose of training DQN is to continuously reduce the difference between the two, so the target needs to be relatively fixed to facilitate convergence. In other words, the eval_net neural network is used to predict the Q estimated value, and the parameters of the eval_net neural network are updated in real time by learning the network parameter data (including state, action and reward, etc.) that interacts with the environment; the target_net neural network is used to predict the Q actual value, and the parameter updates of the target_net neural network are regularly copied from the eval_net neural network.
[0117] The process of generating a deep reinforcement learning model includes initialization of the deep reinforcement learning model and cyclic iteration of the deep reinforcement learning model, where:
[0118] The initialization of the deep reinforcement learning model includes the following steps:
[0119] a) Construct two neural networks, eval_net and target_net, with the same structure as the imitation neural network. The corresponding parameters of the two neural networks are θ and θ′ respectively.
[0120] b) Copy the parameters of the simulated neural network to the eval_net neural network and the target_net neural network.
[0121] c) Determine hyperparameters: learning rate α, discount factor γ, target_net parameter update cycle, memory storage size, etc.
[0122] The iterative cycle of the deep reinforcement learning model includes the following steps:
[0123] a) Get the current cell status s;
[0124] b) According to the current cell state s, select action a, which can be selected through eval_net, where a = argmax a Q(s,a;θ), i.e., the action a performed to make the target optimized cell transfer from the current cell state S to the next cell state S' is determined through the first neural network eval_net;
[0125] c) Execute action a, act on the cell environment (for example, execute action a to optimize the network parameters of the target optimized cell), migrate the environment state to state s' (for example, the target optimized cell transfers from the current cell state s to the next cell state s'), and feedback reward r;
[0126] d) storing the data (s, a, r, s') generated by the above interaction process into memory storage;
[0127] e) calculating a Q estimate, where Q estimate = Q value Q(s,a) corresponding to action a in step b);
[0128] f) Calculate the Q actual value, which is given by the second neural network target_net, where Q actual value = γ*max a′ Q(s′,a′;θ′)+r;
[0129] g) calculating the Q difference, where Q difference = (Q reality - Q estimate), and using the Q difference to define the loss function Loss of the first neural network eval_net;
[0130] h) Solve the parameters θ of the first neural network eval_net when minimizing the loss function Loss;
[0131] i) Every N steps, update the parameters θ′ of target_net by copying the parameters of the first neural network eval_net to the second neural network target_net;
[0132] Repeat steps a) to i) above until the deep reinforcement learning model converges.
[0133] Through the description of the above series of contents, it can be seen that the network optimization method of the embodiment of the present application actually mainly includes 5 functional modules, such as Figure 6 As shown in the figure, the specific functional models are as follows:
[0134] 1) Network optimization experience analysis module: interacts with each target optimization cell, collects network parameter tuning data sources offline at regular intervals, analyzes existing network parameter tuning data sources, and generates network parameter tuning experience data sets;
[0135] 2) Imitation learning module: construct an imitation neural network, apply the imitation learning algorithm, learn the network parameter tuning experience data set, and generate an imitation neural network that maps the strategy from the cell state to the optimal experience action;
[0136] 3) Deep reinforcement learning module: Build a deep reinforcement learning model (also called deep reinforcement learning agent), replicate and imitate the neural network structure and parameter values, so that the deep reinforcement learning agent can obtain the existing network parameter tuning experience. After a limited number of interactions with the real network environment, it can quickly converge to a stable state, and then optimize the network parameters based on the deep reinforcement learning agent;
[0137] 4) Real-time network data analysis module: interacts with each target optimization cell, periodically collects and analyzes network parameters such as communication base station performance indicator data and configuration data, and updates the network environment, cell status and other input data (i.e., network parameter data) required for the cell imitation learning module and deep reinforcement learning module to infer the network optimization strategy for the next cycle.
[0138] 5) Network optimization algorithm selection module: According to the changes in network optimization indicators of the target optimization cell, intelligently select the imitation neural network or deep reinforcement learning model to generate the network optimization strategy. Among them, the imitation neural network is preferred to generate the network parameter optimization strategy, and the network optimization strategy is issued to the corresponding network optimization execution device. If the network optimization indicators are not significantly improved (that is, the network optimization strategy generated by the imitation neural network cannot meet the requirements), the deep reinforcement learning model is selected. After a limited number of interactions between the deep reinforcement learning model and the target optimization cell, the network parameter optimization strategy is generated, and the network optimization strategy is issued to the corresponding network optimization execution device.
[0139] The specific execution process of the network optimization method of the embodiment of the present application can be as follows Figure 7 As shown, the following processing steps are included:
[0140] 1) The offline data acquisition module includes three parts: network parameter tuning data source, network parameter tuning data set and network parameter tuning data incremental update. It can obtain a network parameter tuning data set (i.e., network parameter tuning experience data set) by offline acquisition of network parameter tuning experience data source and parsing and extracting the network parameter tuning experience data source. Each element in the data set is composed of a 5-tuple, and each 5-tuple data includes: a cell identifier that identifies the cell (e.g., a cell identifier of the target optimized cell), a current cell state (e.g., a current cell state of the target optimized cell), a next cell state (e.g., a next cell state of the current cell state of the target optimized cell), an action that transfers the current cell state to the next state (e.g., an action that transfers the current cell state to the next state), and a reward value obtained after the action is taken. The specific mathematical notation is as follows:
[0141] (I c , S t , S t+1 , a t , r t )
[0142] Among them, I c Represents the cell identifier, S t Indicates the current state of the cell, S t+1 Indicates the next state of the cell, a t Indicates that the cell changes from the current state S t Transition to next state S t+1 The action performed, r t Indicates the action taken t Rewards received afterwards.
[0143] It should be noted that the aforementioned incremental update of network parameter tuning data is the content introduced in the following step 9).
[0144] 2) The target cell status and network environment module analyzes the environmental data in real time, that is, by analyzing the cell status data and network environment data of the target optimized cell in real time, the current network parameter data of the target optimized cell is obtained.
[0145] 3) Network optimization algorithm selection module, which selects the imitation neural network or deep reinforcement learning model to generate optimization actions (i.e., generate network optimization strategies) according to the judgment strategy. Among them, the application of imitation neural network is given priority, and the existing network parameter tuning experience data set is fully utilized. When the optimization effect of the imitation neural network cannot meet the optimization requirements, the deep reinforcement learning model is triggered to generate the network optimization actions (i.e., network optimization strategies) of the next cycle. The deep reinforcement learning model integrated with the imitation neural network can converge quickly and avoid adverse effects on the network during the exploration process.
[0146] The imitation learning module includes three parts: imitation learning dataset, imitation neural network construction and imitation learning reasoning application (i.e. imitation neural network reasoning application). These three parts correspond to the following steps 4) to 6) respectively:
[0147] 4) The imitation learning module constructs a learning data set (i.e., training sample data set) that imitates the neural network. Because the existing network parameter tuning experience data set directly gives a current state S with poor performance. t Transfer to the next state S where the performance meets the target t+1 , the action to be performed a t Therefore, we can extract the training sample data set of the imitation neural network from the state to the empirical optimal action from the network parameter tuning experience data set A = {(S i , a i )}. Among them, (S i , a i ) represents a sample, S i is the input x of the model neural network, a i is the output y of the simulated neural network.
[0148] 5) Construct an imitative neural network, that is, to model the imitative neural network by learning the network parameter tuning data set, where the input is the problem cell status and the output is the optimal or suboptimal empirical action. Executing this action can make the problem cell network performance meet the standard.
[0149] 6) Imitation neural network reasoning application: by applying imitation learning algorithm, an imitation neural network is created to learn the empirical data set for network parameter tuning and generate a strategy mapping from cell state to optimal empirical action. This imitation neural network will serve as the basis for the subsequent construction of deep reinforcement learning agents.
[0150] 7)-8): Detailed description of the deep reinforcement learning model, wherein the embodiment of the present application uses DQN (Deep QNetwork) to learn the deep reinforcement learning agent. DQN contains two neural networks, eval_net (i.e., the first neural network) and target_net (i.e., the second neural network), with the same structure but different parameters. The eval_net neural network is used to predict the estimated value of Q, and the parameters of eval_net are updated in real time by learning the data (state, action, and reward) of the interaction with the environment; the target_net neural network is used to predict the actual value of Q, and the parameter updates of target_net are regularly copied from eval_net.
[0151] The specific process of the interaction between the deep reinforcement learning model and the community environment, the imitation neural network, and the network parameter tuning experience data set is as follows: Figure 8As shown, the specific description of the deep reinforcement learning model of steps 7)-8) is as follows:
[0152] 7) Initialization of the deep reinforcement learning model includes the following steps:
[0153] a) Construct two neural networks, eval_net and target_net, with the same structure as the imitation neural network. The corresponding parameters of the two neural networks are θ and θ′ respectively.
[0154] b) Copy the parameters of the simulated neural network to the eval_net neural network.
[0155] c) Determine hyperparameters: learning rate α, discount factor γ, target_net parameter update cycle, memory storage size, etc.
[0156] 8) The cyclic iteration of the deep reinforcement learning model includes the following steps:
[0157] a) Get the current cell status s;
[0158] b) According to the current cell state s, select action a, which can be selected through eval_net, where a = argmax a Q(s,a;θ), i.e., the action a performed to make the target optimized cell transfer from the current cell state S to the next cell state S' is determined through the first neural network eval_net;
[0159] c) Execute action a, act on the cell environment (for example, execute action a to optimize the network parameters of the target optimized cell), migrate the environment state to state s' (for example, the target optimized cell transfers from the current cell state s to the next cell state s'), and feedback reward r;
[0160] d) storing the data (s, a, r, s') generated by the above interaction process into memory storage;
[0161] e) calculating a Q estimate, where Q estimate = Q value Q(s,a) corresponding to action a in step b);
[0162] f) Calculate the Q actual value, which is given by the second neural network target_net, where Q actual value = γ*max a′ Q(s′,a′;θ′)+r;
[0163] g) calculating the Q difference, where Q difference = (Q reality - Q estimate), and using the Q difference to define the loss function Loss of the first neural network eval_net;
[0164] h) Solve the parameters θ of the first neural network eval_net when minimizing the loss function Loss;
[0165] i) Every N steps, update the parameters θ′ of target_net by copying the parameters of the first neural network eval_net to the second neural network target_net;
[0166] Repeat steps a) to i) above until the deep reinforcement learning model converges.
[0167] 9) Incremental update of network parameter tuning experience. With the continuous analysis of the network parameter tuning experience data source, the network parameter tuning experience data set will be incrementally updated. Based on the generated incremental update data set, the storage memory of the imitation neural network and deep reinforcement learning model can be incrementally updated. On the one hand, the newly built intelligent agent can obtain the full amount of network parameter tuning experience. On the other hand, the existing intelligent agent can learn the incrementally updated network parameter tuning experience, which can further accelerate the intelligent agent learning process and reduce the number of iterations of interactive learning with the environment.
[0168] According to the above introduction, it can be seen that the imitation neural network obtained by imitation learning in the embodiment of the present application can make full use of the existing network parameter tuning experience data set; combined with the deep reinforcement learning model of the imitation neural network, it can converge quickly to avoid adverse effects on the network during the exploration process; the continuous incremental update of the network parameter tuning experience data set drives the incremental update learning of the data set of the imitation neural network and the deep reinforcement learning model, which can further accelerate the learning process of the deep reinforcement learning model and reduce the number of interactive iterative learning with the network environment. In addition, the trained neural network can help reduce the number of iterations required to adjust the cell parameters of the wireless network according to the optimization target, and can also help to achieve one-time optimization of the wireless network, that is, it may only be necessary to adjust the network parameters once (rather than iteratively) to achieve the optimization target.
[0169] The present application embodiment provides a network optimization device, such as Fig. 9 As shown, the network capacity prediction device 900 may include: an acquisition module 901, a processing module 902 and an optimization module 903, wherein:
[0170] An acquisition module 901 is used to acquire current network parameter data of a target optimized cell;
[0171] The processing module 902 is used to determine, based on the network parameter data and the network optimization index of the target optimization cell, to generate a network optimization strategy by imitating a neural network or a deep reinforcement learning model according to a preset decision strategy, wherein the imitating neural network is generated by training a predetermined neural network based on a network parameter tuning experience data set of the target optimization cell acquired in advance, and the deep reinforcement learning model is generated based on the imitating neural network;
[0172] The optimization module 903 is used to optimize the network parameters of the target optimization cell according to the network optimization strategy.
[0173] In a possible implementation, the processing model is also used to train a predetermined neural network based on a pre-acquired network parameter tuning experience data set of a target optimized cell to generate an imitation neural network, wherein the processing model is specifically used to:
[0174] A training sample data set is selected from the network parameter tuning experience data set. Each training sample in the training sample data set includes state data and action data. The state data is the cell state S of the target optimized cell. i , the action data is the target optimized cell from the cell state S i Transfer to the next cell state S i+1 The action performed i , i is a natural number;
[0175] The state data of each training sample in the training sample data set is used as the input data of the predetermined neural network, and the predetermined neural network is trained through an imitation learning algorithm until the error between the output data of the predetermined network and the action data corresponding to the state data of each training sample meets the predetermined condition.
[0176] In a possible implementation, each network parameter tuning experience data in the network parameter tuning experience data set is a five-tuple data, and the five-tuple data includes a cell identifier of a target optimized cell, a current cell state of the target optimized cell, a next cell state of the current cell state of the target optimized cell, an action performed by the target optimized cell to transfer from the current state to the next state, and a reward value obtained after performing the action;
[0177] Wherein, when the processing module selects the training sample data set from the network parameter tuning experience data set, it is used to:
[0178] A training sample data set is selected from a network parameter tuning experience data set according to a current cell state of a target optimized cell, a next cell state of the current cell state of the target optimized cell, and an action performed to transfer the target optimized cell from the current cell state to the next cell state.
[0179] In a possible implementation, the deep reinforcement learning model includes a first neural network and a second neural network; wherein the processing module, when generating the deep reinforcement learning model based on the imitation neural network, is used to:
[0180] Constructing a first neural network and a second neural network according to the structure of the imitative neural network, and copying the parameters of the imitative neural network to the first neural network and the second neural network;
[0181] Based on the current cell state S of the target optimized cell, the target parameter value θ of the first neural network is determined by performing a loop iteration on the first neural network, and during the loop iteration, the parameter value θ' of the second neural network is updated by copying the target parameter value θ to the second neural network at predetermined intervals.
[0182] In a possible implementation, when the processing module determines the target parameters of the first neural network by iterating the first neural network based on the acquired current cell state S of the target optimization cell, it is configured to:
[0183] Determine, by means of a first neural network, an action a performed to transfer the target optimized cell from a current cell state S to a next cell state S', and determine a Q estimation value according to the action a;
[0184] Determine the actual value of Q through a second neural network;
[0185] Determine the loss function of the first neural network according to the difference between the actual value of Q and the estimated value of Q;
[0186] The corresponding parameter value when the loss function is minimized is solved, and the parameter value is determined as the target parameter value θ.
[0187] In a possible implementation, the processing module is further configured to:
[0188] Based on the network parameter data and the network optimization indicators of the target optimization cell, it is directly determined to generate a network optimization strategy by imitating the neural network. When the network optimization strategy generated by imitating the neural network cannot meet the requirements, it is determined to generate a network optimization strategy by a deep reinforcement learning model.
[0189] In a possible implementation, the device further includes an updating module, where the updating module is configured to:
[0190] The optimized network parameters of the target optimized cell are obtained, and the optimized network parameters are updated to the network parameter tuning experience data set of the target optimized cell.
[0191] The network optimization device of the embodiment of the present application can execute the network optimization method shown in the above embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.
[0192] The network optimization device of the embodiment of the present application can not only optimize and adjust the network parameters directly based on the imitation neural network, so as to make full use of the existing network parameter tuning experience, but also optimize and adjust the network parameters by using the deep reinforcement learning model constructed on the basis of the imitation empirical neural network, so as to make full use of the advantage of the deep reinforcement learning model that can converge quickly and effectively avoid the adverse effects of the exploration process on the network; by integrating the imitation neural network and the deep reinforcement learning model, deep learning and reinforcement learning are combined to achieve the complementary advantages of the two, thereby providing a solution for the perception decision-making of dynamic optimization of mobile communication networks.
[0193] In an embodiment of the present application, an electronic device is provided, including a memory, a processor and a computer program stored in the memory, and the processor executes the above-mentioned computer program to implement the steps of the method for determining network coverage, which can be implemented compared with the prior art: obtaining the current network parameter data of the target optimization cell; then, based on the network parameter data and the network optimization index of the target optimization cell, according to a preset judgment strategy, determining to generate a network optimization strategy by imitating a neural network or a deep reinforcement learning model, wherein the imitating neural network is generated by training a predetermined neural network based on a network parameter tuning experience data set of the target optimization cell obtained in advance, and the deep reinforcement learning model is generated based on the imitating neural network; then, according to the network optimization strategy, the network parameters of the target optimization cell are optimized. The electronic device can not only directly optimize and adjust the network parameters based on the imitating neural network, thereby making full use of the existing network parameter tuning experience, but also optimize and adjust the network parameters by using the deep reinforcement learning model constructed on the basis of the imitating experience neural network, so as to make full use of the advantage of the deep reinforcement learning model that can converge quickly, and effectively avoid the adverse effects of the exploration process on the network; by integrating the imitating neural network and the deep reinforcement learning model, deep learning and reinforcement learning are combined, and the advantages of the two are complementary, thereby providing a solution for the perception decision of dynamic optimization of mobile communication networks.
[0194] In an alternative embodiment, an electronic device is provided, such as Fig.10 As shown, Fig.10The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.
[0195] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0196] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0197] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.
[0198] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.
[0199] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.
[0200] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.
[0201] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.
[0202] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.
Claims
1. A network optimization method, characterized in that: include: Obtain the current network parameter data of the target optimization cell; Based on the network parameter data and the network optimization index of the target optimized cell, directly determine to generate a network optimization strategy by imitating a neural network; if the network optimization index is not significantly improved, determine to generate a network optimization strategy by a deep reinforcement learning model, wherein the imitating neural network is generated by training a predetermined neural network based on a network parameter tuning experience data set of the target optimized cell acquired in advance, and the deep reinforcement learning model is generated based on the imitating neural network; Optimizing the network parameters of the target optimized cell according to the network optimization strategy; The deep reinforcement learning model includes a first neural network and a second neural network; generating the deep reinforcement learning model based on the imitation neural network includes: Constructing the first neural network and the second neural network according to the structure of the simulated neural network, and copying the parameters of the simulated neural network to the first neural network and the second neural network; Based on the current cell state S of the target optimized cell obtained, the target parameter value θ of the first neural network is determined by looping and iterating the first neural network, and in the process of the looping and iterating, the target parameter value θ is copied to the second neural network at predetermined intervals to update the parameter value θ of the second neural network. ’ .
2. The method according to claim 1, characterized in that The method of training a predetermined neural network based on a network parameter tuning experience data set of the target optimized cell acquired in advance to generate the imitation neural network includes: A training sample data set is selected from the network parameter tuning experience data set, each training sample in the training sample data set includes state data and action data, and the state data is the cell state S of the target optimized cell. i The action data is the target optimized cell from the cell state S i Transfer to the next cell state S i+1 The action performed i , i is a natural number; The state data of each training sample in the training sample data set is used as input data of the predetermined neural network, and the predetermined neural network is trained by an imitation learning algorithm until the error between the output data of the predetermined network and the action data corresponding to the state data of each training sample meets a predetermined condition.
3. The method according to claim 2, characterized in that Each network parameter tuning experience data in the network parameter tuning experience data set is a five-tuple data, and the five-tuple data includes a cell identifier of the target optimized cell, a current cell state of the target optimized cell, a next cell state of the current cell state of the target optimized cell, an action performed by the target optimized cell to transfer from the current state to the next state, and a reward value obtained after performing the action; The step of selecting a training sample data set from the network parameter tuning experience data set includes: A training sample data set is selected from the network parameter tuning experience data set according to the current cell state of the target optimized cell, the next cell state of the current cell state of the target optimized cell, and the action performed to transfer the target optimized cell from the current cell state to the next cell state.
4. The method according to claim 1, characterized in that The step of determining a target parameter of the first neural network by iterating the first neural network based on the acquired current cell state S of the target optimized cell includes: By using the first neural network, it is determined that the target optimized cell is transferred from the current cell state S to the next cell state S ’ The action a performed, and based on the action a, the Q estimate is determined; Determine the Q actual value by the second neural network; Determining a loss function of the first neural network according to a difference between the actual Q value and the estimated Q value; The parameter value corresponding to the minimization of the loss function is solved, and the parameter value is determined as the target parameter value θ.
5. The method according to any one of claims 1 to 4, characterized in that: After optimizing the network parameters of the target optimized cell according to the network optimization strategy, the method further includes: The optimized network parameters of the target optimized cell are obtained, and the optimized network parameters are updated into the network parameter tuning experience data set of the target optimized cell.
6. A network optimization device, characterized in that: include: An acquisition module is used to obtain the current network parameter data of the target optimization cell; A processing module, configured to directly determine to generate a network optimization strategy by imitating a neural network based on the network parameter data and the network optimization index of the target optimized cell, and if the network optimization index is not significantly improved, determine to generate a network optimization strategy by a deep reinforcement learning model, wherein the imitating neural network is generated by training a predetermined neural network based on a network parameter tuning experience data set of the target optimized cell acquired in advance, and the deep reinforcement learning model is generated based on the imitating neural network; An optimization module, used to optimize the network parameters of the target optimized cell according to the network optimization strategy; The deep reinforcement learning model includes a first neural network and a second neural network; generating the deep reinforcement learning model based on the imitation neural network includes: Constructing the first neural network and the second neural network according to the structure of the simulated neural network, and copying the parameters of the simulated neural network to the first neural network and the second neural network; Based on the current cell state S of the target optimized cell obtained, the target parameter value θ of the first neural network is determined by looping and iterating the first neural network, and in the process of the looping and iterating, the target parameter value θ is copied to the second neural network at predetermined intervals to update the parameter value θ of the second neural network. ’ .
7. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Optimization method, device, equipment and medium of Massive MIMO
CN109379752A
System and method for deep learning and wireless network optimization using deep learning
CN110770761A
Mobile network self-optimization method and system, terminal and computer readable storage medium
CN111050330A
Network optimization method and device
CN112188505A
Network intelligent optimization method and device based on channel beam pattern
CN112512069A
