An algorithm deployment method and device of a RIC node based on deep reinforcement learning
By using a deep reinforcement learning-based approach to predict and deploy near real-time RIC nodes, the algorithm set addresses the issue of low service acceptance rates during RIC node deployment, thereby improving service response speed and acceptance rates.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CORP LTD
- Filing Date
- 2022-07-28
- Publication Date
- 2026-07-07
AI Technical Summary
The existing RIC node deployment scheme cannot achieve automatic updates and maintenance of xAPP content, resulting in a decrease in business acceptance rate when business scenarios change.
By employing a deep reinforcement learning-based approach, the current state of the RIC node is determined, and a pre-trained deep reinforcement learning model is used to predict the set of algorithms that need to be deployed. The target set of algorithms is then distributed to the near real-time RIC node to adapt to ever-changing business needs.
It improved business response speed and business acceptance rate, and enabled near real-time automatic updates and maintenance of RIC node algorithms.
Smart Images
Figure CN115328495B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of mobile communication technology, and in particular to an algorithm deployment method and apparatus for RIC nodes based on deep reinforcement learning. Background Technology
[0002] With the deepening of cloud-network convergence, core network functional units have begun to gradually move to the cloud, and research on replacing dedicated network equipment with general-purpose equipment is also progressing steadily. Facing the demands of diversified future network services, inherent design, and open ecosystems, network intelligence is an important path for network evolution.
[0003] The O-RAN (Open Radio Access Network) organization proposed the RAN Intelligent Controller (RIC) for RAN (Radio Access Network) virtualization. The RIC includes Non-RT (Non-Real-Time) RICs and Near-RT (Near-Real-Time) RICs. Non-RT RICs are deployed on nodes with more computing power than Near-RT RICs. Both Non-RT and Near-RT RICs are deployed in a distributed architecture, meaning a Non-RT RIC can connect to one or more Near-RT RICs. Furthermore, Non-RT RICs can distribute trained AI algorithms (or xAPPs) to Near-RT RICs.
[0004] However, given that near real-time RICs are deployed closer to the edge, they have limitations in computing power and storage space, and near real-time RICs need to process business in real time, so it is necessary to deploy relevant AI algorithms in the near real-time RICs in advance.
[0005] In the existing deployment scheme, Near-RT RIC deploys corresponding algorithms based on the initial business needs, but it cannot achieve automatic updates and maintenance of xAPP content. When the business scenario changes, the previously deployed algorithms cannot handle the new business, resulting in a decrease in business acceptance rate. Summary of the Invention
[0006] The purpose of this application is to provide a method and apparatus for deploying algorithms for RIC nodes based on deep reinforcement learning, so as to predict in advance the set of algorithms that need to be deployed for near real-time RIC nodes, save communication overhead, and improve service response speed and service acceptance rate. The specific technical solution is as follows:
[0007] To achieve the above objectives, embodiments of this application provide an algorithm deployment method for RIC nodes based on deep reinforcement learning, applied to non-real-time RIC nodes included in a wireless intelligent controller RIC, wherein the wireless intelligent controller further includes near-real-time RIC nodes, and the method includes:
[0008] Determine the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0009] The node state is input into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision that maximizes the expected acceptance rate. The algorithm deployment decision includes a set of target algorithms that need to be deployed on the near real-time RIC node.
[0010] The target algorithm set is distributed to the near real-time RIC node so that the near real-time RIC node changes the deployed service algorithm set.
[0011] Optionally, the deep reinforcement learning model is trained based on multiple quadruplet data sets, wherein each quadruplet data set includes:
[0012] The first node state of the near real-time RIC node at each decision moment, the decision action at that decision moment, the second node state of the near real-time RIC node after executing the decision action, and the reward value of the decision action;
[0013] The first node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0014] Optionally, the quadruple data can be obtained in the following manner:
[0015] Determine the state of the first node of the near real-time RIC node at the current decision moment;
[0016] The decision action is determined based on the state of the first node, and the set of service algorithms deployed in the near real-time RIC node is changed based on the decision action;
[0017] Calculate the service acceptance rate of the near real-time RIC node between the previous decision time and the current decision time, and determine whether the service acceptance rate is greater than the previously calculated service acceptance rate; if it is greater, determine that the reward value of the decision action is a preset positive reward value; if it is not greater, determine that the reward value of the decision action is a preset negative reward value.
[0018] Optionally, the deep reinforcement learning model is a deep Q-network (DQN) model.
[0019] To achieve the above objectives, embodiments of this application also provide an algorithm deployment device for RIC nodes based on deep reinforcement learning, applied to non-real-time RIC nodes included in a wireless intelligent controller RIC, wherein the wireless intelligent controller further includes near-real-time RIC nodes, and the device includes:
[0020] The first determining module is used to determine the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed in the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0021] The second determining module is used to input the node state into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision with the highest expected acceptance rate, wherein the algorithm deployment decision includes a set of target algorithms that need to be deployed on the near real-time RIC node.
[0022] The distribution module is used to distribute the target algorithm set to the near real-time RIC node, so that the near real-time RIC node can change the deployed service algorithm set.
[0023] Optionally, the deep reinforcement learning model is trained based on multiple quadruplet data sets, wherein each quadruplet data set includes:
[0024] The first node state of the near real-time RIC node at each decision moment, the decision action at that decision moment, the second node state of the near real-time RIC node after executing the decision action, and the reward value of the decision action;
[0025] The first node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0026] Optionally, it also includes an acquisition module for acquiring the quadruple data; the acquisition module includes:
[0027] The determination submodule is used to determine the first node state of the near real-time RIC node at the current decision moment;
[0028] The modification submodule is used to determine a decision action based on the state of the first node, and modify the set of business algorithms deployed in the near real-time RIC node based on the decision action;
[0029] The judgment submodule is used to calculate the service acceptance rate of the near real-time RIC node between the previous decision time and the current decision time, and determine whether the service acceptance rate is greater than the previously calculated service acceptance rate; if it is greater, the reward value of the decision action is determined to be a preset positive reward value; if it is not greater, the reward value of the decision action is determined to be a preset negative reward value.
[0030] Optionally, the deep reinforcement learning model is a deep Q-network (DQN) model.
[0031] To achieve the above objectives, embodiments of this application also provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0032] Memory, used to store computer programs;
[0033] When the processor executes the program stored in memory, it implements the algorithm deployment method steps of any of the above-mentioned deep reinforcement learning-based RIC nodes.
[0034] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the algorithm deployment method steps for any of the deep reinforcement learning-based RIC nodes described above.
[0035] Beneficial effects of the embodiments in this application:
[0036] As can be seen, by applying the algorithm deployment method and apparatus for RIC nodes based on deep reinforcement learning provided in the embodiments of this application, the node state of the near real-time RIC node at the current decision time is determined. The node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time. The node state is input into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision with the highest expected acceptance rate. The algorithm deployment decision includes a target algorithm set that needs to be deployed on the near real-time RIC node. The target algorithm set is then distributed to the near real-time RIC node so that the near real-time RIC node changes the set of service algorithms it has deployed.
[0037] Based on the deep reinforcement learning model, the algorithm set that needs to be deployed for the near real-time RIC node can be predicted in real time according to the real-time changing node state of the near real-time RIC node, so as to adapt to the ever-changing business needs and improve business response speed and business acceptance rate.
[0038] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.
[0040] Figure 1 A flowchart illustrating an algorithm deployment method for RIC nodes based on deep reinforcement learning provided in an embodiment of this application;
[0041] Figure 2 A schematic diagram of a structure for an algorithm deployment device for RIC nodes based on deep reinforcement learning provided in an embodiment of this application;
[0042] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.
[0044] To address the technical problem of low service acceptance rates caused by the inability to predict the algorithm set to be deployed on near real-time RIC nodes in advance in existing technologies, this application provides an algorithm deployment method and apparatus for RIC nodes based on deep reinforcement learning. The method can be applied to non-real-time RIC nodes included in a wireless intelligent controller. The wireless intelligent controller also includes at least one near real-time RIC node. The non-real-time RIC nodes and near real-time RIC nodes are deployed in a distributed architecture, that is, a non-real-time RIC node can be connected to one or more near real-time RIC nodes.
[0045] See Figure 1 , Figure 1 This application provides a flowchart illustrating an algorithm deployment method for RIC nodes based on deep reinforcement learning, which can be applied to non-real-time RIC nodes contained in a wireless intelligent controller RIC, such as... Figure 1 As shown, the method may include the following steps:
[0046] S101: Determine the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0047] In this embodiment, the near real-time RIC node is deployed closer to the edge and needs to process services in real time.
[0048] The near real-time RIC nodes can handle various types of services, such as face recognition and image processing. Correspondingly, the service algorithms can include face recognition algorithms, image processing algorithms, etc.
[0049] Near real-time RIC nodes can receive service requests and inform non-real-time RIC nodes of the received requests. Due to the limitations of computing power and storage space of near real-time RIC nodes, and the need to process services in real time, non-real-time RIC nodes can pre-deploy relevant algorithms to near real-time RIC nodes, and the algorithms can be deployed in xAPP.
[0050] In this embodiment of the application, in order to achieve timely updates of the algorithm set deployed on the near real-time RIC node and maximize the service acceptance rate, a deep reinforcement learning model can be pre-trained, and a decision can be made based on the deep reinforcement learning model to determine the target algorithm set that needs to be cached on the near real-time RIC node.
[0051] Specifically, the deep reinforcement learning model makes decisions based on the node state of the near real-time RIC node to determine the set of business algorithms deployed on the near real-time RIC node.
[0052] Therefore, it is necessary to define the node state of the near real-time RIC node.
[0053] In this embodiment of the application, the deep reinforcement learning model makes a decision based on the node state at each decision time. Therefore, at each decision time, the node state of the near real-time RIC node may include: the set of business algorithms deployed within the near real-time RIC node, and the set of business requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0054] S102: Input the node state into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision with the highest expected acceptance rate, wherein the algorithm deployment decision includes a set of target algorithms that need to be deployed on the near real-time RIC node.
[0055] In one embodiment of this application, the deep reinforcement learning model is the DQN (deep Q network) model, which is essentially a Q-learning model based on deep neural networks.
[0056] Those skilled in the art will understand that after training the DQN model, the model is equivalent to a Q(s, a) function, where s represents the input of the model. In this embodiment, s is the node state of the near real-time RIC node. The model output is the decision action a that maximizes Q(s, a). In this embodiment, a is the algorithm deployment decision, which is the set of target algorithms that need to be deployed on the near real-time RIC node.
[0057] In this embodiment of the application, the goal of maximizing the business acceptance rate is to train the DQN model. Therefore, the Q value in Q(s,a) represents the expected acceptance rate.
[0058] After training the deep reinforcement learning model, inputting the node state into the deep reinforcement learning model will yield the decision action corresponding to the highest expected acceptance rate, which is the algorithm deployment decision with the highest expected acceptance rate.
[0059] S103: The target algorithm set is sent to the near real-time RIC node so that the near real-time RIC node changes the deployed service algorithm set.
[0060] Specifically, once the non-real-time RIC node determines the target algorithm set, it can distribute it to the near-real-time RIC node, thereby enabling the near-real-time RIC node to deploy suitable algorithms in advance, improving service response speed and service acceptance rate.
[0061] As can be seen, by applying the algorithm deployment method for RIC nodes based on deep reinforcement learning provided in the embodiments of this application, the node state of the near real-time RIC node at the current decision time is determined. The node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time. The node state is input into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision with the highest expected acceptance rate. The algorithm deployment decision includes a target algorithm set that needs to be deployed on the near real-time RIC node. The target algorithm set is then distributed to the near real-time RIC node so that the near real-time RIC node changes the set of service algorithms it has deployed.
[0062] Based on the deep reinforcement learning model, the algorithm set that needs to be deployed for the near real-time RIC node can be predicted in real time according to the real-time changing node state of the near real-time RIC node, so as to adapt to the ever-changing business needs and improve business response speed and business acceptance rate.
[0063] In one embodiment of this application, the deep reinforcement learning model is trained based on multiple quadruplet data, each quadruplet data including: the first node state of the near real-time RIC node at each decision time, the decision action at that decision time, the second node state of the near real-time RIC node after executing the decision action, and the reward value of the decision action.
[0064] The first node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0065] Specifically, at the current decision time t, the system state can be quantified as a set. in, This indicates the algorithm that has been trained in the non-real-time RIC node.
[0066] c t This represents the set of service requests that arrive sequentially at the near real-time RIC node between the previous decision time and the current decision time, and can be represented by D. t Each business request is composed of, and is represented as follows:
[0067] c t ={c(t,1),c(t,2),…,c(t,D)} t )}
[0068] θ t The algorithm cached in the near real-time RIC node can be represented as:
[0069] θ t =[θ t [1],θ t [2],…,θ t [c],…,θ t [C] T
[0070] Where, when θ t [c] = 1, indicating that the near real-time RIC node has algorithm c; when θ t [c] = 0 indicates that the c algorithm does not exist in the near real-time RIC node.
[0071] Acquiring the near real-time status of RIC nodes t Then, the deep reinforcement learning model decides what algorithm content needs to be cached at the next time step (t+1), and the corresponding decision is represented as A = {A}. t ,θ t+1}, that is, modifying the existing algorithm of the near real-time RIC node, θ t+1 This represents the set of algorithms cached in the near real-time RIC node after the decision is made.
[0072] In this embodiment of the application, a reward strategy needs to be pre-set during the training process. If the executed decision can improve the business acceptance rate, a positive reward will be obtained; if the executed decision causes the business acceptance rate to decrease, a negative reward will be obtained.
[0073] Specifically, the above quadruple data is obtained using the following steps:
[0074] Step 11: Determine the state of the first node of the near real-time RIC node at the current decision moment.
[0075] Step 12: Determine the decision action based on the state of the first node, and change the set of service algorithms deployed in the near real-time RIC node based on the decision action.
[0076] In deep reinforcement learning, an agent interacts with the environment, records the observed states, actions, and rewards, and uses these experiences to learn a policy function Q(s, a).
[0077] Specifically, in the initial stage, the policy function Q(s, a) is not accurate, but the decision action can still be determined based on the first node state of the near real-time RIC node at each decision moment.
[0078] Step 13: Calculate the service acceptance rate of the near real-time RIC node between the previous decision time and the current decision time, and determine whether the service acceptance rate is greater than the previously calculated service acceptance rate; if it is greater, determine that the reward value of the decision action is a preset positive reward value; if it is not greater, determine that the reward value of the decision action is a preset negative reward value.
[0079] After determining the decision action, the service algorithm deployed on the near-real-time RIC node is changed based on the decision action. Then, the service acceptance rate of the near-real-time RIC node between the previous decision time and the current decision time is calculated. It is determined whether the service acceptance rate is greater than the previously calculated service acceptance rate. If it is greater, the reward value for the decision action is determined to be a preset positive reward value, such as +1; if it is not greater, the reward value for the decision action is determined to be a preset negative reward value, such as -1. The above reward values are only examples and can be adjusted according to the specific environment to adapt to diverse application scenarios.
[0080] The service acceptance rate is the percentage of service requests received by a near real-time RIC node that can be processed. If the near real-time RIC node has an algorithm corresponding to the service request, then the service request can be processed.
[0081] For example, if the near real-time RIC node receives service requests a1, a2, a3, a4, a5, and a6 between the previous decision time and the current decision time, and the algorithms deployed on the near real-time RIC node include A1, A3, and A5, where algorithm A1 is used to process service request a1, algorithm A3 is used to process service request a3, and algorithm A5 is used to process service request a5, then 3 out of the above 6 service requests can be processed, and the service acceptance rate of the near real-time RIC node is 3 / 6 = 50%.
[0082] As can be seen, in this embodiment of the application, if the decision-making strategy can improve the business acceptance rate, it will receive a positive reward; if the business acceptance rate decreases, it will receive a negative reward.
[0083] By continuously executing steps 11-13 above, a large amount of quadruple data can be obtained, denoted as (s, a, r, s'), where s represents the first node state of the near-real-time RIC node at the decision moment, a represents the decision action at the decision moment, s' represents the second node state of the near-real-time RIC node after executing the decision action, and r represents the reward value of the decision action. A memory pool can then be constructed based on the quadruple data.
[0084] In this embodiment of the application, a deep reinforcement learning model can be trained based on a large amount of quadruplet data. Those skilled in the art will understand that the specific process of training a DQN model based on quadruplet data falls within the scope of the prior art.
[0085] Specifically, independent and identically distributed quadruples are selected from the memory pool for training. The loss value is calculated based on the constructed loss function, and the network parameters in the model are updated based on the gradient descent algorithm until the model converges.
[0086] As can be seen, in this embodiment of the application, the DQN model is trained by pre-acquired quadruple data, and based on the DQN model, the set of algorithms that need to be deployed for the near real-time RIC node is predicted in real time according to the node state of the near real-time RIC node, so as to adapt to the ever-changing business needs and improve the business response speed and business acceptance rate.
[0087] This application also provides an algorithm deployment device for RIC nodes based on deep reinforcement learning, see [link to relevant documentation]. Figure 2 , Figure 2 A schematic diagram illustrating the deployment of an algorithm for RIC nodes based on deep reinforcement learning, provided in this application embodiment, is applied to non-real-time RIC nodes included in a wireless intelligent controller RIC. The wireless intelligent controller further includes at least one near-real-time RIC node. The device comprises:
[0088] The first determining module 201 is used to determine the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed in the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0089] The second determining module 202 is used to input the node state into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision with the highest expected acceptance rate, wherein the algorithm deployment decision includes a set of target algorithms that need to be deployed on the near real-time RIC node.
[0090] The distribution module 203 is used to distribute the target algorithm set to the near real-time RIC node so that the near real-time RIC node changes the deployed service algorithm set.
[0091] In one embodiment of this application, the deep reinforcement learning model is trained based on multiple quadruplet data, wherein each quadruplet data includes:
[0092] The first node state of the near real-time RIC node at each decision moment, the decision action at that decision moment, the second node state of the near real-time RIC node after executing the decision action, and the reward value of the decision action;
[0093] The first node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0094] In one embodiment of this application, Figure 2 Based on the illustrated device, it may further include an acquisition module for acquiring the quadruple data; the acquisition module includes:
[0095] The determination submodule is used to determine the first node state of the near real-time RIC node at the current decision moment;
[0096] The modification submodule is used to determine a decision action based on the state of the first node, and modify the set of business algorithms deployed in the near real-time RIC node based on the decision action;
[0097] The judgment submodule is used to calculate the service acceptance rate of the near real-time RIC node between the previous decision time and the current decision time, and determine whether the service acceptance rate is greater than the previously calculated service acceptance rate; if it is greater, the reward value of the decision action is determined to be a preset positive reward value; if it is not greater, the reward value of the decision action is determined to be a preset negative reward value.
[0098] In one embodiment of this application, the deep reinforcement learning model is a deep Q-network (DQN) model.
[0099] As can be seen, the algorithm deployment device for RIC nodes based on deep reinforcement learning provided in the embodiments of this application determines the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed in the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time. The node state is input into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision with the highest expected acceptance rate. The algorithm deployment decision includes the target algorithm set that needs to be deployed in the near real-time RIC node. The target algorithm set is then distributed to the near real-time RIC node so that the near real-time RIC node changes the set of service algorithms it has deployed.
[0100] Based on the deep reinforcement learning model, the algorithm set that needs to be deployed for the near real-time RIC node can be predicted in real time according to the real-time changing node state of the near real-time RIC node, so as to adapt to the ever-changing business needs and improve business response speed and business acceptance rate.
[0101] This application also provides an electronic device, such as... Figure 3 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.
[0102] Memory 303 is used to store computer programs;
[0103] When processor 301 executes a program stored in memory 303, it performs the following steps:
[0104] Determine the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time.
[0105] The node state is input into a pre-trained deep reinforcement learning model to determine the algorithm deployment decision that maximizes the expected acceptance rate. The algorithm deployment decision includes a set of target algorithms that need to be deployed on the near real-time RIC node.
[0106] The target algorithm set is distributed to the near real-time RIC node so that the near real-time RIC node changes the deployed service algorithm set.
[0107] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0108] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0109] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0110] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0111] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described algorithm deployment methods for RIC nodes based on deep reinforcement learning.
[0112] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the algorithm deployment methods for RIC nodes based on deep reinforcement learning in the above embodiments.
[0113] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).
[0114] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0115] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of the algorithm deployment apparatus, electronic device, computer storage medium, and computer program product based on deep reinforcement learning RIC nodes are basically similar to the algorithm deployment embodiments based on deep reinforcement learning RIC nodes, so the descriptions are relatively simple. Relevant parts can be referred to in the descriptions of the algorithm deployment method embodiments based on deep reinforcement learning RIC nodes.
[0116] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A method for deploying RIC nodes based on deep reinforcement learning, characterized in that, The method, applied to non-real-time RIC nodes included in a wireless intelligent controller RIC, wherein the wireless intelligent controller also includes near-real-time RIC nodes, comprises: Determine the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time. The node state is input into a pre-trained deep reinforcement learning model. Based on the node state, the deep reinforcement learning model determines the algorithm deployment decision that maximizes the expected acceptance rate. The algorithm deployment decision includes a set of target algorithms that need to be deployed on the near real-time RIC node. The target algorithm set is distributed to the near real-time RIC node so that the near real-time RIC node changes the deployed service algorithm set. The deep reinforcement learning model is trained on multiple quadruplet data sets, where each quadruplet data set includes: The first node state of the near real-time RIC node at each decision moment, the decision action at that decision moment, the second node state of the near real-time RIC node after executing the decision action, and the reward value of the decision action; The first node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time; During the training process of the deep reinforcement learning model, the objective is to maximize the service acceptance rate. The service acceptance rate is the percentage of service requests received by the near real-time RIC node that can be processed. If the near real-time RIC node has an algorithm corresponding to the service request, then the service request can be processed.
2. The method according to claim 1, characterized in that, The quadruple data is obtained in the following manner: Determine the state of the first node of the near real-time RIC node at the current decision moment; The decision action is determined based on the state of the first node, and the set of service algorithms deployed in the near real-time RIC node is changed based on the decision action; Calculate the service acceptance rate of the near real-time RIC node between the previous decision time and the current decision time, and determine whether the service acceptance rate is greater than the previously calculated service acceptance rate; if it is greater, determine that the reward value of the decision action is a preset positive reward value; If it is not greater than, the reward value of the decision action is determined to be a preset negative reward value.
3. The method according to claim 1, characterized in that, The deep reinforcement learning model is a Deep Q-Network (DQN) model.
4. An algorithm deployment device for RIC nodes based on deep reinforcement learning, characterized in that, The device is applied to non-real-time RIC nodes included in a wireless intelligent controller RIC, wherein the wireless intelligent controller also includes near real-time RIC nodes, and the device comprises: The first determining module is used to determine the node state of the near real-time RIC node at the current decision time. The node state includes: the set of service algorithms deployed in the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time. The second determining module is used to input the node state into a pre-trained deep reinforcement learning model, and based on the deep reinforcement learning model and the node state, determine the algorithm deployment decision with the highest expected acceptance rate, wherein the algorithm deployment decision includes a set of target algorithms that need to be deployed on the near real-time RIC node. The distribution module is used to distribute the target algorithm set to the near real-time RIC node, so that the near real-time RIC node changes the deployed service algorithm set. The deep reinforcement learning model is trained on multiple quadruplet data sets, where each quadruplet data set includes: The first node state of the near real-time RIC node at each decision moment, the decision action at that decision moment, the second node state of the near real-time RIC node after executing the decision action, and the reward value of the decision action; The first node state includes: the set of service algorithms deployed within the near real-time RIC node, and the set of service requests received by the near real-time RIC node between the previous decision time and the current decision time; During the training process of the deep reinforcement learning model, the objective is to maximize the service acceptance rate. The service acceptance rate is the percentage of service requests received by the near real-time RIC node that can be processed. If the near real-time RIC node has an algorithm corresponding to the service request, then the service request can be processed.
5. The apparatus according to claim 4, characterized in that, It also includes an acquisition module for acquiring the quadruple data; The acquisition module includes: The determination submodule is used to determine the first node state of the near real-time RIC node at the current decision moment; The modification submodule is used to determine a decision action based on the state of the first node, and modify the set of business algorithms deployed in the near real-time RIC node based on the decision action; The judgment submodule is used to calculate the service acceptance rate of the near real-time RIC node between the previous decision time and the current decision time, and determine whether the service acceptance rate is greater than the previously calculated service acceptance rate; if it is greater, the reward value of the decision action is determined to be a preset positive reward value; if it is not greater, the reward value of the decision action is determined to be a preset negative reward value.
6. The apparatus according to claim 4, characterized in that, The deep reinforcement learning model is a Deep Q-Network (DQN) model.
7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-3.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-3.