A data processing method and system, device, storage medium

Abstract

The application discloses a data processing method, which comprises the following steps: a first module perceives data of a network environment to obtain a first state; the first state is used for indicating an environment state of the network environment; a second module obtains a first action output by the second module based on the first state and a received reinforcement learning strategy; the first module and the second module are arranged on one node and two nodes of a first node and a second node, and the first node is used for indicating a first controller and the second node is used for indicating a second controller. In addition, the application further discloses a data processing system, equipment and storage medium. The data processing method, system, equipment and storage medium provided by the application can realize multiple functions, thereby better meeting different requirements of different applications.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data processing method, system, device, and storage medium. Background Technology

[0002] Reinforcement learning can be implemented through state-aware modules, action modules, evaluation modules, and reinforcement learning (RL) modules. However, related technologies lack specific deployment methods for these modules. This lack of concrete deployment methods prevents reinforcement learning from achieving a wide range of functionalities, thus hindering its ability to better meet the diverse needs of different applications. Summary of the Invention

[0003] This application mainly provides a data processing method, system, device, and storage medium that can realize multiple functions, thereby better meeting the different needs of different applications.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0006] The first module senses data about the network environment and obtains a first state; the first state is used to indicate the environmental state of the network environment.

[0007] The second module, based on the first state and the received reinforcement learning policy, obtains the first action output by the second module; the first module and the second module are deployed on one and two nodes of the first node and the second node, respectively, with the first node used to instruct the first controller and the second node used to instruct the second controller.

[0008] Secondly, embodiments of this application provide a data processing system, the system comprising: a first module and a second module;

[0009] The first module is used to sense data about the network environment and obtain a first state; the first state is used to indicate the environmental state of the network environment.

[0010] The second module is used to obtain a first action output by the second module based on the first state and the received reinforcement learning policy; the first module and the second module are deployed on one and two nodes of the first node and the second node, respectively, the first node is used to instruct the first controller, and the second node is used to instruct the second controller.

[0011] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data processing method.

[0012] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described data processing method.

[0013] This application provides a data processing method, system, device, and storage medium. The method includes: a first module sensing data from a network environment to obtain a first state; the first state indicating the environmental state of the network environment; a second module obtaining a first action output by the second module based on the first state and a received reinforcement learning policy; the first module and the second module being deployed on one or both nodes of a first node and a second node, respectively, where the first node instructs a first controller and the second node instructs a second controller. This deployment method, by placing the first module and the second module on one or both nodes of the first and second nodes, allows for the implementation of multiple functions, thereby better meeting the diverse needs of different applications. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0015] Figure 2 A schematic flowchart of an optional data processing method provided in an embodiment of this application;

[0016] Figure 3 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0017] Figure 4 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0018] Figure 5 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0019] Figure 6 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0020] Figure 7 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0021] Figure 8 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0022] Figure 9 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0023] Figure 10 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0024] Figure 11 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0025] Figure 12 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0026] Figure 13 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0027] Figure 14 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0028] Figure 15 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0029] Figure 16 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0030] Figure 17 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0031] Figure 18 This is a schematic diagram of an optional structure of a data processing system provided in an embodiment of this application;

[0032] Figure 19 This is a schematic diagram of an optional structure of the data processing system provided in the embodiments of this application;

[0033] Figure 20 This is a schematic diagram of an optional structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of the application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first," "second," and "third" are used only to distinguish different objects and do not represent a specific order of objects, nor are they constituting a chronological order. It is understood that "first," "second," and "third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0038] The data processing method provided in this application embodiment can be applied to a data processing system, which includes a first module and a second module. The first module is used to sense data from the network environment and obtain a first state; the second module is used to determine a first action.

[0039] Here, the first module in the data processing system senses data from the network environment and obtains a first state; the first state is used to indicate the environmental state of the network environment.

[0040] The second module in the data processing system obtains the first action output by the second module based on the first state and the received reinforcement learning policy; the first module and the second module are deployed on one and two nodes of the first node and the second node, respectively, the first node is used to indicate the node where the first controller is located, and the second node is used to indicate the node where the second controller is located.

[0041] As an example, the structure of the data processing system 100 can be as follows: Figure 1 As shown, it includes: a first module 101 and a second module 102.

[0042] Below, in conjunction with Figure 1 The schematic diagram of the data processing system shown illustrates various embodiments of the data processing methods, apparatus, devices, and storage media provided in this application.

[0043] This application provides a data processing method, such as... Figure 2 As shown, it includes the following steps:

[0044] The first module of S201 senses data from the network environment and obtains the first state.

[0045] Here, the first state is used to indicate the environmental state of the network environment.

[0046] The network environment data may include at least one of the following: the number of resources allocated in the resource configuration application and parameters in the load balancing application. The number of resources allocated in the resource configuration application may include: resource blocks (RBs) and resource elements (REs).

[0047] After sensing network environment data, the first module can analyze the data to obtain indicators of the network environment's status. These statuses may include at least one of the following parameters indicative of the wireless environment: Signal-to-Interference plus Noise Ratio (SINR), Reference Signal-Receiving Power (RSRP), Channel Quality Indicator (CQI), Modulation and Coding Scheme (MCS), Block Error Rate (BLER), Physical Resource Block (PRB) occupancy rate of neighboring cells, PRB occupancy rate of the serving cell, and statistics on Acknowledgment (ACK) and Negative Acknowledgment (NACK) messages.

[0048] In one example, the environment state includes: SINR.

[0049] In another example, the environment states include SINR and RSRP.

[0050] In yet another example, the environmental states include: SINR, RSRP, CQI, MCS, BLER, neighboring cell PRB occupancy, serving cell PRB occupancy, ACK, and NACK.

[0051] S202, the second module obtains the first action output by the second module based on the first state and the received reinforcement learning policy.

[0052] Here, the first module and the second module are deployed on one of the first node and the two nodes of the second node. The first node is used to instruct the first controller, and the second node is used to instruct the second controller.

[0053] Reinforcement learning strategies are strategies for choosing which action to perform based on the environmental conditions of a network environment.

[0054] The second module can obtain the first action output by the second module based on the first state and the received reinforcement learning policy.

[0055] In one example, the network environment indicated by the first state includes: environment state 1 and environment state 2, where environment state 1 corresponds to candidate action 1 and environment state 2 corresponds to candidate action 2. If the environment is in environment state 1, candidate action 1 can be selected as the first action; if the environment is in environment state 2, candidate action 2 can be selected as the first action.

[0056] In this embodiment of the application, the first action may include: adjusting the number of resources allocated in the resource configuration application, adjusting one or both of the parameters in the load balancing application.

[0057] In practical applications, the first action may also include other actions, and this application embodiment does not impose any limitations on this.

[0058] Here, for an environmental state, the number of candidate actions corresponding to that environmental state can be one or more, such as two or more candidate actions. This application embodiment does not limit this.

[0059] The first controller can be a non-real-time RAN Intelligent Controller (Non-RT-RIC), and the second controller can be a near-real-time RAN Intelligent Controller (Near-RT-RIC). RAN stands for Radio Access Network.

[0060] In some embodiments, the deployment methods of the first module and the second module include one of the following:

[0061] The first module and the second module are deployed on the first node;

[0062] The first module and the second module are deployed on the second node;

[0063] The first module is deployed on the first node, and the second module is deployed on the second node.

[0064] In one example, the first module and the second module are deployed on the first node.

[0065] In another example, the first and second modules are deployed on the second node.

[0066] In yet another example, the first module is deployed on the first node, and the second module is deployed on the second node.

[0067] In some embodiments, the second module performs at least one of the following functions: receiving action data sent by the third module; the third module is used to manage the first model, the first model including at least: a reinforcement learning model; receiving reinforcement learning policies sent by the fourth module; the fourth module is used to perform performance processing on the first model.

[0068] In one example, the second module receives action data sent by the third module.

[0069] In another example, the second module receives action data sent by the third module, as well as reinforcement learning policies sent by the fourth module.

[0070] Here, the first model may also include one or both of the following: Artificial Intelligence (AI) models and Machine Learning models.

[0071] Performance processing of the first model may include: performance verification, performance testing, performance analysis, and performance optimization of the first model.

[0072] In some embodiments, when the first module is deployed on the first node and the second module is deployed on the second node, the first node sends the trained first model to the second node through a first interface set on the first node; the first model includes at least a reinforcement learning model.

[0073] Here, after the first node sends the trained first model to the second node through the first interface, the second module deployed in the second node can use the trained first model and obtain the first action based on the first state and the reinforcement learning strategy.

[0074] In some embodiments, when the first module is deployed on the first node and the second module is deployed on the second node, the first node sends a pre-trained first model to the second node through a first interface set on the first node, so that the second node modifies the pre-trained first model; the first model includes at least a reinforcement learning model.

[0075] Here, after the first node sends the pre-trained first model to the second node through the first interface, the second node can modify the pre-trained first model according to actual usage requirements. After the pre-trained first model is modified, the second module deployed in the second node can use the modified first model and obtain the first action based on the first state and reinforcement learning strategy.

[0076] In some embodiments, when the first module is deployed on the first node and the second module is deployed on the second node, the first node sends an action selection strategy for determining the first action based on the network environment to the second node through a second interface set on the first node; and / or, when the first module is deployed on the first node and the second module is deployed on the second node, the first node sends an action set to the second node through a second interface set on the first node, the action set including at least two actions.

[0077] In one example, with the first module deployed on the first node and the second module deployed on the second node, the first node can send an action selection strategy based on the network environment to the second node through the second lesson set on the first node, and the first node can send a set of actions to the second node through the second interface set on the first node.

[0078] In another example, where the first module is deployed on the first node and the second module is deployed on the second node, the first node can send an action selection strategy based on the network environment to the second node through the second lesson set on the first node, or the first node can send a set of actions to the second node through the second interface set on the first node.

[0079] Here, the action selection strategy is a strategy based on resources to select the first action. The resources may include at least one of the following: frequency domain resources, time domain resources, and power domain resources.

[0080] In one example, the resources include: frequency domain resources.

[0081] In another example, the resources include frequency domain resources and time domain resources.

[0082] In yet another example, the resources include: frequency domain resources, time domain resources, and power domain resources.

[0083] In some embodiments, the method further includes: the fifth module obtaining a first evaluation result output by the fifth module based on the received first action and the second state corresponding to the first action.

[0084] Here, the first evaluation result is used to indicate whether the selected first action is the target action among at least one action; the target action is the action that can produce the maximum reward under the environmental conditions of the network environment.

[0085] The first evaluation result may include: the first action is the target action in at least one action, or the first action is not the target action in at least one action.

[0086] The fifth module can determine whether the first action is a target action among at least one action based on the received first action and the second state corresponding to the first action.

[0087] In some embodiments, the method further includes: a sixth module obtaining a second evaluation result output by the sixth module based on the received first state and the first action.

[0088] Here, the second evaluation result is used to indicate whether it is worthwhile to select the first action under the environmental conditions of the network environment indicated by the first state.

[0089] The second evaluation result may include: whether it is worthwhile to choose the first action, or whether it is not worthwhile to choose the first action.

[0090] Here, the sixth module can calculate the received first state and first action to form a reward function, a state value function, or an action value function, in order to solve for the optimal value function. The result obtained from solving for the optimal value function is the second evaluation result.

[0091] In this embodiment of the application, the sixth module can also obtain a second evaluation result based on the received first state, first action and first evaluation result.

[0092] Here, the sixth module can calculate the received first state, first action, and first evaluation result to form a reward function, a state value function, or an action value function, in order to solve for the optimal value function. The result obtained by solving for the optimal value function is the second evaluation result.

[0093] In some embodiments, when the fifth module obtains the first evaluation result output by the fifth module based on the first action and the second state corresponding to the first action, the deployment methods of the fifth module and the sixth module include one of the following: the fifth module and the sixth module are deployed on the first node; the fifth module and the sixth module are deployed on the second node.

[0094] In one example, both the fifth and sixth modules are deployed on the first node.

[0095] In another example, both the fifth and sixth modules are deployed on the second node.

[0096] In some embodiments, the method further includes: the fifth module and the sixth module performing at least one of the following functions: receiving a first action and a second state corresponding to the first action sent by the second module; and sending the reinforcement learning policy to the second module.

[0097] Here, after the fifth module receives the first action and the second state corresponding to the first action sent by the second module, the fifth module can obtain the first evaluation result based on the first action and the second state corresponding to the first action.

[0098] Reinforcement learning algorithms are a type of machine learning algorithm characterized by interaction with the environment and dynamic decision-making, exhibiting strong adaptability to complex network environments. Wireless networks are complex heterogeneous networks that support diverse services and application scenarios, including continuous wide-area coverage, high-capacity hotspots, high reliability, low latency, and low-power massive connectivity. Applying reinforcement learning to wireless networks will perform well and achieve good results in dynamic and complex environmental environments.

[0099] Currently, reinforcement learning is mainly applied in wireless networks through independent network elements such as Multi-Access Edge Computing (MEC). MEC, also known as an edge server, is responsible for training the reinforcement learning model and selecting actions. It also updates the action, evaluation, and policy modules. During reinforcement learning inference, the edge server obtains data representing the network state from the wireless network elements and sends configuration parameters reflecting the actions to the corresponding network elements, which can then execute commands based on these parameters.

[0100] According to the relevant content described in the relevant documents, such as Figure 3 The diagram illustrates the deployment methods for reinforcement learning model training and inference in both non-real-time RAN Intelligent Controllers (Non-RT-RIC) and near-real-time RAN Intelligent Controllers (Near-RT-RIC). RAN stands for Radio Access Network.

[0101] like Figure 3 As shown, model training (ML training host) and model inference (ML model host / actor) can respectively realize the training and inference of reinforcement learning models. Figure 3In (a) of the diagram, both model training (ML training host) and model inference (ML model host / actor) are deployed in a Non-RT-RIC. Figure 3 In (b) of the above, both model training (ML training host) and model inference (ML model host / actor) are deployed in Near-RT-RIC. Figure 3 In (c), the model training host is deployed in the Non-RT-RIC, and the model inference host / actor is deployed in the Near-RT-RIC.

[0102] The general lifecycle processes and interface frameworks of Artificial Intelligence (AI) or Machine Learning (ML) in Open Radio Access Network (O-RAN) standards, such as... Figure 4 As shown.

[0103] The framework includes: Data Collection module 401, Data Preparation module 402, AI / ML Training module 403, AI / ML Model Management module 404, AI / ML Continuous Operation module 405, AI / ML Inference module 406, and AI / ML Assisted Solution module 407.

[0104] Figure 4 Some of the functionalities in this architecture can be deployed in one way within the O-RAN two-level intelligent controller (i.e., Non-RT-RIC and Near-RT-RIC) architecture, such as... Figure 5 As shown. Some of the functions include: data preparation, AI / ML training, AI / ML model management, AI / ML continuation operation, and AI / ML inference.

[0105] Here, the data collection module 401 is used to collect data.

[0106] The data preparation module 402 can process data based on application requirements. The data processing includes data preprocessing and data tagging.

[0107] The AI / ML Training module 403 is responsible for training AI models.

[0108] The AI / ML model management module 404 is used to distribute model files. Here, the AI / ML Model Management module 404 is the third module described in the above embodiment.

[0109] AI / ML inference module 406 is used to implement AI model inference.

[0110] The AI / ML Continuous Operation module 405 is used to perform performance verification, performance testing, performance analysis, and performance optimization of AI models.

[0111] Here, the AI / ML Continuous Operation module 405 is the fourth module described in the above embodiments.

[0112] The above-mentioned reinforcement learning algorithms have the following problems when implemented in wireless networks:

[0113] Problem 1: Reinforcement learning is implemented based on independent network elements (e.g., MECs). When implementing wireless applications, these elements need to interact extensively with the wireless network. However, there is no dedicated communication network interface to support this data transmission, leading to issues with the effectiveness of data transmission. Furthermore, since the inference results may be far removed from the wireless network element executing the action when performing model inference on the MEC, the timeliness of data transmission cannot be met, thus affecting the effectiveness of reinforcement learning applications in wireless networks.

[0114] Question 2: The model training mechanism in related technologies is not suitable for machine learning algorithms such as reinforcement learning that require online model training and inference.

[0115] Question 3: The relevant technologies do not address specific deployment methods for the state awareness module, action module, evaluation module, and reinforcement learning (RL) policy update module. However, the data exchange network elements and computational load required for these modules typically vary across different applications. To better meet the needs of different applications, flexible deployment methods need to be designed for each module, thereby supporting the effective deployment of reinforcement learning.

[0116] This application not only solves the above-mentioned problems, but also solves the following problems:

[0117] Question 4: To support the deployment of the action module, evaluation module, and RL policy update module, we propose functional enhancements for the Sequential Minimal Optimization (SMO) algorithm, network elements such as Non-RT-RIC and Near-RT-RIC, as well as the O1 and A1 interfaces. These enhancements include: functional enhancements for AI / ML Model management, AI / ML inference, and AI / ML Continuous Operation in SMO / Non-RT-RIC, and support for AI / ML Continuous Operation in Near-RT-RIC.

[0118] Question 5: To enable flexible implementation of reinforcement learning in wireless networks, we propose multiple data transmission methods based on O1 and A1 interfaces, supporting various reinforcement learning implementation approaches.

[0119] The data processing method provided in this application is described in detail below.

[0120] This application can be implemented based on an O-RAN two-level intelligent controller (Non-RT-RIC and Near-RT-RIC) architecture. Key aspects of this application include the following:

[0121] First, in order to meet the computational demands of reinforcement learning model training and improve the execution efficiency of the inference process of reinforcement learning models, we propose multiple deployment methods for the state awareness module, action module, evaluation module, and RL policy update model required for reinforcement learning on Non-RT-RIC and Near-RT-RIC.

[0122] Second, in order to achieve flexible deployment of reinforcement learning, we propose functional enhancement schemes for SMO / Non-RT-RIC and Near-RT-RIC, as well as functional enhancements for the O1 and A1 interfaces.

[0123] Third, in order to support multiple reinforcement learning implementation methods, model files, policy information and reinforcement information (EI) are transferred between Non-RT-RIC and Near-RT-RIC through O1 and A1 interfaces.

[0124] Reinforcement learning models such as Figure 6 As shown, Agent 601 is the subject that learns and implements strategies. Everything outside the agent that interacts with it can be called the environment. At each time t, the agent is in state s. t Next, select an action based on the RL policy.t Environment 602 responds accordingly to the action and reports a new state S to the agent program. t+1 At the same time, the environment will generate immediate rewards. t+1 The reward or penalty is a specific numerical value, representing the maximum value the agent aims to maximize during action selection. After the immediate reward is received, the agent calculates the reward *v* based on this reward and updates its RL policy accordingly. The agent then selects its next action based on the current state of the environment, maximizing the probability of receiving a positive reward. The selected action is *a*. t+1 It will not only affect the immediate returns, but also the subsequent and eventual cumulative returns.

[0125] like Figure 7 As shown, the reinforcement learning system includes: a state awareness module 701, an action module 702, an evaluation module 703, and an RL policy update module 704. Specifically, the state awareness module 701 is the first module described in the above embodiments, the action module 702 is the second module described in the above embodiments, the evaluation module 703 is the fifth module described in the above embodiments, and the RL policy update module 704 is the sixth module described in the above embodiments.

[0126] Here, the state awareness module is used to sense or collect network environment data and form state values. These state values ​​can be network environment parameters.

[0127] The action module generates actions based on the RL policy and state values ​​(e.g., network environment parameters). The inputs to the action module include the RL policy and state values; the output of the action module includes the action. It also sends state-action value pairs to the evaluation module.

[0128] The evaluation module is used to generate an evaluation result for the action-state value pair based on the state-action value pair, or the state value generated by the action. This evaluation result is the immediate feedback.

[0129] The RL policy update module is used to calculate a reward function, a state value function, or an action value function based on the state-action-instant reward pair, and then solve for the optimal value function. This value function is used to measure the merits of a certain state or action-state, that is, whether the value function is worthwhile for the agent to choose a certain action or perform a certain action in a certain state.

[0130] The O-RAN standard does not provide detailed deployment methods for the state awareness module, action module, evaluation module, and RL policy update module.

[0131] The state-aware module in this application can be deployed in a Non-RT-RIC or an SMO. The deployment of the state-aware module in a Non-RT-RIC will be described in detail below. Since the description of deploying the state-aware module in a Non-RT-RIC also applies to the deployment in an SMO, it will not be repeated here.

[0132] This application proposes the following solutions:

[0133] Deployment Scenario 1: The state awareness module, action module, evaluation module, and RL policy update module are deployed in Non-RT-RIC.

[0134] Non-RT-RIC sends policy information to Near-RT-RIC via the A1 interface.

[0135] Here, the policy information includes: the action 'a' that the network needs to perform, such as: the number of resources allocated in the resource configuration application, the adjustment of parameters in the load balancing application, such as: increasing or decreasing parameters, and the adjustment of the interference decision threshold in interference detection, such as: increasing or decreasing the threshold, etc.

[0136] like Figure 8 As shown, Non-RT-RIC 801 sends policy information to Near-RT-RIC 802 via the A1 interface, and Near-RT-RIC sends actions to base station 803 via the E2 interface. Near-RT-RIC is the second node described in the above embodiment.

[0137] Deployment Scenario 2: The state awareness module, evaluation module, and RL policy update module are deployed in Non-RT-RIC, while the action module is deployed in Near-RT-RIC.

[0138] like Figure 9 As shown, Non-RT-RIC 801 sends state values ​​and RL policy files to Near-RT-RIC 802 through the A1 interface; Non-RT-RIC sends model files to Near-RT-RIC through the O1 / A1 interface; Near-RT-RIC feeds back state-action value pairs to Non-RT-RIC through the A1 interface.

[0139] Deployment Scenario 3: The evaluation module and RL policy update module are deployed in Non-RT-RIC, while the state awareness module and action module are deployed in Near-RT-RIC.

[0140] like Figure 10 As shown, Non-RT-RIC 801 sends RL policy files to Near-RT-RIC 802 via the A1 interface;

[0141] Non-RT-RIC sends model files to Near-RT-RIC via the O1 / A1 interface;

[0142] Near-RT-RIC feeds back state-action value pairs to Non-RT-RIC via the A1 interface.

[0143] Deployment Scenario 4: The state awareness module is deployed in Non-RT-RIC, while the action module, evaluation module, and RL policy update module are deployed in Near-RT-RIC.

[0144] like Figure 11 As shown, Non-RT-RIC 801 sends status values ​​to Near-RT-RIC 802 through the A1 interface, such as status values ​​reflecting network performance, obtained after data processing based on reinforcement learning requirements, including cell PRB utilization, number of cell access users, and cell uplink RSRP value; Non-RT-RIC sends model files to Near-RT-RIC through the O1 / A1 interface.

[0145] Deployment Scenario 5: The state awareness module, action module, evaluation module, and RL policy update module are deployed in Near-RT-RIC.

[0146] like Figure 12 As shown, Non-RT-RIC sends network data to Near-RT-RIC via the O1 / A1 interface, such as parameters reflecting network performance (not state values ​​obtained after processing according to reinforcement learning requirements), including cell PRB utilization, number of users accessing the cell, and cell uplink RSRP value; Non-RT-RIC sends model files to Near-RT-RIC via the O1 / A1 interface.

[0147] The functional enhancements for Non-RT-RIC and Near-RT-RIC will be described in detail below.

[0148] The correspondence between the State Awareness Module, Action Module, Evaluation Module, RL Policy Update Module, and the functional modules in the O-RAN standard is as follows:

[0149] The state-aware module corresponds to Data Preparation in SMO or Non-RT-RIC;

[0150] The action module corresponds to AI / ML inference in Non-RT-RIC and Near-RT-RIC;

[0151] The evaluation module and RL policy update module correspond to AI / ML ContinuousOperation in SMO or Non-RT-RIC.

[0152] In order to enable multiple deployment methods of reinforcement learning, functional modules in the existing SMO / Non-RT-RIC and Near-RT-RIC need to be enhanced.

[0153] After enhancing the functionality of AI / ML Continuous Operation, at least one of the following functions can be achieved:

[0154] Receive reinforcement learning models sent by AI / ML Model Management, wherein the reinforcement learning model includes: an evaluation module and an RL policy update module;

[0155] Implement the deployment of the evaluation module and RL policy update module in the reinforcement learning model;

[0156] Receive status-action value pairs sent by AI / ML Inference;

[0157] Implement the functions of the evaluation module and the RL policy update module, which respectively evaluate the action-state value pairs and update the RL policy;

[0158] Send RL policies to AI / ML Inference.

[0159] After enhancing the functionality of AI / ML Model Management, at least one of the following functions can be achieved:

[0160] The evaluation module and RL policy update module of the reinforcement learning model are sent to the AI / ML Continuous Operation. The sending of the evaluation module and RL policy update module can be done by sending the code that implements the evaluation module and RL policy update module, or by sending it in other ways. This application embodiment does not limit this in any way.

[0161] Send the action module portion of the reinforcement learning model to the AI / ML Inference.

[0162] After enhancing the functionality of AI / ML Inference, at least one of the following functions can be achieved:

[0163] The module that receives reinforcement learning model actions from AI / ML Model Management;

[0164] Receive RL policy file data sent by AI / ML Continuous Operation;

[0165] Send state-action value pairs containing the inference results to AI / ML Continuous Operation.

[0166] For deployment scenario one above, the interactions between various functions in SMO / Non-RT-RIC are as follows: Figure 13 As shown.

[0167] like Figure 13 As shown, the data collection module 401 sends data to the data preparation module 402, the data preparation module 402 sends AI / ML inference data to the AI / ML inference module 406, the AI / ML model management module 404 sends model deployment to the AI / ML inference module 406 and the AI / ML continuation operation module 405, the AI / ML inference module 406 sends state-action value to the AI / ML continuation operation module 405, and the AI / ML continuation operation module 405 sends reinforcement learning policy to the AI / ML inference module 406.

[0168] For deployment scenario two above, the interactions between various functions in SMO / Non-RT-RIC are as follows: Figure 14 As shown.

[0169] like Figure 14 As shown, the data collection module 401 sends data to the data preparation module 402. The data preparation module 402 sends AI / ML inference data to the AI / ML inference module 406 via the A1 interface. The AI / ML model management module 404 sends model deployment to the AI / ML inference module 406 via the O1 / A1 interface. The model management module X1104 sends model deployment to the AI / ML continuation operation module 405. The AI / ML inference module 406 sends state-action value to the AI / ML continuation operation module 405 via the A1 interface. The AI / ML continuation operation module 405 sends reinforcement learning policy to the AI / ML inference module 406 via the A1 interface.

[0170] For deployment scenario three above, the interactions between various functions in SMO / Non-RT-RIC are as follows: Figure 15 As shown.

[0171] like Figure 15As shown, the data collection module 401 sends data to the data preparation module 402, the data preparation module 402 sends AI / ML inference data to the AI / ML inference module 406, the AI / ML model management module 404 sends state-action value to the AI / ML continuation operation module 405 through the A1 interface, the AI / ML continuation operation module 405 sends reinforcement learning policy to the AI / ML inference module 406 through the A1 interface, the model management module 404 sends model deployment to the AI / ML continuation operation module 405, and the model management module 404 sends model deployment to the AI / ML inference module 406 through the A1 / O1 interface.

[0172] After enhancing the functionality of Near-RT-RIC, at least one of the following functions can be achieved:

[0173] Near-RT-RIC supports AI / ML Continuous Operation functionality;

[0174] In Near-RT-RIC, the AI / ML Continuous Operation receives reinforcement learning model evaluation and RL policy update modules from the AI / ML Model Management.

[0175] Example 4: For deployment scenario four above, the interactions between various functions in SMO / Non-RT-RIC are as follows... Figure 16 As shown.

[0176] like Figure 16As shown, the data collection module 401 sends data to the data preparation module 402. The data preparation module 402 sends AI / ML inference data to the AI / ML inference module 406 via the A1 interface. The AI / ML inference module 406 sends state-action values ​​to the AI / ML continuation operation module 405. The continuation operation module 405 sends reinforcement learning policies to the AI / ML inference module 406. The AI / ML model management module 404 sends model deployment to the AI / ML continuation operation module 405 via the O1 / A1 interface.

[0177] Example 5: For deployment scenario five above, the interactions between various functions in SMO / Non-RT-RIC are as follows: Figure 17 As shown.

[0178] like Figure 17 As shown, the data collection module 401 sends data to the data preparation module 402, the data preparation module 402 sends AI / ML inference data to the AI / ML inference module 406, the AI / ML inference module 406 sends state-action value to the AI / ML continuation operation module 405, the continuation operation module 405 sends reinforcement learning policy to the AI / ML inference module 406, the AI / ML model management module 404 sends model deployment to the AI / ML continuation operation module 405 through the O1 / A1 interface, and the AI / ML model management module 404 sends model deployment to the AI / ML continuation operation module 405 through the O1 / A1 interface.

[0179] After enhancing the functionality of AI / ML Model Management, it can support the transformation between different reinforcement learning deployment methods and achieve at least one of the following functions:

[0180] AI / ML Model Management enables the deployment or removal of action module portions in reinforcement learning models for AI / ML inference deployed in SMO / Non-RT-RIC and AI / ML inference deployed in Near-RT-RIC.

[0181] AI / ML Model Management enables the deployment or removal of evaluation modules and RL policy update modules in reinforcement learning models deployed in SMO / Non-RT-RIC and AI / ML Continuous Operation deployed in Near-RT-RIC.

[0182] like Figure 18 As shown, when the reinforcement learning deployment method is deployment method one, the SMO / Non-RT-RIC uses the O1 / A1 interface to send the action module part of the reinforcement learning model to the AI / ML Inference in the Near-RT-RIC through the AI / ML Model Management, and then deletes the action module part of the reinforcement learning model in the AI / ML Inference deployed in the Non-RT-RIC, thus realizing the transformation from deployment scenario one to deployment scenario two.

[0183] The enhanced functionality of the O1 and A1 interfaces will be described in detail below.

[0184] To enable flexible implementation of reinforcement learning in wireless networks, this application proposes various data transmission methods based on O1 and A1 interfaces, including deployment under conditions such as already trained models and pre-trained models; supports the distribution of action space selection strategies to achieve flexible policy modification and updates; and supports the distribution of optional action sets to reduce the decision complexity of the action module.

[0185] Here, the trained model is the first trained model described in the above embodiments, and the pre-trained model is the first pre-trained model described in the above embodiments.

[0186] For model files distributed via the O1 interface, Non-RT-RIC distributes reinforcement learning model files to Near-RT-RIC through the O1 interface. This solution is applicable to deployment scenarios two, three, and four. The distributed reinforcement learning model files contain the trained reinforcement learning models. Furthermore, in deployment scenarios two, three, and four, evaluation and policy update modules are not required in both Non-RT-RIC and Near-RT-RIC.

[0187] For pre-trained reinforcement learning models distributed via the O1 interface, Non-RT-RIC distributes pre-trained reinforcement learning models to Near-RT-RIC via the O1 interface. This solution is applicable to deployment scenarios two, three, and four. The distributed reinforcement learning model file contains the pre-trained reinforcement learning model. This solution is suitable for using existing pre-trained models (reinforcement learning models trained in simulation environments or other network environments / scenarios).

[0188] Regarding the action space selection strategy issued via the A1 interface, the Non-RT-RIC issues the action space selection strategy to the Near-RT-RIC, which then formulates / updates the action space based on the strategy. This solution is applicable to deployment scenarios two, three, and four. The action module selects suitable actions based on the action space selection strategy. The action space selection strategy includes action items and / or action granularity. For example, in resource configuration applications, the action space selection strategy may include frequency domain resources, time domain resources, and power domain resources. The action granularity can be the number of time domain resources, such as the number of RBs.

[0189] Regarding the distribution of optional action sets via the A1 interface, this application applies to deployment scenario two. The Non-RT-RIC distributes the optional action set to the Near-RT-RIC via the A1 interface. This information is part of the policy file transmitted from the policy update module to the action module. The action module in the Near-RT-RIC selects an action from the optional action set based on the policy. This solution can reduce the decision complexity of the action module and facilitates faster convergence of reinforcement learning.

[0190] In summary, the key points of this application are: the deployment method of reinforcement learning, the enhancement of Non-RT-RIC and Near-RT-RIC functions, and the enhancement of O1 / A1 interface data transmission functions.

[0191] The protection points of this application include the following:

[0192] The first and fifth methods for deploying or implementing reinforcement learning based on the O-RAN two-level intelligent controller (Non-RT-RIC and Near-RT-RIC) architecture.

[0193] Second, functional enhancement schemes for SMO / Non-RT-RIC and Near-RT-RIC, as well as functional enhancements for the O1 and A1 interfaces.

[0194] The third and fourth types of O1 / A1 interface data transmission items include the transmission of model files, policy information, and EI information, supporting flexible deployment of reinforcement learning.

[0195] The reinforcement learning deployment method designed in this application, the functional enhancements of SMO / Non-RT-RIC and Near-RT-RIC, and the effects of O1 / A1 interface data transmission may include:

[0196] To achieve the deployment of reinforcement learning based on the O-RAN two-level intelligent controller (Non-RT-RIC and Near-RT-RIC) architecture;

[0197] 2. Online learning of reinforcement learning (including model training and model inference) can be achieved through the functional enhancement schemes of SMO / Non-RT-RIC and Near-RT-RIC;

[0198] Supports state awareness, action, evaluation, and policy update modules based on the high computational demands of reinforcement learning for model training and rapid interaction between network elements;

[0199] The O1 and A1 interfaces support various data transmission methods to enable flexible implementation of reinforcement learning in wireless networks, including deployment under conditions such as pre-trained models and fully trained models; they support the distribution of action space selection policies, enabling flexible policy modification and updates; and they support the distribution of optional action sets to reduce the decision complexity of action modules, which is beneficial for the rapid convergence of reinforcement learning.

[0200] Figure 19 A data processing system 1900 is provided for embodiments of this application. The system 1900 includes: a first module 1901 and a second module 1902.

[0201] The first module 1901 is used to sense data of the network environment and obtain a first state; the first state is used to indicate the environmental state of the network environment.

[0202] The second module 1902 obtains the first action output by the second module based on the first state and the received reinforcement learning policy; the first module and the second module are deployed on one and two nodes of the first node and the second node, respectively, the first node is used to instruct the first controller, and the second node is used to instruct the second controller.

[0203] In some embodiments, the deployment methods of the first module 1901 and the second module 1902 include one of the following:

[0204] The first module 1901 and the second module 1902 are deployed on the first node;

[0205] The first module 1901 and the second module 1902 are deployed on the second node;

[0206] The first module 1901 is deployed on the first node, and the second module 1902 is deployed on the second node.

[0207] In some embodiments, the second module 1902 performs at least one of the following functions:

[0208] The system receives action data sent by a third module; the third module is used to manage a first model, which includes at least a reinforcement learning model.

[0209] The first model receives a reinforcement learning strategy sent by the fourth module; the fourth module is used to perform performance processing on the first model.

[0210] In some embodiments, the system 1900 further includes: a transmitting module, configured to:

[0211] When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends the trained first model to the second node through a first interface set on the first node; the first model includes at least a reinforcement learning model.

[0212] In some embodiments, the sending module is further configured to:

[0213] When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends a pre-trained first model to the second node through a first interface set on the first node, so that the second node modifies the pre-trained first model; the first model includes at least a reinforcement learning model.

[0214] In some embodiments, the sending module is further configured to:

[0215] When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends an action selection strategy based on the network environment to the second node through a second interface set on the first node;

[0216] And / or,

[0217] When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends an action set to the second node through a second interface set on the first node, the action set including at least two actions.

[0218] In some embodiments, system 1900 further includes: a fifth module, configured to:

[0219] Based on the received first action and the second state corresponding to the first action, a first evaluation result is obtained from the output of the fifth module; the first evaluation result is used to indicate whether the selected first action is the target action among the at least one action; the target action is the action that can generate the maximum reward under the environmental conditions of the network environment.

[0220] In some embodiments, system 1900 further includes: a sixth module, configured to:

[0221] Based on the received first state and first action, a second evaluation result is obtained from the output of the sixth module; the second evaluation result is used to indicate whether it is worthwhile to select the first action under the environmental conditions of the network environment indicated by the first state.

[0222] In some embodiments, when the fifth module obtains a first evaluation result output by the fifth module based on the first action and the second state corresponding to the first action, the deployment methods of the fifth module and the sixth module include one of the following:

[0223] The fifth module and the sixth module are deployed on the first node;

[0224] The fifth module and the sixth module are deployed on the second node.

[0225] In some embodiments, the fifth module performs at least one of the following functions:

[0226] Receive the first action and the second state corresponding to the first action sent by the second module;

[0227] The reinforcement learning strategy is sent to the second module.

[0228] It should be noted that the information processing device provided in this application embodiment includes various units that can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0229] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0230] It should be noted that, in the embodiments of this application, if the above-described information processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0231] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data processing method.

[0232] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described data processing method.

[0233] It should be noted that the above description of the storage medium embodiments is similar to the description of the method embodiments described above, and has similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0234] It should be noted that, Figure 20 This is a schematic diagram of a hardware entity of an electronic device according to an embodiment of this application, such as... Figure 20 As shown, the electronic device 2000 includes: a processor 2001, at least one communication bus 2002, at least one external communication interface 2004, and a memory 2005. The communication bus 2002 is configured to enable communication between these components. In one example, the electronic device 2000 further includes: a user interface 2003, which may include a display screen; and the external communication interface 2004 may include standard wired and wireless interfaces.

[0235] The memory 2005 is configured to store instructions and applications executable by the processor 2001, and can also cache data to be processed or already processed (e.g., image data, audio data, and communication data) of the processor 2001 and various modules in the electronic device. It can be implemented by flash memory or random access memory (RAM).

[0236] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of the present application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0237] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0239] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, each functional unit in the embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0241] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0242] Alternatively, if the integrated units described in the embodiments of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0243] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: The first module senses data about the network environment and obtains a first state; the first state is used to indicate the environmental state of the network environment. The second module, based on the first state and the received reinforcement learning policy, obtains the first action output by the second module, wherein the reinforcement learning policy includes the correspondence between the first state and the first action; the deployment methods of the first module and the second module include at least: the first module is deployed on the first node, and the second module is deployed on the second node; the first node is used to instruct the first controller, and the second node is used to instruct the second controller, wherein the first controller is a non-real-time RAN intelligent controller, and the second controller is a near real-time RAN intelligent controller; The first module is a state awareness module, and the second module is an action module. The state awareness module and the action module are modules specific to reinforcement learning.

2. The method according to claim 1, characterized in that, The deployment methods for the first module and the second module also include: The first module and the second module are deployed on the first node; Alternatively, the first module and the second module may be deployed on the second node.

3. The method according to claim 2, characterized in that, The second module implements at least one of the following functions: The system receives action data sent by the third module; the third module is used to manage the first model, which is a reinforcement learning model. The first model receives a reinforcement learning strategy sent by the fourth module; the fourth module is used to perform performance processing on the first model.

4. The method according to claim 1, characterized in that, When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends the trained first model to the second node through the first interface set on the first node; the first model is a reinforcement learning model.

5. The method according to claim 1, characterized in that, When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends a pre-trained first model to the second node through a first interface set on the first node, so that the second node modifies the pre-trained first model; the first model is a reinforcement learning model.

6. The method according to claim 1, characterized in that, When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends an action selection strategy based on the network environment to the second node through a second interface set on the first node; And / or, When the first module is deployed on the first node and the second module is deployed on the second node, the first node sends an action set to the second node through a second interface set on the first node, the action set including at least two actions.

7. The method according to claim 1, characterized in that, The method further includes: The fifth module obtains a first evaluation result based on the received first action and the second state corresponding to the first action; the first evaluation result is used to indicate whether the selected first action is a target action among at least one action; the target action is the action that can produce the maximum reward under the environmental conditions of the network environment.

8. The method according to claim 1, characterized in that, The method further includes: Based on the received first state and first action, the sixth module obtains a second evaluation result output by the sixth module; the second evaluation result is used to indicate whether it is worthwhile to select the first action under the environmental state of the network environment indicated by the first state.

9. The method according to claim 8, characterized in that, When the fifth module obtains the first evaluation result output by the fifth module based on the first action and the second state corresponding to the first action, the deployment methods of the fifth module and the sixth module include one of the following: The fifth module and the sixth module are deployed on the first node; The fifth module and the sixth module are deployed on the second node.

10. The method according to claim 9, characterized in that, The fifth module performs at least one of the following functions: Receive the first action and the second state corresponding to the first action sent by the second module; The reinforcement learning strategy is sent to the second module.

11. A data processing system, characterized in that, The system includes: a first module and a second module; The first module is used to sense data about the network environment and obtain a first state; the first state is used to indicate the environmental state of the network environment. The second module is used to obtain a first action output by the second module based on the first state and the received reinforcement learning policy, wherein the reinforcement learning policy includes the correspondence between the first state and the first action; the deployment of the first module and the second module includes at least the following: the first module is deployed on the first node, and the second module is deployed on the second node; the first node is used to instruct the first controller, and the second node is used to instruct the second controller, wherein the first controller is a non-real-time RAN intelligent controller, and the second controller is a near real-time RAN intelligent controller; The first module is a state awareness module, and the second module is an action module. The state awareness module and the action module are modules specific to reinforcement learning.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method according to any one of claims 1 to 10.

13. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 10.

Images