Wireless access network control device

Abstract

To make it possible to prevent a decrease in learning efficiency and to prevent the amount of learning data transmitted between a device that collects learning data and a device that performs machine learning from increasing too much.SOLUTION: A wireless access network (RAN) control device includes collecting means for controlling the RAN on the basis of a learning model to collect a first learning data from the RAN and determination means for determining the usefulness of first learning data in machine learning for the learning model, and processing means for performing processing to select second learning data to be transmitted to another control device that performs machine learning on the basis of the usefulness of the first learning data from the first learning data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to radio access network (RAN) control techniques. [Background technology]

[0002] Figure 1 shows the control configuration of RAN proposed by the Open Radio Access Network (O-RAN) Alliance. As shown in Figure 1, a first control function that performs long-term control and a second control function that performs short-term control are defined. In O-RAN, the first control function is called the Non-Real-Time RAN Intelligent Controller (Non-RT RIC) and the second control function is called the Near-Real-Time RAN Intelligent Controller (Near-RT RIC).

[0003] The first control function and the second control function are connected via an A1 interface. The second control function controls the central unit (CU) and distributed units (DUs), which are components of the RAN, via an E2 interface. Note that controlling the CU and DU includes notifying and setting various parameter values ​​used by the CU and DU in their processing to the CU and DU, and instructing the CU and DU to execute some operation. Furthermore, in the following description, the term "RAN" is used as a general term for the components. Therefore, for example, "controlling the RAN" means controlling the CU and DU, which are components of the RAN. The first control function, the second control function, and the RAN are further connected via an O1 interface. The O1 interface can be used to transmit traffic data, performance data, fault data, and the like that are measured / detected and accumulated by the RAN.

[0004] Non-Patent Document 1 discloses a configuration that utilizes machine learning for the control of the RAN. Specifically, Non-Patent Document 1 discloses performing machine learning based on various learning data collected from the RAN etc. to generate a learning model, performing inference using the learning model, and controlling the RAN according to the inference result. In one of the multiple configurations disclosed in Non-Patent Document 1, a first control function generates a learning model. Then, a second control function controls the RAN using the learning model generated by the first control function.

[0005] Non-Patent Document 2 discloses distributed reinforcement learning that can obtain high learning efficiency. As shown in FIG. 2, in distributed reinforcement learning, a learning unit performs machine learning based on learning data stored in a storage unit to generate a learning model. The learning unit distributes the learning model to a plurality of execution units that are distributed. Each execution unit is associated with a "control target." The execution unit uses the learning model based on the "current state" of the control target to determine the control content for the control target, and executes control of the control target based on the determined "control content." The execution unit determines the "next state" of the control target as a result of this control, and evaluates the "achievement level" of the control goal. In distributed reinforcement learning, the "control target," "current state," "next state," "control content," and "achievement level" described above are called "Environment," "State," "Next State," "Action," and "Reward," respectively.

[0006] Then, the execution unit calculates an evaluation value called a temporal difference error based on the achievement level, the current state, and the next state. In the following description, the temporal difference error is referred to as a TD error, and its value is referred to as a TD error value. The TD error value is defined as the sum of the "achievement level" obtained as a result of the control and the value determined based on the "next state", minus the value determined based on the "current state" before the control. A large TD error value means that the achievement level of the control goal in the "next state" is low, that is, the accuracy of the output of the learning model based on the "current state" is low, or the output error of the learning model based on the "current state" is large.

[0007] The execution unit stores the "current state", "next state", "control content", and "achievement level" as learning data in the storage unit for the next learning by the learning unit. At this time, the execution unit assigns the TD error value calculated for the learning data to the learning data. As described above, a large TD error value of the learning data means that the accuracy of the output of the learning model based on the input data (current state) to the learning model included in the learning data is low. Therefore, by using learning data with a large TD error value for machine learning, the learning model is significantly updated and the learning effect is improved. For this reason, the learning unit preferentially uses learning data with a larger TD error value among the learning data stored in the storage unit for machine learning. This makes it possible to improve the learning efficiency by the learning unit. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] O-RAN Alliance, "AI / ML workflow description and requirements", O-RAN.WG2.AIML-v01.03, July 2021 [Non-Patent Document 2] D.Horgan,et.al.,"Distributed prioritized experience replay",arXiv:1803.00933.,April 2018 Summary of the Invention [Problem to be solved by the invention]

[0009] FIG. 3 shows a configuration in which distributed reinforcement learning is applied to O-RAN. An execution unit that controls the RAN based on a learning model is arranged in the second control function, and a learning unit that generates a learning model and a storage unit that stores learning data collected by the execution unit are arranged in the first control function. However, in the configuration of FIG. 3, the amount of learning data from each execution unit of the second control function becomes too large. For example, assume that there are 2.5 million user equipments (UEs) in the area covered by the first control function, the size of the learning data acquired from the RAN is 356 bytes per UE, and each execution unit of the second control function in the area covered by the first control function transmits learning data to the first control function at a cycle of 20 times per second. In this case, the first control function receives 142.4 gigabits of data per second, which is not realistic. For example, by lengthening the cycle at which each execution unit of the second control function transmits learning data, the amount of learning data received by the first control function can be reduced, but the learning efficiency may be reduced.

[0010] The present disclosure provides a technology that prevents a decline in learning efficiency and prevents an excessive increase in the amount of learning data transmitted between a device that collects learning data and a device that performs machine learning. [Means for solving the problem]

[0011] According to one embodiment of the present invention, a control device of a radio access network (RAN) includes a collection means for controlling the RAN based on a learning model to collect first learning data from the RAN, a determination means for determining usefulness of the first learning data in machine learning, and a processing means for selecting second learning data from the first learning data based on the usefulness of the first learning data to be transmitted to another control device that generates the learning model. Effect of the Invention

[0012] According to the present disclosure, it is possible to prevent a decrease in learning efficiency and to prevent an excessive increase in the amount of learning data transmitted between a device that collects learning data and a device that performs machine learning. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a control configuration diagram of a RAN according to the background art. [Diagram 2] 1 is a diagram illustrating the configuration of distributed reinforcement learning according to the background art. [Diagram 3] A diagram of the configuration when distributed reinforcement learning is applied to RAN control. [Figure 4] FIG. 2 is a control configuration diagram of a RAN according to an embodiment. [Diagram 5] FIG. 4 is a configuration diagram of a second control device according to the embodiment. [Figure 6] FIG. 2 is a configuration diagram of a first control device according to the embodiment. [Figure 7] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0015] FIG. 4 shows the control configuration of the RAN according to this embodiment. The configuration of FIG. 4 is basically the same as that of FIG. 1. The first control device performs long-period control of the RAN, and is, for example, a device that implements the function of a Non-RT RIC of an O-RAN. The second control device is a device that controls the RAN in a shorter period than the first control device, and is, for example, a device that implements the function of a Near-RT RIC of an O-RAN. The first control device is connected to one or more second control devices. In the example of FIG. 4, the first control device is connected to three second control devices. The first control device performs machine learning to generate a learning model and distributes the learning model to each second control device. Each second control device is associated with a geographical area, and uses the learning model acquired from the first control device to control components of the RAN located in the associated geographical area. Note that the second control device controls a part of the geographical area associated with the second control device in the RAN of the mobile communication network, but the part of the geographical area associated with the second control device in the RAN of the mobile communication network is also referred to as RAN.

[0016] FIG. 5 is a configuration diagram of the second control device. The execution unit 21 acquires a learning model from the first control device, and performs inference based on the learning model to control the RAN. The control objective may be, for example, maximizing the overall throughput or maximizing the average quality of the wireless section. In this case, for example, the second control device may control beamforming so as to achieve the control objective. In addition, the control objective may be, for example, minimizing computer resources used to realize CUs and DUs by virtualization technology. In this case, for example, the second control device may control the placement of CUs and DUs in the RAN.

[0017] For example, the execution unit 21 obtains the output of the learning model by inputting the "current state" of the RAN collected from the RAN to the learning model, and determines the "control content" of the RAN based on this output to control the RAN. The execution unit 21 collects the "next state" of the control target as a result of this control from the RAN, and evaluates the "achievement level" of the control goal. Then, the execution unit 21 generates learning data based on the "current state", the "next state", the "control content", and the "achievement level", determines an evaluation value of the learning data, and assigns it to the learning data. The execution unit 21 stores the learning data to which the evaluation value is assigned in the storage unit 22. In this way, the execution unit 21 has a function of controlling the RAN based on the learning model, a function of collecting learning data, and a function of determining an evaluation value of the learning data. The evaluation value of the learning model indicates the usefulness of the learning model in machine learning for the learning model. Note that high usefulness means that the learning effect of machine learning using the learning data is high. As an example, the above-mentioned TD error value can be used as the evaluation value. In other words, the larger the learning data based on input data to the learning model that increases the error of the learning model output, the larger the value that can be used as the evaluation value.In other words, the larger the learning data based on data that decreases the achievement level of the control goal for RAN, the larger the value that can be used as the evaluation value.

[0018] When the TD error value is used as the evaluation value, the greater the evaluation value assigned to the learning data, the greater the usefulness of the learning data. However, the evaluation value is not limited to the TD error value, and any value that changes depending on, for example, the inference error of the learning model, that is, the error of the output of the learning model, can be used. In this case, the more the learning data based on input data that increases the error of the inference (output of the learning model), the greater the usefulness in machine learning. In other words, the more the learning data based on input data that reduces the accuracy of the inference (output of the learning model), the greater the usefulness in machine learning. Note that the evaluation value of the learning data may indicate that the smaller the value, the greater the usefulness in machine learning.

[0019] The processing unit 23 reads out the learning data to which an evaluation value is assigned that is stored in the storage unit 22, and transmits the learning data to the first control device. In the following description, the "learning data to which an evaluation value is assigned" is simply referred to as "learning data". For example, when the first control device implements a Non-RT RIC and the second control device implements a Near-RT RIC, the processing unit 23 can transmit the learning data to the first control device via the A1 interface or the O1 interface. In this embodiment, the processing unit 23 (second control device) is configured to operate in at least one of the three operation modes described below.

[0020] First operation mode: In the first operation mode, an upper limit value of the transmission speed is set in the processing unit 23, and the processing unit 23 reads out the learning data from the storage unit 22 and transmits it to the first control device so that the transmission speed to the first control device is equal to or lower than the set upper limit value. In other words, the processing unit 23 may transmit the learning data at a transmission speed equal to or lower than the upper limit value while the amount of learning data stored in the storage unit 22 is small, but controls the reading from the storage unit 22 so that the transmission speed does not exceed the upper limit value no matter how much the amount of learning data stored in the storage unit 22 increases. When transmitting the learning data to the first control device, the processing unit 23 reads out the learning data from the storage unit 22 in descending order of usefulness indicated by the evaluation value. For example, when the TD error value is used as the evaluation value, the processing unit 23 reads out the learning data from the storage unit 22 in descending order of the evaluation value.

[0021] By setting an upper limit on the transmission speed for transmission to the first control device, it is possible to prevent the amount of learning data transmitted to the first control device performing machine learning from increasing too much. In addition, since learning data that has been assigned an evaluation value indicating high usefulness in machine learning is preferentially transmitted to the first control device, it is possible to prevent a decrease in the learning efficiency in the first control device.

[0022] Second operation mode: In the second operation mode, a threshold value for the evaluation value is set in the processing unit 23, and the processing unit 23 reads out from the storage unit 22 only learning data to which an evaluation value indicating that the usefulness is higher than the set threshold value is assigned, and transmits the learning data to the first control device. For example, when the TD error value is used as the evaluation value, the processing unit 23 reads out from the storage unit 22 only learning data to which the evaluation value is higher than the threshold value.

[0023] By transmitting only the learning data that satisfies a predetermined condition to the first control device, rather than transmitting all the learning data collected by the execution unit 21 to the first control device, it is possible to prevent the amount of learning data transmitted to the first control device that performs machine learning from increasing too much. Also, since only the learning data that has been assigned an evaluation value indicating that its usefulness in machine learning is higher than a threshold is transmitted to the first control device, it is possible to prevent a decrease in the learning efficiency in the first control device.

[0024] Third operation mode: In the third operation mode, as in the first operation mode, an upper limit value of the transmission speed is set in the processing unit 23, and the processing unit 23 reads out learning data from the storage unit 22 and transmits it to the first control device so that the transmission speed to the first control device is equal to or lower than the set upper limit value. However, in the third operation mode, the processing unit 23 is further set with judgment information for judging the value of the readout probability (probability value) of the learning data based on the evaluation value of the learning data. Note that the judgment information is set so that the probability value of the learning data is higher as the usefulness of the learning data is higher. The processing unit 23 judges the probability value of the readout probability of the learning data based on the judgment information. Then, the processing unit 23 reads out the learning data stored in the storage unit 22 according to the probability value of each piece of learning data. Note that the higher the probability value of the learning data, the higher the probability of being read out from the storage unit 22.

[0025] Therefore, in the third operation mode, unlike the first operation mode, it is probabilistic that less useful learning data stored in the storage unit 22 is read from the storage unit 23 and transmitted to the first control device before more useful learning data stored in the storage unit 22. For example, if machine learning is performed using only useful learning data, the performance of the learning model may deteriorate due to so-called overlearning. In the third operation mode, less useful learning data is also probabilistically transmitted to the first control device, so that deterioration of the performance of the learning model due to overlearning can be suppressed.

[0026] The determination information may be information indicating a formula or a coefficient of the formula for converting the evaluation value into a probability value, or may be information indicating a look-up table (LUT) showing the relationship between the evaluation value and the probability value.

[0027] In any operation mode, it may happen that less useful learning data is not read out from the storage unit 22. To prevent "old" learning data that has been collected for a long time from being used for machine learning, the storage unit 22 may be configured to discard learning data that has been stored in the storage unit 22 for a predetermined period of time.

[0028] FIG. 6 is a configuration diagram of the first control device. The storage unit 12 stores the learning data received from the second control device. The learning unit 11 performs machine learning based on the learning data stored in the storage unit 12 to generate a learning model. At this time, the learning unit 11 can preferentially use the learning data to which a more useful evaluation value is assigned for machine learning. The learning unit 11 transmits the generated learning model to each second control device. In this way, the learning unit 11 has a generation function of performing machine learning to generate a learning model, and a transmission function of transmitting the generated learning model. The generation of learning data by the learning unit 11 and the subsequent transmission to each second control device can be performed at a predetermined cycle or when a predetermined criterion is satisfied. For example, the predetermined criterion can be satisfied when the learning data stored in the storage unit 12 reaches a predetermined amount.

[0029] The control unit 13 controls the operation mode of each second control device (processing unit 23). Fig. 7 is a sequence diagram of a process for controlling the operation mode of the processing unit 23. In S1, the control unit 13 transmits a capability request message to the processing unit 23. In response to the capability request message, the processing unit 23 transmits a capability response message to the control unit 13 in S2. The capability response message has information indicating the operation modes that can be supported (used) by the processing unit 23. The control unit 13 selects the operation mode to actually be used from the operation modes that can be supported by the processing unit 23.

[0030] In S3, the control unit 13 transmits a collection request message including information indicating an operation mode to be actually used to the processing unit 23. The collection request message includes operation information according to the operation mode to be actually used. For example, when the first operation mode is used, the operation information includes information indicating an upper limit of the transmission speed when transmitting the learning data. When the second operation mode is used, the operation information includes information indicating a threshold value. Furthermore, when the third operation mode is used, the operation information includes information indicating an upper limit of the transmission speed when transmitting the learning data and determination information. In S4, the processing unit 23 transmits a collection response message, which is a response to the collection request message, to the control unit 13. Thereafter, the processing unit 23 performs a process of transmitting the learning data to the first control device until the processing unit 23 is notified of the stop of collection from the first control device.

[0031] Furthermore, the control unit 13 can change the operation mode of the processing unit 23 by the collection request message of S3. Furthermore, the control unit 13 can change the operation information used in the operation mode used by the processing unit 23 by the collection request message of S3. For example, when operating in the first operation mode or the third operation mode, the control unit 13 can change the upper limit of the transmission speed depending on the time period and the traffic situation. This can reduce the amount of learning data transmitted to the first control device during a time period with high traffic, and can increase the amount of learning data transmitted to the first control device during a time period with low traffic. Furthermore, when operating in the third operation mode, the control unit 13 can change the determination information depending on whether or not deterioration of the learning performance due to overlearning has occurred, and control the probability value obtained from the evaluation value. This can dynamically suppress deterioration of the learning performance due to overlearning.

[0032] Furthermore, when operating in the second operation mode, the control unit 13 can dynamically update the threshold value according to the amount of learning data transmitted from each second control device. The evaluation value of each learning data increases on average while the inference performance of the learning model is low, and decreases on average as the inference performance of the learning model increases. Therefore, if the threshold value is fixed, the amount of learning data transmitted from the second control device to the first control device may become too small as the inference performance of the learning model increases. Therefore, by dynamically controlling the threshold value, it is possible to prevent the amount of learning data transmitted from each second control device from becoming too small.

[0033] In this embodiment, the operation mode is changed or the operation information used in the operation mode is changed by the collection request message of S3, but a configuration may be adopted in which the operation mode is changed or the operation information used in the operation mode is changed using a message other than the collection request message, such as a "change request message." Furthermore, instead of the control unit 13 setting the operation information in each second control device, a configuration may be adopted in which another device other than the first control device and the second control device, or an operator of a mobile communication network, sets the operation information in the second control device. In this case, the control unit 13 does not need to notify the processing unit 23 of the operation information in S3. In the case of the third operation mode, it is also possible to configure the first control device to set one of the upper limit value and the determination information, and the operator of a mobile communication network, etc. to set the other. Also, for example, when all the processing units 23 are configured with only one operation mode, the processes of S1 and S2 may be omitted. In other words, the control unit 13 does not need to determine the operation modes available in the processing unit 23 and specify one operation mode to be used to the processing unit 23, and the processing unit 23 operates according to one configured operation mode.

[0034] With the above configuration, the first control device can specify the operation mode of the second control device and control the amount of learning data transmitted from each second control device. This makes it possible to prevent a decrease in learning efficiency and to prevent the amount of learning data transmitted from each second control device to the first control device from increasing too much.

[0035] The first control device and the second control device may be realized by one device, for example, one computer. Furthermore, the first control device and the second control device may be realized by a plurality of devices, for example, a plurality of computers, that can communicate with each other. As an example, the learning unit 11, the storage unit 12, and the control unit 13 of the first control device may be implemented in one device, and the three devices that implement the learning unit 11, the storage unit 12, and the control unit 13 may communicate with each other via a network to configure one first control device as a whole. Also, for example, the learning unit 11 and the control unit 13 may be implemented in one device, and the storage unit 12 may be implemented in one device, and these two devices may communicate with each other via a network to configure one first control device as a whole. The same applies to the second control device.

[0036] The present disclosure further provides a program executable by one or more processors, the program including instructions that, when executed by one or more processors of a device, cause the device to function as a first controller and / or a second controller. The present disclosure further provides a non-transitory computer-readable storage medium having the program stored thereon.

[0037] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0038] The above configuration prevents a decline in learning efficiency and prevents the amount of learning data transmitted between the device that collects the learning data and the device that performs machine learning from increasing too much. This makes it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0039] 21: Execution unit, 23: Processing unit

Claims

1. A radio access network (RAN) controller, comprising: A collection means for controlling the RAN based on a learning model and collecting first learning data from the RAN; A determination means for determining usefulness of the first learning data in machine learning for the learning model; a processing means for performing a process of selecting, from the first learning data, second learning data to be transmitted to another control device that performs the machine learning, based on the usefulness of the first learning data; The control device is provided with:

2. An upper limit of the transmission speed is set in the processing means, 2. The control device according to claim 1, wherein the processing means selects the second learning data from the first learning data in descending order of usefulness of the first learning data so that a transmission rate of the second learning data to be transmitted to the other control device is equal to or lower than the upper limit value.

3. The control device according to claim 2 , wherein the upper limit value is set by the other control device.

4. The processing means is set with an upper limit of the transmission speed and determination information for determining a probability value of the selection probability based on the usefulness, the processing means determines the probability value of the first learning data based on the usefulness of the first learning data, and selects the second learning data from the first learning data with a selection probability based on the probability value of the first learning data so that a transmission rate of the second learning data to be transmitted to the other control device is equal to or lower than the upper limit value; The control device according to claim 1 , wherein the probability value based on the determination information is higher as the usefulness is higher.

5. The control device according to claim 4 , wherein at least one of the upper limit value and the determination information is set by the other control device.

6. A threshold value of the usefulness is set in the processing means, The control device according to claim 1 , wherein the processing means selects the first learning data whose usefulness is higher than the threshold as the second learning data.

7. The controller of claim 6 , wherein the threshold is set by the other controller.

8. The control device according to claim 1 , wherein the usefulness of the first learning data is determined based on an error of an output of the learning model based on input data to the learning model, the input data being included in the first learning data.

9. The control device according to claim 8 , wherein the usefulness of the first learning data increases as the error increases.

10. The control device according to claim 1 , wherein the usefulness of the first learning data is higher as a time difference error of the first learning data is larger.

11. The control device of claim 1 , wherein the other control device is a Non-RT RIC.

12. The control device of claim 1 , wherein the control device is a Near-RT RIC.

13. A program, when executed by one or more processors of an apparatus having one or more processors, that causes the apparatus to function as the control device according to any one of claims 1 to 12.

Images