Customer request control system and program
The customer request control system uses reinforcement learning and traffic prediction models to dynamically allocate communication resources by training with pseudo-data, addressing the challenge of real-time resource management in communication networks.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- KDDI CORP
- Filing Date
- 2023-09-22
- Publication Date
- 2026-07-07
AI Technical Summary
Communication carriers face challenges in dynamically determining whether to accept customer requests for communication services due to limited resources, necessitating real-time judgment systems and the integration of reinforcement learning without clear configuration in operational systems.
A customer request control system utilizing a reinforcement learning model trained with pseudo-training data and traffic prediction models to determine acceptability of customer requests, comprising a customer request management DB, pseudo-request generation node, reinforcement learning training node, and reward value determination node.
Enables real-time determination of customer requests based on past network usage information, improving the accuracy and efficiency of resource allocation in communication networks.
Smart Images
Figure 0007886301000001 
Figure 0007886301000002 
Figure 0007886301000003
Abstract
Description
Technical Field
[0001] The present invention relates to a customer request control system and program and so on.
Background Art
[0002] In recent years, communication carriers have been providing high-quality communication services that offer a customer communication resources separated from other customers, such as, for example, a VPN (Virtual Private Network or virtual dedicated line) or a dedicated line.
[0003] In 5G (the fifth-generation mobile communication system) and the like, it has become possible to apply to customers a mobile network constructed of logically separated communication resources in the form of Network Slicing that divides a physical network into a plurality of slices, and expectations for such services are increasing (Non-Patent Document 1).
Prior Art Documents
Non-Patent Documents
[0004]
Non-Patent Document 1
Non-Patent Document 2
Summary of the Invention
Problems to be Solved by the Invention
[0005] However, since communication resources such as routers, servers, and lines possessed by communication carriers are limited, it is difficult to accept requests from all customers at all times. Therefore, it is necessary to dynamically determine whether to provide a communication service usage request received from a customer.
[0006] Until now, decisions regarding the feasibility of providing such services were made by sales and technical personnel involved in the business. However, there is a need to introduce a system that can make real-time judgments and determine whether or not to provide the service.
[0007] Furthermore, in recent years, attempts have been made to apply AI (Artificial Intelligence) technology to control customer requests for communication services, with particular efforts being made to develop methods using reinforcement learning (Non-Patent Document 2).
[0008] Thus, while it has become clear that it is theoretically possible to control customer requests for communication services using reinforcement learning, the specific configuration for introducing reinforcement learning into operational systems has not yet been clearly defined.
[0009] In particular, in reinforcement learning, determining the scenarios for training and the rewards for agent decisions are crucial elements, and it is essential to implement them in actual operational systems.
[0010] Furthermore, the training scenarios (simulated data sets) must be created based on information from past customer requests. In addition, when determining reward values in reinforcement learning, it is necessary to predict customer network usage and ensure that it does not exceed the actual network bandwidth.
[0011] This invention has been made in view of these circumstances, and aims to provide a customer request control system that enables real-time determination of whether a customer request to use a communication network is accepted by creating pseudo-training data from past information and constructing a predictive model for network traffic volume. [Means for solving the problem]
[0012] (1) In order to achieve the above objective, the present invention employs the following means. That is, the customer request control system of the present invention determines whether a customer request to use a communication network is acceptable. , judgment A customer request control system that performs reinforcement learning based on predetermined results, comprising: a customer request management DB that stores past network usage information of customers as customer request information; a pseudo-request generation node that creates pseudo-customer request information based on the network usage information; a reinforcement learning training node that uses a reinforcement learning model to determine whether the customer requests shown in the pseudo-customer request information are acceptable and outputs information indicating the determination result; and a reward value determination node that uses a traffic prediction model based on the amount of traffic when customers have used the network in the past to predict the amount of traffic for each customer and determines a reward value to be assigned to the determination result obtained from the reinforcement learning training node based on the result of the prediction, wherein the reinforcement learning training node performs reinforcement learning training on the reinforcement learning model based on the reward value determined by the reward value determination node.
[0013] (2) The customer request control system of the present invention is further characterized by comprising a customer request determination node that determines whether a customer request to use a communication network is acceptable or not, based on the trained reinforcement learning model and using AI (Artificial Intelligence).
[0014] (3) Furthermore, in the customer request control system of the present invention, the pseudo-request generation node is characterized by calculating statistical information regarding network usage from the network usage information and using the calculated statistical information to calculate the pseudo-customer request information.
[0015] (4) Further, the customer request control system of the present invention further includes a traffic management DB that stores the traffic volume when each customer used the network in the past. The reward value determination node trains a traffic prediction model based on the stored traffic volume and determines the reward value using the trained prediction model.
[0016] (5) Further, the program of the present invention When a customer request to use the communication network is entered, is trained by the customer request control system according to any one of (1) to (4) Using the reinforcement learning model and is characterized by an AI (Artificial Intelligence) determination as to whether a customer request is acceptable. Make the computer function in a way that allows it to perform this task.
Effects of the Invention
[0017] According to the present invention, it is possible to provide a customer request control system that can determine in real time whether a customer request to use a communication network is acceptable.
Brief Description of the Drawings
[0018] [Figure 1] It is a diagram showing a schematic configuration of a customer request control system. [Figure 2] It is a diagram showing an example of the configuration of customer request information. [Figure 3] It is a diagram showing an example of customer request information with a determination result attached. [Figure 4] It is a diagram showing data of the traffic volume used by a customer in the past at a certain point in time in a graph. [Figure 5] It is a sequence diagram showing a procedure for implementing model training of reinforcement learning. [Figure 6] It is a diagram showing a cumulative distribution function (statistical information) indicating the distribution of customer network usage requests. [Figure 7] It is a diagram showing the regional distribution (statistical information) of the transmission node and the reception node used by a customer on the network. [Figure 8]This is a flowchart illustrating the procedure for determining reward values. [Modes for carrying out the invention]
[0019] The inventors focused on the fact that it is not possible to efficiently allocate customer network usage in real time. By creating simulated training data from past information and constructing a predictive model for network traffic volume, they discovered that it is possible to determine in real time whether to accept customer requests to use the communication network, leading to the present invention.
[0020] In other words, the present invention is a customer request control system that determines whether a customer request to use a communication network is acceptable and performs reinforcement learning based on the determination result, comprising: a customer request management DB that stores the customer's past network usage information as customer request information; a pseudo-request generation node that creates pseudo-customer request information based on the network usage information; a reinforcement learning training node that uses a reinforcement learning model to determine whether the customer request shown in the pseudo-customer request information is acceptable and outputs information indicating the determination result; and a reward value determination node that uses a traffic prediction model based on the amount of traffic the customer has used the network in the past to predict the amount of traffic for each customer and determines a reward value to be assigned to the determination result obtained from the reinforcement learning training node based on the result of the prediction, wherein the reinforcement learning training node performs reinforcement learning training on the reinforcement learning model based on the reward value determined by the reward value determination node.
[0021] The embodiments of the present invention will be described in detail below with reference to the drawings. To facilitate understanding of the description, the same reference numerals are used for identical components in the drawings of each embodiment, and redundant explanations are omitted.
[0022] [1] Outline of the Customer Request Control System Figure 1 shows a schematic configuration of a customer request control system. The customer request control system 100 comprises at least the following components: network 1, customer request transmission node 3, customer request determination node 5, customer request management DB 7, pseudo-request generation node 9, reinforcement learning training node 11, traffic management DB 13, and reward value determination node 15.
[0023] Network 1 is a network built and operated by a telecommunications carrier for communication services, and is the network that customers request to use. Examples of network types include, but are not limited to, mobile networks consisting of base stations and mobile core equipment such as 5G, and IP networks consisting of routers such as FTTH (Fiber To The Home).
[0024] The customer request transmission node 3 is a node that transmits information regarding a customer's network usage request (hereinafter also referred to as request information) to the customer request determination node 5. It also receives the determination result from the customer request determination node 5 regarding whether or not to accept the customer's network usage request, and transmits the received determination result to the customer.
[0025] The customer request determination node 5 is a node that determines whether to accept or reject the request information received from the customer request transmission node 3. In this embodiment, it automatically determines whether to accept or reject the request using a learning model trained by a reinforcement learning agent, and sends the determination result to the customer request transmission node 3.
[0026] The Customer Request Management DB7 is a database for storing usage information (also known as past network usage information) sent by customers to date as customer request information. Figure 2 shows an example of the structure of customer request information. As shown in Figure 2, customer request information includes information such as the time the request information was received from the customer, the time the request was canceled, the start time of network usage, the end time of network usage, the bandwidth used, the sending node, and the receiving node.
[0027] The pseudo-request generation node 9 is a node that creates a list of pseudo-customer request information (also simply called pseudo-customer request information) necessary for training reinforcement learning. Based on the customer request information stored in the customer request management DB7, this node creates a list of pseudo-customer request information that is statistically close to the actual customer request information. The pseudo-customer request information has a structure similar to that shown in Figure 2, for example.
[0028] The reinforcement learning training node 11 is a node for training reinforcement learning to determine whether a customer request is acceptable or not. It uses reinforcement learning to determine whether a given customer request is acceptable based on the list of simulated customer request information received from the simulated request generation node 9, and sends the result to the reward value determination node 15. Figure 3 shows an example of customer request information with a determination result. As shown in Figure 3, the simulated customer request information received from the simulated request generation node 9 is assigned an action indicating the determination result: "Accept the request" or "Reject the request." Furthermore, the reinforcement learning training node 11 further trains its reinforcement learning based on the reward value determined by the reward value determination node 15. Details of the reward value determination node 15 will be described later.
[0029] Traffic Management DB13 is a database that stores data on the amount of traffic each customer has used on the network in the past. Figure 4 is an example graph showing the amount of traffic a customer has used in the past at a given point in time.
[0030] The reward value determination node 15 is a node that determines the reward value based on the determination result sent from the reinforcement learning training node 11 by the reinforcement learning agent. In determining the reward value, it is necessary to determine whether the network resources have been exceeded. Therefore, a traffic prediction model learned from past actual traffic volumes obtained from the traffic management DB 13 is used to predict the traffic usage of each customer, and based on the prediction result, it is determined whether the available network resources have been exceeded and the reward value is determined.
[0031] In this way, by using pseudo-training data for reinforcement learning (pseudo-customer request information) created from past customer network usage request information, and a resource prediction model created from the amount of network resources used by past customers, the reward value can be determined to appropriately determine whether the network is available and to accept customer requests to use the communication network in real time.
[0032] [2. Procedure for conducting reinforcement learning model training] Next, we will describe the procedure for conducting reinforcement learning model training in the customer request control system according to this embodiment. Figure 5 is a sequence diagram showing the procedure for conducting reinforcement learning model training.
[0033] First, the pseudo-request generation node requests the customer request management DB to send past customer request information (Figure 2) stored in the database in order to create a list of pseudo-customer request information necessary for reinforcement learning training (step S1). If past customer request information has already been received, it is not necessary to obtain it again, and step S2 can be omitted, and the process can proceed to step S3.
[0034] Next, the customer request management DB retrieves a list of customer request information from within the DB based on a request from the pseudo-request generation node to send customer request information, and sends it to the pseudo-request generation node (step S2).
[0035] Next, the pseudo-request generation node uses the customer request information list obtained from the customer request management DB to calculate statistical information for various types of data. Examples of this statistical information include the cumulative distribution function (CDF) related to the time distribution from the time the customer requested network usage (Figure 2: reception time (t_Request)) to the start time of usage (Figure 2: start time of usage (t_Start)), as shown in Figure 6, and the regional distribution of sending and receiving nodes, as shown in Figure 7.
[0036] Then, using statistical information calculated from a list of customer request information received from customers, a pseudo-list of customer request information is created that is closer to the actual statistical information and used for training reinforcement learning (Step S3). In this way, by creating a pseudo-list of customer request information using statistical information calculated from a list of customer request information actually received from customers, it becomes possible to train reinforcement learning with a customer request information list that is closer to the actual one, and as a result, the accuracy of the resulting training model can be improved.
[0037] Next, the pseudo-request generation node sends the pseudo-customer request information list created in step S3 to the reinforcement learning training node (step S4).
[0038] Next, the reinforcement learning training node makes a decision for each request in the list of simulated customer request information received from the simulated request generation node, determining whether to "accept" or "reject" the request, and assigns a decision result (step S5). The decision is assumed to be performed using a reinforcement learning or deep reinforcement learning model, but is not limited to these. Figure 3 shows an example of a customer request information list to which the decision results have been assigned.
[0039] Next, the reinforcement learning training node sends the result of the customer request information list determined in step S5 to the reward value determination node (step S6).
[0040] The reward value determination node determines the reward value for each point in time, which is the usage time (start time to end time) of each customer request information, based on the customer request information list that has been assigned results determined by reinforcement learning, received from the reinforcement learning training node. In order to determine the reward value, it is first necessary to predict the traffic volume of each customer. To build this prediction model, the reward value determination node requests the traffic management DB to send historical traffic data (step S7). At this point, if the acquisition of historical traffic data and training of the AI model for traffic prediction have been completed recently, that list and AI model may be used, in which case steps S8 and S9 may be omitted and the process may proceed to step S10.
[0041] Next, the traffic management DB retrieves the traffic data requested by the reward value determination node from the database and sends the retrieved traffic data to the reward value determination node (step S8). Figure 4 shows the structure of the traffic data.
[0042] Next, in step S8, the reward value determination node trains an AI model to predict the amount of traffic each customer actually uses, using traffic data obtained from the traffic management DB (step S9). While it is assumed that existing AI models such as LSTM (Long Short-Term Memory) will be used for traffic prediction, it is not limited to these.
[0043] Next, the reward value determination node, using the AI traffic prediction model obtained in step S9, determines whether each customer's request can be fulfilled (step S10) for the customer request information list, which has been received from the reinforcement learning training node and assigned acceptance / rejection results through reinforcement learning. In particular, in the case of a network, network resources (e.g., bandwidth) are limited, and if the total predicted traffic value of all accepted customer request information in a given time period is less than or equal to the upper limit of the network resources, a positive reward value is set. Conversely, if a customer request information exceeds the upper limit of the network resources, a refund will be issued to each customer as a penalty, so a negative reward value is set.
[0044] (Method for determining reward values) Here, we will explain in detail how the reward value is determined. Figure 8 is a flowchart showing the procedure for determining the reward value.
[0045] First, from the list of customer request information obtained from the reinforcement learning training node, the oldest customer request information with a reception time for which the reward value has not yet been determined is obtained (step S10-1).
[0046] Next, in step S5, the acceptance or rejection of each customer request is confirmed (step S10-2). Specifically, first, the action taken for the acquired customer request information is confirmed ("Accept the request" or "Reject the request").
[0047] If the response is "Accept," enter "1" into the variable (action) (step S10-3). On the other hand, if the response is "Reject," enter "-1" into the variable (action) (step S10-4). As will be explained in more detail later, the value entered into this variable (action) will be used as a coefficient when determining the reward value.
[0048] Next, the amount of traffic used by this customer is predicted using the traffic prediction model trained in step S9 (step S10-5).
[0049] Next, the predicted traffic data is added to the traffic prediction graph (step S10-6). The traffic prediction graph is a graph that aggregates the predicted traffic volume for customer request information where the handling status is "Accept" in the customer request information list processed so far.
[0050] Step S10-7 compares the sum of the traffic volumes predicted based on the current customer request information with the current network source volume. In Step S10-7, if the sum of the traffic volumes predicted based on the current customer request information is less than or equal to the upper limit of the network resource volume in the traffic prediction graph at all points in time, the reward base (reward_base) is set to a positive value (100 in this embodiment) (Step S10-8).
[0051] On the other hand, in step S10-7, if the sum of the traffic volume predicted by the current customer request exceeds the upper limit of the network resource volume at any point in the traffic forecast graph, the base of the reward value is set to a negative value (-100 in this embodiment) (step S10-9). In this embodiment, as an example of the value to set as the base of the reward value (reward_base), a positive value of "100" and a negative value of "-100" are used, but it is not limited to these. Also, in this embodiment, a static value is used as the base of the reward value as an example, but other implementations are also conceivable, such as setting the reward value to the actual service usage fee received from the customer, or setting the reward value to the amount refunded to the customer if the network resource volume is exceeded. In such cases, there is an advantage in having a model that is more likely to accept customer requests that are likely to improve revenue.
[0052] Finally, the reward value (reward) is determined by multiplying the reward base (reward_base) by the variable (action) (step S10-10). For example, if something is judged as "Accept" and it is within the network resources, the reward value will be a positive value, which means that the service usage fees that the telecommunications carrier can earn will increase. Similarly, if something is judged as "Reject" and it exceeds the network resource amount, the reward value will also be a positive value due to the multiplication of negatives, which means that the carrier can decide not to process refunds for customers present during that time period due to the resource overrun, and the refunds that the telecommunications carrier has to pay will decrease.
[0053] In this way, the information with the reward value determined in step S10 (S10-1 to S10-10) is sent to the reinforcement learning training node (step S11).
[0054] The reinforcement learning training node trains the model based on the reward value received from the reward value determination node (step S12). Similar to step S5, it is assumed, but not limited to, that the model is trained using reinforcement learning or deep reinforcement learning methods.
[0055] The reinforcement learning training node repeats steps S1 to S12 until a predetermined prediction accuracy is met or until a predetermined number of training iterations are reached.
[0056] Finally, the reinforcement learning training node sends the trained model to the customer request determination node (step S13). The customer request determination node uses this model to determine which customer requests it will actually receive.
[0057] As described above, according to the above embodiment, by determining the reward value using pseudo-training data for reinforcement learning (pseudo-customer request information) created from past customer network usage request information and a resource prediction model created from the amount of network resources used by past customers, it is possible to appropriately determine whether the network is available and to determine in real time whether to accept a customer request to use the communication network. [Explanation of symbols]
[0058] 100 Customer Request Control System 1 Network 3. Customer request sending node 5. Customer Request Determination Node 7 Customer requirement management DB 9. Pseudo-request generation node 11 Reinforcement Learning Training Nodes 13 Traffic Management DB 15 Reward Value Determination Node
Claims
1. A customer request control system that determines whether a customer request to use a communication network is acceptable and performs reinforcement learning based on the determination result, A customer request management database that stores customers' past network usage information as customer request information, A pseudo-request generation node that creates pseudo-customer request information based on the aforementioned network usage information, A reinforcement learning training node that uses a reinforcement learning model to determine whether the customer request shown in the simulated customer request information is acceptable and outputs information indicating the determination result, The system includes a traffic prediction model that uses the traffic volume of customers when they have used the network in the past to predict the traffic volume of each customer, and a reward value determination node that determines the reward value to be assigned to the determination result obtained from the reinforcement learning training node based on the prediction result, A customer request control system characterized in that the reinforcement learning training node performs reinforcement learning training on the reinforcement learning model based on the reward value determined by the reward value determination node.
2. The customer request control system according to claim 1, further comprising a customer request determination node that determines, based on the aforementioned trained reinforcement learning model, whether or not a customer request to use a communication network is acceptable using AI (Artificial Intelligence).
3. The customer request control system according to claim 2, characterized in that the pseudo-request generation node calculates statistical information regarding network usage from the network usage information and calculates the pseudo-customer request information using the calculated statistical information.
4. Furthermore, it includes a traffic management database that stores the amount of traffic each customer has used the network in the past. The customer request control system according to claim 3, characterized in that the reward value determination node trains a traffic prediction model based on the stored traffic volume and determines the reward value using the trained prediction model.
5. A program characterized in that, when a customer request to use a communication network is input, the computer is made to function in such a way that it uses an AI (Artificial Intelligence) to make an AI determination of whether or not the customer request is acceptable, using a reinforcement learning model trained by the customer request control system described in any one of Claims 1 to 4.