System for controlling an execution system including multiple subsystems - Patents.com

Abstract

To improve overall control of a system including multiple subsystems.SOLUTION: A system is disclosed for controlling an execution system that includes multiple subsystems. A storage device stores data for constructing a first model for extracting feature quantities of each input graph. A processor acquires measurement data from each of the multiple subsystems and generates graphs representing the measurement data of each of the multiple subsystems based on the measurement data from each of the multiple subsystems. The processor uses the first model to extract feature quantities from respective graphs and controls the execution system based on the feature quantities.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a system for controlling an executive system that includes multiple subsystems. [Background technology]

[0002] Currently, many systems are composed of multiple heterogeneous subsystems, and the systems are becoming more and more complex. At the same time, there is a demand for efficient operation and control of the entire system composed of these subsystems. Examples of such subsystems include power grids, communication networks, railway and road networks, logistics networks, supply chains, financial transactions, and the flow of people in offices and shopping malls.

[0003] It has also been proposed to analyze or automatically control a system using an AI (Artificial Intelligence) model that uses machine learning. Various machine learning methods have been proposed, and for example, Non-Patent Document 1 discloses a classification method using a graph convolutional network. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Thomas N. Kipf, Max Welling, "Semi-Supervised Classification with Graph Convolutional Networks", ICLR 2017. Summary of the Invention [Problem to be solved by the invention]

[0005] The different subsystems that make up a system have different data domains, such as the type of data that the subsystems target, data structures, systems that acquire data, and KPIs (Key Performance Indicators). Therefore, it is difficult to describe the relationships between the different subsystems and optimize the control of the entire system. [Means for solving the problem]

[0006] A representative example of the present disclosure is a system for controlling an execution system including multiple subsystems, the system including one or more processors and one or more storage devices storing programs executed by the one or more processors. The one or more storage devices store data for constructing a first model that extracts features of each input graph. The one or more processors acquire measurement data from each of the multiple subsystems, generate graphs representing the measurement data of each of the multiple subsystems based on the measurement data from each of the multiple subsystems, extract features from each of the graphs using the first model, and control the execution system based on the features. Effect of the Invention

[0007] According to the representative embodiment of the present invention, it is possible to improve the control of the entire system including a plurality of subsystems. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates an example of a functional configuration of an execution system including a plurality of subsystems and a control system that controls the execution target system. [Diagram 2] 2 illustrates an example of a hardware configuration of a system control unit. [Diagram 3] 13 shows a schematic example of measurement of purchasing behavior by the measurement unit of the subsystem. [Figure 4]13 is a schematic diagram showing an example of measurement of the degree of congestion in a shopping mall by a measurement unit of a subsystem. [Diagram 5] 13 shows a flowchart of an example of processing executed by a system control unit. [Figure 6] 13 shows an example of a graph showing user purchasing behavior over a given period of time, generated from measurement data of a subsystem. [Figure 7] 13 shows an example of a graph showing the degree of congestion for a given period of time, generated based on measurement data of a subsystem. [Figure 8] 13A and 13B are schematic diagrams illustrating an example of a logical configuration of a graph feature extraction unit and an example of a partial logical configuration of a state-action value update unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following, when necessary for convenience, the description will be divided into multiple sections or examples, but unless otherwise specified, they are not unrelated to each other, and one is related to the other as a partial or complete modification, detail, supplementary explanation, etc. Furthermore, in the following, when the number of elements, etc. (including the number, numerical value, amount, range, etc.) is mentioned, it is not limited to the specific number, and may be more or less than the specific number, unless otherwise specified or clearly limited in principle to a specific number, etc.

[0010] A system according to an embodiment of the present specification includes an execution system that executes an action on an environment and a control system that controls the execution system. The execution system includes a plurality of heterogeneous subsystems, each of which has a different KPI (Key Performance Indicator).

[0011] A control system represents each of its multiple subsystems as a graph. A graph is a data type that includes nodes and edges that connect the nodes. Graphs allow for the abstraction of heterogeneous subsystems with different data domains, such as data types, data structures, data acquisition systems, and KPIs, into a common representation.

[0012] The control system uses the graph to optimize the control of the entire system in a system in which subsystems of different data domains are combined. In an execution system in which it is difficult to describe the relationships between multiple subsystems, the control system can balance the KPIs of multiple subsystems and optimize the control of the entire execution system.

[0013] A system according to an embodiment of the present specification will be described below. The system includes an execution system that performs actions on an environment and a control system that controls the execution system. An example of controlling the flow of customers in a shopping mall will be described below. The execution system controls the behavior of customers by performing specific actions in the shopping mall, which is the environment.

[0014] The control disclosed in this specification can be applied to any system as long as the data obtained from the subsystems can be expressed in a graph. The control system disclosed in this specification can be applied to control execution systems including heterogeneous subsystems in environments other than shopping malls, such as power grids, communication networks, railway and road networks, logistics networks, supply chains, and financial transactions.

[0015] As described above, the execution system includes multiple heterogeneous subsystems. Each subsystem has a different KPI and executes a different action. In the example described below, one subsystem measures the purchasing behavior of shoppers in a shopping mall with multiple stores and encourages purchasing behavior by distributing coupons to smartphone apps, etc., in order to increase sales throughout the shopping mall.

[0016] Another example of a subsystem is one that aims to reduce congestion in shopping malls by measuring the congestion level in the aisles and encouraging users to move around in a way that alleviates the congestion by displaying the congestion level on signage.

[0017] The first KPI is the sales of the entire shopping mall, and the other KPI is the congestion level of the aisle. In this way, the two subsystems have different KPIs. A control system according to an embodiment of this specification executes a task of maximizing multiple KPIs, such as increasing sales and reducing congestion, while balancing the multiple KPIs between multiple subsystems. The control system learns optimal control according to this task.

[0018] Fig. 1 shows a schematic diagram of an example of the functional configuration of an execution system including multiple subsystems and a control system that controls the execution target system. The execution system that is the control target includes multiple subsystems, and Fig. 1 shows two subsystems, Subsystem 0 and Subsystem 1, as examples. The execution system may include any number of subsystems.

[0019] Subsystem 0 includes a measurement unit 101 and a control unit 104. Subsystem 1 includes a measurement unit 102 and a control unit 105. An environment 106 is an object on which each of the subsystems acts. In the example described below, the environment 106 corresponds to a shopping mall. A system control unit 103 cooperates with each of the multiple subsystems and controls the entire execution system.

[0020] Measurement units 101 and 102 collect data necessary for improving the KPIs of the corresponding subsystems 0 and 1, respectively, in environment 106. Control units 104 and 105 execute actions in environment 106 that are determined to be necessary for improving the KPIs of the corresponding subsystems, based on the data collected by measurement units 101 and 102, respectively.

[0021] The KPI of subsystem 0 is the sales of the entire shopping mall. The higher the sales, the more favorable the state is evaluated to be. The measurement unit 101 of subsystem 0 includes a POS (Point of Sale) system that uses a point card. The measurement unit 101 collects measurement data such as the date and time of purchase behavior, the store used, the purchase amount, the gender of the user, and the age of the user, by linking the data to a unique ID assigned to the point card. The measurement unit 101 also collects information on the stores in the shopping mall, such as the store type, area, and total sales.

[0022] The control unit 104 of the subsystem 0 includes a smartphone and an application program executed by the smartphone, which is used by each user of the shopping mall. The smartphone can display information transmitted from the system control unit 103, such as discount coupons and store locations.

[0023] The KPI of subsystem 1 is the degree of congestion in the shopping mall. The lower the degree of congestion, the more favorable the state is evaluated to be. The measurement unit 102 of subsystem 1 includes multiple TOF (Time of Flight) sensors installed in the shopping mall.

[0024] The TOF sensor can collect the position of each shopping mall user within the measurement range as measurement data. The user's position obtained by the TOF sensor needs to be understood as the position in the entire shopping mall. The measurement unit 102 holds the position and attitude (direction) of each TOF sensor and a map of the shopping mall (data describing the geometric shapes and attributes of stores and aisles).

[0025] The control unit 105 includes a digital signage and a user's smartphone that executes a smartphone app. The digital signage and smartphone can display information transmitted from the system control unit 103. For example, information that encourages users to move around, such as the congestion situation at each location in a shopping mall, discount coupons with specific attachments, and event information at an event venue, can be distributed.

[0026] The system control unit 103 includes a graph processing unit 107, a measurement data receiving unit 108, a state-action value updating unit 111, an action selecting unit 112, an action-control data converting unit 113, a control data transmitting unit 114, a learning managing unit 115, an error generating unit 116, and a measurement data-reward converting unit 117. The graph processing unit 107 includes a measurement data-graph converting unit 109 and a graph feature extracting unit 110.

[0027] The measurement data receiving unit 108 receives the measurement data obtained by the measurement unit of each subsystem. The measurement data-reward converting unit 117 converts the measurement data obtained by the measurement data receiving unit 108 into a numerical value called reward, which is equivalent to the KPI of each subsystem.

[0028] The measurement data-graph conversion unit 109 converts the measurement data obtained by the measurement data receiving unit 108 into graph data consisting of nodes, edges, and attribute data for the nodes. If the measurement data includes a graph, the conversion process is not necessary.

[0029] The graph feature extraction unit 110 extracts feature quantities from the graph generated by the measurement data-graph conversion unit 109. In one embodiment of this specification, a convolution process is performed on the attribute data of each node in the graph with the attribute data of nodes within a range of a predetermined number of steps, thereby calculating an amount (feature quantity) in which each node reflects the value of an adjacent node. The number of steps is the shortest distance via edges from one node to another node, and the distance to an adjacent node is one step.

[0030] The state action value update unit 111 estimates the value of an action (a combination of control data that can be commanded to a subsystem) that can be taken in a certain state in accordance with the framework of reinforcement learning, and updates (learns) the value of an action taken in a certain state based on the estimation result and the reward obtained as a result of the action. The action selection unit 112 probabilistically selects an action that is expected to obtain a high reward based on the value of an action that can be taken in a certain state in accordance with the framework of reinforcement learning.

[0031] The action / control data conversion unit 113 converts the action selected by the action selection unit 112 into a combination of control data that can be used to instruct the subsystems. The control data transmission unit 114 transmits the control data generated by the action / control data conversion unit 113 to each subsystem.

[0032] The learning management unit 115 manages the overall processing, and in particular evaluates the progress of the learning, and determines whether to inject an error into the measurement data for learning. The error generation unit 116 injects an error into the measurement data used for learning.

[0033] 2 is a schematic diagram showing an example of a hardware configuration of the system control unit 103. The system control unit 103 may have a computer configuration. It includes a processor 151 having computing performance, and a DRAM 152, which is a main storage device providing a volatile temporary storage area for storing programs and data executed by the processor 151. Furthermore, the system control unit 103 includes an auxiliary storage device 154 providing a permanent information storage area using a HDD (Hard Disk Drive), a flash memory, or the like. The DRAM 152, the auxiliary storage device 154, and a combination thereof are each a storage device.

[0034] The system control unit 103 further includes a communication device 153 that performs data communication with other devices including other devices in the system, an input device 155 that accepts operations from a user, and a monitor 156 (an example of an output device) that presents the output results of each process to the user. These components can communicate with each other via a bus. The number of each component of the system control unit 103 is arbitrary, and some of the components, for example, the input device 155 and the monitor 156, may be omitted.

[0035] The functional units of the system control unit 103 described with reference to Fig. 1 can be implemented, for example, by a processor 151 that executes a program including instruction codes. The program for implementing the functional units is stored, for example, in an auxiliary storage device 154. The program executed by the processor 151 and data to be processed are loaded from the auxiliary storage device 154 to the DRAMU 152. The functions within the system may be implemented by circuits for specific functions instead of a processor that operates according to a program.

[0036] The system control unit 103 may be a physical computer system (one or more physical computers) as shown in Fig. 2, or a system built on a group of computing resources (multiple computing resources) such as a cloud platform. The computer system or the group of computing resources includes one or more interface devices (including, for example, a communication device and an input / output device), one or more storage devices (including, for example, a memory (main memory) and an auxiliary storage device), and one or more processors.

[0037] When a function is realized by executing a program by a processor, the defined processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the processor. Processing described using a function as the subject may be considered to be processing performed by a processor or a system including the processor.

[0038] The program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable storage medium (for example, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0039] 1, specifically, the measurement units 101 and 102 and the control units 104 and 105 may each include a computer system that executes necessary processing. Furthermore, at least some of the functions of the measurement units 101 and 102 and the control units 104 and 105 may be implemented in a computer in which the functions of the system control unit 103 are implemented.

[0040] Next, the process flow of the entire system will be described. First, each subsystem measures a specific target in the environment 106. As described above, the measurement unit 101 of subsystem 0 measures the purchasing behavior of shopping mall users. The measurement unit 102 of subsystem 1 measures the degree of congestion in the shopping mall 106.

[0041] Fig. 3 shows a schematic example of measurement of purchasing behavior by the measurement unit 101 of the subsystem 0. Fig. 3 shows a plurality of stores 301 in a shopping mall 200 and an aisle 302 connecting the stores 301. In Fig. 3, the stores 301 are shown as rectangles. In Fig. 3, for ease of illustration, one store and one aisle are indicated by the reference numerals 301 and 302. The stores 301 are lined up along the aisle 302.

[0042] In Fig. 3, arrows indicate movements 303 accompanying a user's purchasing behavior. A user's purchases can be measured by a POS system by referencing the user's use of a point card. In Fig. 3, for ease of illustration, one arrow is indicated by the reference symbol 303. Each arrow 303 indicates two stores 301 where the user made successive purchases. Each arrow 303 indicates that the user made a purchase at the store 301 at the start point of the arrow 303, and then made the next purchase at the store 301 at the end point of the arrow 303.

[0043] Suppose a user uses a point card at a certain store, then moves to the next store 205 and uses the point card again. Arrow 303 indicates the first store and the next store as the start and end points, respectively, and thus can show the user's movement along with the order in which the two stores are used. Note that arrow 303 merely conveniently represents the user's purchase at a store and the associated movement, and does not represent the actual physical movement route between stores.

[0044] Fig. 4 shows a schematic example of the measurement of the degree of congestion in a shopping mall 200 by the measurement unit 102 of the subsystem 1. In Fig. 4, a sector 401 indicates the measurement range of the TOF sensor. A vertex 402 of the sector 401 indicates the viewpoint position of the TOF sensor. For ease of illustration, Fig. 4 shows the measurement range of one TOF sensor as an example with reference numeral 401.

[0045] A sector 401 indicates the measurement range of the TOF sensor. Each TOF sensor is installed so that its measurement range includes a part of the passage 302. Each TOF sensor can continuously measure users within the measurement range 401, and specifically, can measure the number of users at each measurement time, the movement route of each user, and the speed of each user at each measurement time. The TOF sensor may be installed so that its measurement range includes the store.

[0046] As mentioned above, for simplicity, only subsystem 0 and subsystem 1 are illustrated, but many more subsystems may be included. The other subsystems each perform measurement processing in the same manner as subsystem 0 and subsystem 1. In addition, the measurement data obtained in each subsystem is transmitted to the system control unit 103.

[0047] Next, the processing in the system control unit 103 based on the data measured in each subsystem will be described. Figure 5 shows a flowchart of an example of processing executed by the system control unit 103. The system control unit 103 executes feature extraction by graph convolution, decision of actions to be executed for each subsystem, and learning of action values.

[0048] The subsystem executes an action corresponding to the action of the system control unit 103 with respect to the environment 106. To this end, the system control unit 103 determines the action of the subsystem with respect to the environment 106 based on the measurement data.

[0049] Referring to FIG. 5, the learning management unit 115 executes an initialization process (501). The initialization process, for example, sets initial values ​​of parameters of a machine learning model for determining an action. Next, the measurement data receiving unit 108 receives measurement data transmitted from the measurement unit of each subsystem (502). The measurement data-reward conversion unit 117 calculates a reward value for the entire execution system from the measurement data received from the measurement unit of each subsystem (503). The reward is calculated based on the KPI (reward) of each subsystem.

[0050] To explain this more specifically, let us assume that the reward R0 of the subsystem 0 is expressed as the total sales of all the stores in the shopping mall for a certain period of time. The measurement data-reward conversion unit 117 calculates the reward R0 based on the measurement data indicated by the POS system of the subsystem 0.

[0051] Similarly, the reward R1 of subsystem 1 is expressed as the average distance between users within the measurement range of the TOF sensors in the shopping mall. For example, the average distance between users in a certain period of time is calculated in the measurement range of each TOF sensor. The users in the measurement range and the distance between users change over time.

[0052] The calculated average distance is, for example, the average value of the average distances at each time within a certain period. The measurement data-reward conversion unit 117 calculates the average value of the average distances of all the TOF sensors and determines it as the reward. Note that a weight may be assigned to each measurement value of the TOF sensor. In this example, the period for calculating the reward for each of the subsystems 0 and 1 is the same.

[0053] This R1 is calculated based on the measurement data. Similarly, for other subsystems, rewards corresponding to the KPIs of the subsystems are calculated from their respective measurement data. Note that multiple rewards may be defined for one subsystem.

[0054] The total reward R of all subsystems is calculated by the following formula, assuming that there are (N+1) subsystems. R=α0·K0·R0+α1·K1·R1+…+αN·KN·RN

[0055] α(i) is a weighting coefficient for each subsystem, which indicates how much importance is attached to each subsystem. α(i) is a real number between 0 and 1, and α0+α1+...+αN=1. K(i) is a coefficient for normalizing the reward of each subsystem. α(i) and K(i) are preset constants.

[0056] Next, the measurement data-graph conversion unit 109 generates graph data for each subsystem from the measurement data for a certain period received from each subsystem (504). In this example, the period of the measurement data for graph generation is common to all subsystems and is the same as the period for calculating the reward. This enables more appropriate behavior. Note that, depending on the configuration of the execution system, the period for graph generation may differ between subsystems.

[0057] The measurement data measured by the POS system of subsystem 0 as described above may include, for example, purchase date and time, user age, store used, purchase amount, store type, store area, total sales of the store, etc. The measurement data-graph conversion unit 109 uses the values ​​of the measurement data to generate a graph in which stores are nodes and customer movements between stores involving purchases are edges. An attribute vector consisting of a value of 1 or more indicating the attributes of the store, which is generated from the measurement data, is given to the nodes. An edge is defined, for example, for movements by users exceeding a prescribed number.

[0058] FIG. 6 shows an example of a graph 600 that represents a user's purchasing behavior over a given period of time, generated from measurement data of subsystem 0. In FIG. 6, circles 601 and 603 are nodes that represent stores. Lines 604 are edges that represent users' movements between stores. In FIG. 6, some nodes and edges are indicated by symbols as an example. The area 602 shown by dashed lines indicates the node 601 and its adjacent nodes. As described above, attribute data (attribute vector) based on the user's measurement data is assigned to each node.

[0059] The measurement data includes data specific to each store, and is reflected in the attribute data of each node. The attribute data can be expressed as a vector consisting of one or more values ​​obtained from the measurement data. The attribute data of a store node can indicate, for example, the average purchase amount of customers at the store, total sales, average age of customers, store area, store type, etc.

[0060] The measurement data measured by each TOF sensor of subsystem 1 as described above includes the time change of the position of each user within the measurement range within a predetermined period. In addition to the measurement data of each TOF sensor, subsystem 1 can transmit map information of the shopping mall and attribute information of each TOF sensor, for example, information on the position and orientation of the TOF sensor, to system control unit 103. The map information indicates the geometric shapes and attributes of stores and aisles. The map information of the shopping mall and the attribute information of the TOF sensor may be preset in system control unit 103 by a system administrator.

[0061] The graph generated by the measurement data-graph conversion unit 109 may be configured with nodes representing a plurality of representative points set in advance within the passages of a shopping mall and edges representing passages between the representative points. The measurement data-graph conversion unit 109 may define the nodes and edges of the graph from map data of the shopping mall.

[0062] The measurement data-graph conversion unit 109 identifies the measurement range of each TOF sensor from the position and orientation of each TOF sensor and map information, and further calculates the congestion degree of each representative point from the measurement data of each TOF sensor. The congestion degree can be expressed, for example, by the average distance between users as described above.

[0063] The measurement data-graph conversion unit 109 includes the representative points in the attribute data of each node corresponding to the representative points. The attribute data may further include other values ​​indicating the degree of congestion, such as the average number of users, and attributes of the points. The attributes of the points may indicate, for example, that the point is in front of a store with a specific attribute, that it is an intersection, that it is a rest area, that it is an event venue, and the like.

[0064] FIG. 7 shows an example of a graph 700 that represents the degree of congestion in a given period of time, generated based on measurement data from subsystem 1. In FIG. 7, circles 701 and 703 are nodes that represent representative points such as points in front of stores and intersections. Lines 704 are edges that represent passages connecting the representative points. In FIG. 7, some circles and edges are indicated by symbols as an example. The range indicated by dashed line 702 indicates node 701 and its adjacent nodes. As described above, attribute data (attribute vectors) based on the measurement data of users are assigned to each node.

[0065] Returning to FIG. 5, next, the graph feature extraction unit 110 extracts features of the graph data of each subsystem (505). The graph feature extraction unit 110 according to an embodiment of this specification uses a graph convolution network. Graph convolution performs a convolution operation for each node of the graph, specifically, multiplying the attribute data (attribute vector) of adjacent nodes connected to each node by a weight and adding them. Note that attribute data of nodes (nodes of two or more steps) farther than adjacent nodes (nodes of one step) may be included in the convolution operation.

[0066] 6 of the graph data by subsystem 0, the attribute data of the adjacent nodes (nodes within range 602) connected to node 601 are folded into the attribute data of node 601. As a result, the attribute data of the adjacent nodes connected to node 601 is reflected in the attribute data of node 601, and the tendency of the attribute data around node 601, that is, the characteristics, can be found.

[0067] The same is true for the graph data from subsystem 1. Focusing on node 701 in Fig. 7, attribute data of adjacent nodes (nodes within range 702) connected to node 701 is convoluted into the attribute data of node 701. Similar convolution processing is performed on graph data obtained by other subsystems.

[0068] Fig. 8 shows a schematic diagram of an example of the logical configuration of the graph feature extraction unit 110 and an example of the logical configuration of a part of the state-action value update unit 111. The graph feature extraction unit 110 receives graph data based on the measurement data of each subsystem from the measurement data-graph conversion unit 109. In Fig. 8, as an example, the graph data of subsystems 0 and 1 are indicated by reference numerals 801 and 802. The graph data indicates the graph structure and attribute data of each node.

[0069] The graph feature extraction unit 110 inputs the graph data of each subsystem to a corresponding graph convolutional neural network 807. The graph convolutional neural network 807 is an example of a model that extracts the feature amount of the input graph, and data for configuring the model is stored in a storage device.

[0070] In Fig. 8, one graph convolutional neural network is indicated by reference numeral 807 as an example. In the configuration example of Fig. 8, the graph convolutional neural network is composed of a plurality of convolutional layers 805. In Fig. 8, one convolutional layer is indicated by reference numeral 805 as an example.

[0071] Convolution processing by one convolution layer 805 (one convolution processing) convolves attribute data of directly connected adjacent nodes into the attribute data of each node. Multiple convolution layers 805 are connected, and multiple convolution processing is executed for each attribute data. As a result, attribute data of distant nodes that are not directly connected are convolved into the attribute data of each node, and features reflecting the attribute data of nodes distant from each node are extracted.

[0072] Returning to FIG. 5, next, the state action value update unit 111 executes a state action value update process (506). This process corresponds to updating the state action value in the framework of reinforcement learning. The state refers to a feature obtained by convolution processing of the attribute data of the node based on the structure of the graph.

[0073] An action indicates a selection from a group of candidates of control data for each subsystem. One candidate of control data may include, for example, distribution of discount coupons and store locations using a smartphone app, distribution of congestion status for each location in a shopping mall using digital signage and a smartphone app, etc. Different candidates of control data may indicate, for example, different distribution information and / or different distribution destinations.

[0074] The state-action value update involves updating the estimated value or the parameters of the function used for the estimation based on the reward obtained as a result of the action selected in the previous state, so that the estimated value of the value of selecting that action in the previous state matches the reward obtained.

[0075] An example of updating a state action value will be described with reference to Fig. 8. The state action value update unit 111 receives features (vectors) of graph data for each subsystem from the graph feature extraction unit 110. Fig. 8 clearly shows, as examples, a purchasing behavior feature 803 and a congestion degree feature 804. The set of features indicates the current state si.

[0076] The state action value update unit 111 includes a fully connected layer 806, and inputs features from the graph feature extraction unit 110. The fully connected layer 806 outputs Q values ​​of actions a0 to aM in state si, where M is a natural number. FIG. 8 shows, as an example, a Q value (si, a0) 808 of action a0 and a Q value (si, aM) 809 of action aM. The Q value indicates an action value. The state action value update unit 111 updates the parameters of the graph convolution neural network 807 and the neural network of the fully connected layer 806 so as to reduce the difference (error) between the calculated Q value and the actually obtained reward.

[0077] The fully connected layer 806 is an example of a model that determines an index value for determining control data for controlling the execution system based on the feature amount from the graph feature extraction unit 110. Data for configuring the model is stored in a storage device. A model other than the fully connected layer may be used.

[0078] Returning to FIG. 5, next, the behavior selection unit 112 executes a behavior selection process (507). The behavior selection unit 112 selects a highly valuable behavior with a high probability from among the actions that can be taken in the current state. Next, the behavior / control data conversion unit 113 executes a behavior / control data conversion process (508). As described above, the behavior corresponds to the distribution of discount coupons and store locations using a smartphone app, and in order to realize this, the behavior is converted into data compatible with each device.

[0079] Next, the control data transmission unit 114 executes a control data transmission process (509). As a result, the control data is transmitted to the control unit of each subsystem, and each control unit executes a process based on the control data. For example, the above-mentioned distribution of coupons to the smartphone app and display of congestion status on the signage are performed.

[0080] The above series of processes is regarded as one episode, and the learning management unit 115 advances learning by executing multiple episodes. After each episode is completed, the learning management unit 115 performs a learning management judgment process (510). For example, when the number of episodes reaches a predetermined number, or when the rate of change in the sum of rewards of the episodes becomes smaller than a threshold, learning is judged to be complete. If learning is not complete (510: not complete), the flow returns to step 502.

[0081] If the learning is complete (510: complete), the learning management unit 115 executes a robust learning completion determination process (511). The learning management unit 115 determines whether learning has already been performed with an error injected into the measurement data after normal learning. If learning has not been performed with an error injected (511: incomplete), the error generation unit 116 sets an error (512). Thereafter, the flow returns to step 502, and the above-mentioned series of processes are performed with an error injected into the measurement data.

[0082] The error is added to the numerical data in the measurement data. For example, the error can be added to the number of users measured by a TOF sensor. Learning is performed with the error included in the attribute data of the graph. Based on the learning results obtained in the initial learning, learning is performed in a situation where further error has been added, making it possible to learn control that is more robust against error.

[0083] As described above, learning is performed with the error applied, and the learning management unit 115 executes robust learning completion determination processing (511). If it is determined that learning is completed (511: completed), this processing ends. For example, if a predetermined number of episodes is reached or the rate of change in the total sum of rewards for each episode becomes smaller than a threshold, the robust learning processing is determined to be completed. Note that if robust learning is not completed (511: not completed), the error is changed and the processing of the learning and action selection unit is executed again.

[0084] The process shown in Fig. 5 performs machine learning of a learning model while operating the execution system in the real environment 106. After the machine learning shown in Fig. 5 is completed, the system control unit 103 can use the learned model to execute control of the execution system based on the measurement data, or the system control unit 103 may continue without terminating the flow shown in Fig. 5. The learning process by the system control unit 103 may be executed for the real environment 106, or may be executed for the simulated environment 106.

[0085] As a result, the system acquires control rules for each subsystem that maximizes the sum of the subsystem's KPIs while balancing them in the form of rewards.

[0086] In the above embodiment, attribute data is added only to the nodes out of the nodes and edges. In another example, attribute data may also be added to the edges and these may be subjected to convolution processing. In addition, the graph neural network for feature extraction is not limited to the above configuration. For example, by combining a pooling layer or a dropout layer with a convolution layer, it is possible to emphasize features. In addition, a model other than convolution or a neural network may be used to extract the features of graph data.

[0087] In the above example, in the framework of reinforcement learning, data for the execution system is determined based on the action value, and the parameters of the learning model are updated based on the action value and the reward. The system may control the execution system based on features extracted from the graph of the sub-model based on other techniques.

[0088] The system according to an embodiment of the present specification can be applied to control an execution system including subsystems of different measurement data. The measurement data of some of the subsystems may be the same type, and the system according to an embodiment of the present specification may be applied to an execution system including subsystems of the same type of measurement data.

[0089] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0090] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0091] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0092] 101, 102 Measurement unit, 103 System control unit, 104, 105 Control unit, 106 Environment, 107 Graph processing unit, 108 Measurement data receiving unit, 109 Measurement data-graph conversion unit, 110 Graph feature extraction unit, 111 State action value updating unit, 112 Action selection unit, 113 Action-control data conversion unit, 114 Control data transmission unit, 115 Learning management unit, 116 Error generation unit, 117 Measurement data-reward conversion unit, 151 Processor, 152 DRAM, 153 Communication device, 153 Monitor, 154 Auxiliary storage device, 155 Input device, 200 Shopping mall, 205, 301 Store, 302 Passage, 303 User movement, 401 Measurement range, 402 Sensor viewpoint, 600, 700 Graph data, 601, 603, 701, 703 Node, 602, 702 Adjacent nodes, 604, 704 Edges, 803, 804 Features, 805 Convolutional layer, 806 Fully connected layer, 807 Graph convolutional neural network, 808, 809 Q value

Claims

1. 1. A system for controlling an executive system including a plurality of subsystems, comprising: one or more processors; one or more storage devices for storing programs executed by the one or more processors; The one or more storage devices include Data for constructing a first model that extracts features of each input graph; and and storing data for constructing a second model that determines a value of an action for the execution system based on the feature amount; The graph includes nodes, edges connecting the nodes, and attribute data of the nodes; the first model includes a sub-model for each of the plurality of subsystems; The sub-model executes a convolution process on attribute data of the input graph to extract features of the input graph; the first model and the second model are trained so that a difference between a reward for an action on the execution system obtained from measurement data of the execution system and a value of the action is small; The one or more processors: acquiring measurement data from each of the plurality of subsystems; generating a graph representing the measurement data of each of the plurality of subsystems based on the measurement data from each of the plurality of subsystems; Extracting features from each of the graphs of the plurality of subsystems using the sub-model; determining a value of the action based on the features using the second model; A system for selecting an action based on a value of the action, and controlling the execution system with control data corresponding to the selected action.

2. The system according to claim 1, The metrology data of the plurality of subsystems includes metrology data of different types.

3. The system according to claim 1, A system, wherein the measurement data of the multiple subsystems is measured during the same period.

4. The system according to claim 1, The one or more processors: selecting an action for the execution system based on a value of the action; determining a reward for the selected behavior based on measurement data from the plurality of subsystems; The system updates parameters of the first model and the second model based on the value and the reward.

5. The system according to claim 4, The one or more processors update parameters of the first model and the second model using the graph plus error data.

6. The system of claim 1, The execution system controls the flow of people in a shopping mall; the measurement data of a first subsystem of the plurality of subsystems includes data on sales at a store in the shopping mall; The system, wherein the measurement data of a second subsystem in the plurality of subsystems includes data regarding positions of people in the shopping mall.

7. A method for controlling an execution system, the system including a plurality of subsystems, comprising: The system comprises: Data for constructing a first model that extracts features of each input graph; and and storing data for constructing a second model that determines a value of an action for the execution system based on the feature amount; The graph includes nodes, edges connecting the nodes, and attribute data of the nodes; the first model includes a sub-model for each of the plurality of subsystems; The sub-model executes a convolution process on attribute data of the input graph to extract features of the input graph; the first model and the second model are trained so that a difference between a reward for an action on the execution system obtained from measurement data of the execution system and a value of the action is small; The method comprises: the system acquires metrology data from each of the plurality of subsystems; the system generates a graph representing the measurement data of each of the plurality of subsystems based on the measurement data from each of the plurality of subsystems; the system extracts features from each of the graphs using the sub-models; the system uses the second model to determine a value of the action based on the features; the system selecting an action based on a value of the action and controlling the execution system with control data corresponding to the selected action.

Images