Learning device and learning method

Abstract

To execute self-supervised learning capable of accepting a generation strategy of various instances for tabular data in which data of heterogeneous formats such as numeric data, categorical data or the like are mixed.SOLUTION: A learning apparatus is a learning apparatus for constructing a machine learning model for tabular format data, and includes: an arithmetic device that executes predetermined processing; a storage device that can be accessed by the arithmetic device; an upstream task learning unit in which the arithmetic device generates an upstream model relating to an upstream task by self-supervised learning based on neighborhood definition information defining a neighborhood relation of a pair of instances of the tabular format data; and a downstream task learning unit in which the arithmetic device generates a downstream model relating to a downstream task by supervised learning, reinforcement learning or unsupervised learning.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device and a learning method. [Background technology]

[0002] With the recent development of IoT, applications of artificial intelligence, especially machine learning, are expanding in various fields. For example, in the manufacturing industry, the use of robots is expected to increase more than ever before, and attempts are being made to automatically acquire motion plans using reinforcement learning. Also, in the transportation industry, for example, services that predict bus delay times using supervised learning are beginning to be provided.

[0003] On the other hand, building a highly accurate machine learning model often requires a huge amount of training data, but in reality, it is sometimes difficult to collect a sufficient amount of training data due to the difficulty of collecting data in a real environment and labeling it.

[0004] To address this situation, self-supervised learning, a type of pre-learning, is used. Self-supervised learning is a method for pre-learning useful representations without trial and error or labeling on an actual machine by automatically generating training data from input data. One simple method, targeted at deep learning, is to input two highly similar instances into the same neural network and train the resulting representations (features) to approximate each other. Often, the two highly similar instances are created by applying different data augmentation to instances sampled from a dataset. This approach is also called self-supervised representation learning, and whether or not a good representation is obtained depends on the generation strategy for the two highly similar input instances.

[0005] For example, Patent Document 1 describes a learning device that includes a self-learning unit that updates the content of a main conversion process that converts data to be processed into data of a predetermined format by performing self-supervised learning, and a data extension unit that performs a data extension process that generates data to be processed by the main conversion process based on an acoustic time series, wherein the data extension unit performs an acoustic time series extraction process that extracts a partial time series that is a part of the acoustic time series, a duplication process that duplicates the partial time series, and a conversion process that converts one of the partial time series into the other in accordance with a predetermined rule, and the self-learning unit updates the content of the main conversion process by self-supervised learning based on the results obtained by the conversion process. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2022 / 244047 Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 enables self-supervised learning that takes into account similarities over long time intervals. However, the technology described in Patent Document 1 incorporates conversion processing for acoustic time series based on the knowledge of the inventors, and does not target tabular data that contains a mixture of data in different formats, such as numerical data and categorical data.

[0008] For this reason, for example, with regard to data obtained for each bus stop (hereinafter referred to as a stop), it is difficult to obtain a common representation for instances at different stops (represented by discrete values ​​such as IDs or names) of the same bus, or for the same stop of different buses.

[0009] Furthermore, as mentioned above, whether or not a good representation can be obtained depends on the generation strategy for the two input instances that are highly similar. However, the technology described in Patent Document 1 does not take into consideration the fact that a user may search for a trial-and-error strategy for generating instances that bring the representations closer together, or that a computer may search for the strategy mechanically.

[0010] The present invention has been made in consideration of the above-mentioned problems, and aims to enable self-supervised learning that allows for various instance generation strategies for tabular data that contains a mixture of different types of data, such as numerical data and categorical data. [Means for solving the problem]

[0011] The present application includes a number of means for solving at least part of the above-mentioned problems, examples of which are as follows.

[0012] In order to solve the above-mentioned problems, a learning device according to one aspect of the present invention is a learning device for constructing a machine learning model for tabular data, comprising: a calculation device that executes predetermined processing; and a storage device accessible to the calculation device, wherein the calculation device has an upstream task learning unit that generates an upstream model for an upstream task by self-supervised learning based on neighborhood definition information that defines a neighborhood relationship between pairs of instances of the tabular data; and a downstream task learning unit that generates a downstream model for a downstream task by supervised learning, reinforcement learning, or unsupervised learning. [Effects of the Invention]

[0013] According to the present invention, self-supervised learning that allows various instance generation strategies can be performed on tabular data that contains a mixture of different types of data, such as numerical data and categorical data. This makes it possible to acquire representations that take into account not only temporal proximity (similarity) but also positional proximity (similarity), etc.

[0014] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a robot management system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a general computer. [Figure 3] FIG. 3 is a diagram illustrating an example of robot state data. [Figure 4] FIG. 4 is a diagram showing an example of the orthogonal system position and velocity data. [Figure 5] FIG. 5 is a diagram illustrating an example of neighborhood definition information. [Figure 6] FIG. 6 is a diagram illustrating an example of the structure of a self-supervised learning model. [Figure 7] FIG. 7 is a diagram showing an example of the structure of an actor model for reinforcement learning. [Figure 8] FIG. 8 is a diagram illustrating an example of the structure of a critic model for reinforcement learning. [Figure 9] FIG. 9 is a flowchart showing an example of the learning process. [Figure 10] FIG. 10 is a flowchart illustrating an example of an upstream task learning process. [Figure 11] FIG. 11 is a flowchart illustrating an example of downstream task learning processing. [Figure 12] FIG. 12 is a diagram showing an example of a display screen for checking the higher-level task learning loss transition. [Figure 13] FIG. 13 is a diagram showing a display example of the parameter display screen. [Figure 14] FIG. 14 is a diagram showing an example of the configuration of a bus operation control system according to the second embodiment of the present invention. [Figure 15] FIG. 15 is a diagram illustrating an example of the bus state data. [Figure 16] FIG. 16 is a diagram illustrating an example of stop adjacent data. [Figure 17] FIG. 17 is a diagram illustrating an example of neighborhood definition information. [Figure 18] FIG. 18 is a diagram illustrating an example of the structure of a self-supervised learning model. [Figure 19] FIG. 19 is a diagram illustrating an example of the structure of a delay time prediction model. [Figure 20] FIG. 20 is a flowchart illustrating an example of downstream task learning processing. [Figure 21] FIG. 21 is a diagram showing an example of a predicted delay time display screen. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. In all drawings used to explain each embodiment, the same components are generally designated by the same reference numerals, and repeated explanations will be omitted. Furthermore, in the following embodiments, the components (including element steps, etc.) are not necessarily essential unless otherwise specified, or unless they are clearly considered essential in principle. Furthermore, when the terms "consisting of A," "composed of A," "having A," or "including A" are used, other elements are not excluded unless otherwise specified, or unless it is clearly considered that only that element is included. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., these terms include those that are substantially similar or similar to the shape, etc., unless otherwise specified, or unless it is clearly considered otherwise in principle.

[0017] <Robot management system 1 according to the first embodiment of the present invention> FIG. 1 shows an example of the configuration of a robot management system 1 according to a first embodiment of the present invention.

[0018] <Summary> The robot management system 1 includes a learning system 10 related to motion planning (planning of trajectories and gripping) of robot arms in a factory. First, an overview of the overall processing flow of the robot management system 1 including the learning system 10 will be explained. Generally, a simulator corresponding to the robot arm 40 is prepared, and its motion can be planned using a planning engine using random sampling (hereinafter referred to as an existing motion planner). However, there will be some discrepancy between the simulator and the actual machine due to factors such as the tilt of the robot arm 40 when installed, individual differences between the robot arms 40, and deflection due to the weight of the workpiece being gripped.

[0019] The learning system 10 uses data obtained from the actual machine and reinforcement learning to construct a motion planning function that further reduces the deviation. In this case, pre-learning is performed using self-supervised learning that utilizes data obtained using an existing motion planner and simulation data, thereby reducing the number of trial and error steps on the actual machine for reinforcement learning.

[0020] In preparation for self-supervised learning, a time series of robot states (robot state data) for self-supervised learning is collected using an existing motion planner installed in each robot arm 40 and in the planning unit 31 of the robot arm controller 30. It is assumed that multiple robot arms 40 are performing the same type of task asynchronously.

[0021] <Configuration example> Next, we will explain an example of the configuration of the robot management system 1. The robot management system 1 includes a learning system 10, a robot arm control device 30, and a plurality of robot arms 40.

[0022] The learning system 10 and the robot arm control device 30 are each connected to an information network NW1, and communicate various data and information with each other via the information network NW1. The robot arm control device 30 and the robot arm 40 are connected to a control network NW2, and the robot arm control device 30 transmits control signals to the robot arm 40 via the control network NW2, and the robot arm 40 transmits status data to the robot arm control device 30 via the control network NW2.

[0023] Although the robot management system 1 is assumed to be constructed in a closed environment, the information network NW1 and the control network NW2 may include an open WWW (World Wide Web) network as a route between them. Furthermore, if real-time performance can be ensured, the information network NW1 and the control network NW2 may be the same network.

[0024] The learning system 10 includes a learning device 11 and an operation terminal 12. However, the learning device 11 and the operation terminal 12 may be integrated.

[0025] The learning device 11 has the following functional blocks: an overall task management unit 111, an upstream task learning unit 112, a simulation execution unit 113, a downstream task learning unit 114, a display management unit 115, and a learning data management unit 116. The learning device 11 is realized by a general computer such as a personal computer or a server computer.

[0026] 2 shows an example configuration of a general computer 100 that realizes the learning device 11. The computer 100 includes a processor 101 such as a CPU (Central Processing Unit), a memory 102 such as a DRAM (Dynamic Random Access Memory), a storage 103 such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), an input device 104 such as a keyboard, a mouse, or a media drive, an output device 105 such as a display, and a communication module 106 such as an Ethernet (registered trademark) card or a Wi-Fi (registered trademark) adapter.

[0027] The overall task management unit 111, upstream task learning unit 112, simulation execution unit 113, downstream task learning unit 114, display management unit 115, and learning data management unit 116 of the learning device 11 are realized by the processor 101 of the computer 100 executing the program 107 stored in the memory 102.

[0028] The program 107 executed by the processor 101 may be stored in advance in the memory 102, or may be downloaded from a predetermined server or the like via a removable medium (CD-ROM, flash memory, etc.) or a network such as the Internet, stored in the storage 103, which is a non-transitory storage medium, and read out from the storage 103 when needed. For this reason, the learning device 11 (computer 100) should preferably have an interface for reading data from removable media.

[0029] Furthermore, the learning device 11 may be a computer system configured on a single physical computer, or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, multiple programs that realize the functions of the learning device 11 may each operate on separate physical or logical computers, or multiple programs may be combined to operate on a single physical or logical computer.

[0030] Returning to Figure 1, the overall task management unit 111 controls the entire learning device 11.

[0031] The upstream task learning unit 112 performs self-supervised learning using the robot state data 300 (FIG. 3) under the control of the overall task management unit 111. In the self-supervised learning by the upstream task learning unit 112, for example, a network employing the "SimSiam" architecture is used to avoid representation collapse due to stop gradients. This network includes a state encoder that converts the position and velocity obtained from the robot state data 300 into a certain representation (feature quantity), and a predictor (projection head) that predicts another feature quantity from one feature quantity.

[0032] Specifically, the upstream task learning unit 112 prepares a pair of instances (a first instance and a second instance) of the robot state data 300 in accordance with neighborhood definition information 500 (FIG. 5) that is predefined and managed by the learning data management unit 116, and learns a task of predicting the expression of the second instance from the expression (feature) obtained from the first instance (details will be described later).

[0033] The simulation execution unit 113 executes a simulation of future or past robot states under the control of the overall task management unit 111, with a first instance of certain robot state data as the initial state. The simulation result is used as a third instance when predicting the expression of the third instance from the expression (feature) similarly obtained from the first instance. Alternatively, the simulation result is used when learning a task in which the expression of the first instance is predicted from the expression of the third instance.

[0034] Instead of using "SimSiam" for self-supervised learning, other methods such as "BYOL" or "SimCLR" that require negative examples may be used. When negative examples are required, this can be achieved by using, for example, randomly selected instances. In this embodiment, as in "SimSiam," the asymmetric structure of the predictor and the collapse of the representation obtained by the stop gradient are avoided. However, other methods for avoiding collapse, such as utilizing the time average of the encoder as in "BYOL" or decorrelation by whitening, may also be used.

[0035] Furthermore, since the prepared instance pairs have directions in terms of time and position, and a predictor that takes these directions into account is set, in this embodiment, an asymmetric objective function is optimized, rather than a symmetric objective function for two data-augmented instances, as in "SimSiam." This is just one example, and a symmetric objective function may also be optimized for the obtained samples. Furthermore, in this embodiment, the instance pairs that approximate the representations are obtained by sampling or simulation, but a commonly used data augmentation method may also be used, or a combination of both may also be used.

[0036] The downstream task learning unit 114 uses reinforcement learning to learn a new planning engine (RL motion planner) under the control of the overall task management unit 111. The downstream task learning unit 114 uses an actor model and a critic model for reinforcement learning. The actor model takes position, velocity, grip state, and target state as inputs and outputs an action. The critic model takes position, velocity, grip state, target state, and action as inputs and outputs a Q value representing the quality of the action. The reinforcement learning uses a deep deterministic policy gradient (DDPG). The neural networks of both the actor model and the critic model incorporate state encoders learned through self-supervised learning. That is, they use representations of position and velocity learned through self-supervised learning, which is the upstream task. In this embodiment, reinforcement learning is performed separately for each of the multiple robot arms 40. However, in the early stages of learning, the multiple robot arms 40 may be operated in parallel and the learning results may be shared as appropriate. Even in this case, the robot arms 40 are individually trained in the final stages of learning, taking into account the installation and individual differences.

[0037] The display management unit 115 causes the operation terminal 12 to display a UI (User Interface) screen.

[0038] The learning data management unit 116 appropriately stores and manages the robot state data for self-supervised learning, the transition of the loss function during learning of the upstream task and the downstream task, performance indicators such as the difference between the predicted delay time and the actual delay time, pre-trained models, and obtained regression models in the storage 103 of the computer 100 that realizes the learning device 11.

[0039] Operation terminal 12 is realized by a general computer 100 (FIG. 2), similar to study device 11. Operation terminal 12 displays a UI screen and accepts input from user 121 on the UI screen.

[0040] The robot arm control device 30 is realized by a general computer 100 (FIG. 2), similar to the learning device 11. The robot arm control device 30 has the following functional blocks: a planning unit 31, a data collection unit 32, and a control data management unit 33. Each functional block of the robot arm control device 30 is realized by the processor 101 of the computer 100 executing a program 107 stored in the memory 102.

[0041] The planning unit 31 uses a built-in existing motion planner to create a motion plan for the robot arm 40 to collect robot state data 300 for self-supervised learning. Specifically, the planning unit 31 randomly determines the target state, i.e., the arm tip position and grip state on a Cartesian coordinate system, to create an existing motion plan, performs inverse dynamics calculations, controls the joint angles, and creates a motion plan for transitioning the robot arm 40 to the target state.

[0042] The data collection unit 32 collects robot status data 300 from the robot arm 40 operating according to the operation plan. The control data management unit 33 stores the collected robot status data 300 in the storage 103 or the like of the computer 100 that realizes the robot arm control device 30, and manages the data.

[0043] The robot arm 40 employs, for example, an arm with six degrees of freedom and a gripper mounted at the tip thereof. However, the arm may have more than six degrees of freedom, and the arm may have, for example, a suction hand mounted at the tip thereof.

[0044] <Data structure of various data> Next, FIG. 3 shows an example of the data structure of robot state data 300 managed by the control data management unit 33 of the robot arm control device 30. As shown in FIG.

[0045] The robot status data 300 includes the date and time when the data was observed by the robot arm 40, a robot ID for identifying the robot arm 40, the angles of joints 1 to 6 of the robot arm 40, and a gripping state indicating the open / closed state of the gripper mounted on the tip of the arm, all recorded in association with each other. In this embodiment, the robot status data 300 is observed periodically. In the example shown in the figure, the observation period is one second.

[0046] 4 shows an example of the data structure of cartesian position and velocity data 400 generated by the overall task manager 111 based on the robot state data 300. The cartesian position and velocity data 400 corresponds to the tabular data of the present invention. The tabular data is a set of instances made up of tuples made up of multiple pieces of data such as numerical data and categorical data.

[0047] The Cartesian position and velocity data 400 includes, in association with each other, the standardized time (chronological order) at which the data was observed, a robot ID for identifying the robot arm 40, the position (Cartesian coordinate system) of the gripper at the end of the arm, the velocity (Cartesian coordinate system) of the gripper at the end of the arm, and a gripping state indicating whether the gripper is open or closed. The position and velocity (Cartesian coordinate system) of the gripper at the end of the arm are calculated based on the angles (joint coordinate system) of joints 1 to 6 of the robot arm 40. Depending on the type of hand mounted at the end of the arm, the gripping state may include information indicating not only open or closed but also orientation.

[0048] Next, FIG. 5 shows an example of predefined neighborhood definition information 500 managed by the learning data management unit 116.

[0049] The neighborhood definition information 500 is a script written in a domain-specific language that describes neighborhoods for sampling pairs of instances used in self-supervised learning and the assignment of predictors to the neighborhoods.

[0050] In the neighborhood definition information 500, prior to defining the neighborhood, the parameters and variables used to define the neighborhood are declared in advance as "parameter" and "variable." A range can be set for the parameters. This is used for automatic hyperparameter tuning in the process described below. For variables, sampling from a dataset and simulation data using the simulation execution unit 113 of the learning device 11 can be used.

[0051] In the example of Figure 5, four types of neighborhoods N1, N2, N3, and N4 are defined. Because the robot arm operates continuously, it is thought that robot arms that are close in time share similar backgrounds (i.e., preferred expressions that contribute to improving generalization performance). Therefore, neighborhoods N1 and N2 are defined as instances of the same robot arm 40 that are close in time. However, neighborhoods N1 and N2 have different time axis directions, with neighborhood N1 defining instances that are forward in time as neighborhoods, and neighborhood N2 defining instances that are backward in time as neighborhoods.

[0052] Furthermore, the simulation is expected to predict the near future of the robot arm with high accuracy. Therefore, the neighborhood N3 is defined as an instance at a certain time and an instance of the predicted state at a time slightly ahead obtained by the simulation from that time. Note that since general simulators often only handle forward time development, only forward-looking development is defined.

[0053] Furthermore, even if different robots are performing the same task, they will perform almost identical actions. Therefore, if the elapsed time from the start time is the same, they are considered to share a similar background. Therefore, neighborhood N4 is defined as instances within 0.1 seconds of the time (normalized time) when the task start time is set to 0 for different robots.

[0054] Furthermore, in the example of FIG. 5, predictors P1 and P2 are defined, and predictor P1 is assigned to neighborhoods N1 and N3, predictor P2 is assigned to neighborhood N2, and predictor P3 is assigned to neighborhood N4.

[0055] In addition," SimSiam " has only one predictor, in this embodiment, the neighborhood definition information 500 can be used to assign the same predictor or different predictors to different types of neighborhoods. In other words, it is possible to avoid a situation where the task of the predictor becomes too difficult and the expressive power of the resulting expression is poor when a predictor is shared, such as a predictor that predicts the future from the past or a predictor that predicts the past from the future. Furthermore, because the script is separated from the core machine learning program, the user 121 can easily use trial and error to search for the best neighborhood definition for the target problem, explore how to utilize simulation data, and so on.

[0056] <Machine learning model structure> 6 shows an example of a self-supervised learning model 600, which is a machine learning model learned by self-supervised learning by the upstream task learning unit 112. The self-supervised learning model 600 corresponds to the upstream model of the present invention.

[0057] The self-supervised learning model 600 includes an instance pair sampler 601, a state encoder 602, and an embedding amount predictor 603. The instance pair sampler 601 samples an instance pair (instance A, B) based on the neighborhood definition information 500. The state encoder 602 calculates the embedding amount A based on the position and speed of the sampled instance A. The state encoder 602 also calculates the embedding amount B based on the position and speed of the sampled instance B. The embedding amount predictor 603 calculates an estimated embedding amount B by estimating the embedding amount B from the embedding amount A.

[0058] 7 and 8 show examples of a reinforcement learning actor model 700 (FIG. 7) and a reinforcement learning critic model 800 (FIG. 8), which are machine learning models learned by reinforcement learning by the downstream task learning unit 114. The reinforcement learning actor model 700 and the reinforcement learning critic model 800 correspond to the downstream model of the present invention.

[0059] The actor model for reinforcement learning 700 outputs an action to achieve a goal from the position, velocity, grip state, and goal state. The actor model for reinforcement learning 700 includes a state encoder 701 and an actor 702. The state encoder 701 has the same structure as the state encoder 602 of the self-supervised learning model 600, and is initialized using the trained state encoder 602 (the parameters of the state encoder 602 are copied). The state encoder 701 calculates an embedding amount based on the position and velocity. The actor 702 outputs an action based on the embedding amount, grip state, and goal state.

[0060] The actions output by the actor 702 are minute manipulations of the position in the Cartesian coordinate system and manipulations of the gripping state. These manipulations are converted into the joint coordinate system through inverse kinematic calculations by the planning unit 31 of the robot arm control device 30 and then executed.

[0061] The reinforcement learning critic model 800 includes a state encoder 801 and a critic 802. The state encoder 801 has the same structure as the state encoder 602 of the self-supervised learning model 600, and is initialized using the trained state encoder 602 (the parameters of the state encoder 602 are copied). The state encoder 801 calculates an embedding amount based on the position and velocity. The critic 802 outputs a Q value representing the quality of the action based on the embedding amount, the grip state, the target state, and the action output by the actor 702.

[0062] <Learning process by learning device 11> Next, FIG. 9 is a flowchart showing an example of the learning process performed by the learning device 11.

[0063] As a premise for this learning process, it is assumed that the control data management unit 33 of the robot arm control device 30 has already stored and managed the robot state data 300 (FIG. 3) used for self-supervised learning, which has been collected from each robot arm 40 by the data collection unit 32, in the storage 103 or the like.

[0064] The learning process is started in response to a predetermined start operation by the user 121 using the operation terminal 12, for example.

[0065] First, to prepare data to be used in self-supervised learning, the overall task management unit 111 reads out the robot state data 300 under the management of the control data management unit 33 of the robot arm control device 30 and copies it under the management of the learning data management unit 116 of the learning device 11. Then, the overall task management unit 111 converts the values ​​of joint angles 1 to 6, which are the joint coordinate system of the robot state data 300, into a Cartesian coordinate system, and generates Cartesian position and velocity data 400 (FIG. 4) to which velocity is assigned by calculating the difference (step S1).

[0066] Next, the overall task manager 111 generates hyperparameters to be used in the upstream task learning process (self-supervised learning) in step S3 and the downstream task learning process in step S4 (step S2). Specifically, the hyperparameters are generated by randomly sampling delta1, delta2, and delta3, which are parameters described in the neighborhood definition information 500 (FIG. 5), from the described range. Note that in this embodiment, only the above three parameters are targeted for tuning, but various hyperparameters, such as the network structure, the number of mini-batches, the learning rate, and other training conditions, may also be targeted.

[0067] Next, the upstream task learning unit 112 executes an upstream task learning process to generate a state encoder 602 of the self-supervised learning model 600 (FIG. 6) using the orthogonal position and velocity data 400, the neighborhood definition information 500, and the hyperparameters generated in step S2 (step S3).

[0068] FIG. 10 is a flowchart showing an example of the upstream task learning process in step S3.

[0069] First, the upstream task learning unit 112 selects one of the neighborhoods N1 to N4 defined in the neighborhood definition information 500 (step S31).

[0070] Next, the upstream task learning unit 112 samples a pair of instances that match the neighborhood definition selected in step S31 according to the neighborhood definition information 500 and the hyperparameters. If the sampling of instances involves a simulation, the upstream task learning unit 112 first samples one instance from the Cartesian position and velocity data 400, and the sampled instance is used as the initial state. The simulation execution unit 113 generates another instance based on the results of a simulation performed in the future or past at a specified time, and samples these two instances as a pair (step S32). Note that the behavior information of the robot arm 40 required for the simulation can be information on the behavior executed next to or immediately before the sampled instance.

[0071] Next, the upstream task learning unit 112 uses the self-supervised learning model 600 to generate embedding amounts A and B for the position and velocity of the instance pair A and B, and an estimated embedding amount B, which is an estimate of the embedding amount B estimated from the embedding amount A (step S33).

[0072] Next, the upstream task learning unit 112 calculates the loss between the embedding amount B and the estimated embedding amount B using the cosine similarity multiplied by -1 as a loss function, and updates the parameters of the embedding amount predictor 603 and the state encoder 602 regarding the position and velocity A using the backpropagation method (step S34).

[0073] In this embodiment, the loss function used is the cosine similarity multiplied by -1, but other functions such as mean square error may also be used as the loss function. Also, in this embodiment, the loss is related to the embedding amount A and the estimated embedding amount B in consideration of the order, but if the order is not taken into consideration, the embedding amount A and the embedding amount B may be reversed, i.e., the embedding amount A may be estimated from the embedding amount B.

[0074] The embedding amount predictor 603 follows the correspondence between predictors and neighborhoods defined in the neighborhood definition information 500. That is, the neighborhood definition information 500 has three predictors P1 to P3, with the predictor P1 corresponding to neighborhoods N1 and N3, the predictor P2 corresponding to neighborhood N2, and the predictor P3 corresponding to neighborhood N4. Since the predictor can be changed depending on the neighborhood in this way, an error signal for providing an expression for each neighborhood can be sent to the state encoder 602.

[0075] Next, the upstream task learning unit 112 determines whether the learning end condition has been met (step S35). Here, the learning end condition is defined as, for example, repeating the processes of steps S31 to S34 a predetermined number of times (for example, 10,000 times). If the upstream task learning unit 112 determines that the learning end condition has not been met (NO in step S35), the process returns to step S31, and the processes from step S31 onwards are repeated. Thereafter, if the upstream task learning unit 112 determines that the learning end condition has been met (YES in step S35), the upstream task learning process is terminated.

[0076] In this embodiment, data generation by simulation is limited to one side of a pair of instances. However, if different instances can be generated by changing the conditions related to the simulation, two instances may be generated by simulation and paired. In this manner, this embodiment can generate various instances. Furthermore, even for targets for which known data augmentation methods are not clear, such as enlarging, reducing, cropping, etc. for image data, or cutting or combining audio data, self-supervised learning can be performed by sampling and simulation.

[0077] Returning to Figure 9, next, downstream task learning unit 114 copies the parameters in state encoder 602 of self-supervised learning model 600 generated in step S3 to state encoder 701 of reinforcement learning actor model 700 and state encoder 801 of reinforcement learning critic model 800 (Figure 8), and then executes downstream task learning processing for learning reinforcement learning actor model 700 and reinforcement learning critic model 800 (Figure 8) (step S4).

[0078] Fig. 11 is a flowchart showing an example of the downstream task learning process in step S4. Note that the downstream task learning process shown in Fig. 11 is a process related to one robot arm 40. Furthermore, observation of the state of the robot arm and application of actions are performed while the learning device 11, robot arm control computer, and robot arm 40 communicate with each other.

[0079] First, the downstream task learning unit 114 copies the parameters of the state encoder 602 of the self-supervised learning model 600, which have been pre-trained by the upstream task learning process in step S3, to the state encoder 701 of the reinforcement learning actor model 700 and the state encoder 801 of the reinforcement learning critic model 800. Other parameters are initialized using a general method such as normal distribution (step S41).

[0080] Next, the robot arm control device 30 randomly determines the initial state and target state of the robot arm 40, and transitions the state of the robot arm 40 by applying a trajectory plan obtained using an existing motion planner in the planning unit 31 while performing inverse dynamics calculations according to the determined values ​​(step S42).

[0081] Next, the downstream task learning unit 114 uses the reinforcement learning actor model 700 to determine an action from the current state (position, speed, grip state) and the goal state (step S43). Next, the downstream task learning unit 114 applies the determined action to the robot arm 40 to transition the state. Then, the new state is observed and a reward is calculated (step S44). The reward function used in the reward calculation is, for example, +1 if the robot arm approaches the goal state within a specified distance, and -0.01 if it does not.

[0082] Next, the downstream task learning unit 114 calculates a loss using the current state (position, velocity, grip state), goal state, action, and reward, and updates the parameters of the reinforcement learning actor model 700 and the reinforcement learning critic model 800 by backpropagation (step S45). Here, the loss is calculated based on the TD error in reinforcement learning.

[0083] Next, the downstream task learning unit 114 determines whether the episode end condition has been achieved (step S46). Here, the episode end condition is defined as, for example, whether the reward becomes +1, or whether the processing of steps S43 to S45 is repeated a predetermined number of times (for example, 1000 times) even if the reward does not become +1. If the downstream task learning unit 114 determines that the episode end condition has not been achieved (NO in step S46), the processing returns to step S43, and step S43 and subsequent steps are repeated.

[0084] Thereafter, if it is determined that the episode end condition has been achieved (YES in step S46), the downstream task learning unit 114 then determines whether the learning end condition has been achieved (step S47). Here, the learning end condition is defined as the processing from step S42 onwards being repeated a predetermined number of times (for example, 1000 times). If the downstream task learning unit 114 determines that the learning end condition has not been achieved (NO in step S47), the processing returns to step S42, and step S42 onwards are repeated. Thereafter, if it is determined that the learning end condition has been achieved (YES in step S47), the downstream task learning processing is terminated.

[0085] Returning to Figure 9, next, the overall task management unit 111 outputs the transition of the loss of each predictor during self-supervised learning and the success rate of the episode in reinforcement learning as evaluation results to the learning data management unit 116, which then stores these in the storage 103 or the like. The learning data management unit 116 also temporarily saves the trained self-supervised learning model 600, reinforcement learning actor model 700, and reinforcement learning critic model 800 in the storage 103 or the like (step S5).

[0086] Next, the overall task management unit 111 determines whether a predetermined tuning end condition has been met (step S6). Here, the predetermined tuning end condition is defined as, for example, repeating the processes of steps S2 to S5 a predetermined number of times (for example, 100 times), or the evaluation result (the transition of the success rate of episodes) being improved compared to previously created models.

[0087] When the overall task management unit 111 determines that the predetermined tuning end condition has not been achieved (NO in step S6), the process returns to step S2, and the processes in steps S2 to S5 are repeated. After that, when the overall task management unit 111 determines that the predetermined tuning end condition has been achieved (YES in step S6), next, the overall task management unit 111 outputs, as the final model, the model with the best evaluation result (transition of the success rate of episodes) among the models so far to the learning data management unit 116 as the related evaluation result, and the learning data management unit 116 stores this in the storage 103 or the like. Then, the data of the model and the evaluation result temporarily saved in step S5 is deleted from the storage 103 or the like (step S7 in FIG. 9). With this, the learning process ends.

[0088] <UI screen> Next, FIG. 12 shows a display example of the upper task learning loss transition confirmation screen 1000 as a UI screen displayed on the operation terminal 12 by the display management unit 115 when the upstream task learning process in step S3 described above is being executed. The upper task learning loss transition confirmation screen 1000 is for presenting the transition (learning progress) of the upstream task learning process to the user 121.

[0089] The upper task learning loss transition confirmation screen 1000 is provided with a loss transition display area 1001, a "neighborhood definition display" button 1002, and a "parameter display" button 1003.

[0090] The loss transition display area 1001 displays a graph of the loss value versus the number of updates for each predictor defined by the neighborhood definition information 500 used to train the finally selected model stored in the training data management unit 116. In this figure, graphs corresponding to predictors P1 and P2 are displayed, and the graph corresponding to predictor P3 can be displayed by scrolling inside the loss transition display area 1001. By checking the loss transition display area 1001, the user 121 can determine whether or not higher-level task training is progressing smoothly.

[0091] The "Display Neighborhood Definition" button 1002 is a button for displaying the contents of the neighborhood definition information 500 stored in the learning data management unit 116. The "Display Parameters" button 1003 is a button for displaying a parameter confirmation screen 1100 ( FIG. 13 ) that displays the hyperparameters related to the finally selected model stored in the learning data management unit 116 and the parameters used.

[0092] 13 shows an example of a parameter confirmation screen 1100. The parameter confirmation screen 1100 displays a parameter table 1101. The parameter table 1101 displays the values ​​of the parameters in association with their names. For example, the example in the same figure shows that 00:00:00.5, 00:00:00.3, and 00:00:00.1 have been selected as the hyperparameters delta1, delta2, and delta3, and that the learning rate for self-supervised learning (SSL learning rate) and the learning rate for reinforcement learning (RL learning rate) have both been set to 1e-5.

[0093] According to the first embodiment described above, it is possible to perform an operation plan that reduces the discrepancy between the simulator and the actual machine that may occur due to the tilt of the robot arm 40 when it is installed, individual differences between robots, deflection due to the weight of the gripped workpiece, etc. Furthermore, since the upstream task learning is performed prior to the downstream task learning process, the number of trial and error processes on the actual machine for the downstream task learning process can be reduced.

[0094] <Bus operation control system 2 according to a second embodiment of the present invention> Next, Fig. 14 shows an example of the configuration of a bus traffic control system 2 according to a second embodiment of the present invention. Note that, among the components of the bus traffic control system 2, components that are common to the robot management system 1 (Fig. 1) according to the first embodiment of the present invention are given the same reference numerals and their description will be omitted.

[0095] <Summary> The bus traffic control system 2 includes a learning system 20 for predicting delay times for route buses whose routes and timetables are set. First, an overview of the overall processing flow of the bus traffic control system 2 including the learning system 20 will be described.

[0096] In route bus timetables, time intervals may be specified for congested routes or congested time periods, but departure times are often set. In this embodiment, route bus 80 stops at a stop (for example, a bus stop on a bus route) and measures and collects the departure time, average speed for the section, congestion level, and delay time when departing, and transmits this as bus status data in response to a request from bus information management server 70. The congestion level may be calculated based on the weight of the route bus, for example, or may be measured using a motion sensor, camera, or the like installed inside the bus.

[0097] The learning system 20 constructs a delay time prediction function that predicts delay times based on data obtained from route buses 80. In this case, the accuracy of the regression model for delay time prediction is improved by pre-learning using self-supervised learning that utilizes data on route buses on the same route at different times.

[0098] The learning system 20 can also be used to predict delay times for moving objects other than route buses, such as robots that transport goods within a factory or trains that transport people.

[0099] <Configuration example> Next, a description will be given of an example configuration of the bus operation control system 2. The bus operation control system 2 includes a learning system 20, a bus information management server 70, and a plurality of route buses 80.

[0100] The learning system 20 includes a learning device 21 and an operation terminal 12. However, the learning device 21 and the operation terminal 12 may be integrated.

[0101] The learning system 20 and the bus information management server 70 are each connected to an information network NW1, and communicate various data and information with each other via the information network NW1. The bus information management server 70 and each route bus 80 are connected to a WWW network NW3, which enables wireless communication using a mobile phone communication network or the like. Each route bus 80 transmits bus status data to the bus information management server 70 via the WWW network NW3. Bus users 90 can also use an information terminal 91, such as a smartphone, to connect to the bus information management server 70 via the WWW network NW3 and obtain route bus delay time information. Note that a dedicated display terminal may be provided at each stop to display route bus delay time information.

[0102] Like the learning device 11 (FIG. 1), the learning device 21 is realized by a general computer 100 (FIG. 2). The learning device 21 has the following functional blocks: an overall task manager 211, an upstream task learning unit 212, a downstream task learning unit 214, a display manager 215, and a learning data manager 216. The operations of the overall task manager 211, the upstream task learning unit 212, the downstream task learning unit 214, the display manager 215, and the learning data manager 216 are substantially similar to the operations of the overall task manager 111, the upstream task learning unit 112, the downstream task learning unit 114, the display manager 115, and the learning data manager 116 of the learning device 11 (FIG. 1), and therefore descriptions thereof will be omitted as appropriate.

[0103] The overall task management unit 211 controls the entire learning device 21 .

[0104] The upstream task learning unit 212 performs self-supervised learning using bus state data. For self-supervised learning, a network employing the "SimSiam" architecture is used, as in the first embodiment. The network includes a state encoder that converts bus state data into a representation (feature) and a predictor that predicts one feature from another. The specific procedure for self-supervised learning is the same as in the first embodiment.

[0105] The downstream task learning unit 214 uses supervised learning to learn a regression model that predicts delay time. In this embodiment, supervised learning uses a general regression that minimizes the mean square error, but it is also possible to learn a model that estimates distribution and then calculate an expected value or a value that takes risk into account.

[0106] The regression model outputs a predicted delay time for a route bus at the next stop based on the bus status at the three previous stops and the bus status immediately before the preceding bus. In this embodiment, bus status data for the three previous stops and the bus status data immediately before the preceding bus are used as input, but bus status data from the first bus or bus status data for the same route and the same first departure time on the previous day may also be used. Furthermore, if external information on routes, such as road traffic information, is available, it may also be taken into consideration.

[0107] The learning data management unit 216 appropriately stores and manages the transition of the loss function during learning of the upstream task and the downstream task, performance indicators such as the difference between the predicted delay time and the actual delay time, pre-trained models, and obtained regression models in the storage 103 or the like.

[0108] The bus information management server 70 is realized by a general computer 100 (FIG. 2). The bus information management server 70 includes the following functional blocks: a data collection unit 71, a delay time prediction unit 72, a delay time distribution unit 73, and a bus information data management unit 74.

[0109] The data collection unit 71 collects bus status data 1500 (FIG. 15) from the route bus 80. The delay time prediction unit 72 predicts the delay time of the route bus 80 using the delay time prediction model 1900 learned by the learning device 21. The delay time distribution unit 73 displays a predicted delay time display screen 2000 (FIG. 21) showing the predicted delay time of the route bus 80 on the information terminal 91 of the bus user 90. The bus information data management unit 74 stores the collected bus status data 1500 in the storage 103 or the like of the computer 100 that implements the bus information management server 70.

[0110] <Data structure of various data> 15 shows an example of the data structure of bus status data 1500 managed by the bus information data management unit 74 of the bus information management server 70. The bus status data 1500 corresponds to the tabular data of the present invention. The tabular data is a set of instances made up of tuples made up of multiple pieces of data such as numerical data and categorical data.

[0111] The bus status data 1500 includes route ID, bus type (local (local), express, etc.), departure time from the first stop, date, elapsed time, stop ID, average speed, congestion level, and delay time, all of which are recorded in association with each other.

[0112] In this embodiment, route buses operating on a certain date are identified by route ID, bus type, and departure time from the first stop. Route buses with the same route ID but different bus types run the same route, but "express" buses make fewer stops than "regular" buses. Note that buses may be distinguished by different route IDs rather than by bus types such as "express." In this embodiment, route IDs are directional (i.e., route IDs distinguish between uphill and downhill, etc.), but if these distinctions are not made, the direction of travel may be taken into account in addition to the type, such as "regular" or "express."

[0113] 16 shows an example of the data structure of stop adjacent data 1600 managed by the learning data management unit 216. The stop adjacent data 1600 is used when sampling instances for self-supervised learning.

[0114] The stop adjacent data 1600 stores route IDs, bus types, stop ID-A, and stop ID-B in association with each other, with one line of information representing adjacent stops (two stops where the route bus stops sequentially) where buses with the same route ID and bus type are located. For example, this represents that a route bus with route ID "A1" and bus type "regular" stops at stop ID "102" after stop ID "101." The distance between the stops (StopDist) of stop IDs "101" and "102" is defined as 1. Here, the distance between the stops is the value obtained by adding 1 to the number of stops where the route bus does not stop, which exist between the adjacent stops where the route bus stops sequentially.

[0115] Note that even if a stop is not recorded in the stop adjacent data 1600, if it can be reached by tracing the adjacency relationship from the stop adjacent data 1600, the minimum number of times of arrival is the distance between the stops. For example, in the same figure, stop IDs "101" and "103" on a route bus with route ID "A1" and bus type "regular" are not adjacent, but can be traced via stop ID "102," so the distance between the stops of stop IDs "101" and "103" is defined as 2. Note that if the route IDs or bus types are different, it is assumed that the adjacency relationship cannot be traced.

[0116] Next, FIG. 17 shows an example of neighborhood definition information 1700 managed by the learning data management unit 216.

[0117] The neighborhood definition information 1700 is a script written in a domain-specific language that describes neighborhoods for sampling pairs of instances used in self-supervised learning and the assignment of predictors to the neighborhoods.

[0118] In the neighborhood definition information 1700, prior to defining the neighborhood, the parameters and variables used in the neighborhood definition are declared in advance as "parameter" and "variable". This is used for automatic hyperparameter tuning in the process described later. In addition, logic for neighborhood definition can be imported externally using import.

[0119] In the example of FIG. 17, four types of neighborhoods N1 to N4 are defined. Buses traveling on the same route with the same departure time are considered to share similar backgrounds even if they depart on different dates. Therefore, neighborhood N1 defines a bus state in which the distance between stops of a series of stops where a route bus of "regular" bus type stops is 2 or less. Neighborhood N2 defines a bus state in which the distance between stops of a series of stops where a route bus of "express" bus type stops is 1 or less. Because the distance between stops of a "express" bus type bus is greater than that of a "regular" bus type bus, a smaller value of 1 or less than that of a "regular" bus is used. In this manner, distances that take into account the operation of stops can be taken into account. Note that in this embodiment, the distance between stops based on the stop adjacency data 1600 is used, but physical distances (e.g., in units of kilometers) may also be used. Other distances defined by mathematical expressions may also be used.

[0120] Furthermore, buses traveling on the same route with similar departure times are considered to share a similar background. Therefore, neighborhood N3 defines buses with the same route ID and different route buses with departure times from their first stop that are delta1 or less as being in the vicinity of the same stop. Note that this only applies if the dates match. In addition, buses with different departure times that travel near or stop at the same stop at similar times are also considered to share a similar background. Therefore, neighborhood N4 defines buses with the same stop as being in the vicinity of the same route buses on the same date where the absolute value of the difference between the departure time and the sum of the elapsed time (i.e., the time the bus was there) is delta1 or less, regardless of the route ID.

[0121] 17, predictors P1 and P2 are defined, the predictor P1 is assigned to the neighbors N1 and N2, and the predictor P2 is assigned to the neighbors N3 and N4. As described above, in this embodiment, by combining the neighborhood definition information 1700 and the stop neighbor data 1600, it is possible to perform self-supervised learning using one or more of various neighborhood relationships.

[0122] <Machine learning model structure> 18 shows an example of a self-supervised learning model 1800, which is a machine learning model learned by the upstream task learning unit 212. The self-supervised learning model 1800 corresponds to the upstream model of the present invention.

[0123] The self-supervised learning model 1800 includes an instance pair sampler 1801, a state encoder 1802, and an embedding amount predictor 1803. The instance pair sampler 1801 samples an instance pair (instance A, B) based on the neighborhood definition information 1700. The state encoder 1802 calculates an embedding amount A based on the position and speed of the sampled instance A. The state encoder 1802 also calculates an embedding amount B based on the position and speed of the sampled instance B. The embedding amount predictor 1803 calculates an estimated embedding amount B by estimating the embedding amount B from the embedding amount A.

[0124] 19 shows an example of a delay time prediction model 1900, which is a regression model learned by the downstream task learning unit 214. The delay time prediction model 1900 corresponds to the downstream model of the present invention.

[0125] The delay time prediction model 1900 includes a state encoder 1901 and a delay time predictor 1902. The state encoder 1901 has the same structure as the state encoder 1802 of the self-supervised learning model 1800, and is initialized using a trained state encoder 1802. The delay time predictor 1902 outputs a predicted delay time at the next stop s based on the bus states from three stops before to just before (just before) a certain route bus, and the bus state just before (just before) the preceding route bus.

[0126] <Learning process by learning device 21> Next, a description will be given of the learning process by the learning device 21. This learning process is similar to the learning process by the learning device 11 in the first embodiment (FIG. 9), and therefore FIG. 9 will be used and some omissions will be made as appropriate.

[0127] As a premise for this learning process, it is assumed that the bus information data management unit 74 of the bus information management server 70 is already managing bus status data 1500 (Figure 15) to be used for self-supervised learning for a predetermined period (e.g., one month) collected from each route bus 80 by the data collection unit 71.

[0128] The learning process is started in response to a predetermined start operation by the user 121 using the operation terminal 12, for example.

[0129] First, to prepare data for self-supervised learning, the overall task manager 211 reads out the bus state data 1500 managed by the bus information data manager 74 and copies it under the management of the learning data manager 216 of the learning device 21 (step S1).

[0130] Next, the overall task manager 211 generates hyperparameters to be used in the upstream task learning process (self-supervised learning) in step S3 and the downstream task learning process in step S4 (step S2). Specifically, the hyperparameters are generated by randomly sampling delta1, a parameter described in the neighborhood definition information 1700 (FIG. 17), from the described range. Note that in this embodiment, only the aforementioned one parameter is targeted for tuning, but various hyperparameters may also be targeted, such as the network structure, the number of mini-batches, the learning rate, and other training conditions.

[0131] Next, the upstream task learning unit 212 executes an upstream task learning process to generate a state encoder 1802 of the self-supervised learning model 1800 (FIG. 18) using the bus state data 1500, the neighborhood definition information 1700, and the hyperparameters generated in step S2 (step S3).

[0132] An example of the upstream task learning process in step S3 will be described with reference to FIG.

[0133] First, the upstream task learning unit 212 selects one of the neighborhoods N1 to N4 defined in the neighborhood definition information 1700 (step S31).

[0134] Next, the upstream task learning unit 212 samples pairs of instances that match the neighborhood definition selected in step S31 in accordance with the neighborhood definition information 1700 and the hyperparameters (step S32).

[0135] Next, the upstream task learning unit 212 uses the self-supervised learning model 1800 to calculate information on items excluding the delay time of each instance, namely, the route ID, bus type, departure time of the first stop, date, elapsed time, stop ID, average speed, and congestion level, as well as the embedding amount A and embedding amount B, which is an estimate of the embedding amount B estimated from the embedding amount A (step S33).

[0136] Next, the upstream task learning unit 212 calculates the loss of the embedding amount B and the estimated embedding amount B using the cosine similarity multiplied by -1 as a loss function, and updates the parameters of the embedding amount predictor 1803 and the state encoder 1802 regarding the position and velocity A using the backpropagation method (step S34).

[0137] The embedding amount predictor 1803 follows the correspondence between predictors and neighborhoods defined in the neighborhood definition information 1700. That is, there are two predictors P1 and P2, and the predictor P1 is associated with neighborhoods N1 and N2, and the predictor P2 is associated with neighborhoods N3 and N4. Since the predictor can be changed depending on the neighborhood in this way, an error signal for providing an expression for each neighborhood can be sent to the state encoder 1802.

[0138] Next, the upstream task learning unit 212 determines whether the learning end condition has been met (step S35), and if it determines that the learning end condition has not been met (NO in step S35), the process returns to step S31 and the processes from step S31 onwards are repeated. After that, if the upstream task learning unit 112 determines that the learning end condition has been met (YES in step S35), the upstream task learning process is ended.

[0139] In this way, in this embodiment, various instances can be generated. Furthermore, even for targets for which known data extension methods are not clear, such as enlarging, reducing, cropping, etc. for image data, or cutting or combining audio data, and even for data for which mixup, etc., cannot be used because the data contains categorical variables, etc., self-supervised learning can be performed by sampling. Note that, although not used in this embodiment, instances may be generated using simulation, as used in the first embodiment.

[0140] Returning to Fig. 9, next, the downstream task learning unit 214 copies the parameters in the state encoder 1802 of the self-supervised learning model 1800 generated in step S3 to the state encoder 1901 of the delay time prediction model 1900, and then executes downstream task learning processing for learning the delay time prediction model 1900 (step S4).

[0141] FIG. 20 is a flowchart showing an example of the downstream task learning process in step S4.

[0142] First, the downstream task learning unit 214 copies the parameters of the state encoder 1802 of the self-supervised learning model 1800, which has been pre-trained by the upstream task learning process in step S3, to the state encoder 1901 of the delay time prediction model 1900. Other parameters are initialized using a general method such as normal distribution (step S401).

[0143] Next, the downstream task learning unit 214 samples one instance of a certain bus state data and designates it as a base sample. The downstream task learning unit 214 also acquires the instance immediately preceding the base sample and the instance two instances prior to the base sample, among the instances that have the same route ID, bus type, first stop departure time, and date as the base sample. Furthermore, the downstream task learning unit 214 acquires the instance with the longest elapsed time (latest) among the instances that have the same route ID, bus type, and date as the base sample and have the first stop departure time immediately preceding the base sample. Furthermore, the downstream task learning unit 214 defines the number of stops that exist after the base sample as S, samples one value from integers between 1 and S, and sets this as s. These values ​​are combined and used as input to the state encoder 1901 of the delay time prediction model 1900. Furthermore, the delay time of the stop s ahead of the base sample is acquired and set as the output (the predicted delay time of the sth stop). A mini-batch is generated using these input-output pairs as elements (step S402).

[0144] If there is no previous instance, such as the first stop, an instance can be generated by reusing the base sample values ​​for route ID, bus type, first stop departure time, and date, and setting the average speed and congestion level to 0. Similarly, if there is no earlier route bus, an instance can be generated by reusing the base sample values ​​for route ID, bus type, and date, and setting the first stop departure time to 0:00, average speed, and congestion level to 0.

[0145] Next, the downstream task learning unit 214 calculates a predicted delay time from the input mini-batch using the delay time prediction model 1900 (step S403). Next, the downstream task learning unit 214 calculates a loss from the delay time (true value) and the predicted delay time (estimated value), and updates the parameters of the delay time prediction model 1900 by the backpropagation method (step S405). Here, the loss is based on the mean square error.

[0146] Next, the downstream task learning part 214 The downstream task learning unit 114 determines whether the learning end condition has been met (step S405). Here, the learning end condition is defined as, for example, repeating the processes of steps S402 to S405 a predetermined number of times (for example, 1000 times). If the downstream task learning unit 114 determines that the episode end condition has not been met (NO in step S405), the process returns to step S402, and step S402 and subsequent steps are repeated.

[0147] Thereafter, if it is determined that the learning termination condition has been met (YES in step S405), the downstream task learning process is terminated. According to the downstream task learning process, it is possible to learn a model that predicts the delay time at any subsequent stop (s stops ahead) from information about the route bus whose delay time is to be predicted (three stops) and information about the preceding route bus (one stop).

[0148] Return to FIG. 9. Next, the overall task management unit 211 outputs to the learning data management unit 216, as evaluation results, the transition of the loss of each predictor during self-supervised learning and the success rate of episodes in reinforcement learning, and the learning data management unit 216 stores them in the storage 103 or the like. Further, the learning data management unit 216 temporarily stores the learned self-supervised learning model 1800 and the delay time prediction model 1900 in the storage 103 or the like (step S5).

[0149] Next, the overall task management unit 211 determines whether or not a predetermined tuning end condition has been achieved (step S6). If the overall task management unit 211 determines that the predetermined tuning end condition has not been achieved (NO in step S6), the process returns to step S2, and the processes of steps S2 to S5 are repeated.

[0150] After that, when the overall task management unit 211 determines that the predetermined tuning end condition has been achieved (YES in step S6), next, the overall task management unit 211 outputs, as the final model, the model with the best evaluation result (prediction accuracy) among the models so far to the learning data management unit 216 as the related evaluation result, and the learning data management unit 216 stores this in the storage 103 or the like. Then, the data of the models and evaluation results temporarily saved in step S5 is deleted from the storage 103 or the like (step S7 in FIG. 9). Thus, the learning process is completed.

[0151] <UI screen> In the second embodiment, as in the first embodiment, when the upstream task learning process in step S3 described above is being executed, the display management unit 215 can display the upper task learning loss transition confirmation screen 1000 (FIG. 12) and the parameter confirmation screen 1100 (FIG. 13) on the operation terminal 12.

[0152] FIG. 21 shows a display example of the predicted delay time display screen 2000 displayed on the information terminal 91 of the bus user 90 under the control of the delay time distribution unit 73 of the bus information management server 70.

[0153] The predicted delay time display screen 2000 has input fields 2001 to 2003 for selecting and specifying the route ID, bus type, and departure time of the route bus for which the predicted delay time is to be displayed. Furthermore, the predicted delay time display screen 2000 has a status display field 2004 for displaying the stop that the route bus specified by the input into the input fields 2001 to 2003 has just passed, and a predicted delay time display field 2005 that shows the predicted delay time at each stop. The time displayed in the predicted delay time display field 2005 is the value calculated using the delay time prediction model 1900, rounded up to the nearest minute.

[0154] In the figure, bus user 90 has selected route ID "A1," bus type "regular," and first departure time "6:40," and the corresponding route bus has currently passed stop "104," with predicted delay times at stops "105" to "109" being 5 minutes, 3 minutes, 3 minutes, 4 minutes, and 2 minutes, respectively.

[0155] According to the second embodiment, it is possible to predict the delay time at each stop of a route bus. Note that, since upstream task learning is performed prior to downstream task learning for training a regression model for delay time prediction, the accuracy of the regression model can be improved.

[0156] The present invention is not limited to the above-described embodiments and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with or add to the configuration of another embodiment.

[0157] Furthermore, some or all of the aforementioned configurations, functions, processing units, processing means, etc. may be implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the aforementioned configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as programs, tables, and files that implement each function may be stored in memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD. Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0158] 1. Robot Management System 10. Learning System 11. Learning device 111 Overall Task Management Department 112···Upstream task learning part 113 Simulation execution unit 114···Downstream task learning part 115...Display management department 116···Learning Data Management Unit 12. Operation terminal 121 users 2. Bus operation management system 20. Learning System 21. Learning device 211 Overall Task Management Department 212···Upstream task learning part 214···Downstream task learning part 215...Display management department 216···Learning data management section 30. Robot arm control device 31 Planning Department 32 Data collection section 33 Control data management section 40. Robot arm 60 Neighborhood definition information 70 Bus information management server 71 Data Collection Section 72...Delay time prediction section 73 Delay time distribution section 74 Bus Information Data Management Unit 80···Local bus 90 Bus users 91···Information terminal 100...Computer 101 Processor 102...Memory 103 Storage 104 Input Device 105 Output Device 106 Communication module 107···Program 300 Robot status data 400 Cartesian position and velocity data 500 Neighborhood definition information 600···Self-supervised learning model 601···Instance Pair Sampler 602 State Encoder 603 Filling amount predictor 700···Actor model for reinforcement learning 701 Status Encoder 702···Actor 800···Critic model for reinforcement learning 801 State Encoder 802···Critic 1000...Top task learning loss trend confirmation screen 1100 Parameter confirmation screen 1500 Bus status data 1600...Stop adjacent data 1700 Neighborhood definition information 1800···Self-supervised learning model 1801···Instance Pair Sampler 1802 State Encoder 1803 Filling amount predictor 1900···Delay time prediction model 1901 State Encoder 1902 Delay Time Predictor 2000···Predicted delay time display screen

Claims

1. A learning device for constructing a machine learning model for tabular data, A computing device that executes predetermined processing and a storage device that can be accessed by the computing device, an upstream task learning unit configured to generate an upstream model for an upstream task by self-supervised learning based on neighborhood definition information that defines a neighborhood relationship between pairs of instances of the tabular data; the computing device includes a downstream task learning unit that generates a downstream model for a downstream task by supervised learning, reinforcement learning, or unsupervised learning; A learning device comprising:

2. The learning device according to claim 1 , the tabular data has at least one data item representing the date and time of the instance; The neighborhood definition information defines the neighborhood relationship in a time axis direction, The upstream task learning unit selects or generates the instance to be used in the self-supervised learning based on the neighborhood definition information. A learning device characterized by:

3. The learning device according to claim 1 , the tabular data has at least one column indicating the location of the instance; The neighborhood definition information defines the neighborhood relationship in terms of physical distance, The upstream task learning unit selects or generates the instance to be used in the self-supervised learning based on the neighborhood definition information. A learning device characterized by:

4. The learning device according to claim 1 , the computing device has a simulation execution unit that generates the tabular data by simulation, The upstream task learning unit uses the tabular data generated by the simulation execution unit in addition to the observed tabular data for the self-supervised learning. A learning device characterized by:

5. The learning device according to claim 4, the tabular data has at least one column representing the date and time of the instance; The simulation execution unit generates the tabular data by performing a simulation using a certain instance as an initial state. A learning device characterized by:

6. The learning device according to claim 1 , The machine learning model used for the upstream task includes an encoder that converts the instances into a representation, and one or more predictors that convert an output of the encoder into another representation; the self-supervised learning of the machine learning model is to learn parameters of the encoder and the predictor such that, for a pair of instances used in the self-supervised learning, a representation obtained by inputting one of the instances to the encoder and another representation obtained by inputting the other of the instances to the encoder approaches a representation obtained by inputting the other of the instances to the encoder; The neighborhood definition information defines the predictor and the method for selecting the pair of instances. A learning device characterized by:

7. The learning device according to claim 1 , The neighborhood definition information specifies a range of hyperparameters related to generation of instances used in the self-supervised learning, The computing device has an overall task management unit that searches for the hyperparameters in accordance with the ranges of the hyperparameters specified in the neighborhood definition information. A learning device characterized by:

8. The learning device according to claim 2, The tabular data is observation data when a plurality of machines perform work in accordance with the same type of work instruction, The upstream task learning unit selects or generates the pair of instances to be used in the self-supervised learning based on a distance in relative time from when the same type of work instruction is received. A learning device characterized by:

9. The learning device according to claim 3, the tabular data is observation data of a moving body traveling on a predetermined route while stopping at stops on the route and moving to a target; The upstream task learning unit selects or generates the pair of instances to be used in the self-supervised learning based on the distance between the stops on the route. A learning device characterized by:

10. The learning device according to claim 9, The distance between the stops is calculated based on the driving type of the mobile object. A learning device characterized by:

11. A learning method for a learning device for constructing a machine learning model for tabular data, comprising: the learning device includes a calculation device that executes predetermined processing and a storage device that is accessible by the calculation device; The learning method includes: an upstream task learning step in which the computing device generates an upstream model for an upstream task by self-supervised learning based on neighborhood definition information that defines a neighborhood relationship between pairs of instances of the tabular data; a downstream task learning step in which the computing device generates a downstream model for the downstream task by supervised learning, reinforcement learning, or unsupervised learning; A learning method comprising:

Images