Learning apparatus and learning method

Abstract

To execute self-supervised learning capable of accepting a generation strategy of various instances for tabular data in which data of heterogeneous formats such as numeric data, categorical data or the like are mixed.SOLUTION: A learning apparatus is a learning apparatus for constructing a machine learning model for tabular format data, and includes: an arithmetic device that executes predetermined processing; a storage device that can be accessed by the arithmetic device; an upstream task learning unit in which the arithmetic device generates an upstream model relating to an upstream task by self-supervised learning based on neighborhood definition information defining a neighborhood relation of a pair of instances of the tabular format data; and a downstream task learning unit in which the arithmetic device generates a downstream model relating to a downstream task by supervised learning, reinforcement learning or unsupervised learning.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning device and a learning method. [Background technology]

[0002] With the recent development of IoT, applications of artificial intelligence, especially machine learning, are expanding in various fields. For example, in the manufacturing field, the use of robots is expected to increase more than ever before, and attempts are being made to automatically acquire motion plans using reinforcement learning. In the transportation field, for example, services that predict bus delay times using supervised learning are also being provided.

[0003] On the other hand, in order to build a highly accurate machine learning model, a huge amount of training data is often required. However, in reality, it is sometimes difficult to collect a sufficient amount of training data due to the difficulty of collecting data in a real environment and labeling it.

[0004] To deal with such situations, self-supervised learning, a type of pre-learning, is used. Self-supervised learning is a method of pre-learning useful representations without trial and error or labeling on the actual machine by automatically generating training data from input data. One simple method is to input two highly similar instances into the same neural network for deep learning and learn to make the resulting representations (features) closer together. Often, the two highly similar instances are created by applying different data augmentations to instances sampled from a dataset. This approach is also called self-supervised representation learning, and whether or not a good representation can be obtained depends on the generation strategy of the two highly similar instances to be input.

[0005] For example, Patent Document 1 describes a learning device that includes a self-learning unit that updates the content of a main conversion process that converts data to be processed into data of a predetermined format by performing self-supervised learning, and a data extension unit that executes a data extension process that generates data to be processed by the main conversion process based on an acoustic time series, in which the data extension unit performs an acoustic time series extraction process that extracts a partial time series that is a part of the acoustic time series, a duplication process that duplicates the partial time series, and a conversion process that converts one side of the partial time series and the other side in accordance with a predetermined rule, and the self-learning unit updates the content of the main conversion process by self-supervised learning based on the results obtained by the conversion process. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2022 / 244047 Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 enables self-supervised learning that takes into account similarities over long time intervals. However, the technology described in Patent Document 1 incorporates conversion processing for acoustic time series based on the knowledge of the inventors, and does not target tabular data that contains a mixture of data in different formats, such as numerical data and categorical data.

[0008] For this reason, for example, with regard to data obtained for each bus stop (hereafter referred to as a stop), it is difficult to obtain a common representation for instances of the same bus at different stops (represented by discrete values ​​such as IDs or names) or for the same stop on different buses.

[0009] Furthermore, as mentioned above, whether or not a good representation can be obtained depends on the generation strategy for the two highly similar input instances. However, the technology described in Patent Document 1 does not take into consideration the fact that a user may search for a strategy for generating instances that bring the representations closer together through trial and error, or that a computer may search mechanically for such a strategy.

[0010] The present invention has been made in consideration of the above-mentioned problems, and aims to make it possible to perform self-supervised learning that allows for various instance generation strategies for tabular data that contains a mixture of different types of data, such as numerical data and categorical data. [Means for solving the problem]

[0011] The present application includes a number of means for solving at least some of the above-mentioned problems, examples of which are as follows.

[0012] In order to solve the above-mentioned problems, a learning device according to one embodiment of the present invention is a learning device for constructing a machine learning model for tabular data, comprising a calculation device that executes a predetermined process and a storage device accessible to the calculation device, characterized in that the calculation device has an upstream task learning unit that generates an upstream model for an upstream task by self-supervised learning based on neighborhood definition information that defines a neighborhood relationship between pairs of instances of the tabular data, and a downstream task learning unit that generates a downstream model for a downstream task by supervised learning, reinforcement learning, or unsupervised learning. Effect of the Invention

[0013] According to the present invention, it is possible to perform self-supervised learning that allows various instance generation strategies for tabular data that contains a mixture of data in different formats, such as numerical data and categorical data. This makes it possible to acquire expressions that take into account not only temporal neighborhoods (similarity) but also positional neighborhoods (similarity), etc.

[0014] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a robot management system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a typical computer. [Diagram 3] FIG. 3 is a diagram illustrating an example of the robot status data. [Figure 4] FIG. 4 is a diagram showing an example of the orthogonal system position and velocity data. [Diagram 5] FIG. 5 is a diagram illustrating an example of neighborhood definition information. [Figure 6] FIG. 6 is a diagram illustrating an example of a structure of a self-supervised learning model. [Figure 7] FIG. 7 is a diagram showing an example of the structure of an actor model for reinforcement learning. [Figure 8] FIG. 8 is a diagram illustrating an example of the structure of a critic model for reinforcement learning. [Figure 9] FIG. 9 is a flowchart showing an example of the learning process. [Figure 10] FIG. 10 is a flowchart illustrating an example of an upstream task learning process. [Figure 11] FIG. 11 is a flowchart illustrating an example of a downstream task learning process. [Figure 12] FIG. 12 is a diagram showing an example of a display of a higher-order task learning loss transition confirmation screen. [Figure 13] FIG. 13 is a diagram showing a display example of the parameter display screen. [Figure 14] FIG. 14 is a diagram showing an example of the configuration of a bus operation control system according to the second embodiment of the present invention. [Figure 15] FIG. 15 is a diagram illustrating an example of the bus state data. [Figure 16] FIG. 16 is a diagram illustrating an example of the stop adjacent data. [Figure 17] FIG. 17 is a diagram illustrating an example of neighborhood definition information. [Figure 18] FIG. 18 is a diagram illustrating an example of a structure of a self-supervised learning model. [Figure 19] FIG. 19 is a diagram illustrating an example of the structure of a delay time prediction model. [Figure 20] FIG. 20 is a flowchart illustrating an example of the downstream task learning process. [Figure 21] FIG. 21 is a diagram showing a display example of the predicted delay time display screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, a number of embodiments of the present invention will be described with reference to the drawings. In all the drawings for describing each embodiment, the same components are generally given the same reference numerals, and the repeated description thereof will be omitted. In addition, in the following embodiments, the components (including element steps, etc.) are not necessarily essential, except when specifically indicated, or when it is clearly considered essential in principle. In addition, when it is said that "consists of A," "consists of A," "has A," or "includes A," it does not exclude other elements, except when it is specifically indicated that only that element is included. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., it includes those that are substantially similar or similar to the shape, etc., except when it is specifically indicated, or when it is clearly considered not to be the case in principle.

[0017] <Robot management system 1 according to the first embodiment of the present invention> FIG. 1 shows an example of the configuration of a robot management system 1 according to a first embodiment of the present invention.

[0018] <Summary> The robot management system 1 includes a learning system 10 related to motion planning (planning of trajectories and gripping) of a robot arm in a factory. First, an outline of the overall processing flow of the robot management system 1 including the learning system 10 will be described. Generally, a simulator corresponding to the arm is prepared for the robot arm 40, and the motion can be planned using a planning engine using random sampling (hereinafter referred to as an existing motion planner). However, there is a considerable discrepancy between the simulator and the actual machine due to the inclination of the robot arm 40 when it is installed, individual differences of the robot arm 40, deflection due to the weight of the gripped workpiece, etc.

[0019] The learning system 10 uses data obtained from an actual machine and reinforcement learning to construct a motion planning function that further reduces the deviation. In this case, the number of trial and error steps on the actual machine for reinforcement learning is reduced by performing pre-learning through self-supervised learning that utilizes data obtained using an existing motion planner and simulation data.

[0020] In preparation for self-supervised learning, a time series of robot states (robot state data) for self-supervised learning is collected using an existing motion planner mounted on each robot arm 40 and the planning unit 31 of the robot arm control device 30. It is assumed that multiple robot arms 40 are performing the same type of tasks asynchronously.

[0021] <Configuration example> Next, a description will be given of an example of the configuration of the robot management system 1. The robot management system 1 includes a learning system 10, a robot arm control device 30, and a plurality of robot arms 40.

[0022] The learning system 10 and the robot arm control device 30 are each connected to an information network NW1, and communicate various data and information with each other via the information network NW1. The robot arm control device 30 and the robot arm 40 are connected to a control network NW2, and the robot arm control device 30 transmits a control signal to the robot arm 40 via the control network NW2, and the robot arm 40 transmits status data to the robot arm control device 30 via the control network NW2.

[0023] Although the robot management system 1 is assumed to be constructed in a closed environment, the information network NW1 and the control network NW2 may include an open WWW (World Wide Web) network as a route between them. In addition, if real-time performance can be guaranteed, the information network NW1 and the control network NW2 may be the same network.

[0024] The learning system 10 includes a learning device 11 and an operation terminal 12. However, the learning device 11 and the operation terminal 12 may be integrated together.

[0025] The learning device 11 has the following functional blocks: an overall task management unit 111, an upstream task learning unit 112, a simulation execution unit 113, a downstream task learning unit 114, a display management unit 115, and a learning data management unit 116. The learning device 11 is realized by a general computer such as a personal computer or a server computer.

[0026] 2 shows an example of the configuration of a general computer 100 that realizes the learning device 11. The computer 100 includes a processor 101 such as a CPU (Central Processing Unit), a memory 102 such as a DRAM (Dynamic Random Access Memory), a storage 103 such as a HDD (Hard Disk Drive) or SSD (Solid State Drive), an input device 104 such as a keyboard, a mouse, or a media drive, an output device 105 such as a display, and a communication module 106 such as an Ethernet (registered trademark) card or a Wi-Fi (registered trademark) adapter.

[0027] The overall task management unit 111, upstream task learning unit 112, simulation execution unit 113, downstream task learning unit 114, display management unit 115, and learning data management unit 116 of the learning device 11 are realized by the processor 101 of the computer 100 executing the program 107 stored in the memory 102.

[0028] The program 107 executed by the processor 101 may be stored in advance in the memory 102, or may be downloaded from a predetermined server or the like via a removable medium (CD-ROM, flash memory, etc.) or a network such as the Internet, stored in the storage 103, which is a non-transitory storage medium, and read out from the storage 103 when necessary. For this reason, the learning device 11 (computer 100) should preferably have an interface for reading data from removable media.

[0029] Furthermore, the learning device 11 may be a computer system configured on one physical computer, or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, multiple programs that realize the functions of the learning device 11 may each operate on a separate physical or logical computer, or multiple programs may be combined to operate on a single physical or logical computer.

[0030] Returning to Fig. 1, the overall task manager 111 controls the entire learning device 11.

[0031] The upstream task learning unit 112 performs self-supervised learning using the robot state data 300 (FIG. 3) under the control of the overall task management unit 111. In the self-supervised learning by the upstream task learning unit 112, for example, a network employing the "SimSiam" architecture is used to avoid the collapse of the representation due to the stop gradient. This network includes a state encoder that converts the position and speed obtained from the robot state data 300 into a certain representation (feature amount), and a predictor (projection head) that predicts another feature amount from a certain feature amount.

[0032] Specifically, the upstream task learning unit 112 prepares a pair of instances (a first instance and a second instance) of the robot state data 300 in accordance with neighborhood definition information 500 (FIG. 5) that is predefined and managed by the learning data management unit 116, and learns a task of predicting the expression of the second instance from the expression (feature) obtained from the first instance (described in detail later).

[0033] The simulation execution unit 113 executes a simulation of a future or past state of a robot, with a first instance of a certain robot state data as an initial state, under the control of the overall task management unit 111. The result of this simulation is used as a third instance when predicting an expression of the third instance from an expression (feature amount) similarly obtained from the first instance, or when learning a task of predicting an expression of the first instance from an expression of the third instance.

[0034] Instead of using "SimSiam" for self-supervised learning, other methods such as "BYOL" or "SimCLR" that requires negative examples may be used. When negative examples are required, they can be handled by using, for example, randomly selected instances. In addition, in this embodiment, as in "SimSiam", the asymmetric structure by the predictor and the collapse of the expression obtained by the stop gradient are avoided, but a method of avoiding another collapse, such as utilizing the time average of the encoder as in "BYOL" or decorrelation by whitening, may be used.

[0035] In addition, since the prepared instance pair has a direction in terms of time or position, and a predictor that takes the direction into account is set, in this embodiment, an asymmetric objective function is optimized instead of optimizing a symmetric objective function of two data-augmented instances as in "SimSiam". This is just one example, and a symmetric objective function may be optimized for the obtained sample. In addition, in this embodiment, the instance pair that brings the expressions closer together is obtained by sampling or simulation, but a commonly used data augmentation method may be used, or both may be combined.

[0036] The downstream task learning unit 114 learns a new planning engine (RL motion planner) using reinforcement learning under the control of the overall task management unit 111. The downstream task learning unit 114 uses an actor model and a critic model for the reinforcement learning. The actor model takes the position, speed, gripping state, and target state as inputs and outputs an action. The critic model takes the position, speed, gripping state, target state, and action as inputs and outputs a Q value that indicates the goodness of the action. The reinforcement learning uses a deep deterministic policy gradient (DDPG). The neural networks of both the actor model and the critic model include a state encoder learned by self-supervised learning. That is, they use the representation of the position and speed that have been learned by self-supervised learning, which is an upstream task. In this embodiment, reinforcement learning is performed separately for each of the multiple robot arms 40, but in the early stage of learning, the multiple robot arms 40 may be operated in parallel and the learning results may be shared as appropriate. Even in this case, in the final stage of learning, they are individually trained taking into account the installation and individual differences.

[0037] The display management unit 115 causes the operation terminal 12 to display a UI (User Interface) screen.

[0038] The learning data management unit 116 appropriately stores and manages the robot state data for self-supervised learning, the transition of the loss function during learning of the upstream task and the downstream task, performance indicators such as the difference between the predicted delay time and the actual delay time, the pre-trained model, the obtained regression model, etc. in the storage 103 of the computer 100 that realizes the learning device 11.

[0039] Operation terminal 12 is realized by a general computer 100 (FIG. 2), similar to study device 11. Operation terminal 12 displays a UI screen and accepts input from user 121 on the UI screen.

[0040] The robot arm control device 30 is realized by a general computer 100 (FIG. 2), similar to the learning device 11. The robot arm control device 30 has the functional blocks of a planner 31, a data collector 32, and a control data manager 33. Each functional block of the robot arm control device 30 is realized by a processor 101 of the computer 100 executing a program 107 stored in a memory 102.

[0041] The planning unit 31 uses a built-in existing motion planner to create a motion plan for the robot arm 40 to collect robot state data 300 for self-supervised learning. Specifically, the planning unit 31 randomly determines the target state, i.e., the arm tip position and grip state on the Cartesian coordinate system, to create an existing motion plan, performs inverse dynamics calculations, and controls the joint angles to create a motion plan for transitioning the robot arm 40 to the target state.

[0042] The data collection unit 32 collects robot status data 300 from the robot arm 40 that operates according to an operation plan. The control data management unit 33 stores the collected robot status data 300 in the storage 103 or the like of the computer 100 that realizes the robot arm control device 30, and manages it.

[0043] The robot arm 40 employs, for example, an arm with six degrees of freedom and a gripper mounted at the tip thereof. However, the degrees of freedom of the arm may be more than six, and, for example, a suction hand may be mounted at the tip of the arm.

[0044] <Data structure of various data> Next, FIG. 3 shows an example of the data structure of robot state data 300 managed by the control data management unit 33 of the robot arm control device 30. As shown in FIG.

[0045] The robot status data 300 includes the date and time when the data was observed by the robot arm 40, a robot ID for identifying the robot arm 40, angles of joints 1 to 6 of the robot arm 40, a gripping state indicating the open / closed state of the gripper mounted on the tip of the arm, and the like, all recorded in association with each other. Note that in this embodiment, the robot status data 300 is periodically observed. In the case of the figure, the observation period is one second.

[0046] 4 shows an example of the data structure of cartesian position and velocity data 400 generated by the overall task manager 111 based on the robot state data 300. The cartesian position and velocity data 400 corresponds to the tabular data of the present invention. The tabular data is a set of instances made up of tuples made up of multiple pieces of data such as numerical data and categorical data.

[0047] The Cartesian position and velocity data 400 includes, in association with each other, the standardized time (chronological order) at which the data was observed, a robot ID for identifying the robot arm 40, the position (Cartesian coordinate system) of the gripper at the end of the arm, the speed (Cartesian coordinate system) of the gripper at the end of the arm, and a gripping state indicating whether the gripper is open or closed. The position and speed (Cartesian coordinate system) of the gripper at the end of the arm are calculated based on the angles (joint coordinate system) of joints 1 to 6 of the robot arm 40. Depending on the type of hand mounted at the end of the arm, the gripping state may include information indicating the orientation in addition to open or closed.

[0048] Next, FIG. 5 shows an example of predefined neighborhood definition information 500 managed by the learning data management unit 116.

[0049] The neighborhood definition information 500 is a script written in a domain-specific language that describes neighborhoods for sampling pairs of instances used in self-supervised learning and assignment of predictors to the neighborhoods.

[0050] In the neighborhood definition information 500, prior to the definition of the neighborhood, parameters and variables used in the definition of the neighborhood are declared in advance as "parameter" and "variable". A range can be set for the parameters. This is used for automatic hyperparameter tuning in the process described later. For the variables, sampling from the data set and simulation data using the simulation execution unit 113 of the learning device 11 can be used.

[0051] In the example of FIG. 5, four types of neighborhoods N1, N2, N3, and N4 are defined. Since the robot arm operates continuously, it is considered that robot arms that are close in time share a similar background (i.e., a preferable expression that contributes to improving generalization performance). Therefore, the neighborhoods N1 and N2 are defined as instances of the same robot arm 40 that are close in time. However, the neighborhoods N1 and N2 have different time axis directions, and the neighborhood N1 defines an instance forward in time as the neighborhood, and the neighborhood N2 defines an instance backward in time as the neighborhood.

[0052] In addition, the simulation is expected to predict the near future of the robot arm with high accuracy. Therefore, the neighborhood N3 is defined as an instance at a certain time and an instance of the predicted state at a time a little further ahead from that time obtained by the simulation. Note that only forward time development is defined because general simulators often only handle forward time development.

[0053] In addition, even if different robots are performing the same task, they will perform almost the same actions. Therefore, if the elapsed time from the start time is the same, it is considered that they share a similar background. Therefore, for neighborhood N4, we define instances within 0.1 seconds of the time (standardized time) when the task start time is set to 0 for different robots as neighborhood.

[0054] Furthermore, in the example of FIG. 5, predictors P1 and P2 are defined, predictor P1 is assigned to neighborhoods N1 and N3, predictor P2 is assigned to neighborhood N2, and predictor P3 is assigned to neighborhood N4.

[0055] In addition, while "SimSiem" and the like have only one predictor, in this embodiment, the neighborhood definition information 500 can be used to assign the same predictor or different predictors to different types of neighborhoods. In other words, it is possible to avoid a situation in which the task of the predictor becomes too difficult when a predictor is shared, such as a predictor that predicts the future from the past or a predictor that predicts the past from the future, resulting in poor expressiveness of the resulting expression. In addition, since the script is separated from the core program of machine learning, the user 121 can easily try and error to search for the best neighborhood definition for the target problem and to search for the degree of utilization of simulation data.

[0056] <Machine learning model structure> 6 shows an example of a self-supervised learning model 600, which is a machine learning model that is learned by the upstream task learning unit 112 through self-supervised learning. The self-supervised learning model 600 corresponds to the upstream model of the present invention.

[0057] The self-supervised learning model 600 includes an instance pair sampler 601, a state encoder 602, and an embedding amount predictor 603. The instance pair sampler 601 samples an instance pair (instance A, B) based on the neighborhood definition information 500. The state encoder 602 calculates an embedding amount A based on the position and speed of the sampled instance A. The state encoder 602 also calculates an embedding amount B based on the position and speed of the sampled instance B. The embedding amount predictor 603 calculates an estimated embedding amount B by estimating the embedding amount B from the embedding amount A.

[0058] 7 and 8 show examples of an actor model 700 for reinforcement learning (FIG. 7) and a critic model 800 for reinforcement learning (FIG. 8), which are machine learning models learned by the downstream task learning unit 114 through reinforcement learning. The actor model 700 for reinforcement learning and the critic model 800 for reinforcement learning correspond to the downstream model of the present invention.

[0059] The actor model for reinforcement learning 700 outputs an action for achieving a goal from the position, velocity, grip state, and goal state. The actor model for reinforcement learning 700 includes a state encoder 701 and an actor 702. The state encoder 701 has the same structure as the state encoder 602 of the self-supervised learning model 600, and is initialized using the learned state encoder 602 (the parameters of the state encoder 602 are copied). The state encoder 701 calculates an embedding amount based on the position and velocity. The actor 702 outputs an action based on the embedding amount, grip state, and goal state.

[0060] The actions output by the actor 702 are minute manipulations of the position in the Cartesian coordinate system and manipulations of the gripping state. These manipulations are converted into the joint coordinate system through inverse kinematic calculations by the planning unit 31 of the robot arm control device 30 and then executed.

[0061] The critic model 800 for reinforcement learning includes a state encoder 801 and a critic 802. The state encoder 801 has the same structure as the state encoder 602 of the self-supervised learning model 600, and is initialized using the learned state encoder 602 (the parameters of the state encoder 602 are copied). The state encoder 801 calculates an embedding amount based on the position and velocity. The critic 802 outputs a Q value representing the goodness of the action based on the embedding amount, the gripping state, the target state, and the action output by the actor 702.

[0062] <Learning process by learning device 11> FIG. 9 is a flowchart showing an example of a learning process performed by the learning device 11.

[0063] As a premise for this learning process, it is assumed that the control data management unit 33 of the robot arm control device 30 has already stored and managed the robot state data 300 (FIG. 3) used for self-supervised learning, which has been collected from each robot arm 40 by the data collection unit 32, in the storage 103 or the like.

[0064] The learning process is started in response to a predetermined start operation by the user 121 using the operation terminal 12, for example.

[0065] First, the overall task manager 111 reads out the robot state data 300 under the management of the control data manager 33 of the robot arm control device 30 as data preparation for use in self-supervised learning, and copies it under the management of the learning data manager 116 of the learning device 11. Then, the overall task manager 111 converts the values ​​of joint angles 1 to 6, which are the joint coordinate system of the robot state data 300, into an orthogonal coordinate system, and generates orthogonal position and velocity data 400 (FIG. 4) to which velocity has been added by calculating the difference (step S1).

[0066] Next, the overall task manager 111 generates hyperparameters to be used in the upstream task learning process (self-supervised learning) in step S3 and the downstream task learning process in step S4 (step S2). Specifically, the hyperparameters are generated by randomly sampling delta1, delta2, and delta3, which are parameters described in the neighborhood definition information 500 (FIG. 5), from the described range. Note that, although only the above-mentioned three parameters are targeted for tuning in this embodiment, various hyperparameters, such as the network structure, the number of mini-batches, the learning rate, and other training conditions, may be targeted.

[0067] Next, the upstream task learning unit 112 executes an upstream task learning process to generate a state encoder 602 of the self-supervised learning model 600 (FIG. 6) using the orthogonal position and velocity data 400, the neighborhood definition information 500, and the hyperparameters generated in step S2 (step S3).

[0068] FIG. 10 is a flowchart showing an example of the upstream task learning process in step S3.

[0069] First, the upstream task learning unit 112 selects one of the neighborhoods N1 to N4 defined in the neighborhood definition information 500 (step S31).

[0070] Next, the upstream task learning unit 112 samples a pair of instances that match the neighborhood definition selected in step S31 according to the neighborhood definition information 500 and the hyperparameters. When the sampling of the instance involves a simulation, the upstream task learning unit 112 first samples one instance from the orthogonal position and velocity data 400, and the sampled instance is used as an initial state, and the simulation execution unit 113 generates another instance based on the result of a simulation performed in the future or past at a specified time, and samples these two instances as a pair (step S32). Note that, as the behavior information of the robot arm 40 required for the simulation, the behavior information executed next to or immediately before the sampled instance may be used.

[0071] Next, the upstream task learning unit 112 uses the self-supervised learning model 600 to generate embedding amounts A and B for the position and speed of the instance pair A and B, and an estimated embedding amount B which is an estimate of the embedding amount B estimated from the embedding amount A (step S33).

[0072] Next, the upstream task learning unit 112 calculates the loss of the embedding amount B and the estimated embedding amount B using the cosine similarity multiplied by -1 as a loss function, and updates the parameters of the embedding amount predictor 603 and the state encoder 602 regarding the position and speed A by the error backpropagation method (step S34).

[0073] In this embodiment, the loss function used is the cosine similarity multiplied by -1, but other functions such as mean square error may be used as the loss function. Also, in this embodiment, the loss is related to the embedding amount A and the estimated embedding amount B in consideration of the order, but if the order is not taken into consideration, the embedding amount A and the embedding amount B may be reversed, that is, the embedding amount A may be estimated from the embedding amount B.

[0074] The embedding amount predictor 603 follows the correspondence between predictors and neighborhoods defined in the neighborhood definition information 500. That is, the neighborhood definition information 500 has three predictors P1 to P3, and the predictor P1 corresponds to the neighborhoods N1 and N3, the predictor P2 corresponds to the neighborhood N2, and the predictor P3 corresponds to the neighborhood N4. Since the predictor can be changed depending on the neighborhood in this way, an error signal for providing an expression for each neighborhood can be sent to the state encoder 602.

[0075] Next, the upstream task learning unit 112 determines whether or not the learning end condition has been met (step S35). Here, the learning end condition is defined as, for example, repeating the processes of steps S31 to S34 a predetermined number of times (for example, 10,000 times). If the upstream task learning unit 112 determines that the learning end condition has not been met (NO in step S35), the process returns to step S31, and the processes from step S31 onwards are repeated. Thereafter, if the upstream task learning unit 112 determines that the learning end condition has been met (YES in step S35), the upstream task learning process is terminated.

[0076] In this embodiment, data generation by simulation is performed on only one side of a pair of instances, but if different instances can be generated by changing the conditions related to the simulation, two instances may be generated by simulation and paired. In this way, in this embodiment, various instances can be generated. Furthermore, even for targets for which known data extension methods are not clear, such as enlargement, reduction, cropping, etc. for image data, or cutout and synthesis for audio data, self-supervised learning can be performed by sampling and simulation.

[0077] Returning to Fig. 9, the downstream task learning unit 114 then copies the parameters in the state encoder 602 of the self-supervised learning model 600 generated in step S3 to the state encoder 701 of the reinforcement learning actor model 700 and the state encoder 801 of the reinforcement learning critic model 800 (Fig. 8), and executes downstream task learning processing for learning the reinforcement learning actor model 700 and the reinforcement learning critic model 800 (Fig. 8) (step S4).

[0078] Fig. 11 is a flowchart showing an example of the downstream task learning process in step S4. Note that the downstream task learning process shown in Fig. 11 is a process related to one robot arm 40. Also, regarding observation of the state of the robot arm and application of actions, the learning device 11, the robot arm control computer, and the robot arm 40 are assumed to be executed while communicating with each other.

[0079] First, the downstream task learning unit 114 copies the parameters of the state encoder 602 of the self-supervised learning model 600, which have been pre-trained by the upstream task learning process in step S3, to the state encoder 701 of the reinforcement learning actor model 700 and the state encoder 801 of the reinforcement learning critic model 800. Other parameters are initialized using a general method such as normal distribution (step S41).

[0080] Next, the robot arm control device 30 randomly determines the initial state and target state of the robot arm 40, and transitions the state of the robot arm 40 by applying a trajectory plan obtained using an existing motion planner in the planning unit 31 while performing inverse dynamics calculations according to the determined values ​​(step S42).

[0081] Next, the downstream task learning unit 114 uses the actor model 700 for reinforcement learning to determine an action from the current state (position, speed, grip state) and the goal state (step S43). Next, the downstream task learning unit 114 applies the determined action to the robot arm 40 to transition the state. Then, the new state is observed and a reward calculation is performed (step S44). The reward function used in the reward calculation is, for example, +1 when the state approaches the goal state within a specified distance, and -0.01 when it does not.

[0082] Next, the downstream task learning unit 114 calculates a loss using the current state (position, speed, grip state), the goal state, the action, and the reward, and updates the parameters of the actor model 700 for reinforcement learning and the critic model 800 for reinforcement learning by backpropagation (step S45). Here, the loss is calculated based on the TD error in reinforcement learning.

[0083] Next, the downstream task learning unit 114 determines whether or not the episode end condition has been achieved (step S46). Here, the episode end condition is defined as, for example, the reward becoming +1, or the processing of steps S43 to S45 being repeated a predetermined number of times (for example, 1000 times) even if the reward does not become +1. If the downstream task learning unit 114 determines that the episode end condition has not been achieved (NO in step S46), the processing returns to step S43, and step S43 and subsequent steps are repeated.

[0084] Thereafter, when it is determined that the episode end condition has been achieved (YES in step S46), the downstream task learning unit 114 then determines whether or not the learning end condition has been achieved (step S47). Here, the learning end condition is defined as the processing from step S42 onwards being repeated a predetermined number of times (e.g., 1000 times). When the downstream task learning unit 114 determines that the learning end condition has not been achieved (NO in step S47), the processing returns to step S42, and steps S42 onwards are repeated. When it is determined that the learning end condition has been achieved (YES in step S47), the downstream task learning processing is terminated.

[0085] Returning to Fig. 9, next, the overall task management unit 111 outputs the transition of the loss of each predictor during self-supervised learning and the success rate of the episode in reinforcement learning as evaluation results to the learning data management unit 116, which then stores these in the storage 103 or the like. In addition, the learning data management unit 116 temporarily saves the trained self-supervised learning model 600, the reinforcement learning actor model 700, and the reinforcement learning critic model 800 in the storage 103 or the like (step S5).

[0086] Next, the overall task management unit 111 determines whether a predetermined tuning end condition has been met (step S6). Here, the predetermined tuning end condition is defined as, for example, repeating the processes of steps S2 to S5 a predetermined number of times (for example, 100 times) or the evaluation result (the transition of the success rate of episodes) being improved compared to a model created in the past.

[0087] When the overall task management unit 111 determines that a predetermined tuning end condition has not been achieved (NO in step S6), the process returns to step S2, and the processes in steps S2 to S5 are repeated. After that, when the overall task management unit 111 determines that a predetermined tuning end condition has been achieved (YES in step S6), next, the overall task management unit 111 outputs, as the final model, the model with the best evaluation result (transition of the success rate of episodes) among the models so far to the learning data management unit 116 as the relevant evaluation result, and the learning data management unit 116 stores this in the storage 103 or the like. Then, the data of the models and evaluation results temporarily saved in step S5 is deleted from the storage 103 or the like (step S7 in FIG. 9). Thus, the learning process ends.

[0088] <UI screen> Next, FIG. 12 shows a display example of the upper task learning loss transition confirmation screen 1000 as a UI screen displayed on the operation terminal 12 by the display management unit 115 when the upstream task learning process in step S3 described above is being executed. The upper task learning loss transition confirmation screen 1000 is for presenting the transition (learning progress) of the upstream task learning process to the user 121.

[0089] The upper task learning loss transition confirmation screen 1000 is provided with a loss transition display area 1001, a "neighborhood definition display" button 1002, and a "parameter display" button 1003.

[0090] In the loss transition display area 1001, a graph of the loss value versus the number of updates is displayed for each predictor defined by the neighborhood definition information 500 used to train the finally selected model stored in the learning data management unit 116. In the example shown in the figure, graphs corresponding to predictors P1 and P2 are displayed, and the graph corresponding to predictor P3 can be displayed by scrolling inside the loss transition display area 1001. By checking the loss transition display area 1001, the user 121 can determine whether or not the higher-level task learning is progressing smoothly.

[0091] The "Display Neighborhood Definition" button 1002 is a button for displaying the contents of the neighborhood definition information 500 stored in the learning data management unit 116. The "Display Parameters" button 1003 is a button for displaying a parameter confirmation screen 1100 ( FIG. 13 ) that displays the hyperparameters related to the finally selected model stored in the learning data management unit 116 and the parameters used.

[0092] 13 shows a display example of a parameter confirmation screen 1100. The parameter confirmation screen 1100 displays a parameter table 1101. The parameter table 1101 displays the parameter names and their values ​​in association with each other. For example, the example in the figure shows that 00:00:00.5, 00:00:00.3, and 00:00:00.1 have been selected as the hyperparameters delta1, delta2, and delta3, and that the learning rate for self-supervised learning (SSL learning rate) and the learning rate for reinforcement learning (RL learning rate) are both set to 1e-5.

[0093] According to the first embodiment described above, it is possible to make an operation plan that reduces the discrepancy between the simulator and the actual machine that may occur due to the inclination when the robot arm 40 is installed, the individual difference of the robot, the deflection due to the weight of the gripped workpiece, etc. In addition, since the upstream task learning is executed prior to the downstream task learning process, the number of trial and error processes on the actual machine for the downstream task learning process can be reduced.

[0094] <Bus operation control system 2 according to the second embodiment of the present invention> Next, Fig. 14 shows an example of the configuration of a bus traffic control system 2 according to a second embodiment of the present invention. Among the components of the bus traffic control system 2, the components common to the robot control system 1 (Fig. 1) according to the first embodiment of the present invention are given the same reference numerals and the description thereof will be omitted.

[0095] <Summary> The bus traffic control system 2 includes a learning system 20 for predicting delay times for route buses whose routes and timetables are set. First, an overview of the overall processing flow of the bus traffic control system 2 including the learning system 20 will be described.

[0096] In route bus timetables, time intervals may be specified for congested routes or congested time periods, but departure times are often fixed. In this embodiment, route bus 80 stops at a stop (e.g., a bus stop on a bus route) and measures and collects the departure time, average speed for the section, congestion level, and delay time when departing, and transmits them as bus status data in response to a request from bus information management server 70. The congestion level may be calculated, for example, by measuring the weight of the route bus and based on the weight, or may be measured by installing a human sensor, camera, or the like inside the bus.

[0097] The learning system 20 constructs a delay time prediction function that predicts delay times based on data obtained from route buses 80. In this case, the accuracy of the regression model for delay time prediction is improved by pre-learning using self-supervised learning that utilizes data on route buses on the same route at different times.

[0098] The learning system 20 can also be used to predict delay times for moving objects other than route buses, such as robots that transport goods within a factory or trains that transport people.

[0099] <Configuration example> Next, a description will be given of an example configuration of the bus traffic control system 2. The bus traffic control system 2 includes a learning system 20, a bus information management server 70, and a plurality of route buses 80.

[0100] The learning system 20 includes a learning device 21 and an operation terminal 12. However, the learning device 21 and the operation terminal 12 may be integrated together.

[0101] The learning system 20 and the bus information management server 70 are each connected to an information network NW1, and communicate various data and information with each other via the information network NW1. The bus information management server 70 and each route bus 80 are connected to a WWW network NW3 that allows wireless communication using a mobile phone communication network or the like. Each route bus 80 transmits bus status data to the bus information management server 70 via the WWW network NW3. In addition, bus users 90 can use an information terminal 91 such as a smartphone to connect to the bus information management server 70 via the WWW network NW3 and obtain route bus delay time information. Note that a dedicated display terminal may be provided at each stop to display route bus delay time information.

[0102] The learning device 21 is realized by a general computer 100 (FIG. 2) like the learning device 11 (FIG. 1). The learning device 21 has the functional blocks of an overall task manager 211, an upstream task learning unit 212, a downstream task learning unit 214, a display manager 215, and a learning data manager 216. The operations of the overall task manager 211, the upstream task learning unit 212, the downstream task learning unit 214, the display manager 215, and the learning data manager 216 are substantially similar to the operations of the overall task manager 111, the upstream task learning unit 112, the downstream task learning unit 114, the display manager 115, and the learning data manager 116 of the learning device 11 (FIG. 1), and therefore descriptions thereof will be omitted as appropriate.

[0103] The overall task management unit 211 controls the entire learning device 21 .

[0104] The upstream task learning unit 212 executes self-supervised learning using the bus state data. For the self-supervised learning, a network adopting the "SimSiam" architecture is used as in the first embodiment. The network includes a state encoder that converts the bus state data into a certain representation (feature amount), and a predictor that predicts another feature amount from a certain feature amount. The specific procedure of the self-supervised learning is the same as in the first embodiment.

[0105] The downstream task learning unit 214 learns a regression model that predicts the delay time using supervised learning. In the supervised learning of this embodiment, a general regression that minimizes the mean square error is used, but it is also possible to learn a model that estimates the distribution and then calculate an expected value or a value that takes risk into account.

[0106] The regression model outputs a predicted delay time at the next stop for a route bus based on the bus status at the three stops before and the bus status before the preceding bus. In this embodiment, the bus status data at the three stops before and the bus status data before the preceding bus are used as input, but bus status data from the first departure or bus status data for the same route and the same departure time on the previous day may also be used. In addition, if external information on routes such as road traffic information is available, it may be taken into consideration.

[0107] The learning data management unit 216 appropriately stores and manages in the storage 103, etc., the transition of the loss function during upstream task learning and downstream task learning, performance indicators such as the difference between the predicted delay time and the actual delay time, pre-trained models, and obtained regression models.

[0108] The bus information management server 70 is realized by a general computer 100 (FIG. 2). The bus information management server 70 includes the following functional blocks: a data collection unit 71, a delay time prediction unit 72, a delay time distribution unit 73, and a bus information data management unit 74.

[0109] The data collection unit 71 collects bus status data 1500 (FIG. 15) from the route bus 80. The delay time prediction unit 72 predicts the delay time of the route bus 80 using the delay time prediction model 1900 learned by the learning device 21. The delay time distribution unit 73 displays a predicted delay time display screen 2000 (FIG. 21) showing the predicted delay time of the route bus 80 on the information terminal 91 of the bus user 90. The bus information data management unit 74 stores the collected bus status data 1500 in the storage 103 or the like of the computer 100 that realizes the bus information management server 70.

[0110] <Data structure of various data> 15 shows an example of the data structure of bus status data 1500 managed by the bus information data management unit 74 of the bus information management server 70. The bus status data 1500 corresponds to the tabular data of the present invention. The tabular data is a collection of instances made up of tuples made up of multiple pieces of data such as numerical data and categorical data.

[0111] The bus status data 1500 includes route ID, bus type (regular (local), rapid, etc.), departure time from the first stop, date, elapsed time, stop ID, average speed, congestion, and delay time, all of which are recorded in association with each other.

[0112] In this embodiment, route buses operating on a certain date are specified by route ID, bus type, and departure time from the first stop. Route buses with the same route ID but different bus types run the same route, but "express" buses make fewer stops compared to "regular" buses. Note that buses may be distinguished by changing the route ID without having a bus type such as "express". In this embodiment, route IDs are directional (i.e., uphill / downhill, etc. are distinguished by route ID), but if there is no distinction between these, the direction of travel may be taken into consideration in addition to the type such as "regular" or "express".

[0113] 16 shows an example of the data structure of stop adjacent data 1600 managed by the learning data management unit 216. The stop adjacent data 1600 is used when sampling instances for self-supervised learning.

[0114] In the stop adjacent data 1600, route ID, bus type, stop ID-A, and stop ID-B are recorded in association with each other, and one line of information represents adjacent stops (two stops where the route bus stops in sequence) where buses with the same route ID and bus type stop in sequence. For example, it represents that a route bus with route ID "A1" and bus type "normal" stops at stop ID "102" after stop ID "101". The stop distance (StopDist) between stop IDs "101" and "102" is defined as 1. Here, the stop distance is a value obtained by adding 1 to the number of stops where the route bus does not stop that exist between the adjacent stops where the route bus stops in sequence.

[0115] Note that even if a stop is not recorded in the stop adjacent data 1600, if it can be reached by tracing the adjacent relationship from the stop adjacent data 1600, the minimum number of times of arrival is the distance between the stops. For example, in the case of the same figure, stop IDs "101" and "103" on a route bus with route ID "A1" and bus type "regular" are not adjacent, but can be traced via stop ID "102", so the distance between the stops of stop IDs "101" and "103" is defined as 2. Note that if the route IDs or bus types are different, it is assumed that the adjacent relationship cannot be traced.

[0116] Next, FIG. 17 shows an example of neighborhood definition information 1700 managed by the learning data management unit 216.

[0117] The neighborhood definition information 1700 is a script written in a domain-specific language that describes neighborhoods for sampling pairs of instances used in self-supervised learning and the assignment of predictors to the neighborhoods.

[0118] In the neighborhood definition information 1700, prior to the definition of the neighborhood, parameters and variables used in the neighborhood definition are declared in advance as "parameter" and "variable". This is used for automatic hyperparameter tuning in the process described later. In addition, logic for the neighborhood definition can be imported from outside using import.

[0119] In the example of FIG. 17, four types of neighborhoods N1 to N4 are defined. Buses running on the same route with the same departure time are considered to share a similar background even if they are on different dates. Therefore, the neighborhood N1 defines a bus state in which the distance between stops of a series of stops where a route bus of the bus type "normal" stops is 2 or less as the neighborhood. The neighborhood N2 defines a bus state in which the distance between stops of a series of stops where a route bus of the bus type "express" stops is 1 or less as the neighborhood. Since the distance between stops of a bus of the bus type "express" is farther than that of a bus of the bus type "normal" compared to that of a bus of the bus type "normal", a value of 1 or less, which is smaller than that of the bus type "normal", is used. In this way, the distance between stops considering the operation can be taken into account. In this embodiment, the stop distance based on the stop adjacent data 1600 is used, but the physical distance (for example, in km) may be used. In addition, other distances defined by mathematical expressions may be used.

[0120] In addition, buses that run on the same route with similar departure times are considered to share a similar background. Therefore, neighborhood N3 defines the same stop as being in the vicinity for different route buses that have the same route ID and departure times from the starting stop that are delta1 or less. Note that this only applies if the dates match. In addition, buses that run near the same stop or stop at the same stop at a similar time, even if they have different departure times, are also considered to share a similar background. Therefore, neighborhood N4 defines the same stop as being in the vicinity for different route buses on the same date where the absolute value of the difference between the departure time and the sum of the elapsed time (i.e. the time the bus was there) is delta1 or less.

[0121] 17, predictors P1 and P2 are defined, predictor P1 is assigned to neighborhoods N1 and N2, and predictor P2 is assigned to neighborhoods N3 and N4. As described above, in this embodiment, by combining the neighborhood definition information 1700 and the stop neighboring data 1600, self-supervised learning using one or more of various neighborhood relationships can be performed.

[0122] <Machine learning model structure> 18 shows an example of a self-supervised learning model 1800 which is a machine learning model learned by the upstream task learning unit 212. The self-supervised learning model 1800 corresponds to the upstream model of the present invention.

[0123] The self-supervised learning model 1800 includes an instance pair sampler 1801, a state encoder 1802, and an embedding amount predictor 1803. The instance pair sampler 1801 samples an instance pair (instance A, B) based on the neighborhood definition information 1700. The state encoder 1802 calculates an embedding amount A based on the position and speed of the sampled instance A. The state encoder 1802 also calculates an embedding amount B based on the position and speed of the sampled instance B. The embedding amount predictor 1803 calculates an estimated embedding amount B by estimating the embedding amount B from the embedding amount A.

[0124] 19 shows an example of a delay time prediction model 1900, which is a regression model learned by the downstream task learning unit 214. The delay time prediction model 1900 corresponds to the downstream model of the present invention.

[0125] The delay time prediction model 1900 includes a state encoder 1901 and a delay time predictor 1902. The state encoder 1901 has the same structure as the state encoder 1802 of the self-supervised learning model 1800, and is initialized using a trained state encoder 1802. The delay time predictor 1902 outputs a predicted delay time at the s-stop destination from the bus states from three stops before to just before (just before) a certain route bus and the bus state just before (just before) the preceding route bus.

[0126] <Learning process by learning device 21> Next, a description will be given of the learning process by the learning device 21. Since the learning process is similar to the learning process by the learning device 11 in the first embodiment (FIG. 9), the description will be given by using FIG. 9 and omitting appropriate portions.

[0127] As a premise for this learning process, it is assumed that the bus information data management unit 74 of the bus information management server 70 is already managing bus status data 1500 (Figure 15) to be used for self-supervised learning for a predetermined period (e.g., one month) collected from each route bus 80 by the data collection unit 71.

[0128] The learning process is started in response to a predetermined start operation by the user 121 using the operation terminal 12, for example.

[0129] First, as data preparation for use in self-supervised learning, the overall task manager 211 reads out the bus state data 1500 managed by the bus information data manager 74, and copies it under the management of the learning data manager 216 of the learning device 21 (step S1).

[0130] Next, the overall task manager 211 generates hyperparameters to be used in the upstream task learning process (self-supervised learning) in step S3 and the downstream task learning process in step S4 (step S2). Specifically, the hyperparameters are generated by randomly sampling delta1, which is a parameter described in the neighborhood definition information 1700 (FIG. 17), from the described range. Note that, although only the aforementioned one parameter is the tuning target in this embodiment, various hyperparameters, such as the network structure, the number of mini-batches, the learning rate, and other training conditions, may be the targets.

[0131] Next, the upstream task learning unit 212 executes an upstream task learning process to generate a state encoder 1802 of the self-supervised learning model 1800 (FIG. 18) using the bus state data 1500, the neighborhood definition information 1700, and the hyperparameters generated in step S2 (step S3).

[0132] An example of the upstream task learning process in step S3 will be described with reference to FIG.

[0133] First, the upstream task learning unit 212 selects one of the neighborhoods N1 to N4 defined in the neighborhood definition information 1700 (step S31).

[0134] Next, the upstream task learning unit 212 samples pairs of instances that match the neighborhood definition selected in step S31, in accordance with the neighborhood definition information 1700 and the hyperparameters (step S32).

[0135] Next, the upstream task learning unit 212 uses the self-supervised learning model 1800 to calculate embedding amount A and embedding amount B for information on items other than the delay time of each instance, i.e., route ID, bus type, departure time from the first stop, date, elapsed time, stop ID, average speed, and congestion level, as well as an estimated embedding amount B which is an estimate of the embedding amount B estimated from the embedding amount A (step S33).

[0136] Next, the upstream task learning unit 212 calculates the loss of the embedding amount B and the estimated embedding amount B using the cosine similarity multiplied by -1 as a loss function, and updates the parameters of the embedding amount predictor 1803 and the state encoder 1802 regarding the position and speed A by the error backpropagation method (step S34).

[0137] The embedding amount predictor 1803 follows the correspondence between predictors and neighborhoods defined in the neighborhood definition information 1700. That is, there are two predictors P1 and P2, and the predictor P1 is associated with the neighborhoods N1 and N2, and the predictor P2 is associated with the neighborhoods N3 and N4. Since the predictor can be changed depending on the neighborhood in this way, an error signal for providing an expression for each neighborhood can be sent to the state encoder 1802.

[0138] Next, the upstream task learning unit 212 determines whether the learning end condition has been met (step S35), and if it determines that the learning end condition has not been met (NO in step S35), the process returns to step S31 and repeats the processes from step S31 onward. After that, if the upstream task learning unit 112 determines that the learning end condition has been met (YES in step S35), the upstream task learning process is terminated.

[0139] In this manner, in this embodiment, various instances can be generated. In addition, even if the target is one for which known data extension methods are not clear, such as enlargement, reduction, cropping, etc. for image data, or cutout or synthesis for audio data, and even if the data includes categorical variables, etc., and therefore mix-up, etc. cannot be used, self-supervised learning can be performed by sampling. Although not used in this embodiment, instances may be generated using simulation, as in the first embodiment.

[0140] Returning to Fig. 9, next, the downstream task learning unit 214 copies the parameters in the state encoder 1802 of the self-supervised learning model 1800 generated in step S3 to the state encoder 1901 of the delay time prediction model 1900, and then executes downstream task learning processing for learning the delay time prediction model 1900 (step S4).

[0141] FIG. 20 is a flowchart showing an example of the downstream task learning process in step S4.

[0142] First, the downstream task learning unit 214 copies the parameters of the state encoder 1802 of the self-supervised learning model 1800, which has been pre-trained by the upstream task learning process in step S3, to the state encoder 1901 of the delay time prediction model 1900. Other parameters are initialized using a general method such as normal distribution (step S401).

[0143] Next, the downstream task learning unit 214 samples one instance of a certain bus state data and designates it as a base sample. In addition, the downstream task learning unit 214 acquires the instance one instance before on the route and the instance two instances before that among the instances that have the same route ID, bus type, first stop departure time, and date as the base sample. Furthermore, the downstream task learning unit 214 acquires the instance with the longest elapsed time (latest) among the instances that have the same route ID, bus type, and date as the base sample and the first stop departure time one instance before. Furthermore, the downstream task learning unit 214 samples one value from integers between 1 and S, setting this as s, assuming that the number of stops that exist after the base sample is S. These are combined and used as input to the state encoder 1901 of the delay time prediction model 1900. In addition, the delay time of the stop s ahead of the base sample is acquired and set as the output (predicted delay time of the s stop ahead). A mini-batch is generated using these input and output pairs as elements (step S402).

[0144] If there is no previous instance, such as the first stop, an instance can be generated by reusing the base sample values ​​for route ID, bus type, first stop departure time, and date, and setting the average speed and congestion level to 0. Similarly, if there is no preceding route bus, an instance can be generated by reusing the base sample values ​​for route ID, bus type, and date, and setting the first stop departure time to 0:00, and setting the average speed and congestion level to 0.

[0145] Next, the downstream task learning unit 214 uses the delay time prediction model 1900 to calculate a predicted delay time from the input of the mini-batch (step S403). Next, the downstream task learning unit 214 calculates a loss from the delay time (true value) and the predicted delay time (estimated value), and updates the parameters of the delay time prediction model 1900 by the backpropagation method (step S405). Here, the loss is based on the mean square error.

[0146] Next, the downstream task learning unit 114 determines whether or not the learning end condition has been met (step S405). Here, the learning end condition is defined as, for example, repeating the processes of steps S402 to S405 a predetermined number of times (for example, 1000 times). If the downstream task learning unit 114 determines that the episode end condition has not been met (NO in step S405), the process returns to step S402, and steps S402 and after are repeated.

[0147] Thereafter, if it is determined that the learning end condition is met (YES in step S405), the downstream task learning process is terminated. According to the downstream task learning process, a model for predicting the delay time at any subsequent stop (s stops ahead) can be learned from the information of the route bus for which the delay time is to be predicted (three stops) and the information of the preceding route bus (one stop).

[0148] Return to FIG. 9. Next, the overall task management unit 211 outputs the transition of the loss of each predictor during self-supervised learning and the success rate of episodes in reinforcement learning as evaluation results to the learning data management unit 216, and the learning data management unit 216 stores them in the storage 103 or the like. Further, the learning data management unit 216 temporarily stores the learned self-supervised learning model 1800 and the delay time prediction model 1900 in the storage 103 or the like (step S5).

[0149] Next, the overall task management unit 211 determines whether or not a predetermined tuning end condition has been achieved (step S6). If the overall task management unit 211 determines that the predetermined tuning end condition has not been achieved (NO in step S6), the process returns to step S2, and the processes of steps S2 to S5 are repeated.

[0150] After that, when the overall task management unit 211 determines that a predetermined tuning end condition has been achieved (YES in step S6), next, the overall task management unit 211 outputs, as the final model, the model with the best evaluation result (prediction accuracy) among the models so far to the learning data management unit 216 as the related evaluation result, and the learning data management unit 216 stores this in the storage 103 or the like. Then, the models and evaluation result data temporarily saved in step S5 are deleted from the storage 103 or the like (step S7 in FIG. 9). Thus, the learning process is completed.

[0151] <UI screen> In the second embodiment, similar to the first embodiment, when the upstream task learning process in step S3 described above is being executed, the display management unit 215 can display the upper task learning loss transition confirmation screen 1000 (FIG. 12) and the parameter confirmation screen 1100 (FIG. 13) on the operation terminal 12.

[0152] FIG. 21 shows a display example of the predicted delay time display screen 2000 displayed on the information terminal 91 of the bus user 90 under the control of the delay time distribution unit 73 of the bus information management server 70.

[0153] The predicted delay time display screen 2000 is provided with input fields 2001 to 2003 for selecting and specifying the route ID, bus type, and departure time of the route bus for which the predicted delay time is to be displayed. Furthermore, the predicted delay time display screen 2000 is provided with a status display field 2004 for displaying the stop that the route bus specified by the input into the input fields 2001 to 2003 has just passed, and a predicted delay time display field 2005 showing the predicted delay time at each stop. The time displayed in the predicted delay time display field 2005 is the value calculated using the delay time prediction model 1900 rounded up to the nearest minute.

[0154] In the figure, in response to a bus user 90 selecting and specifying route ID "A1," bus type "regular," and first departure time "6:40," the corresponding route bus has currently passed stop "104," and the predicted delay times at stops "105" to "109" are 5 minutes, 3 minutes, 3 minutes, 4 minutes, and 2 minutes, respectively.

[0155] According to the second embodiment, it is possible to predict the delay time at each stop of a route bus. Note that, since upstream task learning is performed prior to downstream task learning for learning a regression model for delay time prediction, the accuracy of the regression model can be improved.

[0156] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with a configuration of another embodiment, or to add a part of the configuration of another embodiment.

[0157] In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing a part or all of them in an integrated circuit. In addition, each of the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function can be placed in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. In addition, the control lines and information lines are shown as those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0158] 1. Robot Management System 10. Learning System 11. Learning device 111 Overall Task Management Department 112...Upstream task learning part 113 Simulation Execution Unit 114...Downstream task learning unit 115...Display management department 116 Learning Data Management Unit 12 Operation terminal 121 users 2. Bus operation management system 20. Learning System 21. Learning device 211 Overall Task Management Department 212...Upstream task learning part 214...Downstream task learning unit 215...Display management department 216 Learning data management unit 30 Robot arm control device 31. Planning Department 32 Data collection section 33 Control data management section 40. Robot arm 60 Neighborhood definition information 70 Bus information management server 71 Data collection section 72... Delay time prediction unit 73...Delay Time Distribution Section 74 Bus Information Data Management Section 80. Local bus 90 Bus passengers 91 Information terminal 100. Computer 101 Processor 102 Memory 103 Storage 104 Input Device 105 Output device 106 Communication module 107 Program 300 Robot status data 400 Cartesian position and speed data 500 Neighborhood definition information 600 Self-supervised learning model 601... Instance Pair Sampler 602 State Encoder 603 Filling amount predictor 700 Actor model for reinforcement learning 701 Status Encoder 702...Actor 800 Critic model for reinforcement learning 801...State Encoder 802...Critic 1000...Top task learning loss transition confirmation screen 1100 Parameter confirmation screen 1500 Bus status data 1600...Stop adjacent data 1700 Neighborhood definition information 1800···Self-supervised learning model 1801... Instance Pair Sampler 1802 State Encoder 1803 Filling amount predictor 1900... Delay time prediction model 1901 Status Encoder 1902 Delay Time Predictor 2000···Expected delay time display screen

Claims

1. A learning device for constructing a machine learning model for tabular data, A computing device that executes a predetermined process and a storage device that can be accessed by the computing device, an upstream task learning unit configured to generate an upstream model for an upstream task by self-supervised learning based on neighborhood definition information that defines a neighborhood relationship between pairs of instances of the tabular data; The computing device includes a downstream task learning unit that generates a downstream model for a downstream task by supervised learning, reinforcement learning, or unsupervised learning; A learning device comprising:

2. The learning device according to claim 1 , the tabular data has at least one data item representing a date and time of the instance; The neighborhood definition information defines the neighborhood relationship in a time axis direction, The upstream task learning unit selects or generates the instance to be used in the self-supervised learning based on the neighborhood definition information. A learning device characterized by:

3. The learning device according to claim 1 , the tabular data has at least one column indicating a position of the instance; The neighborhood definition information defines the neighborhood relationship in terms of physical distance, The upstream task learning unit selects or generates the instance to be used in the self-supervised learning based on the neighborhood definition information. A learning device characterized by:

4. The learning device according to claim 1 , the arithmetic device includes a simulation execution unit that generates the tabular data by simulation, The upstream task learning unit uses the tabular data generated by the simulation execution unit in addition to the observed tabular data for the self-supervised learning. A learning device characterized by:

5. The learning device according to claim 4, the tabular data has at least one column indicating a date and time of the instance; The simulation execution unit generates the tabular data by performing a simulation with a certain instance as an initial state. A learning device characterized by:

6. The learning device according to claim 1 , The machine learning model used for the upstream task includes an encoder that converts the instances into a representation, and one or more predictors that convert an output of the encoder into another representation; The self-supervised learning of the machine learning model learns parameters of the encoder and the predictor such that, for a pair of instances used in the self-supervised learning, a representation obtained by inputting one of the instances to the encoder and another representation obtained by inputting the representation obtained by inputting the other instance to the encoder approaches a representation obtained by inputting the other instance to the encoder; The neighborhood definition information defines the predictor and a method for selecting the pair of instances. A learning device characterized by:

7. The learning device according to claim 1 , The neighborhood definition information specifies a range of hyperparameters related to generation of instances used in the self-supervised learning, The computing device has an overall task management unit that searches for the hyperparameters according to the ranges of the hyperparameters specified in the neighborhood definition information. A learning device characterized by:

8. The learning device according to claim 2, The tabular data is observation data when a plurality of machines perform work according to the same type of work instruction, The upstream task learning unit selects or generates the pair of instances to be used in the self-supervised learning based on a distance in relative time from when the same type of work instruction is received. A learning device characterized by:

9. The learning device according to claim 3, the tabular data is observation data of a moving body moving along a predetermined route while stopping at stops on the route and moving to a target, The upstream task learning unit selects or generates the pair of instances to be used in the self-supervised learning based on a distance between the stops on the route. A learning device characterized by:

10. The learning device according to claim 9, The distance between the stops is calculated based on the driving type of the moving object. A learning device characterized by:

11. A learning method for constructing a machine learning model for tabular data by a learning device, comprising: The learning device includes a calculation device that executes a predetermined process and a storage device that is accessible by the calculation device; The learning method includes: an upstream task learning step in which the computing device generates an upstream model for an upstream task by self-supervised learning based on neighborhood definition information that defines a neighborhood relationship between pairs of instances of the tabular data; a downstream task learning step in which the computing device generates a downstream model for the downstream task by supervised learning, reinforcement learning, or unsupervised learning; A learning method comprising:

Images