Movement prediction device, method, and program
The motion prediction device enhances human-robot collaboration by supplementing motion trajectories with past worker data to ensure accurate prediction of destination points, addressing limitations in existing methods and improving task accuracy and safety.
Patent Information
- Application Number
- PCT/JP2023/017423
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-14
AI Technical Summary
Current motion prediction methods for human-robot collaboration, particularly in tasks like pick-and-place work, often fail to accurately predict trajectories that pass through critical destination points due to limited expressive power of deep learning models and uniform confidence in prediction targets, leading to potential work failures and uncertainties.
The proposed solution involves a motion prediction device that uses past attainment point data and predicted arrival points with corresponding times to supplement motion trajectories, ensuring that the predicted trajectory passes through important destination points by learning from past worker movement trajectories, thereby enhancing prediction accuracy and safety.
This approach improves the accuracy and reliability of predicting destination points, allowing robots to operate at precise positions and timings, ensuring safe and accurate execution of tasks, even in the presence of communication delays.
Smart Images

Figure JP2023017423_14112024_PF_FP_ABST
Abstract
Description
Motion prediction device, method, and program
[0001] One aspect of the present invention relates to a motion prediction device used to predict the motion of a person, for example, while working, and a motion prediction method and program executed by the device.
[0002] For example, when a person repeatedly performs tasks such as tidying up a room or arranging items on shelves, the robot may work in cooperation with the person. In this case, by predicting the movements of the person, the robot can work efficiently and safely by avoiding collisions with the person.
[0003] For example, when a person operates a humanoid or arm-type robot in a remote location to perform tidying up work, it is expected that the negative impact of reduced work efficiency due to communication delays can be reduced by predicting the person's movements, which serve as operating information, and having the remote robot operate in advance.
[0004] Many techniques for predicting human motion have been proposed. For example, a skeletal prediction method uses time-series data (skeletal coordinate series) of a person's past skeletal information extracted from video data, sensor detection data, etc. as input to predict a skeletal coordinate series at a future time. Furthermore, motion prediction methods using the skeletal coordinate series include two types: deterministic prediction, which predicts a single skeletal coordinate series, and probabilistic prediction, which predicts multiple candidate skeletal coordinate series, as described in Non-Patent Document 1, for example.
[0005] More recently, methods have been proposed that improve prediction accuracy by using models that take into account knowledge of the characteristics of human movement in addition to skeletal structures. For example, Non-Patent Document 2 describes a method that utilizes the knowledge that human movement is goal-oriented, moving toward a single point in space. This method predicts in advance the destination point, i.e., the distance a pedestrian will walk within a room shown in the video, based on third-person perspective video information showing the pedestrian, and predicts future skeletal position coordinates using the predicted destination point and a past skeletal coordinate series.
[0006] The use of arrival points is also expected to be effective when predicting the movements of people performing so-called pick-and-place tasks, such as tidying up or arranging displays, in which objects are picked up and placed in designated locations. When a person picks up an object, the arrival point is the "position where the object is picked up," and when they place the object, the arrival point is the "position where the object is placed." By predicting the arrival points of these movements in advance and using them in motion prediction together with time-series data of skeletal information, further improvements in prediction accuracy are expected.
[0007] Yuan, Ye, and Kris Kitani. 2020. “DLow: Diversifying Latent Flows for Diverse Human Motion Prediction.” In Computer Vision - ECCV 2020, 346-64. Springer International Publishing. https: / / doi.org / 10.1007 / 978-3-030-58545-7_20.Cao, Zhe, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. 2020. “Long-Term Human Motion Prediction with Scene Context.” In Computer
[0008] However, when predicting the motion of a person performing pick-and-place tasks, even if the direction of the worker's movement, i.e., the destination point, can be estimated in advance, the predicted trajectory may not pass through the destination point. This is due to two reasons: most motion prediction methods predict the trajectory up to the target time with an equivalent degree of certainty, and the expressive power of the deep learning model used for motion prediction is limited. In various tasks, including pick-and-place tasks, the destination point is an important element corresponding to the position where the object is picked up or placed. Therefore, if a robot is controlled using a trajectory that does not pass through the destination point, the robot-assisted task becomes unreliable and there is a risk of failure.
[0009] The present invention has been made in light of the above circumstances, and aims to provide a technology that improves the accuracy of prediction of the arrival point and improves the reliability and safety of work performed by controlled equipment.
[0010] In order to solve the above problems, one aspect of a motion prediction device or method according to the present invention, when predicting a future motion of a worker, first predicts a sequence of predicted future arrival points of the worker and arrival times at each predicted arrival point based on past arrival point data representing a sequence of past arrival points of the worker, past skeletal information data representing a sequence of past skeletal information of the worker, and at least position information of the worker in the workspace, and outputs the prediction result. Next, a motion trajectory from the current position of the worker to the predicted arrival point based on the prediction result and the past skeletal information data is interpolated based on past motion trajectories of the worker related to a specific task, and output predicted skeletal information data of the worker reflecting the interpolated motion trajectory.
[0011] According to one aspect of the present invention, for example, a motion trajectory of a worker from the worker's current position to a predicted destination is interpolated, and skeletal coordinate series data representing the interpolated motion trajectory is output as data representing a motion prediction result. That is, skeletal coordinate series data based on a motion trajectory interpolated on the assumption that the destination point will be passed is output. This makes it possible to predict a trajectory that will reliably pass through the destination point, which is important in work. Therefore, when the worker and the work target equipment are operated cooperatively or when the work target equipment is remotely controlled, it is possible to operate the work target equipment at an accurate position and with accurate timing, thereby improving the accuracy and safety of work performed by the work target equipment.
[0012] That is, according to one aspect of the present invention, it is possible to provide a technology that improves the accuracy of prediction of the arrival point and improves the reliability and safety of work performed by controlled equipment.
[0013] FIG. 1 is a functional block diagram used to explain the outline of a motion prediction device according to one embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of a robot control system including a motion prediction device according to an embodiment of the present invention. FIG. 3 is a block diagram showing an example of the hardware configuration of the motion prediction device shown in FIG. 2. FIG. 4 is a block diagram showing an example of the software configuration of the motion prediction device shown in FIG. 2. FIG. 5 is a flowchart showing an example of the processing procedure and processing content of motion prediction control executed by a control unit of the motion prediction device shown in FIG. 4. FIG. 6 is a diagram for explaining an example of the operation of a trajectory completion processing unit included in the motion prediction device shown in FIG. 4. FIG. 7A is a diagram showing an example of the operation of a robot grasping an object in a pick-and-place operation. FIG. 7B is a diagram showing an example of the operation of a robot placing an object in a pick-and-place operation.
[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0015] [Summary of the Invention] One aspect of the present invention is to newly provide a trajectory completion unit that is trained in advance using a learning data set that indicates the trajectories of past movements of a worker involved in a specific task, and by using the trained trajectory completion unit to complete the trajectory of the worker's movements between the destination predicted by the destination series prediction unit and the current position of the worker, it is possible to predict the trajectory of movements that will reliably pass through the destination where the worker will perform the task.
[0016] FIG. 1 is a functional block diagram used to explain the outline of an action prediction device according to one aspect of the present invention.
[0017] The motion prediction device 100 according to an aspect of the present invention includes a reaching point sequence prediction unit 200 and a trajectory complementation unit 300 .
[0018] The arrival point series prediction unit 200 has a trained model for predicting arrival points, and inputs data representing a series of the worker's past arrival points, data representing a series of skeletal coordinates representing the worker's past movements, and information representing the worker's position in the workspace into this trained model to predict the worker's future arrival point series and the arrival time at each arrival point.
[0019] The trajectory complementation unit 300 has a movement trajectory complementation model that has been trained in advance using a training data set that represents the trajectories of past movements of a worker involved in a specific task. The trajectory complementation unit 300 inputs the arrival point series predicted by the arrival point series prediction unit 200, the arrival times at each arrival point, and the skeletal coordinate series data that represents the worker's past movements into the trained movement trajectory complementation model, complements the movement trajectory from the present to the predicted arrival point, and outputs data that represents the skeletal coordinate series that reflects the complemented movement trajectory.
[0020] By configuring it in this way, when a worker performs a specific task, it is possible to predict a movement trajectory that will reliably pass through important destination points that will become the work position, and by using this predicted movement trajectory to control the work target equipment, such as a robot, it is possible to perform the work by the robot accurately and safely. [One embodiment] (Configuration example) (1) System Figure 2 is a diagram showing an example of the configuration of a robot control system equipped with a movement prediction device according to one embodiment of the present invention.
[0021] A robot control system according to one embodiment includes a motion prediction device SV that predicts a task trajectory of a worker US. The motion prediction device SV is connected to a camera CM and an administrator terminal TM used by a system administrator or the like as input devices. The motion prediction device SV is also connected to a robot RB to be controlled.
[0022] The robot RB is, for example, a humanoid articulated robot, and performs a predetermined task based on control data that reflects the motion trajectory of the worker US predicted by the motion prediction device SV. The type and configuration of the robot RB may vary depending on the type and content of the task.
[0023] The camera CM captures an image of the work space including the worker US and the object to be worked on from a third-person perspective, and outputs the captured image data of the work space to the action prediction device SV.
[0024] The manager terminal TM is used, for example, to input control commands such as operation start and end to the operation prediction device SV, and to input a learning data set to the learning model provided in the operation prediction device SV.
[0025] The motion prediction device SV may be provided integrally with the robot RB, or may be connected to the robot RB via a network such as a wireless LAN (Local Area Network), and the installation location is not particularly limited.
[0026] (2) Behavior Prediction Device SV FIGS. 3 and 4 are block diagrams showing examples of the hardware and software configurations of the behavior prediction device SV, respectively.
[0027] The behavior prediction device SV is, for example, a personal computer, and includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 4 are connected to the control unit 1 via a bus 5.
[0028] The input / output I / F unit 4 is connected to the camera CM and the administrator terminal TM as input devices, and to the robot RB as a device to be controlled. The input / output I / F unit 4 receives image data of the workspace output from the camera CM, and control commands and learning data sets input from the administrator terminal TM. The input / output I / F unit 4 also outputs movement control data generated by the control unit 1 to the robot RB.
[0029] Note that a wireless interface that adopts a low-power wireless data transmission standard such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) may be used as the input / output I / F unit 4. Using a wireless interface can eliminate the need for signal cables connecting the robot RB, camera CM, and administrator terminal TM.
[0030] The program storage unit 2 is configured by combining, for example, a nonvolatile memory such as a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read only memory (ROM), and stores middleware such as an operating system (OS) as well as application programs required to execute various control processes according to the first embodiment. Hereinafter, the OS and each application program will be collectively referred to as a program.
[0031] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and the storage area includes a destination prediction model storage unit 31, a past data storage unit 32, and a trajectory completion model storage unit 33.
[0032] The arrival point prediction model storage unit 31 stores a learning model for predicting an arrival point series of the worker US. As this arrival point prediction model, for example, a time series prediction model such as Seq-to-Seq or RNN is used.
[0033] The past data storage unit 32 stores data indicating the movement trajectory of the worker US measured in past tasks and data indicating changes in the destination point. The data indicating the movement trajectory may be, for example, time-series data of skeletal coordinates indicating changes in the posture of the worker US. Furthermore, the data indicating changes in the destination point may be time-series data of three-dimensional coordinates in the workspace.
[0034] The trajectory interpolation model storage unit 33 stores a trajectory interpolation model for interpolating a movement trajectory from the current position of the worker US to a predicted arrival point. As the trajectory interpolation model, for example, a time series prediction model using a neural process or a Gaussian process is used.
[0035] The control unit 1 includes, as processing function units used to implement one embodiment of the present invention, an input data acquisition processing unit 11, a destination point sequence prediction processing unit 12, a trajectory complement model learning processing unit 13, a trajectory complement processing unit 14, and a motion prediction data output processing unit 15.
[0036] Each of the processing units 11 to 15 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 15 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).
[0037] In a prediction phase in which the movement of the worker US is predicted, the input data acquisition processing unit 11 acquires video data output from the camera CM, which is an image of a work space including the worker US and a work target object captured from a third-person perspective, via the input / output I / F unit 4. Then, the input data acquisition processing unit 11 detects the three-dimensional position coordinates of the worker US and the work target object in the work space from the acquired video data.
[0038] In the prediction phase, the arrival point series prediction processing unit 12 inputs the three-dimensional position coordinates of the worker US and the work object detected by the input data acquisition processing unit 11, and the past skeletal coordinate series data and arrival point series data of the worker US read from the past data storage unit 32, into a trained arrival point prediction model stored in the arrival point prediction model storage unit 31. The arrival point prediction model predicts a series of future arrival points of the worker US and the arrival time at each arrival point based on the above input data, and outputs data indicating the prediction result to the trajectory complementation processing unit 14. Note that an example of the prediction process of the arrival point series will be described in the operation example.
[0039] In the learning phase, the trajectory completion model learning processing unit 13 learns the trajectory completion model using an input / output data set for learning input from, for example, the administrator terminal TM, and stores the learned trajectory completion model in the trajectory completion model storage unit 33. As the input / output data set for learning, an input / output data set generated based on a movement trajectory of a worker US involved in a specific task is used.
[0040] The trajectory interpolation processing unit 14 inputs, into a trajectory interpolation model, prediction data including a future arrival point series and arrival times at each arrival point output from the arrival point series prediction processing unit 12, and past skeletal coordinate series data read from the past data storage unit 32. The trajectory interpolation model interpolates a movement trajectory from the current point to the predicted arrival point based on the input data. The trajectory interpolation processing unit 14 then outputs skeletal coordinate series data reflecting the movement trajectory interpolated by the trajectory interpolation model. An example of the movement trajectory interpolation processing will be described in the operation example.
[0041] The motion prediction data output processing unit 15 outputs the skeletal coordinate series data output from the trajectory complement processing unit 14 as motion prediction data for the worker US from the input / output I / F unit 4 to the robot RB.
[0042] (Example of Operation) Next, an example of operation of the operation prediction device SV configured as above will be described.
[0043] Here, an example will be described in which pick-and-place work is performed, in which an object is grasped and placed in a predetermined location, such as in tidying up, displaying, etc. In pick-and-place work, when the worker US performs the action of grasping an object, the "position where the object is grasped" is the destination point, and when the worker US performs the action of placing the object, the "position where the object is placed" is the destination point.
[0044] (1) Learning Phase When the learning phase is set by an instruction from the administrator terminal TM, for example, the behavior prediction device SV executes a process of learning a trajectory interpolation model. Note that, as the prediction model for the arrival point series can use an existing trained model, a description thereof will be omitted here.
[0045] The control unit 1 of the motion prediction device SV, under the control of the trajectory complement model learning processing unit 13, acquires input / output data sets for learning from, for example, the administrator terminal TM, and learns the trajectory complement model using the acquired input / output data sets.
[0046] The input / output data set for learning is generated based on the past motion trajectories of the worker US involved in a specific task. In this example, since the target is pick-and-place work, the input / output data set for learning is generated based on motion trajectory data including the points reached when the worker US performed pick-and-place work in the past, i.e., the three-dimensional position coordinates where the worker US actually grasped and placed an object. By using this input / output data set to train a trajectory completion model in advance, it becomes possible to predict the motion trajectory of the worker US specialized in pick-and-place work.
[0047] The trajectory complement model learning processing unit 13 repeatedly learns the trajectory complement model using the above-mentioned multiple input / output data sets for learning, and then stores the learned trajectory complement model in the trajectory complement model storage unit 33, and ends the processing.
[0048] (2) Prediction Phase FIG. 5 is a flowchart showing an example of the processing procedure and processing content of the operation prediction process executed by the control unit 1 of the operation prediction device SV in the prediction phase.
[0049] (2-1) Acquisition of Input Data When a request to start motion prediction is input from the manager terminal TM and the control unit 1 of the motion prediction device SV detects this request to start motion prediction in step S10, first in step S11, under the control of the input data acquisition processing unit 11, the control unit 1 acquires video data of the workspace captured by the camera CM via the input / output I / F unit 4. Then, the input data acquisition processing unit 11 detects the three-dimensional position coordinates of the worker US and each object in the workspace from the acquired video data.
[0050] Note that instead of the video data captured by the camera CM, depth data representing the positions of the worker US and the object being worked on in the work space may be acquired from, for example, the manager terminal TM.
[0051] (2-2) Prediction of Arrival Point Series Next, under the control of the arrival point series prediction processing unit 12, the control unit 1 of the operation prediction device SV executes the process of predicting the arrival point series when the worker US performs pick-and-place work as follows.
[0052] That is, first, in step S12, the arrival point series prediction processing unit 12 reads out past skeletal coordinate series data of the worker US from the past data storage unit 32, and in step S13, reads out past arrival point series data of the worker US from the past data storage unit 32. Then, in step S14, the arrival point series prediction processing unit 12 inputs the read out past skeletal coordinate series data and arrival point series coordinate data, as well as the three-dimensional position coordinates of the worker US and each object in the workspace acquired by the input data acquisition processing unit 11, into the trained arrival point prediction model stored in the arrival point prediction model storage unit 31.
[0053] The skeletal coordinate series data input into the arrival point prediction model does not necessarily have to be the entire body skeleton, but may be skeletal coordinate series data of only the left and right arms or the upper body, or only the body parts related to the pick-and-place operation.
[0054] The arrival point can be expressed as the three-dimensional coordinates of the center position of the object to be worked on. However, if the position where the object is to be placed is predetermined for the work, the combination of the object's placement position can be expressed using a one-hot expression. Note that it is not necessary to use all of the data input to the arrival point prediction model; it is sufficient to select data according to the model configuration.
[0055] In the arrival point prediction model, a process is performed to predict the worker US's future arrival point series and the arrival time at each arrival point based on the past skeletal coordinate series data and arrival point series coordinate data, and the three-dimensional position coordinates of the worker US and each object.
[0056] For example, in a reach prediction model, the reach of the current action is estimated from the video data or depth data of the workspace using a deep generative model such as a Variational Auto Encoder (VAE), and then a sequential prediction model such as a Markov model is used to recursively predict the reach of the future action.
[0057] Other possible arrival point prediction models include a method of recursively predicting a future arrival point series by repeatedly using a past arrival point to sequentially predict the next arrival point, or a method of directly predicting a future arrival point series using a past arrival point series.
[0058] Among these, the former existing method for sequentially predicting arrival points includes, for example, a method such as a Markov model, in which the transition probability from one arrival point to the next is learned in advance from other task data and used for prediction. The latter method for directly predicting arrival point sequences includes, for example, a method using an existing deterministic deep learning-based time series prediction model such as Seq-to-Seq, and a probability-based time series prediction method such as Gaussian process regression or neural process. This method is described in detail in, for example, Non-Patent Document 1.
[0059] In step S15, the arrival point series prediction processing unit 12 outputs prediction data representing the arrival point series obtained by the arrival point prediction model and the arrival time at each arrival point to the trajectory interpolation processing unit 14.
[0060] (2-3) Complement of Movement Trajectory Next, the control unit 1 of the movement prediction device SV executes the complementation process of the movement trajectory under the control of the trajectory complementation processing unit 14 as follows.
[0061] That is, in step S16, the trajectory complement processing unit 14 inputs the predicted data representing the arrival point series and the arrival time at each arrival point output from the arrival point series prediction processing unit 12, and the past skeleton coordinate series data read out from the past data storage unit 32, into the learned trajectory complement model stored in the trajectory complement model storage unit 33.
[0062] In step S17, the trajectory complement model complements the movement trajectory from the current position to the predicted destination point based on the input destination point series, predicted data representing the arrival time at each destination point, and past skeleton coordinate series data.
[0063] FIG. 6 shows an image of the above-mentioned locus interpolation process. In this example, the function f 0 Using an input / output data set LD representing a known motion trajectory according to the function f 0 Learn an approximate representation of the function f 0 A probabilistic representation of f Z and then use this trained prediction model to calculate the current position x C , y C The predicted destination x G , y G unknown position x between t , y t This shows the case where the trajectory in
[0064] Possible trajectory completion models include existing deep learning-based time series models such as RNN and Seq-to-Seq, which are used in existing time series prediction. Also possible are models that perform probabilistic predictions that take into account the uncertainty of human movement by using Bayesian-based methods such as Gaussian processes or deep Bayesian-based methods such as neural processes. Probabilistic predictions can output not only plausible action sequences, but also all possible action sequence candidates and values representing their probabilities.
[0065] Then, the trajectory complementation processing unit 14 outputs skeletal coordinate series data representing the motion trajectory complemented by the trajectory complementation model.
[0066] (3) Output of Motion Prediction Data Finally, in step S18, under the control of the motion prediction data output processing unit 15, the control unit 1 of the motion prediction device SV outputs the skeletal coordinate series data output from the trajectory completion processing unit 14 as motion prediction data from the input / output I / F unit 4 to the robot RB.
[0067] (4) Example of operation of robot RB The robot RB performs a series of operations related to pick-and-place work, for example, based on data representing a pre-specified work type and work content, and the operation prediction data output from the operation prediction device SV.
[0068] 7A and 7B show an example of a pick-and-place operation by a robot RB. First, as shown in Fig. 7A, the robot RB grasps an object (e.g., a cup) Q at a position Pa on a table. Next, as shown in Fig. 7B, the robot RB places the grasped cup Q at a predetermined position Pb on a shelf.
[0069] (Operation and Effect) As described above, in one embodiment, in the learning phase, a trajectory completion model is trained using an input / output data set representing the past movement trajectory of the worker US involved in a specific task. Then, in the prediction phase, first, the three-dimensional position coordinates of the worker US and each object in the workspace shown by the video data acquired from the camera CM, as well as past skeletal coordinate series data and arrival point series coordinate data, are input into the arrival point prediction model to predict the arrival point series and the arrival time at each arrival point. Next, the predicted arrival point series and arrival times at each arrival point, and the past skeletal coordinate series data are input into the trained trajectory completion model to predict the movement trajectory from the current position to the predicted arrival point, and skeletal coordinate series data representing the predicted movement trajectory is output.
[0070] Therefore, the motion trajectory of the worker US from the current position of the worker US to the predicted destination is complemented, and skeletal coordinate series data representing the complemented motion trajectory is output as data representing the motion prediction result. That is, skeletal coordinate series data based on the motion trajectory complemented on the assumption that the destination point will be passed is output. This makes it possible to predict a trajectory that reliably passes through the "object grasping position" and "object placement position," which are important in pick-and-place work, for example. Therefore, when the worker US and the robot RB are operated cooperatively or when the robot RB is remotely controlled, it is possible to operate the robot RB at the correct position and with the correct timing, thereby improving the accuracy and safety of pick-and-place work by the robot RB.
[0071] [Other Embodiments] (1) In the embodiment, the case where past skeleton coordinate series data and arrival point series data are stored in the past data storage unit 32 has been described as an example, but they may be acquired from an external storage device. Furthermore, the training data set for the trajectory prediction model may be generated based on past arrival point series data when the task is performed correctly.
[0072] (2) The motion prediction device may be installed on the web or cloud, or may be built into the robot control device as a motion prediction function unit, in addition to being installed at the work site. Furthermore, various modifications can be made to the installation location of the motion prediction device, the configuration of the processing units, the processing procedures and contents of each processing unit, the type and content of the work, the type and configuration of the equipment to be worked on, and the like, without departing from the spirit of the present invention.
[0073] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.
[0074] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.
[0075] DESCRIPTION OF SYMBOLS 100...Motion prediction device 200...Arrival point series prediction unit 300...Trajectory completion unit SV...Motion prediction device RB...Robot CM...Camera TM...Administrator terminal 1...Control unit 2...Program storage unit 3...Data storage unit 4...Input / output I / F unit 5...Bus 11...Input data acquisition processing unit 12...Arrival point series prediction processing unit 13...Trajectory completion model learning processing unit 14...Trajectory completion processing unit 15...Motion prediction data output processing unit 31...Arrival point prediction model storage unit 32...Past data storage unit 33...Trajectory completion model storage unit
Claims
1. A motion prediction device for predicting future motion of a worker, comprising: an arrival point series prediction processing unit that predicts a series of predicted arrival points of the worker in the future and the arrival time at each predicted arrival point based on past arrival point data representing a series of past arrival points of the worker, past skeletal information data representing a series of past skeletal information of the worker, and at least positional information of the worker in a workspace, and outputs information representing the prediction result; and a trajectory complementation processing unit that complements a motion trajectory from the current position of the worker to the predicted arrival point based on the information representing the prediction result and the past skeletal information data, based on the past motion trajectory of the worker related to a specific task, and outputs skeletal information prediction data of the worker in which the complemented motion trajectory is reflected.
2. The motion prediction device according to claim 1, wherein the trajectory completion processing unit has a trajectory completion model that has been trained in advance using an input / output data set that represents the past motion trajectories of the worker involved in the specific task, inputs information representing the prediction result and the past skeletal information data into the trajectory completion model to complete the motion trajectory from the worker's current position to the predicted arrival point, and outputs skeletal information prediction data of the worker that reflects the completed motion trajectory.
3. The movement prediction device according to claim 1, wherein the trajectory completion processing unit comprises: a learning processing unit that, in a learning phase, learns a trajectory completion model using an input / output data set representing the past movement trajectory of the worker related to the specific task; and a prediction processing unit that, in a prediction phase, inputs information representing the prediction result and the past skeletal information data into the trajectory completion model to complete the movement trajectory from the current position of the worker to the predicted arrival point, and outputs skeletal information prediction data of the worker reflecting the completed movement trajectory.
4. The motion prediction device according to claim 2, wherein the trajectory complement processing unit uses, as the trajectory complement model, a time series prediction model that performs probabilistic prediction taking into account uncertainty of the motion of the worker.
5. A motion prediction method for predicting future motions of a worker using an information processing device, comprising: a step of predicting a series of predicted arrival points of the worker in the future and the arrival time of each predicted arrival point based on past arrival point data representing a series of past arrival points of the worker, past skeletal information data representing a series of past skeletal information of the worker, and at least positional information of the worker in the workspace, and outputting information representing the prediction result; and a step of complementing a motion trajectory from the current position of the worker to the predicted arrival point based on the information representing the prediction result and the past skeletal information data, based on the past motion trajectory of the worker involved in a specific task, and outputting skeletal information prediction data of the worker in which the complemented motion trajectory is reflected.
6. A program for causing a processor included in the motion prediction device to execute at least one of the processes executed by each processing unit included in the motion prediction device according to any one of claims 1 to 4.
Citation Information
Patent Citations
Early prediction of an intention of a user's actions
US20180032868A1
Learning device, learning method, and program
WO2022176196A1
