Pick-and-place motion prediction device, prediction method, and prediction program

Abstract

This pick-and-place operation prediction device has an arrival point series prediction unit and an operation prediction unit. The arrival point series prediction unit predicts a future arrival point series using a prediction model produced through machine learning, the prediction model accepting, as inputs, a past arrival point series and at least one of a past skeletal coordinate series and video information pertaining to a third-party viewpoint relating to a worker performing pick-and-place work. The operation prediction unit predicts a future skeletal coordinate series of the worker using a prediction model produced through machine learning, the prediction model accepting, as inputs, the past skeletal coordinate series and the future arrival point series that is predicted by the arrival point series prediction unit.

Description

Technical Field

[0001] The present invention relates to a pick-and-place operation prediction device, a prediction method, and a prediction program.

Background Art

[0002] When a person repeatedly performs tasks such as room cleaning or display work, it is conceivable to provide support by robot technology. In this case, by predicting the movement of a person who is the operator, the work can be advanced efficiently, or a collision with the person can be avoided and the work can be executed safely.

[0003] In addition, when a person operates a humanoid robot or an arm-type robot located at a remote location to perform a cleaning task, it is expected to reduce the adverse effect of a decrease in work efficiency due to communication delay by predicting the movement of the person that is the operation information and causing the robot at the remote location to perform a pre-action.

[0004] Many techniques for predicting human movements have been proposed so far. In many skeleton prediction methods, the time-series information (skeleton coordinate series) of past human skeleton information extracted from images, sensors, etc. is input, and the skeleton coordinate series at a future time is predicted. In recent years, a technique has emerged that improves the prediction accuracy by improving the model in consideration of findings on the characteristics of human movements other than the skeleton. In Non-Patent Document 1, the finding that human movement is goal-oriented toward a point in space is utilized. From the video information from the third-person perspective in which pedestrians are reflected, the arrival point, that is, how far in the room reflected in the video the pedestrian will walk, is predicted in advance, and the future skeleton position coordinates are predicted using the predicted arrival point and the past skeleton coordinate series.

[0005] The use of destination points is also expected to be effective when predicting the movements of people who repeatedly perform pick-and-place tasks, such as tidying up or displaying items, where they grasp objects and place them in designated locations. When a person grasps an object, the "position where the object is grasped" becomes the destination point, and when a person places an object, the "position where the object is placed" becomes the destination point. Similar to Non-Patent Document 1, it is expected that predicting these destination points in advance and using them in conjunction with skeletal coordinate sequences for motion prediction will improve prediction accuracy. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Z. Cao, H. Gao, K. Mangalam, Q.-Z. Cai, M. Vo, and J. Malik, "Long-Term Human Motion Prediction with Scene Context", Computer Vision - ECCV 2020, Cham, 2020, pp. 387-404.<URL:https: / / doi.org / 10.1007 / 978-3-030-58452-8_23> . [Overview of the project] [Problems that the invention aims to solve]

[0007] There are two challenges when predicting the movements of a person performing a pick-and-place task using the method described in Non-Patent Document 1. These two challenges will be explained below with reference to Figure 5. Figure 5 is a schematic diagram illustrating the two challenges in predicting the movements of a person performing a pick-and-place task.

[0008] The first issue (Issue 1) is that prediction accuracy decreases in the transition zone between actions. In pick-and-place operations, there is a transition between actions with different destinations, such as "grabbing a plate" → "carrying it to the shelf." Between the predicted time and the predicted time in the future, changes in the direction of movement occur that cannot be narrowed down using past observation information (skeletal coordinate series, video information), and changes in the direction of movement that cannot be narrowed down using the skeletal coordinate series or the destination of the current action occur, resulting in a decrease in the accuracy of action prediction in the transition zone between actions.

[0009] The second challenge (Challenge 2) is the difficulty in predicting movement when observational information is insufficient. Even within the same destination, it is difficult to narrow down the movement until sufficient time-series observational information, such as the input skeletal coordinate sequence and video information, is obtained, as the movement continues for a certain period of time.

[0010] This invention has been made in view of the above circumstances, and its purpose is to provide a pick-and-place motion prediction device, prediction method, and prediction program that can quickly and accurately predict the movements of a person performing pick-and-place work. [Means for solving the problem]

[0011] One aspect of the present invention is a pick-and-place motion prediction device. The pick-and-place motion prediction device comprises a destination sequence prediction unit and a motion prediction unit. The destination sequence prediction unit receives video information from a third-party perspective of a worker performing a pick-and-place operation, at least one of past skeletal coordinate sequences, and past destination sequences as input, and uses a machine learning prediction model to predict future destination sequences. The motion prediction unit receives past skeletal coordinate sequences and the future destination sequences predicted by the destination sequence prediction unit as input, and uses a machine learning prediction model to predict the worker's future skeletal coordinate sequences.

[0012] One aspect of the present invention is a pick-and-place motion prediction method. The pick-and-place motion prediction method includes the steps of: receiving third-party perspective video information of a worker performing a pick-and-place operation, a past skeletal coordinate sequence, and a past destination sequence as input, and using a machine learning prediction model to predict a future destination sequence; and receiving a past skeletal coordinate sequence and a future destination sequence as input, and using a machine learning prediction model to predict the worker's future skeletal coordinate sequence.

[0013] One aspect of the present invention is a pick-and-place operation prediction program. The pick-and-place operation prediction program causes a computer having a processor and a memory device to perform at least some of the functions of the components of the pick-and-place operation prediction device described above. [Effects of the Invention]

[0014] According to the present invention, a pick-and-place motion prediction device, prediction method, and prediction program are provided that can quickly and accurately predict the movements of a person performing pick-and-place work. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 is a block diagram showing the functional configuration of a pick-and-place motion prediction device according to an embodiment. [Figure 2] Figure 2 is a block diagram showing the hardware configuration of a pick-and-place motion prediction device according to an embodiment. [Figure 3] Figure 3 is a flowchart showing the process flow for predicting a future skeletal coordinate sequence performed by the pick-and-place motion prediction device according to the embodiment. [Figure 4] Figure 4 is a schematic diagram illustrating the process flow for predicting a future skeletal coordinate sequence performed by the pick-and-place motion prediction device according to the embodiment. [Figure 5] Figure 5 schematically illustrates the challenges in conventional examples of predicting the movements of people performing pick-and-place tasks.

Embodiment for Carrying out the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0017] 〈Configuration Example〉 (Functional Configuration) First, referring to FIG. 1, the functional configuration of the pick-and-place operation prediction device according to the embodiment will be described. FIG. 1 is a block diagram showing the functional configuration of the pick-and-place operation prediction device according to the embodiment.

[0018] The pick-and-place operation prediction device 10 is a device that predicts the future body movements of a person performing a pick-and-place operation. The pick-and-place operation is an operation of grasping an object to be operated, moving it, and placing the object. Such an operation includes, for example, tidying up work. Hereinafter, for the sake of convenience, a person performing a pick-and-place operation is referred to as an operator. The body movements of the operator are represented by the operator's skeleton coordinate series. That is, the skeleton coordinate series means body movements.

[0019] The input information to the pick-and-place operation prediction device 10 is the operator's past skeleton coordinate series, video information and depth information from a third-person perspective, and past reach point series. The past skeleton coordinate series is time-series information of the operator's past skeleton information extracted from video, sensors, etc. The video information from a third-person perspective is video information in which the operator and the entire work space are shown, and the depth information is information indicating the positions of each object in the work space. The past reach point series is time-series information of the reach points in past pick-and-place operations. That is, the past reach point series is time-series information of the "grasped position" and "placed position" of the object by the operator. The expression method of the past reach point series can be represented by the three-dimensional position coordinates of the center position of the object or the one-hot expression of the combination of the candidate positions for placing the object.

[0020] In addition, the output information from the pick-and-place motion prediction device 10 is the future skeleton coordinate series of the operator. The future skeleton coordinate series is time-series information of the operator's skeleton information at a time prior to the prediction time for performing motion prediction. The future skeleton coordinate series is used, for example, by a robot that cooperates with the operator. Alternatively, the future skeleton coordinate series is used for remote operation of a humanoid robot or an arm-type robot located at a remote location.

[0021] The pick-and-place motion prediction device 10 includes an input unit 20, a reach point series prediction unit 30, a motion prediction unit 40, and an output unit 50.

[0022] The input unit 20 receives the operator's past skeleton coordinate series, video information and depth information from a third-person perspective, and the past reach point series input from the outside to the pick-and-place motion prediction device 10. The input unit 20 passes the received past skeleton coordinate series, video information and depth information from a third-person perspective, and the reach point series to the reach point series prediction unit 30. The input unit 20 also passes the received past skeleton coordinate series to the motion prediction unit 40.

[0023] The operator's past skeleton coordinate series of the input information may relate not only to the operator's whole body skeleton but also to only some body parts related to the pick-and-place operation, such as the operator's left and right arms and only the upper body.

[0024] The reach point series prediction unit 30 receives, as inputs, video information and depth information from a third-person perspective, the past skeleton coordinate series, and the past reach point series from the input unit 20, and predicts the future reach point series using a prediction model based on machine learning. The reach point series prediction unit 30 has a model storage unit 31 that stores the prediction model. The reach point series prediction unit 30 passes the predicted future reach point series to the motion prediction unit 40.

[0025] In pick-and-place operations, the destination is the "grabbing position" when an object is grasped, and the destination is the "placing position" when the object is carried or placed. The destination can be represented using the 3D position coordinates of the object's center. The future destination sequence is a sequence of the 3D position coordinates of the "grabbing position" and "placing position" where the worker will grasp the object in the future. If the "placing position" is predetermined during the operation, it can be represented using a one-hot encoding of the combination of "placing positions". In predicting the future destination sequence, it is not always necessary to use all of the input information; selection should be made according to the prediction model being used.

[0026] In one example, the destination sequence prediction unit 30 recursively predicts future destination sequences by repeatedly performing sequential predictions of the next destination using the previous destination. Existing methods for sequentially predicting destinations include methods like the Markov model, which learn the transition probability from one destination to the next from other working data in advance and use it for prediction. Alternatively, as described in Non-Patent Document 1, a method can be considered in which the destination of the current operation is estimated using a deep generative model such as a Variational Auto Encoder (VAE) from video information and depth information, and then the destination of future operations is recursively predicted using a sequential prediction model such as the Markov model.

[0027] In another example, the destination sequence prediction unit 30 directly predicts future destination sequences using past destination sequences. Possible methods for directly predicting destination sequences include using existing deterministic deep learning-based time series prediction models such as Seq-to-Seq, or probability-based time series prediction methods such as Gaussian process regression or Neural Process.

[0028] The motion prediction unit 40 receives the worker's past skeletal coordinate sequence from the input unit 20 and the future destination sequence from the destination sequence prediction unit 30 as input, and uses a machine learning prediction model to predict the worker's future skeletal coordinate sequence. The motion prediction unit 40 has a model storage unit 41 for storing the prediction model. The motion prediction unit 40 passes the predicted future skeletal coordinate sequence to the output unit 50.

[0029] The prediction model used by the motion prediction unit 40 could be an existing deep learning-based time series prediction model such as RNN or Seq-to-Seq, which are used in existing framework predictions.

[0030] Furthermore, methods for performing probabilistic predictions that take into account the uncertainty of human movement can be considered by using Bayesian-based methods such as Gaussian processes or deep Bayesian-based methods such as Neural Processes. By performing probabilistic predictions, not only the most plausible motion sequence but also all possible motion sequence candidates and their probability values ​​can be provided to external devices such as robots. As an example of how these motion sequence candidates and probability values ​​can be used, they can be used to generate safer robot motion paths when a robot is working in cooperation with a human worker.

[0031] The output unit 50 receives a future skeletal coordinate sequence from the motion prediction unit 40 and outputs it to the outside of the pick-and-place motion prediction device 10. In one example, the future skeletal coordinate sequence output from the pick-and-place motion prediction device 10 is used to control a robot that works in cooperation with an operator performing a pick-and-place operation. In another example, the future skeletal coordinate sequence is used to remotely control a humanoid robot or an arm-type robot that performs a pick-and-place operation in a remote location.

[0032] (Hardware configuration) Next, the hardware configuration of the pick-and-place operation prediction device 10 will be described. For example, the pick-and-place operation prediction device 10 is composed of a personal computer or a server computer, etc.

[0033] Figure 2 is a block diagram showing the hardware configuration of the pick-and-place motion prediction device 10 according to the embodiment. As shown in Figure 2, the pick-and-place motion prediction device 10 includes a processor 61, a ROM (Read Only Memory) 62, a RAM (Random Access Memory) 63, an auxiliary storage device 64, and an input / output interface 65.

[0034] The processor 61, ROM 62, RAM 63, auxiliary storage device 64, and input / output interface 65 are electrically connected to each other via a bus 66, and data is exchanged via the bus 66.

[0035] The processor 61 is composed of a general-purpose hardware processor, such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit). The processor 61 controls the entirety of the ROM 62, RAM 63, auxiliary storage device 64, and input / output interface 65.

[0036] ROM62 is a non-volatile memory that constitutes part of the main memory. ROM62 non-temporarily stores the startup program required when the processor 61 starts up. The processor 61 starts up by executing the program in ROM62. ROM62 is, for example, composed of EPROM (Erasable Programmable Read Only Memory) and stores various startup settings in addition to the startup program.

[0037] RAM63 is a volatile memory that constitutes part of the main memory. RAM63 temporarily stores the program necessary for processing by the processor 61 and the data necessary for executing the program. The processor 61 executes the program in RAM63, performs calculations on the data in RAM63, and stores the calculation results in RAM63.

[0038] The auxiliary storage device 64 consists of non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The auxiliary storage device 64 non-temporarily stores programs executed by the processor 61 and data necessary for program execution. The processor 61 reads the programs and data from the auxiliary storage device 64 into the RAM 63 and executes various functions by running the programs.

[0039] The input / output interface 65 is connected to an external input device 71 and an output device 72, etc., enabling the input of information from the input device 71 and the output of information to the output device 72. For example, the input / output interface 65 may be a wired interface or a wireless interface. A wired interface includes a port to which a device is connected. A wireless interface includes Bluetooth®, WiFi®, etc. The input / output interface 65 constitutes an input section 20 and an output section 50.

[0040] The input device 71 includes equipment that inputs video information, depth information, past skeletal coordinate sequences, and past destination sequences to the pick-and-place motion prediction device 10. For example, the input device 71 includes a receiving device. The input device 71 may also include a disk drive, keyboard, mouse, touch panel, etc. The input device 71 is not limited to these and may include any other input device. The output device 72 includes equipment that outputs a future skeletal coordinate sequence. For example, the output device 72 includes a transmitting device. The output device 72 may also include a disk drive, display, etc. The output device 72 is not limited to these and may include any other output device. The input device 71 and the output device 72 may be configured as an input / output device 73 that has the functions of both.

[0041] A program stored non-temporarily in the auxiliary storage device 64 is provided to the computer, for example, via a computer-readable recording medium 74 on which the program is stored non-temporarily. Such a recording medium 74 is called a non-temporarily computer-readable recording medium. Non-temporarily computer-readable recording media include disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memory.

[0042] A program stored non-temporarily in the auxiliary storage device 64 is read into and stored non-temporarily via the input device 71, which is a disk drive, and the input / output interface 65, if the recording medium 74 is a disk, or via the input / output interface 65, which is a port, if the recording medium 74 is semiconductor memory. Alternatively, the program may be stored on a server on a network, downloaded from the server, and stored non-temporarily in the auxiliary storage device 64.

[0043] The program stored non-temporarily in the auxiliary storage device 64 includes a pick-and-place operation prediction program. The pick-and-place operation prediction program is a program that causes the computer constituting the pick-and-place operation prediction device 10 to execute the functions of the input unit 20, the destination sequence prediction unit 30, the operation prediction unit 40, and the output unit 50.

[0044] At startup, the processor 61 executes a program in the ROM 62 and loads the OS into the RAM 63 to start up. Under the control of the OS, the processor 61 monitors instruction inputs and the connection of external devices. The processor 61 also sets up a program area and a data area in the RAM 63 under the control of the OS. In response to an instruction input to start the pick-and-place motion prediction device 10, the processor 61 loads the pick-and-place motion prediction program from the auxiliary storage device 64 into the program area of ​​the RAM 63, and also loads the prediction model and data necessary for executing the pick-and-place motion prediction program from the auxiliary storage device 64 into the data area of ​​the RAM 63. The processor 61 calculates the data in the data area according to the pick-and-place motion prediction program and writes the calculation results to the data area. Through these operations, the processor 61, RAM 63, auxiliary storage device 64, and input / output interface 65 work together to execute the functions of the input unit 20, the destination sequence prediction unit 30, the motion prediction unit 40, and the output unit 50 of the pick-and-place motion prediction device 10.

[0045] <Example of operation> Next, referring to Figure 3, we will describe the process by which the pick-and-place motion prediction device 10 predicts the future physical movements of the worker, i.e., the skeletal coordinate sequence. Figure 3 is a flowchart showing the flow of the process by which the pick-and-place motion prediction device 10 predicts the future skeletal coordinate sequence according to the embodiment. In other words, Figure 3 is a flowchart of the pick-and-place motion prediction method according to the embodiment.

[0046] In step S1, the destination sequence prediction unit 30 obtains video information from a third-person perspective, depth information, past skeletal coordinate sequences, and past destination sequences from the input unit 20.

[0047] In step S2, the destination sequence prediction unit 30 takes the video information and depth information acquired in step S1, along with the past skeletal coordinate sequence and past destination sequence as input, and uses a machine learning prediction model to predict the future destination sequence.

[0048] In step S3, the motion prediction unit 40 obtains the operator's past skeletal coordinate sequence from the input unit 20, and also obtains the future destination sequence predicted in step S2 from the destination sequence prediction unit 30.

[0049] In step S4, the motion prediction unit 40 takes the past skeletal coordinate sequence and the future destination sequence obtained in step S3 as input and uses a machine learning prediction model to predict the worker's future skeletal coordinate sequence, i.e., body movement.

[0050] Figure 4 schematically shows the process flow by which the pick-and-place motion prediction device predicts the worker's future skeletal coordinate sequence based on third-person perspective video information, depth information, past skeletal coordinate sequence, and past destination sequence, after going through the processes of steps S1 to S4 described above.

[0051] <effect> In the pick-and-place motion prediction device 10 according to this embodiment, the destination sequence prediction unit 30 takes video information and depth information from a third-party viewpoint, as well as past skeletal coordinate sequences, and past destination sequences as input, and uses a machine learning prediction model to predict future destination sequences. The motion prediction unit 40 takes the worker's past skeletal coordinate sequences and the future destination sequences predicted by the destination sequence prediction unit 30 as input, and uses a machine learning prediction model to predict the worker's future skeletal coordinate sequences. Because the prediction made by the motion prediction unit 40 is based on future destination sequences as well as past skeletal coordinate sequences, it can narrow down the physical movements of the worker performing the pick-and-place operation at an early stage. In other words, it can improve the accuracy of motion prediction even at the timing of motion changes and at times when conventional past observation information (skeletal coordinate sequences, video information) cannot be used to narrow down the movements. That is, the accuracy of motion prediction in motion change sections is improved. For this reason, the pick-and-place motion prediction device 10 according to this embodiment can predict the worker's future physical movements quickly and with high accuracy.

[0052] When the pick-and-place motion prediction device 10 according to this embodiment is applied to a robot that works in cooperation with an operator performing a pick-and-place operation, the robot can predict the operator's physical movements and act accordingly, enabling efficient work execution and safe work execution by avoiding collisions with the operator.

[0053] When the pick-and-place motion prediction device 10 according to this embodiment is applied to a humanoid robot or an arm-type robot that performs pick-and-place work in a remote location, it is expected that failures due to malfunctions in the preliminary execution of remote work will be reduced, and the time required to confirm whether the robot is moving in the direction desired by the operator will be shortened, thereby shortening the overall work time for remote work.

[0054] In this embodiment, the pick-and-place motion prediction device 10 is described as a computer having a processor 61 and a storage device (ROM 62, RAM 63, and auxiliary storage device 64), where the storage device stores a pick-and-place motion prediction program, and the processor 61 executes the pick-and-place motion prediction program to predict the physical movements of a worker performing a pick-and-place task. However, some functions of the pick-and-place motion prediction program executed by the processor 61 may be implemented in combination with programs already recorded in the computer, or they may be implemented using hardware such as a PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), or GPU (Graphic Processing Unit).

[0055] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention. [Explanation of symbols]

[0056] 10…Pick and place motion prediction device 20...Input section 30…Achievement point series prediction unit 31...Model memory unit 40…Motion prediction unit 41...Model memory unit 50…Output section 61… Processor 62...ROM 63...RAM 64…Auxiliary storage device 65… Input / Output Interface 66... ​​Bus 71…Input devices 72…Output device 73… Input / Output Devices 74…Recording media

Claims

1. A destination sequence prediction unit receives video information from a third-party perspective of a worker performing a pick-and-place operation, at least one of the past skeletal coordinate sequences, and a past destination sequence as input, and uses a machine learning prediction model to predict the future destination sequence. A motion prediction unit receives the aforementioned past skeletal coordinate sequence and the aforementioned future destination sequence predicted by the destination sequence prediction unit as input, and uses a machine learning prediction model to predict the future skeletal coordinate sequence of the worker. A pick-and-place motion prediction device having the following features.

2. The destination sequence prediction unit receives depth information as input in addition to the video information, and uses the prediction model to predict the future destination sequence. The pick-and-place motion prediction device according to claim 1.

3. The aforementioned destination sequence prediction unit predicts the future destination sequence recursively by repeatedly performing sequential predictions of the next destination using the previous destination. The pick-and-place motion prediction device according to claim 1.

4. The destination sequence prediction unit directly predicts the future destination sequence using the past destination sequence. The pick-and-place motion prediction device according to claim 1.

5. The motion prediction unit performs a probabilistic prediction of the future sequence of destinations, taking into account the uncertainty of the worker's movements. The pick-and-place motion prediction device according to claim 1.

6. The process involves receiving third-party video information of a worker performing a pick-and-place operation, along with past skeletal coordinate sequences and past destination sequences as input, and using a machine learning-based predictive model to predict future destination sequences. The steps include: receiving the past skeletal coordinate sequence and the future destination sequence as input, and using a machine learning prediction model to predict the worker's future skeletal coordinate sequence; A pick-and-place operation prediction method having the following characteristics.

7. A computer having a processor and memory, To perform at least some of the functions of the components of the pick-and-place motion prediction device described in any one of claims 1 to 5, A pick-and-place operation prediction program.

Images