Motion prediction device, method and program

Abstract

To provide an action prediction device, a method, and program that can accurately predict what follows the action of a photographed person considering the situation of surroundings.SOLUTION: In an action prediction device 1 predicting the following action based on a video of the action of a person, a video acquisition unit 10 acquires frame images capturing a person targeted to action prediction obtained by photographing the person for a predetermined time. An action analysis unit 20 analyzes the action of the person up to the present. A destination point estimation unit 30 estimates or acquires a destination point of the action of the person. A representation extraction unit 40 extracts the potential representation of the following action based on the action of the person up to the present and the destination point. An action prediction unit 50 predicts the following action of the person based on the action up to the present, the destination point, and the potential representation. A destination point acquisition unit 30 includes an object detection unit 301 and a direction estimation unit 302, and estimates the position of an object detected in the direction of movement of the person as the destination point.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an apparatus, method and program for predicting the continuation of a human action based on a video of the human action, and more particularly to an apparatus, method and program for predicting a human action by taking into account the surrounding circumstances. [Background technology]

[0002] The ability to predict detailed human movements, such as limb movements, several seconds in advance is required in fields such as quickly detecting dangerous behavior and controlling robots and actuators that work in cooperation with people.

[0003] Non-Patent Document 2 discloses a technology that detects the skeletal position of a person from a 2D image and predicts the future skeletal position from the past skeletal positions using LSTM (Long Short Term Memory). While Non-Patent Document 2 makes it easy to predict simple movements, it is necessary to add additional restrictions on the postures that a person can take, making it difficult to impose strict constraints.

[0004] Non-Patent Document 1 discloses a technology that uses a CVAE (Conditional Variational Autoencoder) model that employs a GRU (Gated Recurrent Unit) as an encoder-decoder module instead of an LSTM. CVAE can learn to output a plausible next action based on a latent representation randomly sampled from a standard normal distribution, with past actions as a condition.

[0005] Furthermore, Non-Patent Document 1 imposes diversity on the latent representation so that multiple diverse prediction results are output as subsequent actions, taking into account the fact that there are diverse possibilities for future human actions. Non-Patent Document 1 applies various linear transformations to randomly sampled latent variables based on given past actions, and gives them to CVAE as new latent representations. This makes it possible to predict plausible actions that cover diverse future possibilities.

[0006] Patent Document 1 discloses a technology that predicts a person's behavior by reflecting the characteristics of each person. Patent Document 1 analyzes the behavioral state and the person's identifier of a photographed person, and generates behavior prediction information based on rules that are collated with the obtained person information. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2013-186556 A [Non-patent literature]

[0008] [Non-Patent Document 1] Yuan, Ye and Kitani, Kris (2020). Dlow: Diversifying latent flows for diverse human motion prediction. ECCV [Non-Patent Document 2] Erwin Wu and Hideki Koike: FuturePose - Mixed Reality Martial Arts Training Using Real-Time 3D Human Pose Forecasting With a RGB Camera, WACV(2019) [Non-Patent Document 3] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788, doi: 10.1109 / CVPR.2016.91. [Non-Patent Document 4] Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich, "Simple Online and Realtime Tracking with a Deep Association Metric", ICIP2017 [Non-Patent Document 5] Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", CVPR2017 [Non-Patent Document 6] Yang, Ceyuan and Xu, Yinghao and Shi, Jianping and Dai, Bo and Zhou, Bolei, "Temporal Pyramid Network for Action Recognition", CVPR2020 Summary of the Invention [Problem to be solved by the invention]

[0009] In Non-Patent Document 1, it is possible to predict a variety of appropriate actions as a continuation of a given action. However, there are cases where actions with low probability are predicted due to lack of consideration of the surrounding situation. For example, when making an autonomous robot refer to the predicted action results to make an action plan, it is desirable to have the robot concentrate on predicting actions with high probability according to the surrounding situation. However, Non-Patent Document 1 presents more predicted results than necessary, including actions with low probability, which may hinder the autonomous robot's action plan.

[0010] Although Patent Document 1 predicts behavior that reflects the characteristics of each individual, there is a possibility that the predicted behavior will be inconsistent with the situation because it does not take into consideration the surrounding circumstances.

[0011] An object of the present invention is to provide a motion prediction device, method, and program that can solve the above technical problems and accurately predict the subsequent motions of a photographed person while taking into account the surrounding circumstances. [Means for solving the problem]

[0012] In order to achieve the above object, the present invention is characterized in that a motion prediction device for predicting a subsequent motion of a person based on a video of the person performing the motion has the following configuration.

[0013] (1) The system is equipped with a means for analyzing a person's movements up to the present, a means for acquiring a destination point of the movements, a means for extracting a latent expression of a subsequent movement based on the person's movements up to the present and the destination point, and a means for predicting the person's subsequent movement based on the person's movements up to the present, the destination point, and the latent expression.

[0014] (2) The means for acquiring the destination point of the movement estimates the destination point based on the person's movement up to the present.

[0015] (3) The means for acquiring the destination point includes a means for detecting an object from an image and a means for estimating a direction of movement of a person based on the image, and the position of the object detected in the direction of movement is estimated to be the destination point. Effect of the Invention

[0016] (1) Accurate predictions are possible because the system takes into account the surrounding circumstances and predicts the next movements of the person being photographed.

[0017] (2) The destination point is related to motion, and since the destination point is estimated based on the motions up to the present, the destination point can be estimated with high accuracy.

[0018] (3) The destination point depends on the direction of movement, and since the position of an object detected in the direction of movement is estimated as the destination point, the destination point can be estimated with even higher accuracy. [Brief description of the drawings]

[0019] [Figure 1] 1 is a functional block diagram showing a configuration of a main part of an action prediction device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram showing a configuration of an expression extraction unit. [Diagram 3] 13 is a flowchart showing an operation of an expression extraction unit. [Figure 4] FIG. 2 is a functional block diagram showing a configuration of an operation prediction unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of the main parts of a motion prediction device 1 according to an embodiment of the present invention, and the main components include a video acquisition unit 10, a motion analysis unit 20, a destination point estimation unit 30, an expression extraction unit 40, and a motion prediction unit 50.

[0021] Such a three-dimensional shape reconstruction device 1 can be configured by implementing applications (programs) that realize each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

[0022] In this embodiment, for a single action of one person, the subsequent action following that action is predicted based on the analysis results of the past actions up to the present, and here, in particular, the prediction of a scene in which a person is about to pick up an object (static object) is explained as an example. Note that if there are multiple people to be the subject of action prediction, individual predictions can be made by repeating the same process for each person.

[0023] The video acquisition unit 10 uses an RGB camera or a depth camera to capture a target person for motion prediction for a predetermined time (e.g., 3 seconds) to acquire frame images of the person. Alternatively, a video file in which a person has been captured in advance for a predetermined time may be acquired.

[0024] The motion analysis unit 20 analyzes the acquired video to detect and track people, and estimates the person's current motion based on the results of the detection and tracking. A deep learning model such as YOLO (You Only Look Once) disclosed in Non-Patent Document 3 can be used for person detection, and DeepSORT disclosed in Non-Patent Document 4 can be used for person tracking. This makes it possible to accurately track the same person across multiple frames.

[0025] The motion analysis unit 20 further performs posture estimation processing on each frame image in which a person is captured. For posture estimation, a method of outputting the positions of skeletal key points in a video as postures, such as OpenPose disclosed in Non-Patent Document 5, can be used. The positions of key points can be expressed in two-dimensional coordinates or three-dimensional coordinates.

[0026] The output of OpenPose and other tools is usually 2D coordinates, so additional processing is required to express it in 3D coordinates, such as triangulation (when the input has multiple viewpoints), adding depth information (when the input has a depth image), or fitting a 3D model of a human.

[0027] The motion analysis unit 20 estimates the motion of the person by combining the results of all frames. As a result of the analysis If the number of frames is T, the number of skeletal key points is J, and the posture is expressed in three-dimensional coordinates, the data format of the motion analysis results will be a T × 3J matrix.

[0028] In addition to the above analysis results, the behavior of a person may also be analyzed and labeled, and an action label may be attached to the action analysis result. The action label can be estimated, for example, by using a deep learning model disclosed in Non-Patent Document 6.

[0029] The destination point estimation unit 30 includes an object detection unit 301 and a direction estimation unit 302. Motion analysis resultsBased on this, the object that is the target of the action is detected, and the point that the person is aiming for is estimated based on the direction of the person's movement.

[0030] When predicting the action of a person picking up an object as in this embodiment, several categories of objects that a person can pick up are given in advance to object detection unit 301, and object detection is performed on objects belonging to those categories using the method described in Non-Patent Document 3.

[0031] In addition to the action of picking up an object, similar processing can be performed by defining objects related to the target action in advance. For example, if the action of opening a door is predicted, the detection target is a handle, and if the action of sitting down is predicted, a chair, etc. If the action analysis unit 20 has recognized an action label, the detection target may be limited to objects related to the action label.

[0032] If an object is detected, the direction of the person's movement is used as a clue to identify the object that is the target of the movement. The direction estimation unit 302 calculates the person's direction of movement as the frame average of the moving direction of the center coordinate of the body, and identifies the target object as the object closest to the person on an extension of the moving direction.

[0033] The estimated object position is expressed using the same coordinate system as the coordinate system expressing the person's movement. In the case of two-dimensional coordinates, the coordinates are the center coordinates of the area output by the detector, and in the case of three-dimensional coordinates, the three-dimensional coordinates of the center of the object are obtained by triangulation, depth image, fitting of a predefined object model, etc. The estimated object position is sent to the expression extraction unit 40 at the subsequent stage as the destination point of the movement together with the person's movement.

[0034] Furthermore, in the case of an action of picking up an object, since it is considered that the way of picking up an object changes depending on the type of object, such as a cup or a bag, the type of object may also be sent to the downstream expression extraction unit 40. The type of object can be represented by a numerical value such as an ID, and by using the detector disclosed in Non-Patent Document 3, the ID can be output together with the detection result.

[0035] When the type of object is taken into consideration, for example, the object ID may be converted into a one-hot vector representation and combined with the destination point data before performing subsequent processing. Note that a configuration for inputting the destination point may be provided instead of the destination point estimation unit 30, so that the destination point can be manually input.

[0036] The expression extraction unit 40 extracts a plurality of valid latent expressions z by inferring latent expressions of future actions to be continued by the person using an approximate probability distribution based on the estimation results of the person's action x and destination point y.

[0037] 2 is a functional block diagram showing the configuration of the expression extraction unit 40, in which a person's current movement x is input to a gated regression unit (GRU) 401, and a destination point y is input to a multilayer perceptron (MLP) 402. The GRU 401 weights the person's movement x according to each time, and then outputs it to the downstream multilayer perceptron 403.

[0038] FIG. 3 is a flowchart showing a procedure by which the expression extraction unit 40 extracts a latent expression z based on the estimation results of a person's movement x and a destination point y. Here, the person's movement x and the destination point y are assumed to be expressed in three-dimensional coordinates.

[0039] In step S1, we extract K vector values ​​ε = {ε 1 ,…,ε K} is randomly sampled and provided to the conversion unit 405. In step S2, each MLP 403 randomly samples a person's motion x∈R T×3J By encoding K complex nonlinear functions φ (e.g., pre-trained neural networks), we obtain K matrix-vector pairs A k ,b k Calculate (k=1,…,K).

[0040] In step S3, the downstream conversion unit 405 applies the K vector values ​​ε to the following equation (1) to obtain K vector values ​​δ (δ1 ~δ k )

[0041] δ k =A k ε k +b k (1)

[0042] In step S4, the MLP 402 encodes the destination point y by L complex nonlinear functions ψ (e.g., pre-trained neural networks) and generates L matrix-vector pairs C l ,d l Calculate (l=1,…,L).

[0043] In step S5, the latter conversion unit 406 applies the K vector values ​​δ to the following equation (2) to obtain K·L latent representations Z={Z 1, 1 ,…,Z K, L}.

[0044] Z k,1 =C 1 δ 1 +d 1 … Z k,L =C L δ k +d L (2)

[0045] The nonlinear function φ in step S2 and the nonlinear function ψ in step S4 are learned collectively on a data set including pairs of a person's motion x and a destination point y. The person's motion x in the data set is separated at intermediate frames and treated as an "observed motion" and a correct "continued motion." The learning method may be similar to that in Non-Patent Document 1. Specifically, the objective function of the following equation (3) is minimized using an optimization method such as stochastic gradient descent.

[0046] L Recon +L Div +L KL (3)

[0047] Here, L Recon is the reconstruction error, L Divindicates an energy function that promotes diversity in the prediction results. If the prediction results of K·L actions obtained by inputting the latent representation z to the action prediction unit 50 described later are expressed by the following equation (4) and the correct answer is expressed by the following equation (5), then the L Recon ,L Div can be calculated using the following equations (6) and (7), where λ is set to an appropriate constant.

[0048]

number

[0049] L KL denotes the average Kullback-Leibler distance between the probability distribution of the K·L latent representations z and N(0,I). k, l The probability distribution of N(0,I) is A k ,b k and C. l ,d l This can be expressed as a probability distribution obtained by affine transformation in the following equation (8).

[0050]

number

[0051] The Kullback-Leibler distance between the above equation (8) and N(0,I) is D k,l Then, L KL is calculated by the following equation (9): Note that since the output of the motion predictor 50 is used in this learning, the motion predictor 50 must be trained beforehand.

[0052]

number

[0053] The order of steps S2, 3 and steps S4, 5 may be interchanged. In that case, the symbols in the above equations will be interchanged as appropriate, for example by sampling L vector values ​​in step S1.

[0054] Steps S2 and S3 and steps S4 and S5 may be performed simultaneously to encode the person's movement x and destination point y using K·L complex nonlinear functions (e.g., pre-trained neural networks), calculate K·L matrix-vector pairs A, b, and obtain the latent representation z by a single transformation of the vector value ε.

[0055] The obtained latent representation z can be regarded as a value sampled from a probability distribution conditioned on the person's movement x and the object's position y, and a total of K·L latent representations z are sent to the subsequent movement prediction unit 50 as latent representations of subsequent movements.

[0056] The action prediction unit 50 predicts a number of appropriate actions of a person based on the person's actions x up to the present, the destination point y, and the latent expressions z. For example, the person's action x, the destination point y, and K·L latent expressions z are input to a complex nonlinear function ρ (such as a neural network trained in advance), and K·L predictions of subsequent actions x' are output. An example of a model in which the complex nonlinear function is a neural network is shown in FIG. 4.

[0057] GRU501 weights the past actions x of the person up to the present according to each time. MLP502 encodes the destination point y by L complex nonlinear functions ψ and generates L matrix-vector pairs C l ,d l (l=1, ..., L). The combination module 503 combines the outputs of the GRU 501 and the MLP 502 with the latent representation z and outputs the combined output to the downstream GRU 504.

[0058] The GRU 504 infers a predicted motion x' from x, y, and z using a nonlinear function ρ. The learning of the nonlinear function ρ is performed on a data set including pairs of a person's motion x and a destination point y, and an ideal state may be estimated by using a method similar to CVAE, which is a deep generative model based on Non-Patent Document 1, and allowing information on the objective variable to be taken into account in generating latent variables.

[0059] GRU504 considers an objective function with two terms: the reconstruction error of the prediction x', the approximate probability distribution of the latent representation z, and the Kullback-Leibler distance with N(0,I), and minimizes it using optimization methods such as stochastic gradient descent.

[0060] In the above embodiment, both the expression extraction unit 40 and the action prediction unit 50 consider the destination point y, but only one of them may be considered. Specifically, either the process of steps S4 and S5 performed by the expression extraction unit 40 or the input of the destination point y to the action prediction unit 50 may be omitted.

[0061] According to the above embodiment, the subsequent motion of a person can be accurately predicted based on the video of the person moving, so that an inexpensive and convenient motion prediction system can be provided to many people regardless of geographic or economic disparity. As a result, it is possible to contribute to the achievement of Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations. [Explanation of symbols]

[0062] 1...motion prediction device, 10...image acquisition unit, 20...motion analysis unit, 30...destination point estimation unit, 40...expression extraction unit, 50...motion prediction unit, 401, 501, 504...GRU, 402, 403, 404, 502...MLP, 405, 406...conversion unit, 503...combination module

Claims

1. A motion prediction device for predicting a subsequent motion of a person based on a video of the person performing the motion, comprising: a means for analyzing a video of a person in motion, estimating each of the person's current postures, and outputting a sequence of the estimated postures as a motion analysis result; A means for estimating a destination point of the person's movement based on the result of the movement analysis; A means for extracting a latent expression of a subsequent motion of the person based on the motion analysis result and a destination point; and a means for predicting a subsequent action of the person based on the result of the action analysis, the destination point and the latent expression.

2. The motion prediction device described in Claim 1, characterized in that the means for outputting the motion analysis results analyzes a video of a person moving, detects, tracks and estimates the posture of the person, and outputs a series of postures of the person up to the present.

3. The motion prediction device as described in claim 1 or 2, characterized in that, instead of the means for estimating the destination point, it is provided with a means for acquiring a manually input destination point.

4. The means for estimating a destination point includes: A means for detecting an object from the video; means for estimating a moving direction of a person based on the image; 3. The motion prediction device according to claim 1, wherein the position of the object detected in the moving direction is estimated as a destination point.

5. The means for estimating the destination point is notified of a category of the object in advance; The motion prediction device according to claim 4 , wherein the means for detecting an object detects an object belonging to the category.

6. The means for outputting the motion analysis result identifies a person's motion and outputs a motion label of the identification result; The action prediction device according to claim 4 , wherein the means for estimating the destination point detects an object corresponding to the action label.

7. A motion prediction method in which a computer predicts a subsequent motion of a person based on a video of the person performing the motion, comprising: Analyzing a video of a person in motion to estimate each of the person's current postures, and outputting a sequence of the estimated postures as a motion analysis result; Estimating a destination point of the person's movement based on the movement analysis result; Extracting a latent expression of a subsequent motion of the person based on the motion analysis result and the destination point; A motion prediction method, comprising: predicting a subsequent motion of the person based on the motion analysis result, a destination point, and a latent expression.

8. A motion prediction program for predicting a subsequent motion based on a video of a person performing a motion, a step of analyzing a video of a person in motion, estimating each of the person's current postures, and outputting a sequence of the estimated postures as a motion analysis result; estimating a destination point of the person's movement based on the movement analysis result; extracting a latent expression of a subsequent motion of the person based on the motion analysis result and the destination point; and predicting the person's subsequent actions based on the results of the action analysis, the destination point, and the latent expression.

Images