Prediction device, program, and learning method

Abstract

To provide an estimation device capable of precisely and flexibly performing diverse movement estimate.SOLUTION: An estimation device 10 for estimating and generating a future movement X from a past movement X generates a future movement in which an operation category is reflected on time series by decoding the past movement, a latent code z, and a movement category L set on time series for each of future time by an operation decoding part 13. The multiple future movements are generated by generating the multiple latent codes, and by changing timing of transition of an operation category on the corresponding time series for each of multiple transition codes.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a prediction device that predicts future actions from past actions, a program for the device, and a learning method. [Background technology]

[0002] Predicting a person's future movements is useful for optimizing the control of cars and robots, avoiding danger, etc. To make predictions, in addition to capturing a person's trajectory using cameras and sensors, it is also common to capture multiple skeletal points such as hands and feet to predict more detailed movements.

[0003] One future is not enough to consider possibilities, but past actions of people X 1:t For future action Y t+1:T Here, as shown below, the subscript 1:t of X indicates past frames X(k) (frames up to the current time t) spanning time k=1, 2, ..., t-1, t, and the subscript t+1:T of Y indicates future frames Y(k) spanning time k=t+1, t+2, ..., T-1, T. However, if it is clear from the context, they may be omitted and written as past frame X, future frame Y, etc. Hereinafter, this notation will be used for frames representing actions. Past action frame X 1:t ={X(k)|k=1,2,…,t-1,t} (sometimes abbreviated as X) Future Action Frame Y t+1:T ={Y(k)|k=t+1,t+2,…,T-1,T} (sometimes abbreviated as Y)

[0004] Methods for predicting multiple actions often use a conditional variational autoencoder (CVAE).

[0005] Non-Patent Document 1 and Non-Patent Document 2 use CVAE for motion estimation.

[0006] The VAE (variational autoencoder) on which CVAE is based has an encoder and decoder like a normal autoencoder, and is normalized so that the latent space is continuous. During inference, latent code z can be sampled from the latent space and z can be decoded, with z's that are close in distance producing similar decoded results. However, this similarity is not uniform across distance, and the extent to which the difference in behavior depending on the distance greatly depends on the model and training parameters.

[0007] CVAE adds conditions to the encoder input and decoder input of VAE, making it easier to obtain decoding results under specified conditions. In inferring diverse actions, by inputting past action X as a condition and selecting and decoding multiple different latent codes z, it is possible to predict multiple different future actions Y for past action X.

[0008] Although Non-Patent Documents 1 and 2 apply CVAE to motion prediction, Non-Patent Document 3 discloses details of general CVAE without limiting its application.

[0009] Patent Document 1 does not use CVAE or VAE, but uses a recurrent neural network (RNN) with a general encoder and decoder to predict motion and motion labels from video. This method estimates joint positions from past videos, estimates motion characteristics based on optical flow from past videos, and encodes these to obtain a latent state. The current motion label is estimated from that state. Furthermore, a decoder predicts future posture and future motion labels from the current motion label and state. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2021-144679 [Non-patent literature]

[0011] [Non-Patent Document 1] Yuan Ye and Kitani Kris, "Dlow: Diversifying latent flows for diverse human motion prediction", ECCV 2020. [Non-patent document 2] Sedegh Aliakbarian, Fatemesh Saleh, Lars Petersson, Stephen Gould Mathieu Salzmann, "Contextually plausible and diverse 3D human motion prediction", ICCV 2021. [Non-patent document 3] Diederik P. Kingma, Kingma, Danilo J. Rezende, Shakir Mohamed and Max Welling, "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014. Summary of the Invention [Problem to be solved by the invention]

[0012] The above-described conventional techniques have had a problem in that they cannot make better use of various motion predictions by making them more precise and flexible.

[0013] In Non-Patent Documents 1 and 2, only past movements are input as conditions for CVAE, and the category of the predicted movement, such as the type of movement, person, or attribute, cannot be specified. Because of this, if you want to predict only a desired category, many unnecessary predictions outside of that category will be output. Furthermore, even if a prediction for a desired category is output with a low probability, it does not necessarily mean that there is diversity within that category.

[0014] On the other hand, it is thought that decoding specifying the category of an action will be possible by inputting labels such as the category of the action in addition to past actions as the conditions specified during learning in Non-Patent Documents 1 and 2. However, even if it becomes possible to predict the desired category with a high probability, it is not possible to control the diversity within it.

[0015] Furthermore, a decoder that can specify an action category cannot be used in common for conventional applications that do not specify an action category. In other words, multiple decoders must be trained and prepared separately for each different input condition to the decoder. This leads to problems such as increased training time, increased memory requirements for storing trained models, and inconsistent results due to different decoder outputs.

[0016] Furthermore, Patent Document 1 only predicts one future action and does not support diverse predictions.

[0017] In view of the above-mentioned problems of the prior art, an object of the present invention is to provide a prediction device that can perform various motion predictions precisely and flexibly, and also to provide a corresponding program and learning method. [Means for solving the problem]

[0018] To achieve the above object, the present invention provides a prediction device that predicts and generates future actions from past actions, and generates future actions that reflect the action categories on the time series by decoding the past actions, latent codes, and action categories set on the time series for each future time.The present invention also provides a program that causes a computer to function as the prediction device.

[0019] The present invention also provides a prediction device that is composed of a deep learning network and that predicts and generates future actions from past actions, and that generates future actions that reflect the action categories on the time series by decoding the past actions, latent codes, and action categories set on the time series for each future time. The method for learning parameters of the deep learning network in the prediction device is characterized in that a learning label, past actions, and future actions are input into an encoder to generate a latent code, the latent code generated by the encoder, the learning label, and past actions are input into the prediction device to predict a future action, and the parameters are learned by evaluating the error between the predicted future action and the learning future action. [Effects of the Invention]

[0020] According to the prediction device of the present invention, by decoding the action categories set on the time series, future actions that reflect the action categories on the time series can be generated, thereby enabling precise prediction, and by also utilizing the latent code at this time, prediction can be made flexibly. According to the learning method of the present invention, the parameters of the prediction device can be learned. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a functional block diagram of a prediction device according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating the configuration of a deep learning network of a prediction device. [Figure 3] FIG. 10 is a diagram illustrating an example of generation of time-varying labels by a label generating unit. [Figure 4] FIG. 10 is a schematic image diagram of a future action label. [Figure 5] FIG. 1 is a configuration diagram of a learning device according to an embodiment. [Figure 6] FIG. 10 is a diagram showing a configuration for learning independently using an individual feature extraction unit or a configuration in which a final layer is added to the individual feature extraction unit. [Figure 7] FIG. 10 is a diagram showing an example of a GUI for specifying a label and an example of a GUI for making a prediction that narrows down the previous prediction result. [Figure 8] FIG. 1 is a diagram illustrating an example of a hardware configuration of a general computer. DETAILED DESCRIPTION OF THE INVENTION

[0022] 1 is a functional block diagram of a prediction device 10 according to one embodiment, which includes a latent code extraction unit 11, a label generation unit 12, a label feature extraction unit 121, an action decoding unit 13, a behavior prediction unit 14, and an individual feature extraction unit 15. The prediction device 10 of this embodiment can have a minimum configuration for performing action prediction, including a label generation unit 12 in addition to the latent code extraction unit 11 and the action decoding unit 13, which are based on the same framework as in Non-Patent Document 1.

[0023] Here, as described above, the past behavior input to the prediction device 10 is expressed as X 1:t The predicted future behavior output by the prediction device 10 is Y ^ t+1:T Let the correct answer be Y t+1:T Let K be the number of motions to be predicted. The subscripts 1:t indicate past frames, and t+1:T indicate future frames, but are often omitted.

[0024] In addition to the minimum configuration described above, the prediction device 10 may further include a label feature extraction unit 121, a behavior prediction unit 14, and an individual feature extraction unit 15, each of which is shown enclosed in parentheses in FIG. 1 as an optional additional configuration.

[0025] The prediction device 10 can be constructed using a deep learning network, and FIG. 2 is a diagram showing the configuration of the deep learning network of the prediction device 10. (That is, although FIGS. 1 and 2 show the configuration of the same prediction device 10, FIG. 1 shows a simplified functional block diagram, and FIG. 2 shows a schematic diagram of the deep learning network that realizes some of the functional blocks in FIG. 1.)

[0026] Hereinafter, each part of the prediction device 10 shown in FIGS. 1 and 2 will be described in detail.

[0027] First, the latent code extraction unit 11 extracts K latent codes z (K≧2) and outputs these K latent codes z to the motion decoding unit 13. The internal processing of this unit will be described. To be precise, the latent codes are individually identified as "z k (1≦k≦K)”, which will be abbreviated as “z” to represent any one of them. As shown in FIG. 2, the latent code extraction unit 11 can be realized either as a first configuration including a random number generator 111 and a mapping unit 112, or as a second configuration including only an independent sampling unit 113.

[0028] In the first configuration, the random number generator 111 samples ε (corresponding to a seed value) as a random number following a Gaussian distribution as shown in the following equation (111), and the mapping unit 112 samples the parameter ψ as shown in the following equation (112). k Let Tψ be a mapping function k By this, K latent codes z k In the second configuration, the independent sampling unit 113 may sample z K times independently from the prior distribution p as shown in the following equation (113).

[0029]

number

[0030] Although it is expected that the diversity will be lower compared to the first configuration, a fairly wide variety of movements can still be decoded when the second configuration is adopted. Note that, in this embodiment, it is possible to limit the categories of movements predicted as the output of the prediction device 10, but sampling with the second configuration tends to limit the categories to be decoded more easily than with the first configuration.

[0031] Next, the label generation unit 12 receives a user specification and generates a label L=L of a desired category to be decoded for each of the K predicted actions. k t+1:T (1≦k≦K) is output to the motion decoding unit 13. (Note that when the label feature extraction unit 121 is additionally used as an optional configuration, the output label L of the label generation unit 12 is set to L k t+1:T is input to the label feature extraction unit 121, and the feature h l k t+1:T (1≦k≦K) is output to the motion decoding unit 13.)

[0032] Here, the label-specifiable behavior types are defined in advance as categories. For example, if there are five behavior categories: [walking, running, boxing, throwing, and conversational gestures], and the desired future category to be decoded by the user is "walking," the label L is assigned [1, 0, 0, 0, 0] repeatedly for the number of predicted frames to obtain the time series label L. k t+1:T If all K future actions are specified as "walk", the same K time series labels are used.

[0033] Similarly, for "running" label L should be [0,1,0,0,0], for boxing label should be [0,0,1,0,0], for "throwing" label should be [0,0,0,1,0], and for "gestures during conversation" label should be [0,0,0,0,1], which should be output repeatedly for the number of predicted frames.

[0034] Furthermore, depending on the user's discretion, the desired label may be composed of a single category as described above, or may be composed of a mixture (blend) of multiple categories. If multiple categories can be output (if the user desires to output a mixture of multiple categories), a label vector can be specified as a weight when blending each category so that the sum of the label values ​​of all categories equals 1. For example, if either "walking" or "running" is acceptable, [0.5,0.5,0,0,0] can be used, and if any action is acceptable, [0.2,0.2,0.2,0.2,0.2] can be used repeatedly for the number of predicted frames to generate a label. If you want to decode K actions, give the same time-series label K times and generate L k t+1:T (1≦k≦K) is sufficient.

[0035] Furthermore, although the above explanation is an example of specifying time-invariant labels for times t+1, t+2, ..., T of the motion prediction target, the label generation unit 12 can also specify a different label for each frame to be predicted according to user specification and generate time-variant labels. By utilizing this property, it is also possible to explicitly add variety to the transition start timing of motion types in predicted motion.

[0036] For example, as shown in Fig. 3, an example of generation of time-varying labels by the label generation unit 121, when a value [0,1,0,0,0] meaning "run" is specified for the label L, the transition timing may be automatically varied and output by the label generation unit 121. For example, for one sample (e.g., k=1), the label "run" is repeatedly specified for the number of predicted frames from the timing of the t+1 frame, and for another sample (e.g., k=2), the label "run" is specified from the timing of the t+2 frame, and the label before that is specified as "none (e.g., value [0.2,0.2,0.2,0.2,0.2])". The label "none" can function as a label corresponding to the continuation of a past action into the future.

[0037] In this way, by setting various transition start timings in advance, such as t+1, t+2, t+4, etc., the label generating unit 121 generates various time series labels L k t+1:T (1≦k≦K) can be generated automatically.

[0038] As described above, the label generation unit 121 can generate labels of constant values ​​(constant values ​​of a single category or mixed categories) across times t+1, t+2, ..., T for the motion prediction target, according to user specifications, or time-varying labels that transition to a single category specified by the user at various transition times, and output these to the motion decoding unit 13.

[0039] As described above, by adding the label feature extraction unit 121 as an optional configuration, the labels generated by the label generation unit 121 may be converted into feature quantities and then output to the motion decoding unit 13 .

[0040] The label feature extraction unit 121 can be configured by an MLP (Multilayer Perceptron) or the like, and extracts features h from the label L. l If the label L is composed of the same label that does not change over time, then the h l The time series label feature h l k t+1:T (1≦k≦K) can be obtained. Similarly, in the case of time-varying features, the time-series label feature h l k t+1:T (1≦k≦K) can be obtained.

[0041] The configuration using the label feature extraction unit 121 is optional, so it can be omitted to obtain the label L k t+1:T It is also possible to input the feature directly to the decoder (motion decoding unit 13) without extracting the feature from the L k t+1:T(That is, the label feature extraction unit 121 corresponds to a configuration in which the feature extraction process performed by the motion decoding unit 13 when the label feature extraction unit 121 is not used is performed as preprocessing outside the motion decoding unit 13. By configuring it as an external configuration, it becomes possible to adjust the network configuration and parameters in the external configuration, and it is expected that better feature extraction will be performed.) Thereafter, the feature h l k t+1:T Label L k t+1:T Since it can be handled in the same way as

[0042] The above describes a case where a user manually assigns a desired label to the label generation unit 12 as an input label. However, as another embodiment, by utilizing the optional behavior prediction unit 14, an automatic label automatically output by the behavior prediction unit 14 based on behavior prediction may be input to the label generation unit 12 instead of a manual label.

[0043] That is, the content of this automatic label is the past action X input to the prediction device 10. 1:t The predicted future behavior Y ^ t+1:T The motion category of the prediction device 10 is calculated based on the actual motion Y ^ t+1:T The categories of the automatic labels can be set in advance to be the same as the categories of the manual labels. For example, if the categories of the behavior types in the manual labels are set to five types: [walking, running, boxing, throwing, gestures during conversation], the automatic labels can be output in the same categories.

[0044] 1 and 2, both manual labels and automatic labels are depicted as inputs to the label generating unit 12, but depending on the embodiment, only manual labels or automatic labels may be input.

[0045] The action prediction unit 14 receives the input of the 1:t frame (input action X 1:tor a normal video representation of this) 1:t or some of the information may be used. 1:t It can be other types of information such as video or audio at the same time. Prediction from other types of information is possible. 1:t Unlike predictions based only on the video, other information such as gaze, attributes, and conversation can also be used, which may make it easier to predict behavior that changes frequently. In particular, there are many existing methods for predicting behavior from video, and the behavior prediction unit 14 may use any of the existing methods.

[0046] Some examples of behavior prediction that can be used in the behavior prediction unit 14 will be introduced.

[0047] Non-Patent Document 4 makes it possible to caption future actions from video. Non-Patent Document 5 makes it possible to predict whether a pedestrian will run out into the road based on a bounding box estimated from video, the video context, posture, and vehicle speed. Non-Patent Document 6 predicts actions from video. In doing so, it also predicts whether the future is predictable. If the predictability is high, a more specific prediction (lower-level prediction) is made, and if not, an abstract prediction (higher-level prediction) is automatically made. For example, an abstract prediction might be "greeting," and specific predictions might be "shaking hands," "high-fiving," or "bowing." If the prediction result is "greeting," it can be determined that other specific actions at lower levels, such as "shaking hands," "high-fiving," or "bowing," are also likely.

[0048] [Non-Patent Document 4] Mehrdad Hosseinzadeh and Yang Wang, "Video Captioning of Future Frames", WACV, 2021. [Non-Patent Document 5] Kotseruba et al., "Benchmark for Evaluating Pedestrian Action Prediction", WACV, 2021. [Non-patent Document 6] Didac Suris, Ruoshi Liu and Carl Vondrick, "Learning the Predictability of the Future", CVPR, 2021.

[0049] In addition, when there are multiple behavior prediction results by the behavior prediction unit 14, the label generation unit 12 can automatically generate labels using these multiple results. For example, if a running prediction is obtained with 40% and a walking prediction with 60%, and K=10, four time series labels may be generated from the running behavior and six time series labels may be generated from the walking behavior. In hierarchical behavior prediction such as that in Non-Patent Document 6, lower-level specific behaviors may be used as multiple prediction results.

[0050] The generation of time-series labels in the label generation unit 12, which receives the prediction results from the action prediction unit 14, may be a combination of the above-mentioned method of diversifying transition timing and the method of using multiple action prediction results. Figure 4 shows a schematic image diagram of a future action label generated using such a combination of labels. This method can also be used when manual labels are input to the label generation unit 12. That is, the label generation unit 12 can automatically set various transition timings for a single manually specified category, and for multiple manually specified categories, it can generate labels as shown in Figure 4 by combining cases where transitions are made to each category and cases where the transition timing within each category is varied.

[0051] The action decoding unit 13 extracts the K latent codes z and the past actions X 1:t and K time series labels L k t+1:T Decoder G θ and predicts K actions as shown in the following equation (13).

[0052]

number

[0053] Here, inside the decoder, motion features are extracted from past motions X. For example, we use trained gated recurrent units (GRU cells) to extract motion features from past motions X. 1:t From operation characteristics h x h x and z are time series labels L t+1:T Unlike the previous example, there is no time series data with future changes, so the same value is used in every future frame. x The same value is used K times. x and each z k and time series label L k t+1:T Combine them to create future action Y ^k t+1:T can be decoded.

[0054] The prediction device 10 can predict future actions with a variety of labels as described above by learning the parameters of its deep learning network in advance. This learning method will be explained below. The use of the optional personal feature extraction unit 15 in the prediction device 10 will be described later.

[0055] In this embodiment, as in Non-Patent Document 1, the motion predictor is trained using CVAE, and then the mapping function is trained using CVAE or the like. When using the procedure of equation (113) to extract latent codes, the method for training the mapping function is not essential, so the explanation of training the mapping function will be omitted. (Note that even when using the mapping functions of equations (111) and (112), a separate, pre-set mapping function may be used.) The training of the motion predictor will be briefly explained below.

[0056] FIG. 5 is a block diagram of a learning device 40 according to one embodiment. The learning device 40 includes an encoder 20 having networks NW21-25, a decoder 10 having the same configuration as the prediction device 10 of FIGS. 1 and 2, and an error evaluation unit 30 having networks NW31-33. Similar to a typical deep learning framework, the learning device 40 uses learning data to perform error forward propagation in the order of the encoder 20, the decoder 10, and the error evaluation unit 30 to evaluate the error between the future action Y^ predicted from the input past action X and the correct future action Y. The parameters of the decoder 10 (prediction device 10) are repeatedly updated by performing error backpropagation in the opposite direction to the forward propagation using this evaluation error, thereby learning the parameters of the decoder 10. Note that the functional block configuration for parameter updating is not shown in FIG. 5.

[0057] As mentioned above, the motion predictor (decoder 10, prediction device 10) can be trained using a CVAE (Conditional Variational Autoencoder). In a similar framework to Non-Patent Document 1, the past motion X is applied to the conditions of the encoder 20 and the decoder 10. 1:t The input to the encoder 20 is the correct answer for the future action Y t+1:T The output of the decoder 10 is the predicted future motion Y^ t+1:T This becomes:

[0058] In this embodiment, a label L is further added as a condition for the encoder 20 and the decoder 10. When the training data set has a transition of behavior, the label of the condition is added as a time series label L. 1:t and L t+1:T However, even if training is done using a dataset without motion transitions, the decoder will not be able to generate time series labels L t+1:T As in the case of prediction, the encoder 20 may extract features from the label L using the network 21 such as an MLP.

[0059] That is, in the encoder 20, as shown in the figure, the label L is input directly or after feature extraction in the network 21 such as an MLP, to the network 25 of the MLP. 1:t The results were analyzed by treating the GRU cells with NW22 and the MLP and RNN with NW23. x ,h i Obtain these features h x ,h i Enter the answer Y into NW25 of the MLP. t+1:T is processed by NW24 of GRU cells to obtain future feature h, which is then input to NW25 of MLP.

[0060] The NW25 of the MLP uses the above label L (or features extracted from label L) and each feature h x ,h i , h as input to generate distribution parameters, the mean μ and variance σ, and output them to the latent code extraction unit 11 (independent sampling unit 113) of the decoder 10. Note that the NW 23, such as MLP or RNN, is an optional component in the encoder 20, and when the individual feature extraction unit 15 is used in the decoder 10, this NW 23, such as MLP or RNN, is additionally used as a corresponding component on the encoder 20 side.

[0061] The processing of the decoder 10 during learning is the same as that in the trained prediction device 10, except that the prior distribution p of the independent sampling unit 113 is determined by the mean μ and variance σ, which are distribution parameters output by the encoder 20, and the latent code z is obtained by equation (113) (the latent code z is obtained as "z = μ + ε * σ" using a small random number ε). Therefore, a duplicated explanation will be omitted. t+1:T is the past action X 1:t Or Label L 1:t Alternatively, the output result of the behavior prediction unit 14 may be provided as an input.

[0062] The error evaluation unit 30 evaluates the predicted motion Y output by the decoder 10 using the same framework as in Non-Patent Document 1. ^The prediction error is the MSE (Mean Square Error) between the correct answer Y and the predicted value Y. The encoder 20 and the decoder 10 can be trained using backpropagation in the same way as in the normal CVAE so as to minimize the combined error of the prediction error and the KLD (Kullback-Leibler Divergence).

[0063] The networks NW31, 32, and 33 in the error evaluation unit 30 are optional components when the individual feature extraction unit 15 is used in the decoder 10.

[0064] Regarding the network architecture of the learning device 40, except for the label generation unit 12, the same architecture as that shown in Non-Patent Document 1 as shown in FIG. 5 can be used, but is not limited to this.

[0065] As an optional configuration for explicitly considering the characteristics of an individual, an individual feature extraction unit 15 may be used. As described above, when the individual feature extraction unit 15 is used, the learning device 40 can further use a network 23 such as an MLP or RNN in the encoder 20, and networks NWs 31, 32, and 33 in the error evaluation unit 30.

[0066] In this case, as shown in the figure, the input condition h of the personal feature is input to the NW 25 of the encoder 20 and the decoder (GRU Cells MLP) in the operation decoding unit 13 of the decoder 10. i can be added without considering personal characteristics. x In learning using only the decoder 10, there is a problem that the predicted action by the decoder 10 tends to have different personal characteristics from the input past action. i This problem can be addressed by extracting features such as individual movement, angle, and posture.

[0067] As shown in the figure, X1:t and Y corresponding to the individual t+1:T The individual feature extraction unit 15, which serves as an individual classifier that receives input and outputs an individual, is trained in advance, and the individual feature amount h iThe individual feature extraction unit 15 can be configured with MLP, RNN, etc., and extracts h from X and Y using a classifier excluding the final layer (generally a fully connected layer) of the classifier. i By calculating this, it becomes easier to predict individuals who are not present at the time of learning. (Note that if the individual feature extraction unit 15 is configured with the final layer of the classifier included, the individual feature extraction unit 15 will calculate the individual feature h i By excluding the final layer, the individual feature extraction unit 15 outputs the individual feature h i can be output.)

[0068] Furthermore, for the motion Y^ predicted by the decoder 10, a trained classifier may be used to calculate the error related to classification (typically a cross-entropy error), and error backpropagation may be performed to add the individual identification error Ei to the prediction error. During error backpropagation, it is advisable to fix the weight parameters of the classifier. For example, training may be performed by minimizing the weighted sum of the individual identification error, prediction error, and KL divergence, or training may be performed by alternating between (a) minimizing only the individual identification error and (b) minimizing the weighted sum of the prediction error and KL divergence. This makes it more likely that the motion predicted by the decoder will have the same individual characteristics as the input past motion.

[0069] Using a classifier without a final layer, h obtained from Y^ i And h calculated from X i You can also calculate the MSE of the classifier. At this time, the weight parameters of the classifier are not fixed, but are updated and trained, making it easier to predict people who are not in the training.

[0070] The learning of the classifier (individual feature extraction unit 15) may be performed separately from the learning of the label generation unit 12 and the like, using a configuration of the learning device 40 in which only the individual feature extraction unit 15 as shown in FIG. 6 is used, or a configuration in which NWs 31 and 32 of the error evaluation unit 30 are added to the individual feature extraction unit 15, or the like, or the individual feature extraction unit 15 may be trained in parallel with the learning of the decoder 10 and encoder 20 including the label generation unit 12 and the like, by alternately updating parameters, within the overall configuration of the learning device 40.

[0071] As described above, the prediction device 10 trained by the learning device 40 of this embodiment makes it easier to limit the category of the action to be predicted. In other words, predictions can be almost limited to the desired category, and output noise can be significantly reduced. In addition, the output can be controlled so that the prediction results within the desired category are diverse. Various supplementary examples, additional examples, alternative examples, etc. will be described below.

[0072] (1) As an example of an application of the prediction device 10 according to an embodiment of the present invention, when remote communication, which may involve transmission delays, is conducted using an avatar (an avatar that captures the movements of a remote user), the prediction device 10 can predict the future movements of the avatar and appropriately compensate for the delay. Alternatively, it can prepare natural and diverse movements of avatars, such as those of a mob of spectators, without the need to capture the actual movements of the remote user, thereby ensuring a sense of realism in the virtual space where avatar communication takes place. This enables smooth remote communication and eliminates the need for users to actually travel to remote locations for meetings, etc. (travel to remote locations is not necessarily required). Therefore, such an application of the embodiment of the present invention can reduce carbon dioxide emissions by saving the energy resources required for user travel, thereby contributing to Goal 13 of the United Nations' Sustainable Development Goals (SDGs), "Take urgent action to combat climate change and its impacts."

[0073] (2) The motion data X, Y, etc. (including motion data for learning in the learning device 40) input and output to the prediction device 10 may be any type of data representing the motion of a person, etc. For example, information on the 2D or 3D positions or movements of multiple key points (often joints) that indicate the skeleton, or various other information obtained at the time of each frame may be used. Therefore, when the positions of joints, etc. defined in this motion data X, Y, etc. are rendered as frame images at each time t and played back and displayed as a video, the motion of the person, etc. at each time t is reproduced as a video.

[0074] (3) X 1:t and Y t+1:T h may use 3D coordinates, Euler angles, or other pose representations. i A plurality of features may be input simultaneously. For example, when 3D coordinates are input, it is easy to extract posture and individual movement features, and when Euler angles are input, it is easy to extract individual angle features, so both features may be used. Also, the output of the classifier (individual feature extraction unit 15) may be the same skeletal 3D coordinates and angle information X as the input. In other words, the latent space of the autoencoder becomes hi. X 1:t and Y t+1:T The classifier (individual feature extraction unit 15) that receives the input and identifies an individual may use the same model except for the network architecture and input layer of the label feature extraction unit 121.

[0075] (4) Usage example The prediction device 10 of this embodiment allows users to specify labels, making it possible to efficiently predict actions in a desired category. For example, it can efficiently predict what actions are likely to occur if a person runs out into the road. Furthermore, since labels and timing can be specified, the device is also suitable for creating CG (computer graphics) animations. It is also possible to select from multiple presented actions. Furthermore, when a personal identifier is used as the label content, different actions can be predicted depending on the person who specified it. For example, it is possible to input a student's past actions, output the student's future actions, and present what future actions a teacher would take, making it possible to use the device for skill learning. Furthermore, in predictions involving many action transitions, it is expected that prediction accuracy can be improved by combining the device with behavior prediction.

[0076] As mentioned above, the label generation unit 12 can automatically generate labels according to user specifications, but the user may also explicitly specify the desired labels from among these automatically generated labels. An example of such a GUI (Graphical User Interface) is shown in Example Ea of Figure 7. In addition to past behaviors as input, the user can also specify the number of predictions they wish to make.

[0077] Furthermore, it is possible to make a narrowed down prediction by using the previous prediction result output by the prediction device 10. Example Eb in FIG. 7 shows an example of a GUI for making such a narrowed down prediction. For example, suppose a user who is a CG animation creator uses this motion predictor. If the user determines that the displayed prediction results do not bring out a result close to the image they want to use, the user can instruct the prediction device 10 to that effect, and the display can be switched to the initial screen, example Ea, without performing a narrowed down search.

[0078] On the other hand, suppose that a prediction result (displayed in the form of a moving animation video or the like) that is close to the image the user wants to use appears among the displayed prediction results, and this is the first result (Result 1) in Example Eb. It is preferable to display not only the prediction result but also the label that derived the result, as this makes it easier for the user to narrow down the search.

[0079] In this case, as further shown in Example Eb, by checking Result 1 and specifying the transition start timing range, the user can make a prediction with the transition start timing changed based on Result 1. In this case, as further shown below Result 1, it is preferable to display an interface such as a slider for narrowing down the start transition timing.

[0080] The prediction device 10 can further narrow down the predictions by generating labels under these conditions using the label generation unit 12. In this way, by repeatedly specifying labels and checking the results of the predicted movements, the CG animation creator can narrow down the options and select a prediction that is closer to the image they have in mind.

[0081] Rather than using the displayed latent code as is, the user may specify a nearby latent code (seed for refinement). While latent spaces typically have many dimensions, dimensionality reduction techniques such as t-SNE (t-distribution stochastic neighbor embedding) can be used to reduce the average latent code to two or three dimensions and display it in the GUI. (The covariance of the latent code does not need to be displayed.) The user can specify the range for resampling the seed by drawing a circle (ellipse) or line around a preferred result. For example, if the user feels that a state between Results 1 and 2 is preferable, the user can specify a line passing through Results 1 and 2. This is because latent codes sampled from the line between Results 1 and 2 often decode intermediate states between Results 1 and 2. When the predict button is pressed, the latent code extraction unit 11 resamples z within that range, decodes the behavior, and displays the result.

[0082] The range specified by the user is dimension-reduced data, so it is restored to the original dimension and then decoded. For example, if a line is specified, z 1 and z 2 A line on an R-dimensional vector passing through t·z 1 +(1-t)·z 2 (0≦t≦1) If a circle is specified around a particular result (e.g., result 1), then the size of ε' (ε'~p(ε')) is changed accordingly, and z=μ 1 +ε' σ 1 Here, μ 1 , σ 1 indicates the mean and covariance when the result 1 is obtained. By decoding the multiple latent codes z obtained, a narrowed down prediction can be made, which can be redisplayed on the GUI.

[0083] After receiving the user's specification via the GUI as described above, the latent code extraction unit 11 samples z within the specified range. Then, the label generation unit 12 performs internal processing by uniformly allocating transition start timings within the specified transition start timing range using the allocated number of samples, and generates multiple time series labels with different transition start timings. The latent code z is the same as the code z that output the result 1. 1 The specified category L after the transition will also be the same as Result 1.

[0084] When making a prediction by specifying a category without using the previous prediction result, a latent code z different from the previous one is used. k Simply select (1≦k≦K) to make a prediction and display it. This GUI can of course be used for the first prediction as well as for the second and subsequent refined predictions.

[0085] (5) FIG. 8 is a diagram illustrating an example of the hardware configuration of a general computer device 60. The prediction device 10 and the learning device 40 of this embodiment can be realized as one or more computer devices 60 having such a configuration. When each device is realized using two or more computer devices 60, information required for processing may be transmitted and received via a network. The computer device 60 includes a CPU (Central Processing Unit) 61 that executes predetermined instructions, a GPU (Graphics Processing Unit) 62 as a dedicated processor that executes some or all of the CPU 61's execution instructions in place of or in cooperation with the CPU 61, a RAM 63 as a main memory device that provides a work area for the CPU 61 (and the GPU 62), a ROM 64 as an auxiliary memory device, a communication interface 65, a display 66, an input interface 67 that accepts user input via a mouse, keyboard, touch panel, or the like, and a bus BS for transmitting and receiving data among these.

[0086] Each functional unit of the prediction device 10 and the learning device 40, and the prediction method and learning method executed by each device, can be realized by a CPU 61 and / or a GPU 62 that reads from a ROM 64 and executes a predetermined program corresponding to the function of each unit or step. Both the CPU 61 and the GPU 62 are types of arithmetic devices (processors). Here, when display-related processing is performed, a display 66 also operates in conjunction with the CPU 61 and the GPU 62, and when communication-related processing related to data transmission and reception is performed, a communication interface 65 also operates in conjunction with the CPU 61 and the GPU 62.

[0087] The GUI shown in FIG. 7 may be displayed on a display 66, and a designation input from the user may be received at an input interface 67. [Explanation of symbols]

[0088] 10... Prediction device (decoder), 11... Latent code extraction unit, 12... Label generation unit, 121... Label feature extraction unit, 13... Action decoding unit, 14... Action prediction unit, 15... Individual feature extraction unit 40... learning device, 20... encoder, 30... error evaluation unit

Claims

1. A prediction device that predicts and generates a future action from a past action, a prediction device that receives as input the past actions, latent codes given as numerical values ​​of the actions, and action categories set on a time series for each future time, generates and outputs future actions that reflect the action categories on the time series, A prediction device that is equipped with a computer, which realizes the process of obtaining the output from the input by having the computer execute a program of a trained deep learning network.

2. The prediction device described in Claim 1, characterized in that the computer generates the latent code over multiple times by executing a program, and generates the future actions over multiple times by changing the timing of the transition of the action category on the corresponding time series for each of the multiple transition codes.

3. The prediction device described in Claim 1, characterized in that the computer accepts the specification of the type of the action category as a manual specification by executing a program.

4. A prediction device as described in Claim 1, characterized in that the computer automatically specifies the type of action category as the action category of a future action estimated from the past action by executing a program.

5. When the computer executes the program, extracting a feature from the past action that identifies an actor performing the past action; The prediction device according to claim 1 , wherein the future action is generated based on the past action, a latent code given as a numerical value of the action, the action category, and the features.

6. When the computer executes the program, generating a plurality of the latent codes, and generating a plurality of the future actions based on each of the plurality of latent codes; accepting a user's selection from the plurality of generated future actions; The prediction device according to claim 1, wherein a future action is generated by regenerating the future action using a latent code corresponding to the selected future action or a latent code in the vicinity of the selected latent code.

7. A prediction device as described in Claim 6, characterized in that the computer executes a program to provide a graphical user interface for accepting selections by the user.

8. A prediction device as described in Claim 6, characterized in that the computer, by executing a program, provides a graphical user interface for accepting the specification of a latent code corresponding to the selected future action or a latent code in the vicinity of that latent code.

9. The latent code is defined as a value in a high-dimensional space, When the computer executes the program, The prediction device according to claim 8, wherein the graphical user interface is provided by displaying the latent code after reducing its dimension to two or three dimensions, and accepting specification of a range of the latent code in the reduced dimension state.

10. A program that causes a computer to function as the prediction device according to any one of claims 1 to 9.

11. A prediction device that is configured with a deep learning network and predicts and generates future actions from past actions, A method for learning parameters of a deep learning network in a prediction device that generates future actions that reflect action categories on a time series based on the past actions, latent codes given as action numerical values, and action categories set on a time series for each future time, comprising: A latent code is generated based on the learning label, past actions, and future actions, and the latent code is given as a numerical value of the action. The generated latent code, the learning label, and the past action are input to the prediction device to predict a future action; A learning method comprising: learning the parameters by evaluating an error between the predicted future motion and a future motion for learning.

Images