Prediction device, learning method and program

Abstract

To provide a prediction device capable of predicting a variety of motions more accurately and flexibly.SOLUTION: A prediction device for predicting a plurality of future motions from past motions is provided, the prediction device being configured to perform processing 61 for encoding the past motions to a plurality of latent codes in a category for specified future motions, and processing 64 for decoding each of the plurality of latent codes to obtain a result of predicting a plurality of future motions in the specified category corresponding to the past motions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a prediction device that predicts future actions from past actions, and a learning method and program therefor. [Background technology]

[0002] Predicting a person's future movements is useful for optimizing the control of cars and robots, avoiding danger, etc. To make predictions, in addition to capturing a person's trajectory using cameras and sensors, it is also common to capture multiple skeletal points such as hands and feet to predict more detailed movements.

[0003] One future is not enough to consider possibilities, but past actions of people X 1:t For future action Y t+1:T Here, as shown below, the subscript 1:t of X indicates past frames X(k) (frames up to the current time t) spanning time k=1, 2, ..., t-1, t, and the subscript t+1:T of Y indicates future frames Y(k) spanning time k=t+1, t+2, ..., T-1, T. However, if it is clear from the context, they may be omitted and written as past frame X, future frame Y, etc. Hereinafter, this notation will be used for frames representing actions. Past action frame X 1:t ={X(k)|k=1,2,…,t-1,t} (sometimes abbreviated as X) Future Action Frame Y t+1:T ={Y(k)|k=t+1,t+2,…,T-1,T} (sometimes abbreviated as Y)

[0004] Methods for predicting multiple actions often use a conditional variational autoencoder (CVAE).

[0005] Non-Patent Document 1 and Non-Patent Document 2 use CVAE for motion estimation.

[0006] The VAE (variational autoencoder) on which CVAE is based has an encoder and decoder like a normal autoencoder, and is normalized so that the latent space is continuous. During inference, latent code z can be sampled from the latent space and decoded, with z being close to each other resulting in similar decoded results. However, this similarity is not uniform across distance, and the extent to which a large distance results in different behavior depends heavily on the model and training parameters.

[0007] CVAE adds conditions to the encoder input and decoder input of VAE, making it easier to obtain decoding results under specified conditions. In inferring diverse actions, by inputting past action X as a condition and selecting and decoding multiple different latent codes z, it is possible to predict multiple different future actions Y for past action X.

[0008] Although Non-Patent Documents 1 and 2 apply CVAE to motion prediction, Non-Patent Document 3 discloses details of general CVAE without limiting its application.

[0009] Patent Document 1 does not use CVAE or VAE, but uses a recurrent neural network (RNN) with a general encoder and decoder to predict motion and motion labels from video. This method estimates joint positions from past videos, estimates motion characteristics based on optical flow from past videos, and encodes these to obtain a latent state. The current motion label is estimated from that state. Furthermore, a decoder predicts future posture and future motion labels from the current motion label and state. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2021-144679 [Non-patent literature]

[0011] [Non-Patent Document 1] Yuan Ye and Kitani Kris, "Dlow: Diversifying latent flows for diverse human motion prediction", ECCV 2020. [Non-patent document 2] Sedegh Aliakbarian, Fatemesh Saleh, Lars Petersson, Stephen Gould Mathieu Salzmann, "Contextually plausible and diverse 3D human motion prediction", ICCV 2021. [Non-patent document 3] Diederik P. Kingma, Kingma, Danilo J. Rezende, Shakir Mohamed and Max Welling, "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014. Summary of the Invention [Problem to be solved by the invention]

[0012] The above-described conventional techniques have had a problem in that they cannot make better use of various motion predictions by making them more precise and flexible.

[0013] In Non-Patent Documents 1 and 2, only past movements are input as conditions for CVAE, and the category of the predicted movement, such as the type of movement, person, or attribute, cannot be specified. Because of this, if you want to predict only a desired category, many unnecessary predictions outside of that category will be output. Furthermore, even if a prediction for a desired category is output with a low probability, it does not necessarily mean that there is diversity within that category.

[0014] On the other hand, it is thought that decoding specifying the category of an action will be possible by inputting labels such as the category of the action in addition to past actions as the conditions specified during learning in Non-Patent Documents 1 and 2. However, even if it becomes possible to predict the desired category with a high probability, it is not possible to control the diversity within it.

[0015] Furthermore, a decoder that can specify an action category cannot be used in common for conventional applications that do not specify an action category. In other words, multiple decoders must be trained and prepared separately for each different input condition to the decoder. This leads to problems such as increased training time, increased memory requirements for storing trained models, and inconsistent results due to different decoder outputs.

[0016] Furthermore, Patent Document 1 only predicts one future action and does not support diverse predictions.

[0017] In view of the above-mentioned problems of the conventional technology, an object of the present invention is to provide a prediction device that can perform various motion predictions more precisely and flexibly, and a learning method and program therefor. [Means for solving the problem]

[0018] To achieve the above object, the present invention provides a prediction device for predicting multiple future actions from past actions, the prediction device comprising: a process for encoding the past actions into multiple latent codes based on categories for specified future actions; and a process for decoding the multiple latent codes to obtain prediction results of multiple future actions based on the specified categories corresponding to the past actions. The present invention also provides a learning method and a program corresponding to the prediction device. The learning method is a method for learning parameters of a deep learning network used by a prediction device, which learns parameters of a first network that encodes the past actions into multiple latent codes based on a category for a specified future action, and parameters of a second network that decodes the multiple latent codes to obtain prediction results of multiple future actions based on the specified category, corresponding to the past actions; when learning the parameters of the first network, learning is performed using at least a first reconstruction error (Er) calculated by comparing the prediction results of multiple future actions obtained by processing past actions for learning with the first network and the second network with a ground-truth future action for learning; when learning the parameters of the second network, learning is performed using at least a label pose error (El) calculated by comparing the prediction results of multiple future actions obtained by processing past actions for learning with the first network and the second network with a pseudo-ground-truth future action for learning; and for each prediction result of the multiple future actions, searching for the pseudo-ground-truth future action from learning data that falls into the specified category as one similar to the prediction result. [Effects of the Invention]

[0019] According to the prediction device of the present invention, a specified category is taken into consideration when making a prediction, and the multiple latent codes are encoded to obtain multiple prediction results of future actions by decoding the multiple latent codes, so that various motion predictions can be made more precise and flexible by taking the specified category into consideration.Furthermore, according to the learning method of the present invention, parameters used in the prediction device can be learned. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a functional block diagram of a prediction device according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a deep learning network structure according to an embodiment of the prediction device. [Figure 3] FIG. 2 is a functional block diagram of a prediction device according to another embodiment different from that of FIG. 1. [Figure 4] FIG. 2 is a functional block diagram of an embodiment of a first learning device for learning parameters of a motion decoding unit. [Figure 5] FIG. 10 is a functional block diagram of a second learning device according to an embodiment for learning parameters of a latent chord extraction unit. [Figure 6] FIG. 5 is a diagram showing the deep learning network structure of the first learning device of FIG. [Figure 7] FIG. 6 is a diagram showing the deep learning network structure of the second learning device of FIG. [Figure 8] FIG. 5 is a functional block diagram of a first learning device according to another embodiment different from that of FIG. 4. [Figure 9] 10A and 10B are diagrams illustrating a schematic example of calculation of a reconstruction error and calculation of a category-specified posture error. [Figure 10] FIG. 1 is a diagram illustrating an example of a hardware configuration of a general computer device. DETAILED DESCRIPTION OF THE INVENTION

[0021] 1 is a functional block diagram of a prediction device 60 according to one embodiment. The prediction device 60 is configured with a deep learning network and includes at least a mapping function parameter generation unit 62, a mapping function unit 63, a motion decoding unit 64, and a random number generator RG. The prediction device 60 of this embodiment follows the same method as in Non-Patent Document 1 and further performs additional processing that takes categories into consideration, thereby enabling a variety of motion predictions in desired categories.

[0022] The user can specify the desired category as a label L related to the future to be decoded. For example, L can be an action category or a personal attribute, but here we will explain it as an action category. (As mentioned above, we will use X to denote past actions.) 1:t , the predicted future behavior is Y ^ t+1:T , the correct answer is Y t+1:T Let us assume that we want to predict K (K≧2) diverse actions. The subscripts 1:t indicate past frames, and t+1:T indicate future frames, but are often omitted.

[0023] FIG. 2 is a diagram showing a deep learning network structure according to an embodiment of each of these units 62, 63, 64, and RG that the prediction device 60 includes.

[0024] As shown in FIG. 1 by enclosing functional blocks in curly brackets {}, the prediction device 60 may further include a feature extraction unit 67 as a configuration for additionally performing preprocessing for the mapping function parameter generation unit 62. (The processing of this feature extraction unit 67 will be described later when explaining the second learning device 92 (FIG. 5).) The prediction device 60 may further include a feature calculation unit 65 and a feature emphasis unit 66 as configurations for additionally performing intermediate processing between the mapping function unit 63 and the motion decoding unit 64. (When these additional components 65 and 66 are not used, the motion decoding unit 64 receives the output from the mapping function unit 63, as indicated by line L63. When these additional components 65 and 66 are used, the mapping function unit 63 outputs to the feature calculation unit 65, and then the motion decoding unit 64 receives the output from the feature emphasis unit 66, as indicated by line L66.)

[0025] 1, the mapping function parameter generating unit 62, the mapping function unit 63, the random number generator RG, and the feature extracting unit 67, the feature calculating unit 65, and the feature emphasizing unit 66, which can be optionally added, constitute a latent code extracting unit 61. The latent code extracting unit 61 extracts the past action X 1:t and the label L (and optional personal feature emphasis instructions) as input, and the latent code zk to the motion decoding unit 64.

[0026] The processing of each part in FIG. 1 will be described in detail below.

[0027] First, the latent code extraction unit 61 extracts K latent codes z (K≧2), and outputs these K latent codes z to the operation decoding unit 64 via either the line L63 or the line L66. The internal processing will now be described.

[0028] The mapping function parameter generating unit 62 receives the label L of the action category (specified by the user) and the past action X. 1:t (provided by the user) and a set of K parameters ψ 1 , ψ 2 , ···ψ K is output to the mapping function unit 63. k It is expressed as (1≦k≦K), where K is specified in advance as 2 or more. The processing of this mapping function parameter generation unit 62 can be expressed as a function Qr(L,X) as in the following equation (62).

[0029]

number

[0030] That is, the parameter ψ is calculated by the function Qr. k (1≦k≦K) is the size n z ×n z A real square matrix A k and n z a real vector b k It can be calculated as: n z is the number of dimensions of the latent space. As shown in Figure 2, the function Qr that handles the processing of the mapping function parameter generator 62 is a GRU cell (gated recurrent unit) that receives inputs L and X at the front end and generates intermediate output h L After obtaining x (operational feature), this intermediate output h LThis can be achieved by further inputting x into an MLP (multilayer perceptron).

[0031] The mapping function unit 63 generates the K parameters ψ obtained by the mapping function parameter generation unit 62. k and the output ε (corresponding to the so-called seed value) of the random number generator RG, which samples from the Gaussian distribution p as "ε~p(ε)", are used as inputs, and by applying a mapping function by affine transformation as shown in the following equation (63), K latent codes z k (1≦k≦K).

[0032]

number

[0033] The latent code z obtained by the mapping function part 63 k (1≦k≦K) may be output directly to the motion decoding unit 64 as shown by line L63, or may be output to the feature calculation unit 65 and subjected to additional feature emphasis processing by the feature calculation unit 65 and the feature emphasis unit 66, and then output to the motion decoding unit 64 as shown by line L66.

[0034] The action decoder 64 detects past actions X 1:t and latent code z k (1≦k≦K) as input, and the decoder G θ By giving K future actions Y ^k (1≦k≦K) is predicted and output.

[0035]

number

[0036] The past action X input to the action decoder 64 1:t and the past action X input to the latent code extraction unit 61 1:t is the same thing.

[0037] Decoder G constituting the operation decoding unit 64 θ As shown in the network structure in Figure 2, inside the network, action features are extracted from past actions X. For example, action features h are extracted from past actions X using trained GRU cells (gated recurrent units). x The motion feature h x and each z k The future action Y is calculated by combining the two and inputting them into a decoder consisting of GRU cells (gated recurrent units) on the front side and MLP (multilayer perceptron) on the back side. ^k can be decoded.

[0038] In this embodiment, the latent code z is output by the latent code extraction unit 61, which has learned the parameters of the deep learning network using a learning method described later. k By making the above prediction in the action decoding unit 64, it is possible to decode an action with a limited category, such as maintaining a desired category or transitioning to a desired category, regardless of the past action X.

[0039] For example, suppose the label L of an action category includes "walking," "running," "throwing," "punching," "gestures," etc. When the label L of the action category is "walking," if the past action is walking, the future will remain "walking," and if the past action is other than walking, such as "running," "throwing," "punching," or "gestures," the future will transition to "walking."

[0040] An additional embodiment will be described below in which a feature calculation unit 65 and a feature emphasis unit 66 are further used to add emphasis instruction processing to the content of the label L in the action prediction by the prediction device 60.

[0041] Here, the label L may be a label for identifying an individual and a label for a category of an action to be predicted, which may be input simultaneously. That is, the label L may be configured as L=(L1, L2), where L is a first label L1 for identifying an individual and a second label L2 for an action category. In this case, the feature calculation unit 65 and the feature emphasis unit 66 accept a user-input designation of whether or not to emphasize a feature. The user may input the degree of emphasis, or may input what to emphasize.

[0042] For example, when feature emphasis is input, the feature calculation unit 65 calculates how the specified individual's motion differs from the average motion in the latent space. For example, when there are six people from A to E (when L1 = A to E), the mapping function unit 63 calculates z for six people by mapping the same ε using Qr(A, X), Qr(B, X), ..., Qr(E, X). k Calculate each of them and calculate z for 6 people. k The average latent code z k μ If the specified individual is person A, the latent code z output by person A is calculated. k The average latent code z k μ The difference between these two is taken as the feature vector z of A's actions. k F =z k -z k μ Let's say.

[0043] The feature emphasis unit 66 calculates a K-dimensional feature weight vector α·z related to the degree of emphasis. k F , modulates the latent code by the following equation (66), and outputs it to the motion decoding unit 64. z k = z k +α·z k F …(66)

[0044] Note that this equation (66) can be expressed as z k Regarding the value z before update k [更新前]Substitute into the right side and set the updated value z as the value of the left side that is equal to this. k [更新後] Here, z on the right side k F Regarding z k F =z k [更新前] -z k μ is required. z k [更新後] = z k [更新前] +α·z k F …(66')

[0045] The value of the α element means a feature weight, where if it is positive, the individual features are emphasized, and if it is negative, the individual emphasis is eliminated. The feature weight α may be preset, or if there is an instruction for the degree of feature emphasis (individual feature emphasis instruction in Figure 1), the feature emphasis unit 66 may set a value of α according to that instruction. For example, when the degree of emphasis is specified as 0 to 0.5 times, the feature weight α may be set to a random number within the range of 0 to 0.5. Setting both positive and negative values ​​to α, or various other values, makes it possible to create even more diversity, which is particularly effective when creating CG where the emphasis of movement features is desired. The latent code z, which performs feature emphasis in the latent space, is k When this is decoded, the output is a motion that emphasizes / reduces (minimizes) the motion characteristics of person A. Also, since predicted motions tend to be averaged, setting the degree of emphasis of motion characteristics, i.e., the value of α, to 0 or higher has the effect of preventing the reduction of personal characteristics in the predicted motion.

[0046] 3 is a functional block diagram of a prediction device 60 according to another embodiment different from that of FIG. 1. The prediction device 60 of FIG. 3 includes a selective latent code extraction unit 71 and an action decoding unit 64. The selective latent code extraction unit 71 further includes an unlabeled latent code extraction unit 72, a (labeled) latent code extraction unit 61, and a selection unit 74. This (labeled) latent code extraction unit 61 is the same as the latent code extraction unit 61 of FIG. 1.

[0047] That is, the prediction device 60 in Fig. 3 corresponds to the latent code extraction unit 61 in Fig. 1 replaced with a selective latent code extraction unit 71. In other words, the prediction device 60 in Fig. 3 is the latent code extraction unit 61 of the prediction device 60 in Fig. 1, to which an unlabeled latent code extraction unit 72 and a selection unit 74 are added to form the selective latent code extraction unit 71.

[0048] According to the selective latent code extraction unit 71 in Fig. 3, since the action decoding unit 64 is fixed for learning, the unlabeled latent code extraction unit 72 that does not specify the action category label L as disclosed in Non-Patent Document 1 and the (labeled) latent code extraction unit 61 of this embodiment can be used together. The selection unit 74 selects, according to a user input, whether to use the unlabeled latent code extraction unit 72 or the (labeled) latent code extraction unit 61 as the latent code extraction unit to be used, and outputs the latent code z obtained by the selected latent code extraction unit (72 or 61). k is output to the motion decoding unit 64.

[0049] That is, when the label L of the action category is input and decoding is desired, the output of the (labeled) latent code extraction unit 61 is selected by the selection unit 74, and when the label L of the action category is not input, the output of the unlabeled latent code extraction unit 72 is selected by the selection unit 74. The outputs of these (unlabeled, labeled) latent code extraction units (72 or 61) may be changed as appropriate.

[0050] For example, latent code z k and selects either z k Alternatively, the mapping function parameter generator 62 in the (labeled) latent code extractor 61 may output a mapping function parameter, and the selector 74 may perform an affine transformation from ε to obtain z kAlternatively, the function Qr itself for generating parameters for the mapping function may be output. Note that Qr(X) trained without label input and Qr(X,L) trained with label input are prepared in advance. Instead of Qr(X,L), Qrl(X) for label L may be trained separately. If the action category L is not specified, Qr(X) is used. If the action category L is specified, Qr(X,L) is selected.

[0051] Also, the decoder G in the operation decoding unit 64 θ (z, X) is common regardless of whether or not a label is used, and a future action Y^ is predicted from the latent code z and the past action X. The latent code extraction unit 72, which does not specify a label, is used when natural prediction is desired without limiting the action category L, while the latent code extraction unit 61, which specifies a label, is used when prediction is desired by specifying the action category L. Since the action decoding unit 64 is the same, it is easy to match the latent codes of both units. For example, if the latent codes output from the unlabeled latent code extraction unit 72 and the (labeled) latent code extraction unit 61 are the same, the same future action can be output. Furthermore, calculations such as likelihood using μ or Σ from the latent code z can be performed using the same criteria. Furthermore, even when the action category L is specified, the selection unit 74 may select latent codes at a predetermined ratio from the latent codes output from the unlabeled latent code extraction unit 72 and the latent codes output from the (labeled) latent code extraction unit 61 (using a latent code obtained by weighting both latent codes at a predetermined ratio).

[0052] The configuration of the prediction device 60 in Fig. 3 described above is suitable for the case where, during or after the learning of the deep learning network constituting the prediction device 60 described next, the user checks whether the learning is being performed appropriately or conducts a comparative experiment on the presence or absence of the label L. That is, the selection unit 74 can accept a user's specification of which (unlabeled or labeled) latent code extraction unit (72 or 61) to use, or whether to use the weighted sum of both of these latent codes.

[0053] Below, we will explain a method for learning the parameters of the deep learning network that constitutes the prediction device 60 of one embodiment (the parameters of the function Qr of the mapping function parameter generation unit 62 in the latent code extraction unit 61 and the parameters of the operation decoding unit 64).

[0054] FIG. 4 is a functional block diagram of one embodiment of a first learning device 91 for learning parameters of the motion decoding unit 64. The first learning device 91 includes a motion encoding unit 81, a sampling unit 82, and a motion decoding unit 64 (the parameter learning target), and updates the parameters of the motion decoding unit 64 by applying an existing method such as backpropagation using a reconstruction error Er evaluated by processing in this order.

[0055] Figure 5 is a functional block diagram of a second learning device 92 according to one embodiment for learning the parameters (mapping function Qr) of the latent code extraction unit 61. The second learning device 92 has the same configuration as the prediction device 60 of Figure 1, and updates the parameters of the latent code extraction unit 61 by applying an existing method such as backpropagation using the reconstruction error Er', diversity error Ed, label pose error El, etc., which are evaluated by performing processing in the same order as the prediction device 60 of Figure 1.

[0056] When the second learning device 92 updates the parameters of the latent code extraction unit 61, the parameters of the action decoding unit 64 in the second learning device 92 are fixed as values ​​during learning (or learned values). First, the parameters of the action decoding unit 64 are learned by the first learning device 91, and then the parameters of the latent code extraction unit 61 are learned by the second learning device 92, thereby obtaining the final learning results of these parameters. The completion of learning in each of the first learning device 91 and the second learning device 92 can be determined by any existing method, such as whether the number of repetitions has reached a threshold or whether the evaluation error is determined to be smaller than a threshold.

[0057] FIG. 6 is a diagram showing the deep learning network structure of the first learning device 91 in FIG. 4, and FIG. 7 is a diagram showing the deep learning network structure of the second learning device 92 in FIG. 5. The details of the learning methods by the first learning device 91 and the second learning device 92 will be explained in order below. Note that the learning method of this embodiment follows a similar framework to Non-Patent Document 1. First, the first learning device 91 uses CVAE to learn parameters of the motion decoding unit 64 as a motion predictor. Then, the second learning device 92, which has a motion decoding unit 64 set with these learning parameters fixed, uses CVAE or the like to learn a mapping function Qr. However, learning is performed by evaluating various errors that take into account the label L, which is not considered in Non-Patent Document 1.

[0058] The first learning device 91 performs learning through the following process.

[0059] The past action X is used as learning data for the condition of the action encoding unit 81 during parameter learning. 1:t and its future true behavior Y t+1:t and label L. (Here, the true future action Y t+1:t is prepared to maintain the behavior according to the label L or to transition from the past behavior to the behavior according to the label L.) CVAE is used, and the behavior encoding unit 81 outputs the mean μ and covariance matrix Σ representing the latent state to the sampling unit 82.

[0060] The sampling unit 82 samples the latent code z from the latent state (mean μ and covariance matrix Σ) and outputs it to the motion decoding unit 64. That is, as in Figures 1 and 2, a random number ε is generated by the random number generator RG from a Gaussian distribution corresponding to the latent state, and the latent code z is sampled using the following equation as a substitute for equation (63) (equation (63) cannot be used at the time of learning in the first learning device 91 because the function Qr is unknown). z=μ+ε·Σ

[0061] In the parameter learning process, the action decoding unit 64 compares the sampled latent code z with the past action X 1:tis given as input, and the future action Y is predicted using the parameters during learning. ^ t+1:T will be output.

[0062] The first learning device 91 receives the correct answer Y of the future action. t+1:T and predicted action Y ^ t+1:T The reconstruction error Er (prediction error Er) is calculated using the MSE (Mean Square Error) between the prediction error Er and the regularization term KL. Here, similar to the general CVAE disclosed in Non-Patent Document 3, the motion encoding unit 81 and the motion decoding unit 64 are trained to minimize the combined error "Er-KL" (combined error obtained by subtracting the regularization term KL from the prediction error Er) of the prediction error Er and the Kullback-Leibler Divergence (KLD) corresponding to the regularization term KL. As described in Non-Patent Document 3, the regularization term KL is a term for bringing z closer to a normal distribution, and as the term KL becomes smaller, the combined error becomes larger, so it acts as a penalty term.

[0063] By repeating the above steps, the weight parameters of the motion encoding unit 81 and the motion decoding unit 64 are learned. Note that in the first learning device 91, the individual functional blocks that evaluate errors and update parameters are not shown to simplify the drawing.

[0064] As shown in FIG. 6 , the deep learning network structure of the first learning device 91 is an example of the network architecture of the action encoding unit 81. 1:t Past operational characteristics h x GRU Cells (gated regression units) extract the label L or label features h from the label L. L MLP and GRU Cells (which are the same as the feature extraction unit 67 in Fig. 1 and Fig. 5, and are therefore shown as the feature extraction unit 67 in Fig. 6) that extract the future action Y t+1:T Future operation characteristics h y GRU Cells extract these past and future features h x and h y and label feature h LAn MLP (multilayer perceptron) can be used to output latent variables from inputs that combine past actions X (or labels L). 1:t Past operational characteristics h x GRU cells extract the latent codes z and h x It is possible to use a combination of GRU cells in the front stage and MLP cells in the back stage, which decode the above frame by frame.

[0065] In this way, in this embodiment, the label L is further utilized under the same framework as in Non-Patent Document 1 etc. In other words, by inputting the label L of the action category to the action encoding unit 81 and learning it, it becomes easier to separate the latent states by the label L.

[0066] On the other hand, if we assume that the label L of the action category is also input to the action decoding unit 64 for learning, it is thought that separating the latent states by the label will become even easier. However, if the label L of the action category is input to the action decoding unit 64 for learning, the label L of the action category will also be required for the action decoding unit 64 (prediction device 60 in FIG. 1) at the time of prediction, so it is not possible to switch the latent code extraction unit between with and without label input as shown in FIG. 3. Furthermore, depending on the action category L, it may be difficult to reduce the reconstruction error during decoding, resulting in a decrease in quality.

[0067] Therefore, in one embodiment, it is sufficient to input the label L of the action category only to the action encoding unit 81 and have it learn. The label L may be a personal identification, or may be a label that combines a label indicating the action category and a label indicating the person. On the other hand, in a situation where there are no difficulties in reducing errors as described above, as another embodiment, the label L may also be input to the action decoding unit 64 (h in FIG. 6). x and z) and further input a label L to the decoder.

[0068] In view of the above considerations, FIG. 8 shows a functional block diagram of a first learning device 91 according to another embodiment that can achieve the same effect as when a label L is input to the motion decoding unit 64, without inputting the label L to the motion decoding unit 64. The first learning device 91 in FIG. 8 further includes a classifier 83 that automatically identifies the label L, as an additional configuration to the configuration in FIG. 4, and this classifier 83 is also subject to parameter learning. The classifier 83 is trained to identify the label L of the latent code obtained by the sampling unit 82, and as a result, learning can be promoted so that the latent codes obtained by the sampling unit 82 are appropriately separated and arranged in the latent space according to the label L.

[0069] In this way, even if the motion decoding unit 64 does not have a label input L, it becomes easier to separate latent states using labels. In this case, similar to the framework of existing methods such as Non-Patent Document 1, it is preferable to alternate between learning using a configuration in which the motion encoding unit 81, the sampling unit 82, and the motion decoding unit 64 are connected (the configuration of FIG. 4 ) and learning using a configuration in which the motion encoding unit 81, the sampling unit 82, and the classifier 83 according to this embodiment are connected (the configuration in FIG. 4 where the motion decoding unit 64 is removed and the classifier 83 is added). The classifier 83 may be an MLP or the like. One-hot vectors may be used as labels, and cross-entropy error may be used as the error Es for classification, but this is not limiting because classification is not the purpose. For example, MSE error of vectors may be used, and the vectors may not be one-hot. Multiple classifiers 83 may be used. For example, a first classifier for personal IDs and a second classifier for motion categories may be prepared.

[0070] The second learning device 92 performs learning through the following process. As explained above, the configuration of the second learning device 92 in Fig. 5 is the same as that of the prediction device 60 in Fig. 1, and so basically the same processes as those performed during prediction are performed during learning, but as part of the learning process, a process is added in which error evaluation is performed using learning data and the parameters of the function Qr are updated.

[0071] That is, the motion decoding unit 64 that has been trained by the first learning device 91 is placed in the second learning device 92, and the mapping function parameter generating unit 62 (function Qr) is trained. At this time, it should be noted that the weight parameters of the motion decoding unit 64 are fixed and are not updated.

[0072] The second learning device 92 also uses a CVAE, but it does not have to be a CVAE as disclosed in Non-Patent Document 1. The latent code extraction unit 61 of the second learning device 92 extracts K latent codes z. As in the explanation of FIG. 1 etc., this latent code is called z k (1≦k≦K). In this case, the mapping function parameter generating unit 62 (function Qr given by parameters during learning) is 1:t is input, and K sets of parameters ψ for the mapping function are output by processing equation (62) etc., in the same way as explained in FIG. 1 etc. As explained in FIG. 1 etc., these parameters are k (1≦k≦K). The mapping function unit 63 samples ε from a Gaussian distribution and converts ε into K latent codes z k The motion decoding unit 64 converts the latent code z k and past action X 1:t is input and the future action Y ^ t+1:T (Similar to FIG. 1, the latent code z k may be the result of either the processing route of line L63 or L66.)

[0073] The mapping function parameter generation unit 62 (function Qr) receives input of the label L of the action category. In this case, as preprocessing, features may be extracted from the label L of the action category by a feature extraction unit 67 such as MLP, or a one-hot vector may be directly input to the mapping function parameter generation unit 62. A past action X is also input to this function Qr to obtain Qr(X,L). Alternatively, a different Qr may be learned for each label. That is, a Qr may be learned for each label, such as Qr1(X) for the first label and Qr2(X) for the second label. However, unlike conventional methods, it is necessary to learn for each label using a loss function, which will be described later.

[0074] The second learning device 92 learns the future action Y ^ t+1:T The predicted results are used as learning data to determine the correct future action Y t+1:T (prepared to follow the label L), the network Qr (function Qr) is trained. If we assume that the training is performed using a method similar to that described in Non-Patent Document 1, the training will be performed to minimize the loss function L(φ) in the following equation (92'). (Note that, as with the first training device 91, the individual functional blocks that perform error evaluation and parameter update for the second training device 92 are not shown in the drawing to simplify the diagram.)

[0075]

number

[0076] where L KL is the KL divergence, and L KL The more we minimize E, the lower the diversity but the higher the likelihood. d is the diversity loss, and the predicted K actions Y k The larger the pairwise distance between two actions (1≦k≦K), the smaller the value, and the more diverse the action is. r is the reconstruction error, and the closest distance to the correct answer is Yk It is not enough to calculate for all K samples, but it is good if one of them is close to the correct answer. The smaller the value, the better the prediction result Y k is closer to the correct answer.

[0077] However, in the learning of the conventional method of equation (92'), the prediction result falls outside the range of the label L of the specified action category. One reason for this is thought to be that the label L is not taken into consideration at all when learning Qr(X). Therefore, in this embodiment, a loss function that takes the label L into consideration is introduced as in equation (92), and learning of Qr is performed so as to simultaneously increase diversity.

[0078]

number

[0079] That is, L KL is the KL divergence as in Non-Patent Document 1, and L KL The more we minimize E, the lower the diversity but the higher the likelihood. d is a loss related to diversity, as in Non-Patent Document 1, and the predicted K actions Y k The larger the pairwise distance between two actions (1≦k≦K), the smaller the value, and the more diverse the actions are judged to be.

[0080] On the other hand, Er' and El are errors introduced in this embodiment. d , λ r , λ l is a hyperparameter. The reconstruction error Er' and the label pose error El in this embodiment will be described below. Note that the reconstruction error Er' in this embodiment is different from the reconstruction error Er in the conventional method.

[0081] For the reconstruction error Er', the future correct action Y t+1:T Without using the pseudo-real action M t+1:TIt is advisable to search from other training data. This makes it easier to extract latent codes including the operation transition in the natural label direction. The pseudo true operation is, for example, the last frame (X t ) of the past operation and the frame M t that is most similar to X t within the specified label, which is found by means of MSE or the like, and the posture M t+1:T of the frames after that (after t + 1) is taken as the pseudo true operation. (Note that since the time of the frame that is most similar does not necessarily coincide with the time t of the frame (X t ), generally the frame that is most similar is the frame M τ at some time τ, and the pseudo true operation is M τ+1:τ+T-t . However, from the viewpoints of variable notation and simplification and clarification in the description, the search result that τ = t for this most similar frame time τ is described as if it were the case.)

[0082] A schematic diagram of the calculation process of the reconstruction error Er' is shown in Example Ea of FIG. 9. The reconstruction error may be averaged over all frames from t + 1 to T, or may be averaged over the frames from t' to T at the latter part of the frames (t + 1 ≤ t' ≤ T). Since the pseudo true operation is likely to have a non - smooth connection between the past and the future, the switching part between X 1:t and M t+1:T is not calculated as an error and it is desirable to set it with t + 1 < t'. This reconstruction error Er' uses only the smallest error among the K samples, similar to the conventional reconstruction error Er.

[0083] Furthermore, a new label posture error El is provided. Also here, in order to have diversity, the pseudo true operation within the specified label is determined. The following First Embodiment and Second Embodiment are possible.

[0084] In the First Embodiment, a plurality of pseudo true operations are selected from within the specified operation category so as to have diversity within the specified label range. First, the frame M within the specified operation that is closest to the last frame Y^ k T of the predictionk T Detect it. (Note that, as described above, the time of the closest frame to be detected is not necessarily T, and generally it is M at time τ k τ However, from the perspective of simplifying and clarifying the explanation, etc., regarding this detection time τ, it will be explained as if τ = T. ) Include several frames before that frame as a pseudo true operation M k t'':T Let it be (t < t'' ≤ T). Predicted Y^ k t+1:T For the last group of frames Y^ k t'':T and the group of frames M in the specified operation category k t'':T The pose error El of the label can be calculated by MSE etc. of the pose. A schematic diagram of this calculation is shown in Example Eb of FIG. 9. This error may be calculated for each of the K samples. That is, for each k = 1, 2,..., K, one frame M similar to one last frame Y^ k T is detected, and one corresponding frame range M k T can be evaluated by MSE. At this time, as an additional process process, the data detected as the most similar at a certain k = k1 may be excluded from the search targets that will be the most similar at subsequent k = k2 ≠ k1.

[0085] Here, since there is also information on operations that cannot be determined only by the pose, it is desirable that t'' ≠ T. (That is, it is desirable that the pseudo true operation M k t'':T is composed of two or more times and includes operation information on the time axis as video information. ) On the other hand, since there are many transitions to poses farther from Er', it is better to set t' < t''. That is, as shown in Examples Ea and Eb of FIG. 9, by setting the frame time length when evaluating the pose error El of the label to be shorter than the frame time length when evaluating the reconstruction error Er', the pose error El of the label can be made to emphasize the evaluation error on the farther future side in comparison with the reconstruction error Er'. Also, for the final frame Y^ k ​​​T Starting from frame M^ k T However, this is not limiting, and M may be detected starting from any frame after t+1.

[0086] The second embodiment is a method of extracting K or more actions for a period of t:T frames from a specified action category and using them as pseudo-true actions. For example, 5 × K actions are randomly extracted, and actions with a pairwise distance lower than a certain level are removed to determine K or more pseudo-true actions. That is, for example, (1) thin out those with a small pairwise distance (mutually similar) from the 5K actions to obtain, for example, 3K actions that are distributed sparsely to some extent, and (2) search for the most similar action to each of the predicted K actions (within the time t:T range) from these 3K actions (the search may be performed with the added constraint that the most similar actions do not overlap), and calculate the label pose error El as the sum of the K MSEs.

[0087] As described above, the prediction device 60 trained by the training method of this embodiment can perform a variety of motion predictions after receiving the designation of the label L. Various supplementary examples, additional examples, alternative examples, etc. will be described below.

[0088] (1) The learning method of this embodiment may be combined with the technique of Non-Patent Document 2. That is, a technique that extends the learning technique used when the first learning device 91 learns the action decoding unit 64 may be applied. In this technique, not only the encoder and decoder for future actions but also the encoder and decoder for past actions c are learned simultaneously. When predicting, μ and σ are obtained using the former encoder, and μc and μσ are obtained using the latter encoder, and prediction can be performed using the combined z. Specifically, first, the condition c, which is the past action, is encoded, and the mean μc and covariance σc can be output. After calculating the tentative latent code zc (=μc+σc·ε) obtained from the condition, the latent code z for decoding the future action is calculated (z=z k (each) is calculated according to the following formula:

[0089]

number

[0090] This makes it easier to extract a latent code z that better reflects the conditions. In other words, in Non-Patent Document 2, the true p(z|c) is used instead of p(z) as the prior distribution for extracting z. This improves quality and enables implicit decoding of future actions that are consistent with the context.

[0091] (2) The prediction device 60 of this embodiment can be used in the following ways, for example.

[0092] The prediction device 60 allows you to specify a label L, so you can efficiently predict the behavior of a desired category. For example, you can efficiently predict what behavior is likely to occur if someone were to run out into the road. In addition, because you can specify a label L, it is also suitable for creating CG animations. You can also select from multiple labels presented.

[0093] Furthermore, when a personal identifier is used as the label content, different actions can be predicted depending on the designated person. For example, by inputting a student's past actions, the system can output the student's future actions and also show what future actions the teacher would take, which is expected to be useful for skill learning.

[0094] Furthermore, in an embodiment in which the prediction device 60 additionally utilizes a feature calculation unit 65 and a feature emphasis unit 66, predictions can be made that emphasize or reduce the movement features of an individual. In the case of creating CG animations where movements need to be emphasized or predictions involving a variety of movement transitions, it is expected that prediction accuracy can be improved by combining this with behavior prediction.

[0095] (3) The motion data X, Y, etc. (including motion data for learning) input and output to the prediction device 60 may be any type of data representing the motion of a person, etc. For example, information on the 2D or 3D positions or movements of multiple key points (often joints) that indicate the skeleton, or various other information obtained at the time of each frame may be used. Therefore, when the positions of joints, etc. defined in this motion data X, Y, etc. are rendered as frame images at each time t and played back and displayed as a video, the motion of the person, etc. at each time t is reproduced as a video.

[0096] (4) FIG. 10 is a diagram showing an example of the hardware configuration of a general computer device 30. Each device shown in FIGS. 1, 3 to 5, etc. can be realized as one or more computer devices 30 having such a configuration. When each device is realized using two or more computer devices 30, information required for processing may be transmitted and received via a network. The computer device 30 includes a CPU (Central Processing Unit) 31 that executes predetermined instructions, a GPU (Graphics Processing Unit) 32 as a dedicated processor that executes some or all of the CPU 31's execution instructions in place of or in cooperation with the CPU 31, a RAM 33 as a main memory device that provides a work area for the CPU 31 (and GPU 32), a ROM 34 as an auxiliary memory device, a communication interface 35, a display 36, an input interface 37 that accepts user input via a mouse, keyboard, touch panel, etc., and a bus BS for transmitting and receiving data among these.

[0097] 1, 3 to 5, etc., and the prediction methods and learning methods executed by these devices can be realized by a CPU 31 and / or a GPU 32 that reads from a ROM 74 and executes a predetermined program corresponding to the function of each unit or step. Both the CPU 31 and the GPU 32 are types of arithmetic devices (processors). When display-related processing is performed, a display 36 also operates in conjunction with the CPU 31 and the GPU 32, and when communication-related processing related to data transmission and reception is performed, a communication interface 35 also operates in conjunction with the CPU 31 and the GPU 32.

[0098] (5) As an example of application of the prediction device 60 according to an embodiment of the present invention, when remote communication involving transmission delays is performed using an avatar (an avatar that captures the movements of a remote user), the prediction device 60 can predict the future movements of the avatar and appropriately compensate for the delay. Alternatively, it can prepare natural and diverse movements for avatars that represent audience members, etc., without the need to capture the actual movements of the remote user, thereby ensuring a sense of realism in the virtual space where avatar communication takes place. This enables smooth remote communication and eliminates the need for users to travel to remote locations for meetings, etc. (travel to remote locations is not necessarily required). Therefore, such application examples of the present invention can reduce carbon dioxide emissions by conserving the energy resources required for user travel, thereby contributing to Goal 13 of the United Nations' Sustainable Development Goals (SDGs), "Take urgent action to combat climate change and its impacts." [Explanation of symbols]

[0099] 60...prediction device, 61...latent code extraction unit, 62...mapping function parameter generation unit, 63...mapping function unit, 65...feature calculation unit, 66...feature emphasis unit, 64...motion decoding unit, 72...unlabeled latent code extraction unit, 74...selection unit 91...first learning device, 92...second learning device, 81...motion encoding unit, 82...sampling unit

Claims

1. A prediction device for predicting a plurality of future actions from past actions, encoding the past actions into a plurality of latent codes under categories for designated future actions; and obtaining prediction results of a plurality of future actions under the specified category corresponding to the past actions by decoding each of the plurality of latent codes.

2. The specified category includes information specifying an identifier of an operating subject, 2. The prediction device according to claim 1, wherein in the process of encoding into the plurality of latent codes, characteristics of the subject of the action in the specified category are emphasized or reduced according to a user specification.

3. a first process for encoding the past actions into a plurality of latent codes under categories for designated future actions; a second process that encodes the past behavior into a plurality of latent codes without accepting the designation of the category; 2. The prediction device according to claim 1, further comprising: a process for obtaining prediction results of a plurality of future actions corresponding to the past actions under the specified category or independent of the category by decoding the plurality of latent codes obtained by the selected first process or second process, respectively.

4. Executing a process of obtaining prediction results of the plurality of future actions by a pre-trained deep learning network; The prediction device according to claim 3 , wherein the first process and the second process use the same parameters of the deep learning network.

5. A method for learning parameters of a deep learning network used by the prediction device according to claim 1, comprising: a first parameter of a first network that encodes the past actions into a plurality of latent codes under a category for a specified future action; and learning second parameters of a second network that is responsible for a process of obtaining prediction results of a plurality of future actions under the specified category corresponding to the past actions by decoding each of the plurality of latent codes; The learning order is to learn the second parameter first, and then learn the first parameter; When learning the second parameter, learning is performed using at least a first reconstruction error (Er) calculated by comparing prediction results of a plurality of future actions obtained by processing past actions for learning using a network in which an encoder is arranged at the upstream side of the second network with a correct future action for learning; When learning the first parameter, learning is performed by using at least a label posture error (El) calculated by comparing prediction results of a plurality of future actions obtained by processing past actions for learning using a network in which the second network, to which the learned second parameter has been set, is disposed downstream of the first network, with a pseudo-correct future action for learning; A learning method characterized by searching for the pseudo-correct future action from learning data that falls into the specified category as being similar to the prediction result for each of a plurality of future actions.

6. The learning method described in claim 5, characterized in that learning data corresponding to the specified category for searching for the pseudo-correct future action is prepared by excluding data determined to be similar to each other.

7. A program that causes a computer to function as the prediction device according to any one of claims 1 to 4.

Images