Movement generation device, movement generation model learning device, and program

Abstract

To provide a movement generation device, a movement generation model learning device, and a program that enable movement generation in response to an event at a frame level.SOLUTION: The movement generation device includes an encoder that obtains a latent variable and a decoder that generates a movement from the latent variable. The latent variable is given constraints so as to transition for each event to achieve frame-level event editing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a motion generation device, a motion generation model learning device, and a program technology. [Background technology]

[0002] There is a technology that generates actions according to input text (see, for example, Non-Patent Document 1). The technology described in Non-Patent Document 1 generates actions corresponding to input text by learning the relationship between text and actions using learning data composed of paired data of text and actions. As a result, the technology described in Non-Patent Document 1 can generate time-series data including multiple actions (for example, "A person was walking. He suddenly stopped and picked up an object") based on text. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Chuan Guo, Shihao Zou, Xinxin Zuo, et al., “Generating Diverse and Natural 3D Human Motions from Text”, in Proc. CVPR, 2022 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, the editing of each event individually at the frame level is not taken into consideration, and there is a problem in that it is not easy to respond to the generation of actions according to events at the frame level.

[0005] In view of the above circumstances, an object of the present invention is to provide a motion generation device, a motion generation model learning device, and a program that are capable of generating motions in response to events at the frame level. [Means for solving the problem]

[0006] One aspect of the present invention is a motion generation device comprising an encoder that determines latent variables and a decoder that generates motions from the latent variables, wherein the latent variables are constrained to transition for each event in order to realize frame-level event editing.

[0007] One aspect of the present invention is a motion generation model learning device, in which learning data is pair data of text and motion, the device comprising: a motion encoder that estimates a first latent variable from the motion; a decoder that reconstructs a motion from the estimated first latent variable; a text encoder that estimates a second latent variable from the text; and a learning unit, wherein the learning unit optimizes the first latent variable and the second latent variable by simultaneously performing the following steps: learning so that an input motion of the motion encoder and an output motion of the decoder approach each other when a motion is reconstructed from the first latent variable; and learning so that the first latent variable and the second latent variable approach each other when the second latent variable is estimated.

[0008] One aspect of the present invention is a program that causes a computer of a motion generation device to find latent variables and generate motions from the latent variables, the latent variables being constrained to transition for each event in order to realize frame-level event editing. Effect of the Invention

[0009] The present invention makes it possible to generate actions according to events at the frame level. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a configuration of a motion generating device according to an embodiment. [Diagram 2] FIG. 13 illustrates an event representation. [Diagram 3] FIG. 1 is a diagram illustrating an overview of an example of a learning method for an action generation model in an embodiment. [Figure 4]FIG. 11 is a diagram illustrating an example of a method for learning an action generation model in the embodiment. [Diagram 5] This is a conceptual image of cd 1:L and c'd 1:L. [Figure 6] FIG. 1 is a diagram showing the steps of a motion generation method using a learned motion generation model in an embodiment. [Figure 7] FIG. 11 is a diagram showing a processing procedure for generating an action from a text in the embodiment. [Figure 8] FIG. 11 is a diagram showing a processing procedure for reconstructing an action from an action in the embodiment. [Figure 9] 11A and 11B are diagrams illustrating an example of an operation created from text using the method of the embodiment. [Figure 10] FIG. 13 is a diagram showing an action edited by the method of the embodiment using [cpri 1, cpri 1, ..., cpri L'-1, cpri L'-1, cpri L'] obtained from an event expression sequence estimated from text cpri 1:L'. [Figure 11] FIG. 1 illustrates the original operation. [Figure 12] This is a diagram in which the segment from the 55th to 68th frames of the original motion in FIG. 11 is expanded to the 55th to 110th frames. [Figure 13] FIG. 13 is a diagram showing the result of editing the characteristics of an event using the “edit event characteristics” method. [Figure 14] FIG. 13 illustrates a motion segment with only one event representation added. [Figure 15] FIG. 15 is a diagram showing the result of inserting the event of FIG. 14 into the 146th frame of FIG. 11 using the "insertion of new event" method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. 1 is a diagram showing an example of the configuration of a motion generating device according to the present embodiment. As shown in FIG. 1, the motion generating device 1 includes, for example, a motion generation model learning device 2, an acquisition unit 3, a motion generating unit 4, and an output unit 5. The movement generation model learning device 2 includes, for example, a movement generation model 21, a learning unit 22, and a storage unit . The action generation model 21 includes, for example, an encoder 211 and a decoder 214 . The encoder 211 includes, for example, a text encoder 212, a motion encoder 213, and a latent variable predictor 219. The latent variable prediction unit 219 includes, for example, an attention network 215, a posterior distribution network 216, a prior distribution network 217, and a latent representation network 218.

[0012] The text encoder 212 obtains feature vectors of words from the text. The action encoder 213 obtains an event latent variable (first latent variable) from the action data. The decoder 214 estimates a motion sequence from the event latent variables obtained by the motion encoder 213. The attention network 215 estimates text information from word feature vectors and the like. The posterior distribution network 216 obtains the mean and standard deviation of the latent variable (second latent variable) from the latent variable (first latent variable) estimated by the motion encoder 213 and text information and the like. The prior distribution network 217 obtains the mean and standard deviation of a latent variable (second latent variable) from text information or the like. The latent representation network 218 obtains a posterior distribution event latent variable and a posterior distribution end point variable from the latent variable (second latent variable) estimated by the posterior distribution network 216 and text information, etc. Alternatively, the latent representation network 218 obtains a prior distribution event latent variable and a prior distribution end point variable from the latent variable (second latent variable) estimated by the prior distribution network 217 and text information, etc.

[0013] The acquisition unit 3 acquires motion data and text that is an explanation of the motion data. The learning unit 22 uses the text and the motion data to learn the motion generation model 21 . The storage unit 23 stores programs, mathematical expressions, predetermined values, predetermined threshold values, etc., used during learning and action generation. The motion generation unit 4 generates motions using a trained motion generation model 21. The output unit 5 outputs to an external device the information (motion sequence) generated by the motion generation unit 4. The external device is, for example, a personal computer, a control device for controlling motion, or the like.

[0014] Details of the processing of each unit will be described later in the learning method and action generation method.

[0015] The motion generating device 1 and the motion generation model learning device 2 are configured using a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) and a memory. The motion generating device 1 functions as, for example, an acquisition unit 3, a motion generating unit 4, and an output unit 5 by the processor executing a program. The motion generation model learning device 2 functions as, for example, an encoder 211 and a decoder 214 by the processor executing a program. Note that all or part of the functions of the motion generating device 1 and the motion generation model learning device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above programs may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (for example, an SSD: Solid State Drive), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above programs may be transmitted via an electric communication line.

[0016] (Event Expression) Fig. 2 is a diagram showing an event representation. As shown by reference symbol g10 in Fig. 2, the motion of an avatar is composed of frames, and each frame represents a pose. In the embodiment, the motion of an avatar is semantically divided into "events" (reference symbols g11 to g13), and an event is changed only when it cannot be predicted from the previous frame and past events. In this embodiment, the event representation of each frame can be obtained and used to infer events from text or actions, or to edit events to generate new motions. This motion editing can include, for example, changing the event time, changing the event characteristics, or adding new events. In addition, the embodiment can obtain events in an unsupervised manner.

[0017] [Learning a motion generation model] Fig. 3 is a diagram showing an outline of an example of a method for learning a motion generation model in this embodiment. Fig. 4 is a diagram showing an example of a method for learning a motion generation model in this embodiment.

[0018] The training data consists of L frames. i Operation data x consisting of 1:L i (The following formula (1)) (symbol g101) and text T explaining its operation i (Code g102) pair data (x 1:L i ,T i ) are used (N≧1) (i∈{1,...,N}). In the following description, for simplicity, the subscript i will be omitted. The learning data is acquired by the acquisition unit 3.

[0019]

number

[0020] The maximum length of the latent variables that can be estimated from the text is L max (L max ≧L). In order to accommodate this length, the input motion data x1:L padding is done for x 1:Lmax For example, padding is used to fill in the missing length (L max -L), you can insert a specific value (such as 0 or the average value), or you can insert a value that is the same as the end point, or you can insert a value that wraps around the end point.

[0021] I. Motion Encoder First, the motion encoder 213 will be described. Data Encoder 213 (E data ) is the operation data x 1:Lmax (Code g101) to latent variable c d 1:Lmax (Code g103) and latent variable e d 1:Lmax After estimating (symbol g103), we apply constraints on the number of switches and d 1:Lmax That is, the latent variables are constrained to transition for each event in order to realize frame-level event editing. The motion encoder 213 further selects the value of the latent variable e d 1:Lmax By determining the end of the operation based on the value of d 1:L =(following equation (2))(symbol g104) is calculated using the following equation (3).

[0022]

number

[0023]

number

[0024] where c d 1:Lmax is a latent variable related to the event expression (hereafter, called the event latent variable), and e d 1:Lmax is the frame l(∈{1,…,L maxAn indicator (binary 0 or 1, hereinafter referred to as the end point variable) indicating whether the event has ended at}). That is, when l < L, e d l = 0, and otherwise e d l = 1. For c d 1:L , let c’ d 1:L =(c’ d 1 ,…,c’ d L ) be defined as follows. For example, as shown in Figure 5, when the value of c d 1:L (point g151) changes significantly, the motion encoder 213 causes the value of c’ d 1:L (point g152) to switch. Figure 5 is a conceptual image diagram of c d 1:L and c’ d 1:L . The horizontal axis represents time, and the vertical axis represents the representative value. That is, when there is a large change in the value of c d l , then c’ d l = c d l , and otherwise c’ d l = c’ d l-1 . According to this definition, (c’ d 1 ,…,c’ d L ) can be represented as (c d i1 ,…,c d iL ) (when l = 1, i1 = 1; when 1 < l ≤ L, i1 ∈ {l, i l-1}).

[0025] An indicator (i1,…,i d 1:L ) representing the switching timing of c’ L) is a loss function R(L,i 1:L ) (where R is a calligraphy character). For example, it can be calculated by optimizing the following equations (4) to (6).

[0026]

number

[0027]

number

[0028]

number

[0029] In equation (6), d(i,j)=dist(Dec(c d j ,x 1:i-1 ),x i ) is the motion data x of frame i i and the data Dec(c) of the frame i of the motion reconstructed by the motion encoder 213 and the decoder 214 (decoder; Dec) d j ,x 1:i-1 ) is the distance between the nodes. For example, any distance criterion such as L1 distance or L2 distance may be used as the distance index. Also, τ is the switching penalty. The learning unit 22 performs the case-discrimination step in the formula (6) by using i' l While imposing constraints (τ) on the switching of the value of l ) is optimized to minimize

[0030] R(L,i' 1:L ) (where R is a calligraphy character) is not differentiable and is not easy to optimize. Therefore, in the implementation, a smoothing parameter γ is introduced to satisfy the following equation (7), and R(L,i 1:L ) (where R is a calligraphy character)s (where L is a calligraphy character) (code g121) (first loss function) is optimized. For example, the loss function L s (where L is a calligraphy character) is calculated by dynamic programming using the following equations (8) to (9).

[0031]

number

[0032]

number

[0033]

number

[0034] Loss function L s (where L is a calligraphy character) is calculated based on the reconstruction loss d(i,j). For this reason, the learning unit 22 calculates the reconstruction loss L r (where L is a calligraphy character) (code g122) (second loss function) is also optimized at the same time. That is, the loss function L s is the loss function for the first latent variable, and the loss function L r (where L is a calligraphy letter) is the loss function for the decoder 214 reconstruction.

[0035]

number

[0036] In addition, in formula (10), x' i =Dec(c' d i ,x' 1:i-1 ).

[0037] II. Decoder Next, the decoder 214 will be described. The decoder 214 (Dec) converts the event latent variable c′ obtained by the action encoder 213 into d 1:L =(c' d 1 ,…,c' d L ) to the action sequence x' 1:L =(x'1,…,x' L For example, the motion x' at frame t is estimated. t is calculated by the following equation (11).

[0038]

number

[0039] In formula (11), c' d t represents the event latent variable for frame t, and x' 1:t-1 represents the action sequence before frame t. For example, the model can be constructed as follows: 1:t-1 The position encoding vector included in the feature vector is added to this and processed by a Recurrent Neural Network (RNN, Long Short-Term Memory (LSTM), etc.), and then the processing result is compared with c' d i Alternatively, a configuration in which the combined result is input to a multi-layer perceptron (MLP) may be used.

[0040] III. Text Encoder Next, the text encoder 212 will be described. Text Encoder 212 text ) is a feature vector w of a word from a text T (code g102). 1:M =(w 1 ,…,w M ) (symbol g107) is calculated using the following equation (12).

[0041]

number

[0042] In formula (12), M represents the number of words in the text T. For example, the feature vector w m The vector can be obtained by inputting two pieces of data, namely, word embeddings obtained from a trained language model (e.g., Global Vectors for Word Representation (GloVe)) and a positional encoding vector representing the word position, into a multi-layer perceptron.

[0043] IV. Attention Network Next, the attention network 215 will be described. Attention Network 215(F attn ) is the feature vector of a word w 1:M , and c post 1:t-1 and e post 1:t-1 (code g108) plus the position encoding vector gives the text information w attn t (Code g109) is estimated as shown in the following equation (13).

[0044]

number

[0045] V. Posterior Distribution Networks Next, the posterior distribution network 216 will be described. Posterior network 216 (N post ) is the text information w attn t (Code g109), the posterior distribution event latent variable c in frame t-1 post t-1 and the posterior distribution endpoint variable e post t-1(symbol g108'), and the event latent variable c' derived from the action in frame t (obtained by the action encoder). d t and the end variable e d t From (code g110), the latent variable z post t The average μ post t and standard deviation σ post t (Code g111) is calculated using the following equation (14).

[0046]

number

[0047] Latent variable z post t For example, the average μ post t , standard deviation σ post t Normal distribution of N post t (μ post t ,σ post t ) where N is the number of calligraphy characters.

[0048] VI. Prior Distribution Networks Next, the prior distribution network 217 will be described. Prior network 217 (N pri ) is the text information w attn t (Code g109), the posterior distribution event latent variable c in frame t-1 post t-1 and the posterior distribution endpoint variable e post t-1 From (code g108'), the latent variable z pri t The average μ pri t and standard deviation σ pri t(Code g112) is calculated using the following equation (15).

[0049]

number

[0050] Latent variable z pri t For example, the average μ pri t , standard deviation σ pri t Normal distribution of N pri t (where N is a calligraphy letter)(μ pri t ,σ pri t ) is sampled.

[0051] VI. Latent Representation Network Latent Representation Network 218 (representation network;D rep ) is the latent variable z estimated by the posterior distribution network 216. post t (Code g113) and text information w attn t (Code g109), the posterior distribution event latent variable c in frame t-1 post t-1 and the posterior distribution endpoint variable e post t-1 From (symbol g108'), the posterior distribution event latent variable c post t and the posterior distribution endpoint variable e post t (Code g114) is calculated using the following equation (16).

[0052]

number

[0053] VII. Loss Function Next, the loss function used in learning will be described. The loss function mentioned above (L s (Code g121) (where L is a calligraphy letter) and L r In addition to the loss function (where L is a calligraphy letter) (code g122), the following four loss functions are used: The third loss function is the action-derived event latent variable c' d t (symbol g104), and the text-derived event latent variable c post t Loss function L that expresses the difference with (sign g114) diff (where L is a calligraphy character) (code g123). Loss function L diff (where L is a calligraphy character) is defined, for example, by the following formula (17):

[0054]

number

[0055] In formula (17), ||·|| is a norm and represents a distance in a metric space. The subscript number of ||·|| represents the value of the norm, and "2" is the 2-norm (Euclidean norm). Although the L2 distance is used as an index of distance in the above description, any distance measure such as L1 distance may be used. diff (where L is a calligraphy character) to minimize c' d t and c post t This helps students to get closer to each other and learn common concepts between actions and text.

[0056] The fourth loss function is the prior distribution of the latent variables N pri t (where N is a calligraphy character) (code g124) and the posterior distribution N post t(where N is a calligraphy character) (code g125) KL (Code g126). Loss function L KL (where L is a calligraphy character) is defined, for example, by the following formula (18).

[0057]

number

[0058] In equation (18), KL is the Kullback-Leibler (KL) Divergence. Here, an example is shown in which the Kullback-Leibler divergence is used as an index of the distribution distance, but other distribution distance measures may be used. KL By minimizing pri t (where N is a calligraphy character) and the posterior distribution N post t (where N is a calligraphy letter) should be closer.

[0059] The fifth loss function is the loss function L for predicting the frame length of the action. e (where L is a calligraphy character) (code g127). Loss function L e (where L is a calligraphy character) is defined, for example, by the following formula (19).

[0060]

number

[0061] The sixth loss function is the sequence of latent event variables c d 1:L (Code g103) and c' d 1:L (Code g104) and the text-derived event latent variable sequence c post 1:LFor (symbol g114), we use a loss function L that makes the motion and text closer together when they are obtained from data in which the motion and text correspond (pair data), and makes them move apart when they are obtained from data in which the motion and text do not correspond. a (where L is a calligraphy character) (code g128). For example, let c d1 1:L1 , c' d1 1:L1 , c post 1:L1 (The number of frames is L1.) The event latent variables obtained from the actions other than the paired data are c' d2 1:L2 (The number of frames is L2.) The loss function L a is defined by the following equation (20).

[0062]

number

[0063] The learning unit 22 calculates the loss function L a (where L is a calligraphy character), we minimize the corresponding action and text-derived event latent variables (c d1 1:L1 and c post 1:L1 , and c' d1 1:L1 and c post 1:L1 ) are closer to each other, while the event latent variables (c' d2 1:L2 and c post 1:L1 , and c' d1 1:L1 and c' d2 1:L2 ) to separate. Although the case where the L2 distance is used as an index of distance has been described here, any distance measure such as the L1 distance may be used. Also, instead of using all four terms (the two denominator terms and the two numerator terms), one or more of the terms may be used. Overall loss function L total(where L is a calligraphy character) is a combination of the loss functions mentioned above and is expressed as follows:

[0064]

number

[0065] During learning, the learning unit 22 uses the overall loss function L total (where L is a calligraphy character). In equation (21), λ recon , λ r , λ KL , λ e , λ emb and λ a is a hyperparameter that determines the weight of the loss function. Although the above describes the case where all the loss functions are used, optimization may be performed using one or more parts of the loss functions.

[0066] [How to generate motion using a trained motion generation model] First, the generation of motion from text will be described. FIG. 6 is a diagram showing the procedure of the method of generating motion using a trained motion generation model in this embodiment. FIG. 7 is a diagram showing the processing procedure of generating motion from text in this embodiment. As shown in FIG. 6 and FIG. 7, when generating motion from text, the motion generation model 21 may be configured to include a text encoder 212, a decoder 214, an attention network 215, a prior distribution network 217, and a latent representation network 218. Action generation from text (input: text T, output: action x') 1:L’ The procedure for the above is as follows. Note that the following process is performed after the learning of each of the above-mentioned parts is completed.

[0067] (Step S101) The motion generator 4 uses the text encoder 212 to generate a word feature vector w from the input text T (code g201). 1:M (Code g202) is calculated.

[0068] (Step S102) The action generation unit 4 uses the attention network 215 to calculate the feature vector w 1:M (symbol g202), and the output sequence c from frame 1 to frame t-1 of the latent representation network 218. pri 1:t-1 and e pri 1:t-1 (symbol g203) plus the position encoding vector included in the feature vector, the text information w attn t (Code g204) is estimated.

[0069] (Step S103) The action generation unit 4 uses the prior distribution network 217 to generate text information w attn t (Code g204), the prior distribution event latent variable c in frame t-1 pri t-1 and the prior distribution endpoint variable e pri t-1 From (code g203'), the latent variable z pri t The average μ pri t and standard deviation σ pri t (symbol g205). The motion generator 4 calculates, for example, the average μ pri t , standard deviation σ pri t Normal distribution of N pri t (μ pri t ,σ pri t ) (where N is a calligraphy character) (code g206), the latent variable z pri t Request.

[0070] (Step S104) The motion generation unit 4 uses the latent representation network 218 to calculate the latent variable z pri t (Code g206), text information w attnt (Code g204), the prior distribution event latent variable c in frame t-1 pri t-1 and the prior distribution endpoint variable e pri t-1 From (symbol g203'), the prior distribution event latent variable c pri t and the prior distribution endpoint variable e pri t (Code g207) is calculated.

[0071] (Step S105) The action generation unit 4 uses the decoder 214 to calculate the prior distribution event latent variable c pri t (Code g208) and the motion sequence x' from frame 1 to frame t-1 1:t-1 From (code g209), action x' in frame t t Estimate.

[0072] (Step S106) The motion generation unit 4 repeats the processes of steps S102 to S105 from t=1 to t=L max In the course of the repeated calculation, the motion generation unit 4 calculates the prior distribution end point variable (e pri ) t When x' becomes equal to or greater than a predetermined threshold, the motion is deemed to be completed, and the repeated calculation is terminated. The frame at this time is designated as L'. The output unit 5 outputs the generated motion sequence x' 1:L’ (Code g210) is output to an external device. The external device is, for example, a personal computer, a control device that controls the operation, etc.

[0073] Next, the reconstruction of a motion from a motion will be described. Fig. 8 is a diagram showing the processing procedure of the reconstruction of a motion from a motion in the embodiment. As shown in Fig. 8, when reconstructing a motion from a motion, the motion generation model 21 may be configured from a motion encoder 213 and a decoder 214. Reconstructing Actions from Actions (Input: Action x 1:L ,Output: Action x ’1:L’ The procedure for the above is as follows. Note that the following process is performed after the learning of each of the above-mentioned parts is completed.

[0074] (Step S201) The motion generator 4 uses the motion encoder 213 to generate a motion x 1:L From (code g301), the event latent variable c' d 1:L’ (Code g302) is found.

[0075] (Step S202) The action generation unit 4 uses the decoder 214 to generate an event latent variable c′ d t and the motion sequence x' from frame 1 to frame t-1. 1:t-1 From (code g303), action x' in frame t t Estimate.

[0076] (Step S203) The motion generation unit 4 repeats the process of step S202 from t=1 to t=L'. Note that the end frame L' of the repeated calculation is e obtained in the process of the calculation of step S201. d t The output unit 5 outputs the generated motion sequence x' 1:L’ (Code g304) is output.

[0077] [How to edit motion using a trained motion generation model] Since the event representation is considered to be related to the event captured in the corresponding frame, we use the corresponding event latent variable (c pri 1:L’ , works x 1:L If is the input, then c' d 1:L’ ) to output action x' 1:L’ The following three examples of editing actions are explained below.

[0078] I. Editing the number of frames in an event In the embodiment, the latent variables of an event are restricted in the number of value switches, so that they take similar values ​​within the same event. Therefore, if you want to increase the number of frames for each event, you can increase the number of latent variables for the corresponding event, and if you want to decrease the number of frames for each event, you can decrease the number of latent variables for the corresponding event. In the conventional method, when the latent variables fluctuate greatly within the same event, increasing or decreasing the number of events significantly changes the way the variables fluctuate, making it difficult to generate smooth movements. In contrast, according to the method of the embodiment, the latent variables fluctuate little within the same event as described above, so that increasing or decreasing the number of events has little effect on the way the variables fluctuate, and it is possible to generate the same type of movement smoothly. For example, in the case of the movement "walking," it is possible to increase or decrease the stride length while maintaining the movement of the feet and hands.

[0079] II. Editing Event Characteristics In the embodiment, since the latent variables take different values ​​for each event, the characteristics of the events can be changed by changing the values ​​of the latent variables, and as a result, the characteristics of the actions can be changed. For example, if there are two events, it is possible to smoothly change two different actions by smoothly interpolating the two event latent variables corresponding to those events.

[0080] III. Inserting a new event In the embodiment, since the latent variable takes a different value for each event, it is possible to insert a new event by inserting an event having a specific value. For example, a new event can be inserted between an event corresponding to the action of "walking" and an event corresponding to the action of "stopping".

[0081] [evaluation] In order to evaluate whether the method of the embodiment can learn event expressions at the frame level and generate actions according to events at the frame level, the following evaluation was performed. As a comparative example, a conventional method for generating actions from text (Non-Patent Document 1) was used. The evaluation target was the effectiveness of the above-mentioned "editing the number of frames of an event". First, texts containing the word "walk" were collected from the test data, and actions were generated from these texts using the method of the comparative example. The quality of the generated actions was evaluated when the generated actions were edited using the method of the comparative example and the method of the embodiment. The evaluation was performed on the generated actions when the frame length of the latent variable was doubled.

[0082] In the comparative example, the fragment code c^ is extracted from the text T. s 1:T Then, operate p^ 1:T’ As a method to double the frame length of the latent variables, the fragment code c^s 1:T Duplicate the code snippet c'^ twice s 1:2T-1 =(c^ s 1 ,c^ s 1 ,…,c^ s T-1 ,c^ s T-1 ,c^ s T ) was performed by double the length of the snippet code c'^ s 1:2T-1 The motions generated from were used for evaluation. In contrast, in the method of the embodiment, the latent variable of the event predicted from the input operation is c' d 1:L In the case of , double it (c'' d 2L-1 =c' d 1 ,c' d 1 ,…,c' d L-1 ,c' d L-1 ,c'd L ) was used for evaluation.

[0083] As an evaluation index, the quality was evaluated on a 5-point scale based on whether the number of steps in the “walking” action in the action in which the length of the latent variable was doubled, compared to the original action, using the method of the comparative example or the method of the embodiment.

[0084] The difference between the evaluation score of the comparative example method and the evaluation score of the method of the embodiment was 0.551 ± 0.098 (± and after indicate a 95% confidence interval) (positive values ​​indicate that the method of the embodiment is better, and negative values ​​indicate that the method of the comparative example is better). Thus, the evaluation results showed that the method of the embodiment is significantly better than the method of the comparative example. This means that the method of the embodiment is more capable of increasing the number of steps in a manner that is more intuitive than the method of the comparative example. In other words, it can be said that the method of the embodiment is more suitable for generating actions in response to events at the frame level than the method of the comparative example.

[0085] (Reference evaluation example) FIG. 9 is a diagram showing an example of a motion created from text by the method of the embodiment. The text is "person walking at an average pace forward, swaying arms and torso with a sense of swagger." FIG. 10 shows the motion example created from text by the method of the embodiment. pri 1:L’ The event expression sequence estimated from [c pri 1 ,c pri 1 ,…,c pri L’-1 ,c pri L’-1 ,c pri L’9 and 10 are diagrams showing movements edited by the method of the embodiment using [ ]. In Figs. 9 and 10, each movement diagram is a movement at each time (t). Note that the event time of the edited movement is expected to be twice that of the original movement. In Figs. 9 and 10, the dotted lines represent the right arm and right leg, the dashed and dotted lines represent the left hand and left leg, and the solid lines represent the head and torso. The method of the embodiment can increase the number of steps by changing the duration of the event as shown in Figs. 9 and 10, and is successful in lengthening the duration of the walking movement.

[0086] FIG. 11 shows the original operation. 11, time t=1 represents the operation of event 1, times t=8 and 15 represent the operation of event 2, times t=22 and 29 represent the operation of event 3, time t=36 represents the operation of event 4, times t=43 and 50 represent the operation of event 5, and times t=57 and 64 represent the operation of event 6. In addition, times t=71 and 75 represent the operation of event 7, times t=85 and 92 represent the operation of event 8, time t=99 represent the operation of event 9, time t=106 represent the operation of event 10, times t=113, 120 and 127 represent the operation of event 11, and time t=134 represent the operation of event 12. In addition, time t=141 represents the operation of event 13, time t=148 represents the operation of event 14, time t=155 represents the operation of event 15, times t=162 and 169 represent the operation of event 16, time t=176 represents the operation of event 17, time t=183 represents the operation of event 18, and time t=190 represents the operation of event 19.

[0087] FIG. 12 is a diagram in which the segment from the 55th to 68th frames (event 6) of the original movement in FIG. 11 is expanded to the 55th to 110th frames. 12, time t=1 represents the operation of event 1, times t=8 and 15 represent the operation of event 2, times t=22 and 29 represent the operation of event 3, time t=36 represents the operation of event 4, times t=43 and 50 represent the operation of event 5, and times t=57, 64, 71, 78, 85, 92, 99, and 106 represent the operation of event 6. In addition, times t=113 and 120 represent the operation of event 7, and times t=127 and 134 represent the operation of event 8. Furthermore, time t=141 represents the operation of event 9, time t=148 represents the operation of event 10, times t=155, 162, and 169 represent the operations of event 11, time t=176 represents the operation of event 12, time t=183 represents the operation of event 13, time t=190 represents the operation of event 14, time t=197 represents the operation of event 15, and times t=204 and 211 represent the operations of event 16. Furthermore, time t=218 represents the operation of event 17, time t=225 represents the operation of event 18, and time t=232 represents the operation of event 19.

[0088] Fig. 12 is a diagram showing the result of editing the number of frames of the action event in Fig. 11 using the method of "editing the number of frames of an event". The method of the embodiment makes it possible to extend the length of the event segment while maintaining the nature of the action in Fig. 11, as shown in Fig. 12.

[0089] Fig. 13 is a diagram showing the result of editing the characteristics of an event using the "editing the characteristics of an event" method. Specifically, it shows a motion generated using a sequence of event latent variables obtained by interpolating two event latent variables corresponding to the upper left and lower right motions. The method of the embodiment is capable of smoothly interpolating between the two motions as shown in Fig. 14.

[0090] FIG. 14 is a diagram showing a motion segment with only one event expression (event A) being added.

[0091] FIG. 15 is a diagram showing the result of inserting event A (33 frames) of FIG. 14 into the 146th frame of FIG. 11 using the "insertion of a new event" method. 16, time t=1 represents the operation of event 1, times t=8 and 15 represent the operation of event 2, times t=22 and 29 represent the operation of event 3, time t=36 represents the operation of event 4, times t=43 and 50 represent the operation of event 5, and times t=57 and 64 represent the operation of event 6. In addition, times t=71 and 78 represent the operation of event 7, times t=85 and 92 represent the operation of event 8, time t=99 represent the operation of event 9, time t=106 represent the operation of event 10, times t=113, 120 and 127 represent the operation of event 11, and time t=134 represent the operation of event 12. Furthermore, time t=141 represents the operation of event 13, times t=148, 155, 162, 169, and 176 represent the operations of the inserted new event (event A), time t=183 represents the operation of event 14, time t=190 represents the operation of event 15, time t=197 represents the operation of event 16, and time t=204 represents the operation of event 17. Furthermore, times t=211 and 218 represent the operations of event 18, and time t=225 represents the operation of event 19. The method of the embodiment makes it possible to insert new events while maintaining existing events, as shown in FIG.

[0092] In the action generation device 1 and action generation model learning device 2 configured in this manner, when learning the relationship between text and action, a constraint is imposed on the latent variable representing the common concept of text and action, such that transitions are allowed only when switching is necessary. This makes it possible for the motion generation device 1 and the motion generation model learning device 2 to express the above-mentioned event mechanism (switching occurs only when a frame that cannot be predicted from past frames and past events appears) in the latent variable space of the motion generator. This makes it possible for the motion generation device 1 and the motion generation model learning device 2 to perform motion generation according to events at the frame level, such as changing the number of frames of an event by adjusting the length of the latent variable in the time direction, changing the characteristics of an event by replacing a latent variable, and inserting a new event by inserting a new latent variable between latent variables.

[0093] Furthermore, the motion generation device 1 and the motion generation model learning device 2 are able to learn event expressions at the frame level, and therefore are able to generate motions according to events at the frame level. In addition, embodiments can obtain an event representation for each frame and use it to infer events from text or actions, or edit events to generate new motions, such as by modifying event times, modifying event features, or adding new events. In addition, embodiments can obtain events in an unsupervised manner.

[0094] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0095] The present invention can be applied to devices that perform machine learning for the purpose of generating actions, devices that generate actions, etc. [Explanation of symbols]

[0096] 1...motion generation device, 2...motion generation model learning device, 3...acquisition unit, 4...motion generation unit, 5...output unit, 21...motion generation model, 22...learning unit, 23...memory unit, 211...encoder, 212...text encoder, 213...motion encoder, 214...decoder, 215...attention network, 216...posterior distribution network, 217...prior distribution network, 218...latent representation network, 219...latent variable prediction unit

Claims

1. An encoder for finding latent variables; a decoder for generating actions from the latent variables; The latent variables are constrained to transition for each event in order to realize frame-level event editing.

2. The encoder determines the latent variables from text; The action generating device according to claim 1 , which generates an action from the text.

3. The encoder determines the latent variables from the motions; The motion generating device according to claim 1 , which reconstructs a motion from the motions.

4. The number of frames of the event is changed by adjusting the length of the latent variable in the time direction; or varying the characteristics of the event by varying the value of the latent variable; The action generating device according to claim 1 , further comprising: a step of: inserting a new event by inserting a new latent variable between the first latent variable and the second latent variable, thereby editing the object to be generated.

5. The training data is a pair of text and action data. a motion encoder for estimating a first latent variable from the motions; a decoder for reconstructing motion from the estimated first latent variables; a text encoder for estimating a second latent variable from the text; A learning unit, The learning unit is learning such that, when a motion is reconstructed from the first latent variable, an input motion of the motion encoder and an output motion of the decoder are close to each other; learning the first latent variable and the second latent variable to approach each other when the second latent variable is estimated; and optimizing the first latent variable and the second latent variable by simultaneously performing the learning. A motion generation model learning device.

6. The learning unit is 6. The action generation model learning device according to claim 5, wherein, when learning the latent variables, learning is performed while imposing a constraint on switching of the latent variables that allows transition only when switching is necessary.

7. The learning unit is a first loss function for the first latent variable; and a second loss function for the reconstruction of the decoder; and a third loss function that represents a difference between a first latent variable determined by the motion encoder and the second latent variable determined by the text encoder; A fourth loss function representing a distribution distance between a prior distribution and a posterior distribution of the latent variable; and a fifth loss function for predicting a frame length of the motion; a sixth loss function that causes the first sequence of latent variables and the second sequence of latent variables to approach each other when the sequences are obtained from data in which the action and the text correspond, and causes the sequences to move apart when the sequences are obtained from data in which the action and the text do not correspond; 6. The action generation model learning device according to claim 5, wherein optimization is performed so as to minimize at least one of the loss functions, and the first latent variable and the second latent variable are optimized.

8. A computer of the motion generating device Find the latent variables, generating an action from the latent variables; The latent variables are constrained to transition on an event-by-event basis to enable frame-level event editing. program.

Images