How to learn

Abstract

To efficiently perform learning by taking state change into consideration in performing learning by using time series attitude information.SOLUTION: A learning method that uses a sequence as learning data calculates task errors and contrastive errors in contrastive learning by using a query and a key, performs parameter learning of an encoder f, a decoder g, first extractor ψ and a second extractor φ by using these errors, sets a sequence piece being a portion of a sequence in respective sequences in determining a positive example and a negative example from the sequence, searches a similar sequence piece similar to the sequence piece from the sequence, makes an attitude located in a future from the sequence piece only by prescribed width a positive example, and makes an attitude located in a future from the searched similar sequence piece only by the prescribed width and being dissimilar to the positive example a negative example.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a learning method and a method for expanding learning data when learning using time-series posture information. [Background technology]

[0002] <General motion prediction technology> FIG. 1 shows the configuration of a typical predictor P1 related to human motion prediction technology. The predictor P1 uses past (up to the present) time-series posture information X 1:t When you input the future time series posture information X^ t+1:t+Δt Here, the time is expressed as an integer number in the same way as is customary for specifying the frame time of a video. 1:t The posture-related information from time 1 to t frame is output as X^ t+1:t+Δt denotes the predicted pose-related information from t+1 to t+Δt frames.

[0003] Specifically, the elements of X, which are the input and output of the predictor P1, are at least the 2D or 3D positions or movements of multiple key points (often joints, so hereinafter, the explanation is given for joints) that indicate the skeleton, and various other information obtained at the time of the frame is input, and the same type of information as this input is output. To give an example of the operation image of such a predictor P1, for example, if the predictor P1 is input with time-series posture information and speed of a person walking monotonously, it will output time-series posture information of the person walking monotonously in the future, and if it is input with a running scene, it will output a scene of the person running in the future. This motion predictor P1 is often trained using deep learning.

[0004] In this motion prediction, as in Non-Patent Document 1, an encoder-decoder model is often used, and the predictor P1 in Figure 1 is also configured as an encoder-decoder model, with a predictor P1 having an encoder E1 on the front stage and a decoder D1 on the rear stage.

[0005] It is necessary to train the parameters of the encoder E1 and the decoder D1, which are deep learning networks that configure the predictor P1, in advance. As shown in FIG. 1 as a schematic diagram of process M1, in this training, the loss of the predictor P1 is calculated based on the future information X t+1:t+Δt and the information output by the encoder-decoder model is X^ t+1:t+Δt The loss is generally defined as the mean square error (MSE) between the input X and the training parameters are trained to minimize this loss. 1:t The latent space h output by the encoder E1 for (i.e., the intermediate data h, which is an embedding vector in the latent space and is input to the decoder D1) undergoes representation learning in the process of reducing this loss. This representation learning can be said to be learning to improve prediction and reconstruction.

[0006] <Conventional method of route prediction using contrastive learning> Meanwhile, in recognition technologies such as image recognition, a technique called contrastive learning has been attracting attention. Figure 2 shows a schematic diagram of the use of contrastive learning in image recognition. In contrastive learning, positive and negative examples (as multiple keys) are automatically created for an anchor image (as a single query), and representation learning is performed to be able to distinguish between positive and negative examples. Positive examples are often created by data augmentation, and negative examples often use data with different label values. Learning is performed so that similar things are close to each other in the latent space h, and different things are far from each other in the latent space h. It is known that high-quality representation learning is possible even with a small dataset.

[0007] In the example of Figure 2, the anchor image, the positive example image, and the negative example image are input to the encoder E2 consisting of a deep learning network, and the output is the query q and the positive example key k obtained in the latent space. + , negative example key k - In the learning process M2, the query q and the positive key k +The similarity between query q and negative key k is high (the distance is small). - The parameters of the encoder E2 are trained so that the similarity to the anchor image is low (the distance is large). If the anchor image is, for example, a human image, the positive example image can be prepared by data augmentation by subjecting this human image to color conversion, geometric conversion, noise superposition, etc., and the negative example image can be prepared as, for example, a cat image with a different label from the human image.

[0008] Generally, contrastive learning is used for recognition but not for prediction, but there are examples where contrastive learning is used for path prediction. In Non-Patent Document 2, contrastive learning is used in combination with an encoder-decoder model for encoder representation learning.

[0009] Fig. 3 shows a schematic diagram of the contrastive learning technique of Non-Patent Document 2. Non-Patent Document 2 uses contrastive learning to address the issue that it is not possible to robustly predict the path of each individual (agent) in a scene in which multiple people exist as a multi-agent problem (for example, the result of predicting a path that is socially difficult to accept, such as causing agents to collide with each other, may be obtained).

[0010] In the contrast learning of this method, negative example data of human paths (trajectories) is created in scenes with multiple people. With this method, cases where people collide or are likely to collide become negative examples, and these can be created automatically. This method makes it possible to use negative examples in path prediction by treating it as a trajectory classification problem. Figure 3 shows a schematic diagram of the overall process P10 of the contrast learning in this method, divided into four processes P11 to P14.

[0011] First, as a sample setting process P11 for contrastive learning, trajectory data S[i]={s 1 ,…,s T} [i]Prepare. As a framework for the learning method, specify the intermediate time t (1 < t < T), and for a certain agent i, the trajectory {s 1 ,…,s t}, [i] from that up to time t and the trajectory {s t+1 ,…,s T} [i] from time t + 1 to T are targeted for prediction. Since the trajectory data S[i] = {s 1 ,…,s T} [i] exists in its entirety from time 1 to T, the trajectory {s 1 ,…,s t} [i] is used as the input for learning, and the predicted values {ŝ t+1 ,…,ŝ T} [i] of the trajectory after time t + 1 are obtained, and loss evaluation is performed by comparing with the actual correct trajectory {s 1 ,…,s t} [i] , and the parameters of the deep learning network for path prediction can be learned.

[0012] In the same method, during this learning, in order to realize a robust trajectory prediction so that the predicted trajectory is not socially unacceptable such as people colliding with each other, contrast learning is used. In the process P11 of Figure 3, as an example, the trajectory data S[i] exists only for three people with i = 0, 1, 2. The agent (oneself) whose trajectory is to be predicted for loss evaluation during learning is i = 0, and the other agents (others) are i = 1, 2. Each trajectory indicated by a symbol in the figure is as follows. Note that for each trajectory, the history up to the present is drawn as a solid line arrow, and the future trajectory is drawn as a dashed line, and they are schematically distinguished. Trajectory A0h = {s 1 ,…,s t} [0] … Trajectory of oneself i = 0 up to time t (present) Trajectory A0f = {s t+1 ,…,s T} [0] … Trajectory of oneself i = 0 after time t + 1 (future) Trajectory A1h={s 1 ,…,s t} [1] …The trajectory of the first person i=1 up to time t (present) Trajectory A1f={s t+1 ,…,s T} [1] …The trajectory of the first person i=1 from time t+1 onwards (future) Trajectory A2h={s 1 ,…,s t} [2] …The trajectory of the second person i=2 up to time t (present) Trajectory A2f={s t+1 ,…,s T} [2] …The trajectory of the second person i=2 from time t+1 onwards (future)

[0013] In this method, when predicting a future trajectory A0f from one's own past trajectory A0h, in addition to using the already prepared future trajectory A0f as the correct answer, in order to apply contrastive learning, the future trajectories A1f and A2f of others (two people in the example in process P11 of Figure 3, but not limited to two people) are used to determine negative and positive examples as follows. ● Positions on other people's future trajectories A1f, A2f ... are considered as negative examples (If you were to move to that position, you would be in the same position as someone else and would collide with them.) ●Nearby positions on other people's future trajectories A1f, A2f ... are considered as negative examples (If you were to move to that position, you would be in close proximity to other people and there would be a risk of collision.) ● Your position on your future trajectory A0f and its neighboring positions ... are considered as positive examples (The prepared trajectory data D[i] (i=0,1,2) is collision-free over the entire time period 1 to t to T.)

[0014] In this way, in order to apply the contrastive learning framework in process P11, positive and negative examples are set as described above, and the trajectory data is encoded (feature extraction process P12), similarity calculation process P13, and classification process P14 are performed. (Note that, as is clear from the above definition, in this positive and negative example setting process P11, it is possible to automatically set from the prepared trajectory data D[i] (i=0, 1, 2) on the premise that there is no collision between agents in this trajectory data.)

[0015] Specifically, for the agent i (i=0 in the example of step P11 in the figure, but hereafter it will be a general i), the following equation is used: , As shown above, after encoding the past state with f, we apply ψ consisting of an MLP (multi-layer perceptron) and output the query. h t i =f(s 1:t , i) … (i is agent, s 1:t is the trajectory of agent i up to the present time t, i.e., s 1:t ={s 1 ,…,s t } [i] ) query=ψ(h t i ) …

[0016] On the other hand, the ground truth (positive example) after Δt seconds in the trajectory data of the agent i set in process P11 is compared with the collision examples (negative examples) and collision-like examples (negative examples) after Δt seconds in the trajectory data of the other agent i. t+Δt i Then, we sample the data as follows: <c>Output the key key as shown below. key=Φ(s t+Δt i , Δt) … <c>

[0017] The above formula< / c> < / c> , , <c>The process by [reference] is schematically shown as the feature extraction process P12 in FIG. 3.

[0018] Next, as also schematically shown in FIG. 3, in the similarity calculation process P13 and the classification process P14, learning is performed to increase the similarity of the key output from the grand truth (positive example) with respect to the query and decrease the similarity of the key output from the negative example (collision, samples likely to collide) with respect to the query by similarity calculation based on the cosine similarity between the query and the key obtained in the feature extraction process P12.

[0019] In the same method of Non-Patent Document 2, learning is performed to minimize both the classification error (contrast loss, contrast error) by contrast learning described in FIG. 3 above and the prediction error (task loss, task error) in normal trajectory prediction. FIG. 4 shows an outline of the learning procedure for minimizing both the classification error and the prediction error in Non-Patent Document 2.

[0020] In step S41, learning data and the future time difference Δt to be predicted are set. That is, as also illustrated in the process P11 of FIG. 3, movement trajectory data of a plurality of persons at times 1 to T are prepared, and a time t (1 < t < T) in the middle thereof and a future time t + Δt (1 < t < t + Δt < T) to be predicted are set. After calculating the individual losses of steps S41, S42, S43, and S44 according to the setting of this step S41, error backpropagation is performed using the total error obtained from these individual losses in step S45 to learn the parameters of the deep learning network, and this is repeated until convergence.

[0021] Step S41 corresponds to the process P11 of FIG. 3 and sets negative examples (and positive examples) for contrast learning. Step S42 corresponds to the process P11 of FIG. 3 and the sample in the lower row, and in step S43, after passing through the processes P12, P13, and P14 of FIG. 3, the contrast loss L by contrast learning contrastive In step S44, a predictor P1 of a general encoder-decoder network as shown in FIG. 1 is used to predict the position at future time t+ΔT from the trajectory from time 1 to t (trajectory up to the current time t), and the predicted result is compared with the correct answer (correct example) to calculate the task loss L task Calculate (f,g).

[0022] In step S45, parameters of the encoder f, the decoder g, the query extractor ψ, and the key extractor Φ are learned so as to minimize the total error L given by the following equation: L=L task (f,g) + λ L contrastive (f,ψ,Φ) Here, λ is a hyperparameter. The task loss L calculated in step S44 task minimizes the prediction error in the encoder-decoder model and uses the contrast loss L obtained in step S43. contrastive serves to distinguish between defined positive and negative examples. [Prior art documents] [Non-patent literature]

[0023] [Non-Patent Document 1] Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. "Convolutional sequence to sequence model for human dynamics" in CVPR, pp.5226-5234, 2018. [Non-Patent Document 2] Yuejiang Liu, Qi Yan, Alexandre Alahi,"Social NCE: Contrastive Learning of Socially-aware Motion Representations" in ICCV, 2021.(https: / / arxiv.org / pdf / 2012.11717.pdf) Summary of the Invention [Problem to be solved by the invention]

[0024] The above-mentioned conventional predictions based on learning do not take into account changes in the state of the action (animation). The state of the action will be described below.

[0025] Events that involve major changes in motion, such as "walking," "running," and "stopping," are called "states." Although not directly related to realistic motion prediction technology, in the field of 3D animation generation, there are motion clips that express various aspects of a "state," and these are blended together for use. For example, even within the state of "walking," there are clips of various walking styles, such as "normal walking," "brisk walking," "walking backwards," and "walking as if being pulled by something," and these are smoothly connected and blended for use. Here, actions that correspond to clips in the same state are called "child states." Figure 5 shows an example of such states and child states that can be expressed in a tree structure.

[0026] By the way, a predictor trained by deep learning is likely to output incorrectly when this "child state" changes. In other words, there are cases where the child state to be predicted after Δt frames is different even though the child state continues without much change, or cases where the child state to be predicted after Δt frames does not change even though there should be a change in the child state. For example, when changing from "normal walking" to "walking as if being pulled", the output may predict "normal walking" even though a sign of being pulled is input as past information. Conversely, even if the "normal walking" continues, the output may predict "walking as if being pulled". When there is a similar child state transition in the learning data, there are many errors in transitioning even though there is no sign of it.

[0027] Depending on the learning method, it is thought that similar problems are likely to occur not only with child state changes, but also with state changes such as the transition from "walking" to "running."

[0028] We believe this is due to inappropriate or insufficient encoding of state and child state changes in the representation learning of the encoder's latent space. In other words, there was a problem in that the encoder's representation learning was not performed in a way that would make it less likely for the motion predictor to misdetect (or more likely to correctly detect) changes in the state and child states, thereby improving prediction accuracy.

[0029] In general, data augmentation and contrastive learning are effective in improving the quality of expression learning. However, such data augmentation and contrastive learning methods have not been proposed for human motion prediction. Regarding contrastive learning, in Non-Patent Document 2, negative example data of a person's movement path is created on the condition that a collision has occurred or is about to occur, but this movement path is defined as a time series of two-dimensional coordinates that represent only one representative position (center of gravity, etc.) of the person, and the time-series posture data (time series of the person's pose at each time) that represents the person's movement is completely different data from such a movement path. In other words, since the negative example creation in Non-Patent Document 2 is specialized for two-dimensional movement paths, it is not possible to apply the negative example creation in Non-Patent Document 2 to time-series posture data whose data content is completely different. In other words, in the conventional technology, there is no method of creating negative example data for motion prediction to improve the prediction of motion using posture data.

[0030] In view of the above problems of the conventional technology, the first object of the present invention is to provide a learning method that can efficiently learn by taking into account changes in state and child state when learning using time-series posture information. In view of the above problems of the conventional technology, the second object of the present invention is to provide a learning method that can efficiently learn using the expanded data by performing data expansion that takes into account changes in state and child state when learning using time-series posture information. In view of the above problems of the conventional technology, the third object of the present invention is to provide an expansion method that can effectively expand data by taking into account changes in state and child state when learning using time-series posture information. [Means for solving the problem]

[0031] In order to achieve the first object, the present invention provides a learning method that uses a plurality of sequences, which are posture time series information, as learning data, and that converts the sequences into a latent space by applying an encoder, predicts future values ​​of the sequences by applying a decoder to the latent space, calculates a task error from a difference between the future values ​​and a correct answer in the sequence, applies a first extractor to the latent space to convert it into a query, determines positive and negative examples from the sequences, applies a second extractor to the positive and negative examples to convert them into keys, and calculates a contrast error in contrast learning using the query and the key. and learning parameters of the encoder, the decoder, the first extractor, and the second extractor using the task error and the contrast error, and determining positive examples and negative examples from the sequences includes: in each sequence, setting a sequence fragment that is a part of the sequence, searching within the sequence for a similar sequence fragment that is similar to the sequence fragment, determining a posture that is located a predetermined width in the future from the sequence fragment as a positive example, and determining, among postures that are located the predetermined width in the future from the searched similar sequence fragment, those that are dissimilar to the positive example as negative examples.

[0032] In order to achieve the second object, the present invention provides a learning method that uses a plurality of sequences, which are posture time-series information, as learning data, the method comprising: applying an encoder to the sequence to convert it into a latent space; predicting a future value of the sequence by applying a decoder to the latent space; calculating a task error from a difference between the future value and a correct answer in the sequence; converting the sequence into a query by applying a first extractor to the latent space; determining positive examples and negative examples from the sequence; converting the positive examples and negative examples into keys by applying a second extractor to the sequence; calculating a contrast error in contrast learning using the query and the key; and learning parameters of the encoder, the decoder, the first extractor, and the second extractor using the task error and the contrast error. In order to expand the learning data, the method comprises the steps of: converting a first sequence piece (X 1:t ) and finds a second sequence fragment (X t+1:t+N ) and The first sequence piece (X 1:t ), followed by the second sequence piece (X t+1:t+N ) modulated the operation of the third sequence piece (S t+1:t+Δτ ) to obtain training data by extending the one sequence, and 1:t ) and the third sequence piece (S t+1:t+Δτ ) is connected to the sequence, the fourth sequence piece (S tm: t' ) and set the fourth sequence piece (S tm: t' The last time (t') of the first sequence piece (X 1:t ), the fourth sequence piece (S tm: t' ) and is positioned a predetermined width (Δt) in the future from the second sequence piece (X t+1:t+N The second feature is to classify poses belonging to the negative example (Cd) as negative examples.

[0033] In addition, in order to achieve the third object, the present invention provides a method for expanding learning data when multiple sequences that are posture time series information are used as learning data, the method comprising the steps of: targeting one sequence, searching for a first sequence piece within the sequence that is determined to have little state change, setting a second sequence piece as a continuation of the first sequence piece within the sequence, and obtaining data extending the one sequence by connecting a third sequence piece in which the movement in the second sequence piece is modulated after the first sequence piece. Effect of the Invention

[0034] According to the first feature, when learning using time-series posture information, positive and negative examples are automatically set taking into account changes in state and child states, thereby enabling efficient learning in a contrastive learning framework. According to the second feature, when learning using time-series posture information, negative examples are automatically set using extended data in which it is known that changes in state and child states have occurred due to data modulation, thereby enabling efficient learning in a contrastive learning framework. According to the third feature, when learning using time-series posture information, data modulation allows effective data expansion taking into account changes in state and child states. [Brief description of the drawings]

[0035] < / c> [Figure 1] FIG. 1 is a diagram showing a configuration of a general predictor related to human movement prediction technology. [Diagram 2] FIG. 1 is a diagram illustrating the use of contrastive learning in image recognition. [Diagram 3] FIG. 1 is a diagram illustrating the contrastive learning technique of Non-Patent Document 2. [Figure 4] FIG. 1 is a diagram showing an overview of a learning procedure by minimizing both classification error and prediction error in Non-Patent Document 2. [Diagram 5] FIG. 2 is a diagram showing examples of states and sub-states that can be represented by a tree structure. [Figure 6] 1 is a functional block diagram of a motion analysis device according to an embodiment. [Figure 7] 1 is a flowchart of an operation of a motion analysis device according to an embodiment. [Figure 8] 1 is a flowchart of a learning method in a motion analysis device according to an embodiment. [Figure 9] FIG. 9 is a diagram showing a detailed flow of step S82 in FIG. 8 according to one embodiment. [Figure 10] FIG. 10 is a diagram showing a schematic example for explaining the process content of each step in FIG. 9 (setting of positive examples and negative examples for contrastive learning). [Figure 11] 1 is a flowchart of a data augmentation method according to one embodiment. [Figure 12] 1A to 1C are diagrams illustrating schematic examples for explaining processing contents related to a data augmentation method. [Figure 13] 13 is a flowchart of positive example and negative example pose setting according to the third embodiment, taking advantage of the fact that a synthesis sequence is known. [Figure 14] FIG. 2 is a diagram showing functional block configurations (only input / output relationships) of a state transition presence / absence discriminator, an action discriminator, and a presentation device. [Figure 15] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] 6 is a functional block diagram of a motion analysis device 40 according to an embodiment. As shown in the figure, the motion analysis device 40 includes a feature extraction unit 10 configured with an encoder f and a first extractor ψ, a generation unit 20 configured with a decoder g, a sample setting unit 30, a learning feature extraction unit 31 configured with a second extractor φ, a contrastive learning unit 32, a task learning unit 33, an update unit 34, and an extension unit 35. (Note that, as will be described later, during inference, the feature extraction unit 10 can use only the encoder f on the front stage and omit the first extractor ψ on the rear stage, and in this case, the feature extraction unit 10 may be a conversion unit 10 that handles the conversion process to the latent space h.)

[0037] FIG. 7 is a flowchart of the operation of the motion analysis device 40 according to an embodiment. Below, an overview of the operation of each unit in FIG. 6 will be given while explaining each step in FIG. 7. In step S1, learning data to be used in the learning process in the next step S2 is prepared by manual work or by collecting data from a database already constructed, and input to the sample setting unit 30, and then the process proceeds to step S2. As the learning data, a large number of time-series posture information (hereinafter referred to as "sequences") X[i]=X 1:T[i] Prepare (i=1,2,3,…).

[0038] Each of these sequences X[i] is the same type of data as the time-series posture information described with reference to FIG. 1, and includes at least information on the 2D or 3D positions or movements of multiple joints at each time t=1, 2, ..., T[i]. In other words, when the positions of the joints defined in this sequence X[i] are rendered as frame images at each time t and played back as a video, the movements of a person or the like at each time t=1, 2, ..., T[i] are reproduced as an image. For example, 1000 sequences X[1], X[2], ..., X

[1000] are prepared as capturing various scenes of a person walking, and the next 1000 sequences X

[1001] , X

[1002] , ..., X

[2000] are prepared as capturing various scenes of a person running. In this way, each individual sequence X[i] is prepared as the time-series posture information X of a single human (or a robot or animal as long as the joints can be defined). 1:T[i] A large number of sequences X[i] (i=1, 2, 3, ...) may be prepared as being composed of a single state as shown in Fig. 5. (That is, when each sequence X[i] is rendered as a frame image of the joint positions and played in a video format as described above, it becomes a so-called motion clip video, but the length T[i] (number of frames T[i]) of the video can be any length.)

[0039] Furthermore, each individual sequence X[i] prepared is composed of a single state, but may or may not include one or more changes in the child state illustrated in Fig. 5. For example, the sequence X[1] for i=1 may be composed of a single state of "walking" and a single child state in which "normal walking" continues from start to finish, while the sequence X[2] for i=2 may also be composed of a single state of "walking", but may start with "normal walking" and switch to "brisk walking" midway through, resulting in two child states.

[0040] In step S1, as an additional process, all or a part of the many pieces of training data X[i] (i=1, 2, ...) prepared above may be input to the extension unit 35, and the training data expanded by the extension unit 35 may be input to the sample setting unit 30. Details of the data expansion method by the extension unit 35 will be described later.

[0041] In step S2, the learning data prepared in step S1 is used by using a contrastive learning method to learn parameters of the feature extraction unit 10, the generation unit 20, and the learning feature extraction unit 31, which are configured as a deep learning network in the motion analysis device 40, and then the process proceeds to step S3. In step S3, as an inference process, the feature extraction unit 10 and the generation unit 20, to which the learned parameters in step S2 have been set, generate the sequence S to be inferred. 1:t The encoder f converts the input sequence S into a latent space h (embedding vector h in the latent space), and the decoder g decodes the latent space h to obtain the input sequence S 1:t The future value of S t+1:t+Δt The predicted value S^ t+1:t+Δt Then, the flow of FIG. 7 ends.

[0042] The inference process in step S3 is the same as the prediction process of a future sequence using the encoder-decoder model of the existing method described with reference to FIG. 1:t The encoder f and decoder g process the sequence over the next Δt time periods and obtain the predicted value S^ t+1:t+Δt or the input sequence S 1:t A sequence S for only one time period after t+1:t+1 By repeating the process of predicting, the predicted value S^ of the sequence for Δt time periods in the future is obtained. t+1:t+Δt (Note that in the following, such a sequence of only one time point will be referred to as a "pose" S, which corresponds to a frame of one time point in a video.) t+1 =S t+1:t+1 Define it as:

[0043] That is, the first time, the input sequence S 1:t The pose S after one time is obtained by processing the encoder f and the decoder g. t+1 The predicted value S^ t+1 The second time, the input sequence is {S 2:t ,S^ t+1 } to find the pose S after two times. t+2 The predicted value S^ t+2 The third time, the input sequence is {S 3:t ,S^ t+1 ,S^ t+2 } to find the pose S three times later. t+3 The predicted value S^ t+3 The output is repeated Δt times, and in the Δtth time, the input sequence is {S Δt:t ,S^ t+1 ,S^ t+2 …, S^ t+Δt-1 } to find the pose S after Δt. t+Δt The predicted value S^ t+Δt By outputting the predicted sequence S^ t+1:t+Δt The time series of predicted poses for each future time obtained by the repeated calculations {S^ t+1 ,S^ t+2 …, S^ t+Δt }. (Note that by repeating the output of one future time pose Δt times, the predicted sequence S^ t+1:t+Δt In order to simplify the drawing by showing only the main flow, only a part of the process flow for obtaining the predicted pose S^ at a future time output by the decoder g is shown in FIG. t+k (The flow of reading (k=1,2,…,Δt-1) again as input to the encoder f is omitted in Figure 6.)

[0044] As shown above in the functional block configuration of FIG. 6 and the flow of FIG. 7, the motion analysis device 40 of this embodiment uses a contrastive learning framework similar to that of Non-Patent Document 2 to learn parameters of a deep learning network, and uses an encoder f and a decoder g as an inference process to calculate an input sequence S 1:t From the future predicted value S^ t+1:t+Δt Here, while the data handled in Non-Patent Document 2 was the two-dimensional movement trajectories of multiple agents (i.e., the two-dimensional movement trajectory of a single representative point such as the center of gravity of a person), the data handled in this embodiment is a sequence that is time-series posture information of a single person, etc. (i.e., time-series information representing the positions and movements of multiple joints in a two-dimensional or three-dimensional space of a person, etc.), and although both are related to the movements of a person, etc., the specific data handled is completely different. Therefore, a unique method in this embodiment is required to utilize contrastive learning for sequences. Details of the contrastive learning in this embodiment (details of step S2 in FIG. 7) will be further described below.

[0045] The encoder f and the decoder g can be configured as a deep learning network similar to the configuration described above in Fig. 1, and the first extractor ψ and the second extractor φ can also be configured as an MLP (multilayer neural network) that is a deep learning network similar to the configuration described above in Fig. 3. In the following description, the symbols "f, g, ψ, φ" representing these deep learning networks are used as reference symbols meaning components as functional blocks as shown in Fig. 6, and also when expressing the processing content by the deep learning network as a function that obtains an output from an input, or when expressing the parameters of the deep learning network (parameters that are subject to sequential updating during learning), these symbols "f, g, ψ, φ" are used as variable names meaning function names or network parameters.

[0046] FIG. 8 is a flowchart of a learning method in the motion analysis device 40 according to an embodiment, and shows an example of details of step S2 in FIG. 7. When the flow in FIG. 8 is started, in step S80, the sample setting unit 30 prepares a batch B[k] to be used for learning in the kth time (k=1, 2, ...; k=1 at the start of the flow) from the learning data prepared in step S1 in FIG. 7, and then the flow proceeds to step S81. The kth batch B[k] is set to a sequence X[1], X[2], ..., X[N[k]] extracted from the learning data in total of N[k]. Note that, as a structure of the flow in FIG. 8, in the kth processing, learning is performed using this batch B[k]={X[1], X[2], ..., X[N[k]]} in steps S81 and S82, and parameters f, gφ, and φ of the encoder f, decoder g, first extractor ψ, and second extractor ψ, which are the deep learning network configuration in the motion analysis device 40, are repeatedly updated.

[0047] In step S81, the sample setting unit 30 sets positive and negative examples for contrastive learning and a correct answer for task learning from the batch B[k], and then calculates each loss (contrast loss and task loss) by processing each subsequent unit (feature extraction unit 10, generation unit 20, feature extraction unit for learning 31, contrastive learning unit 32, and task learning unit 33), and then proceeds to step S82. As shown in Fig. 8, this step S81 is configured to include steps S82, S83, and S84, and the processing content at each step is summarized as follows.

[0048] <Step S82> The sample setting unit 30 sets positive and negative examples by determining positive and negative poses from the batch B[k]. To set positive and negative examples, a unique method in this embodiment is used to enable contrastive learning. (Note that the positive examples are X, as will be described later in detail.) t+Δt Set the negative example as S t+Δt Set etc.)

[0049] <Step S83> Using the positive examples and negative examples set in step S82, the feature extraction unit 10 generates a query using the encoder f and the first extractor ψ (i.e., query=ψ(h)=ψ(f(X 1:t )) and in the learning feature extraction unit 31, one positive example X t+Δt and multiple negative examples S t+Δt A plurality of keys (corresponding to positive examples and negative examples) are generated from the query and the key, and a contrast error is calculated in the contrast learning unit 32 using the corresponding query and key as input, and output to the update unit 34. The contrast error calculated from the query and the key may be calculated as a cross entropy error or the like, as in Non-Patent Document 2 and other general contrast learning methods.

[0050] It is expected that the latent space h transformed by the trained encoder f will be closer to the values ​​corresponding to positive examples and farther from the values ​​corresponding to negative examples due to the contrast error. Similarly, it is expected that the predicted values ​​generated by the trained decoder g will be closer to the ground truth and the values ​​corresponding to positive examples and farther from the values ​​corresponding to negative examples due to the parallel learning by task error.

[0051] <Step S84> This step S84 can be executed separately and in parallel with the above steps S82 and S83. In step S84, the sample setting unit 30 selects a sequence X[k] from the batch B[k] as a prediction source. 1:t and input it to the encoder f, and the encoder f calculates the latent space h=f(X 1:t ) and this latent space h is decoded by the decoder g to obtain the future predicted value X^ t+1:t+Δt =g(h) to the task learning unit 33. Furthermore, the sample setting unit 30 outputs the prediction source sequence X 1:t The correct answer for the future prediction result X t+1:t+Δt The task learning unit 33 also outputs the obtained predicted value X^ t+1:t+Δt and the corresponding correct answer X t+1:t+Δt The task error is calculated from the above and output to the update unit 34.

[0052] With regard to step S81 which is made up of steps S82, S83 and S84, the details of step S82 for setting positive examples and negative examples will be described later.

[0053] In addition, in step S82, the positive example X t+Δt and negative example S t+Δt By using a common Δt representing the future width on the time axis when setting, and a common Δt representing the predicted future width on the time axis when calculating the predicted value in step S84, it is possible to calculate the contrast error and task error based on a common batch B[k].

[0054] In step S85, the update unit 34 performs error backpropagation using a total error determined as a weighted sum of the contrast error calculated by the contrast learning unit 32 in step S83 for the batch B[k] and the task error calculated by the task learning unit 33 in step S84, thereby updating the parameters of the encoder f, the decoder g, the first extractor ψ, and the second extractor φ, and then the process proceeds to step S86. The error backpropagation process itself may be performed in the same manner as is usually performed in deep learning, and in this embodiment, error backpropagation may be performed in the following three paths, and the corresponding ones (those present on the paths) of the parameters f, g, ψ, and φ may be updated. 1st path: Decoder g → Encoder f Second path: first extractor ψ → encoder f Third path: Second extractor φ

[0055] Note that the first path is the inverse of the path used to calculate the task error, and the second and third paths are the inverses of the two parallel paths used to calculate the contrast error, respectively. When backpropagating the errors in the update unit 34, the total error calculated in step S85 can be used as the weighted sum of these two errors.

[0056] Alternatively, as a variant, instead of minimizing the overall error as in some adversarial learning techniques, parameters may be updated alternately by using only the task error when backpropagating the error in the first pass, and using only the control error when backpropagating the error in the second and third passes. When learning alternately and updating parameters, a ratio may be specified, or the control error may be backpropagated only when a control error exists.

[0057] In FIG. 6, the process flow in which the update unit 34 updates the parameters f, g, ψ, and φ by performing error backpropagation is omitted from the perspective of simplifying the drawing.

[0058] In step S86, the update unit 34 judges whether the learning is completed by a convergence judgment, and if it is completed, the process proceeds to step S87, and if it is not completed, the process returns to step S80, and the above learning process is repeated by setting the batch B[k+1] for the k+1th time, which is the next time after the kth time. The convergence judgment in step S86 may be judged by any existing method in deep learning, and may be judged, for example, by determining that the change from the previous (k-1th) value due to the kth update result of each parameter f, g, ψ, φ is small by a threshold judgment, or by determining that the total error calculated in step S85 is small by a threshold judgment, or by determining that the value of k, which is the number of repetitions, has reached a predetermined upper limit.

[0059] In step S87, the parameters f, g, ψ, and φ for which the convergence judgment has been made in the kth iteration are output as learning results, and these parameters are set in the encoder f, the decoder g, and the first extractor ψ (and the second extractor φ) to enable the inference process (step S3 in FIG. 7), and the flow in FIG. 8 is terminated.

[0060] <First embodiment of negative example pose setting> Hereinafter, details of step S82 (steps S82, S83, S84) in Fig. 8, the details of which will be described later, will be described. Fig. 9 is a diagram showing a detailed flow of step S82 according to one embodiment, and this step S82 can be realized by three steps S821, S822, S823. As already explained regarding step S82, the subject of operations of step S82 and the three steps S821, S822, S823 constituting it is the sample setting unit 30, so below only the processing contents of the three steps S821, S822, S823 will be explained, and reference to the subject of operations will be omitted.

[0061] 10 shows a schematic example for explaining the process contents (setting of positive and negative examples for contrastive learning) of each step S821, S822, and S823 in FIG. 9. In FIG. 10, an arbitrary sequence X[i] included in the batch B[k] used in the k-th learning process is a sequence X[i] spanning times 1 to T. 1:T As being such, the processing content for this entire sequence X 1:T is divided into example EX1 to EX4 and shown. In the following description, the example of this Figure 10 is referred to as an explanatory example.

[0062] <Step S821> As shown in example EX1, for the entire sequence X 1:T as a part of it, for example, from the first part, an input target (past) sequence segment X 1:t+Δt is set (t + Δt < T), and a sequence segment S 1:t+Δt similar to the front side part (past part) X 1:t of the target sequence segment X 1:t is searched from sequences other than the target sequence X 1:t+Δt . Note that in the example of Figure 10 including example EX1, the sequence segment X 1:t+Δt is set in the first part of the entire sequence X 1:T as an example, so the search location as other than this becomes the future side sequence part X t+Δt+1:T . Generally, an arbitrary partial sequence X 1:T within the entire sequence X A:B (1 ≦ A < B ≦ T) is used as a sequence segment, and the search location can be the remaining part (partial sequences X 1:A-1 and X B+1:T located in front and behind respectively). Hereinafter, regarding the sequence segment X A:B that can be an arbitrary position like this, using the first part X 1:t+Δt as in example EX1, the case where the search range becomes the future part X t+Δt+1:T will be described as an example.

[0063] Note that for the entire sequence under consideration, a sequence (a segment that forms a part of the entire sequence) consisting of a part of that sequence is called a sequence segment.

[0064] This search is a method similar to template matching in the case of an image. Using the front side part X 1:t of the target sequence segment as a template, the future range X t+Δt+1:T While moving, the position where the similarity at each position is judged to be high by the threshold value (there may be any number of such positions), is called the similar sequence piece S 1:t (Note that similar sequence fragment S 1:t is in the future range X t+Δt+1:T Since it is inside the "S 1:t =X a:a+t-1 " (t+Δt+1≦a and a+t-1≦T) holds true.)

[0065] Similarity can be calculated by using joint-related errors such as MPJPE (Mean Per Joint Position Error), and the lower the MPJPE, the more similar the two are. Other errors used in motion prediction, such as MJE (Mean Joint Error), MAnE (Mean Angular Error), and PCK (Percentage of Correct Keypoints), can also be used.

[0066] That is, the whole sequence X 1:T Pose X at any time b in b is considered to be a vector that lists the joint coordinates, etc. (Pose X b is a vector X that lists the data values ​​of each joint position that constitutes the pose. b ) and the front part X 1:t The data consists of vectors like the following: X 1:t =(X 1 ,X 2 ,…,X t )

[0067] The data X that is the result of arranging this vector 1:t Future range X as a template a:a+t-1 The difference when searching for "X a:a+t-1 -X 1:t " is as follows, so the error (magnitude of difference) of the joint coordinates, etc. is evaluated from the difference using the MPJPE etc., and the search results for the parts that are judged to be similar because the error is small are called similar sequence pieces S 1:t We can obtain it as follows. X a:a+t-1 -X 1:t =(X a -X 1 ,X a+1 -X 2 ,…,X a+t-1 -X t )

[0068] In the example EX2 of FIG. 10, as a result of searching for the example EX1, two similar sequence fragments S 1:t An example is shown in which

[0069] If the search is not exhaustive and there is periodicity, such as walking, it is possible to use that knowledge to perform the search. For example, when walking, the foot changes periodically from right foot to left foot to right foot, so by using that periodicity, for example, by limiting the search position to only the intermittent positions for each period, the amount of calculations can be reduced compared to a full search.

[0070] <Step S822> Front side portion X of the sequence of interest 1:t Pose X after Δt from the final frame t t+Δt (That is, the sequence of interest X 1:t+Δt The last pose of X t+Δt ) and the similar sequence fragment S of the search result in step S821. 1:t For each of the similar sequence pieces S 1:t Pose S after Δt from frame t t+Δt The similarity is calculated.

[0071] The calculation of the similarity between poses in step S822 may be the same as the calculation of the similarity between sequence pieces in step S821, and can be calculated using an error used in motion prediction such as MPJPE, with the smaller the error, the higher the similarity. (That is, since a sequence with a time length of 1 is a pose, the similarity can be calculated in a similar manner.)

[0072] <Step S823> Front part X of the sequence piece of interest 1:t and the similar sequence piece S evaluated in step S822 above. 1:t Pose X after Δt t+Δt , S t+Δt If the similarity between pose X and pose X is low, t+Δt , S t+Δt If it is determined that S is dissimilar, t+Δt All of the positive examples are X t+Δt (That is, the sequence of interest X 1:t+Δt The last pose of X t+Δt ) and set it as one.

[0073] In addition, the sequence of interest X 1:t+Δt It is desirable to find many poses that can be set as negative examples. If it is determined that the number of set negative examples is small, the following additional processing may be performed to increase the number of negative examples. That is, by setting |Δs|<Δt, the number of poses set as negative examples is increased by Δs. t+Δt A pose S located nearby on the time axis t+Δt+Δs You can also add X to the negative examples. t+Δt is a positive example.

[0074] Examples EX3 and EX4 are shown as examples following Examples EX1 and EX2 of FIG. 10. In Example EX3, two similar sequence fragments S 1:t For each of the poses S after Δt t+Δt Pose X t+Δt In the example EX4, the one located earlier on the time axis is determined to be dissimilar, and the corresponding pose S t+Δt is set as a negative example, and the positive example X t+Δt is shown to be set.

[0075] In addition, as a premise (assumption) for the processing of steps S821, S822, and S823 in FIG. 9, the entire sequence X 1:T The content of the sequence X corresponds to a motion clip of the same state with a mixture of substates. 1:T Set the start position of the sequence of interest X 1:t+Δt (As mentioned above, the target sequence fragment X can generally be set at any position, not just the start position. A:B ) should be set to a length that is not too long so that it corresponds to a single child state (i.e., so that there is no transition to another child state in the middle).

[0076] Attention sequence piece X 1:t+Δt Whether or not a state transition exists is automatically determined by a state transition discriminator 51 described later, and the state transition discriminator 51 automatically determines whether or not a state transition exists in a batch B[k]. 1:t+Δt Alternatively, only those states corresponding to a single child state may be used for learning.

[0077] The above process is performed assuming that the above preconditions are met, starting from the start position of the sequence piece of interest X 1:t+Δt The length of time t when setting the sequence of interest X A:B The process can be fully automated by specifying the position and length of similar sequence fragment S and Δt in advance. 1:t Search for pose S after Δt. t+Δt The similarity of the two can be calculated, and negative example data can be automatically selected based on the result of the similarity. t+1 ~S t+Δt This has the advantage that it is possible to easily specify negative example data without the trouble of manually creating transition data. On the other hand, when sequences containing transitions are used as negative examples, it has the advantage that the child state changes can be more effectively reflected in the learning.

[0078] In the process of FIG. 9 and the explanatory example of FIG. 10, the entire sequence X 1:T Immediately after the start of the sequence of interest X 1:t+Δt However, as explained above, the entire sequence X 1:T The purpose of this process is to automatically estimate that there is a change in the child state from the middle of the sequence and automatically set positive and negative examples. Therefore, as mentioned above, the sequence fragment of interest is the sequence fragment X located immediately after the start position. 1:t+Δt In general, the sequence fragment of interest X A:B can be used.

[0079] In this embodiment, as described above, positive examples in which the child state does not change and negative examples in which the child state changes are automatically set, and learning is performed to minimize the contrast error and task error in the framework of contrastive learning. Therefore, the finally trained predictor (encoder f and decoder g) is a predictor that can predict the input sequence X 1:t For each, the future predicted value X^ is calculated appropriately so that the state and child states do not change. t+1:t+Δt (Note that, as will be described later in Supplementary Note (2), this embodiment can also perform learning when the child state or the like changes by using the above-described method as is.)

[0080] <Second embodiment of negative example pose setting> As already indicated in the heading, the above-mentioned automatic negative example pose setting in the sample setting unit 30 according to the flow of Fig. 9 is the first embodiment. Next, a second embodiment of this automatic negative example pose setting will be described. The first and second embodiments may be combined to enable multiple negative example poses to be set, or negative example poses may be set using only one of the embodiments.

[0081] In the second embodiment, a predictor (encoder f and decoder g) is used to predict the input X 1:t The predicted value X^ output for t+Δt and the ground truth (correct answer) X t+Δt If the similarity of is lower than a certain threshold (the error is high), the predicted value X^ t+Δt is used as a negative example. The similarity can be evaluated using MPJPE or similar as described above. Note that the input X 1:t It is preferable that is training data rather than test data.

[0082] The predictor (encoder f and decoder g) may be one that is in the middle of learning according to the flow of FIG. 8, or one that has already been separately trained in advance.

[0083] When using something that is in the middle of learning, for example, in the repeated process shown in Figure 8, the case X^ where the error in the previous epoch (previous batch B[k-1]) is greater than a certain threshold t+Δt Then, during the learning of the current epoch (current batch B[k]), the representation learning may be performed using that case as a negative example. (In this case, the common sequence X 1:T If there exists a common sequence X 1:T In the case where the error is large in the evaluation result of the previous batch B[k-1], t+Δt 8 are performed in parallel for the current epoch (current batch B[k]), so that the error in step S84, which is performed in parallel, is large in the case X^ t+Δt may be used.

[0084] In addition, the pose X^ has a large error. t+Δt Instead, the predicted sequence X^ has a large error. t+1:t+Δt may be used as a negative example.

[0085] In conventional backpropagation, even if it is impossible to escape from an error-prone pattern, the negative examples act as a penalty greater than that of conventional backpropagation, and it is expected that representation learning will enable the model to escape from negative examples while learning.

[0086] In order to carry out the processing of the second embodiment described above, the task error evaluation result in the task learning unit 33 and the case X^ where the task error is large are required. t+Δt To simplify the drawing, however, the flow of this process is omitted from FIG. 6.

[0087] <Data extension method by extension unit 35 (can be added to step S1 in FIG. 7)> The following describes a data expansion method using the expansion unit 35, the details of which will be described later. Note that since the expansion unit 35 is the subject of operation in this data expansion method, the following description will not mention this subject of operation, and will only describe the processing content in the data expansion method.

[0088] Fig. 11 is a flowchart of a data augmentation method according to an embodiment. Fig. 12 is a diagram showing a schematic example for explaining the processing contents of Fig. 11 (and Fig. 13 described later) as processing contents related to the data augmentation method, and will be referred to appropriately in the following description.

[0089] In the data augmentation method according to the flow in Fig. 11, as shown in the example EX6 in Fig. 12, the original sequence X = X 1:T By synthesizing or modulating the child states in 1:T' By automatically generating the above, it is possible to achieve data expansion and increase the amount of training data that involves changes in the child state.

[0090] When the flow of FIG. 11 is started, in step S101, an original sequence X (=X 1:T ) in which the child state does not change much 1:t+N (⊂X), and in step S102, the first half X 1:t , latter half X t+1:t+N (Note that in the following steps, the latter half of this length N, X t+1:t+N is modulated to create a section of length Δτ, but it is preferable to set N>>Δτ with respect to the length before and after the modulation.)

[0091] Note that the original sequence section X where the child state does not change 1:t+N Whether or not the state transition occurs may be automatically determined by a state transition existence discriminator 51 described later. Also, as will be described later as supplementary note (2), regardless of the presence or absence of a change in the child state, any original sequence section X 1:t+N However, in the following description, it is assumed that a child state that does not change is selected.

[0092] In step S103, the first half X 1:t is copied and stored as the first half of the composite frame S, and in step S104, the second half X t+1:t+N Modulation is applied only to the modulated X t+1:t+N Part or all of the sequence in the latter part of the sequence S t+1:t+Δτ Let us assume that.

[0093] In step S105, the first half component part X copied in step S103 is 1:t and the latter part S modulated in step S104. t+1:t+Δτ Let S be a composite sequence S = (S 1:t ,S t+1:t+Δτ ), the flow of FIG. 11 ends. 1:t is S 1:t =X 1:t and the latter part S of the composite sequence S t+1:t+Δτ is S t+1:t+Δτ ≠X t+1:t+Δτ It becomes.

[0094] The modulation in step S104 can be performed by approximating the trajectory of each joint of the person with a curve and gradually shrinking or expanding the curve in the time direction, increasing or decreasing the amplitude in the spatial direction, etc. The change in amplitude or speed (in the time direction) should be set within the range of possible changes observed from the actual measured value.

[0095] Also, in step S104, instead of or in addition to modulation, synthesis may be performed. For synthesis, a blending technique in 3D animation can be used. Different child states may be blended. When creating a transition in 3D animation, synthesis can be performed by clarifying the transition start timing.

[0096] Modulation and synthesis are added from t+1, which is the start time of the latter half. By presetting a number of predetermined methods for the modulation and / or synthesis in step S104, which are defined in the form of a predetermined function or parameter, it becomes possible to automatically generate a number of synthesized sequences S from one original sequence X, and by adding a number of automatic sequences S and using them for learning, learning can be performed effectively by data expansion.

[0097] The flow of FIG. 11 is executed for all original sequences X to which the data augmentation method is applied. If it is known in advance that the entire original sequence X is composed of the states and child states of a single action, the selection of step S101 can be omitted and the original sequence section X 1:t+N We may set X to match the original sequence X (the whole sequence).

[0098] <Third embodiment of (positive and) negative example pose setting...when using synthetic sequences> When learning is performed using a synthetic sequence prepared by the above data augmentation method, the first or second embodiment described above may be used, but additional flag information indicating that it is a synthetic sequence (synthetic sequence S at the time of synthesis=(first half S 1:t , second half S t+1:t+Δτ ) that includes information to distinguish the non-composite first half from the composited second half, it is possible to set positive and negative example poses according to the third embodiment, where it is known that the sequence is a composite sequence. This additional flag information, etc., can include information on the original sequence X used to compose the composite sequence S (association information between the composite sequence S and the original sequence X), as shown in cases Cb and Cd (FIG. 12) described later, to set positive and negative example poses.

[0099] FIG. 13 is a flowchart of a process for setting positive and negative example poses according to the third embodiment, taking advantage of the fact that the synthesis sequence is known. Each step of the process will be described below.

[0100] When the flow of FIG. 13 is started, in step S201, an input sequence range for learning is selected in the composite sequence S (which is known to be a composite sequence by the above-mentioned flag information, etc.). This range is the “interesting sequence piece X” in the first embodiment. 1:t+Δt Front part of X 1:t " and the start time t m Range S from to end time t' tm:t' is considered to have been selected.

[0101] In the example EX7 in Figure 12, the selected input S tm:t' In particular, the input S tm:t' The final time t' of the composite sequence S = (first half S 1:t , second half S t+1:t+Δt ) depending on whether the position is in the first half or the second half of the example EX7. In step S202 and thereafter, positive and negative examples are set according to the case classification shown in this example EX7, and further according to the case classification and consideration into four cases Ca, Cb, Cc, and Cd as shown in example EX8.

[0102] In step S202, the input S tm:t' Regarding the time t' of the final frame, if t' is a frame prior to t (t'≦t), the process proceeds to step S203, and if not (if t'>t), the process proceeds to step S204.

[0103] In step S203, the input sequence S tm:t' (At this time, S tm:t '=X tm:t' ) does not contain any indication of change, so the modulated or synthesized S t'+Δt can be used as a "negative example". Here, we assume that in the synthetic sequence S, the input sequence S tm::t' The pose S ahead of the predicted future width Δt t'+Δt is a composite sequence S = (first half S 1:t , second half S t+1:t+Δt The synthesis process and Δt are set so that the signal belongs to the latter half of the

[0104] On the other hand, as shown in case Cb of FIG. 12, in step S203, the sequence X is the same as the original sequence X. t'+Δt is set as a "positive example." Note that, based on the above assumption, t'+Δt≧t+1. In other words, the child state transition start timing is t+1.

[0105] In step S204, the input sequence S in the composite sequence S is tm:t' The final frame t' of the input sequence S is t+1 or later (t'≧t+1). tm:t' contains the signs of change. t+1:t' Therefore, as shown in case Cc in Fig. 12, the part S, which is a modulated or synthesized pause located after the premonition, t'+Δt can be used as "positive examples" or "augmented data" for both contrastive training and training using the prediction error of encoder f and decoder g. (Note that this "augmented data" can be used to add correct answer data when training using the prediction error of encoder f and decoder g.)

[0106] On the other hand, as shown in case Cd of FIG. 12, in step S204, t'+Δt is the input S tm:t' It is discontinuous with respect to t and can be used as a "negative example" for contrastive learning. m ≦t. (That is, the input sequence S tm:t' Since the child state of is changed between times t and t+1, it is assumed that the child state after the change will continue at the future time t'+Δt. t'+Δt is a child state before the change and is discontinuous, which contradicts this assumption, so it is considered a "negative example.")

[0107] As described above, in the third embodiment using the data augmentation method, not only data augmentation but also negative examples can be created using synthetic data, and a lot of data that contributes to expression learning regarding child state transition boundaries can be created. It is expected that the boundaries of child state transitions can be correctly captured by the above data augmentation (including negative example data creation).

[0108] As shown in the upper part of the table in Example EX8 in Figure 12, instead of or in addition to the pause, the sequence S t'+1:t'+Δ t or X t'+1:t'+Δt may be a positive example, a negative example, or augmented data.

[0109] As described above, according to each embodiment of the present invention, when performing learning taking into account a task or contrastive learning using a sequence that is time-series posture information, it is possible to perform efficient learning even with limited learning data, taking into account the presence or absence of changes in the state or child states within the sequence, and to perform data expansion so as to enable efficient learning taking into account the changes.

[0110] In other words, it is possible to learn high-quality representations to reduce errors in motion predictions that include child state transitions. In particular, by identifying negative examples of motion posture sequences, it is possible to make effective use of small amounts of training data. In addition, by expanding data and creating negative examples and inputting them, it is possible to pad out a large amount of training data and improve the quality of representation learning near the boundaries. In addition, when negative examples are future postures, it is possible to easily learn without sequence synthesis.

[0111] Various supplementary and alternative examples are explained below.

[0112] (1) As an application example of the encoder f and the decoder g learned in the embodiment of the present invention, when remote communication in which transmission delays may occur is performed by an avatar, it is possible to realize smooth remote communication by predicting the future actions of the avatar and appropriately correcting the delays, and it is possible to actually omit the user from traveling to a remote location for a meeting, etc. (travel to a remote location is not necessarily required). Therefore, such an application example of the embodiment of the present invention can reduce carbon dioxide emissions by saving energy resources required for user movement, which makes it possible to contribute to Goal 13 of the Sustainable Development Goals (SDGs) led by the United Nations, "Take urgent action to combat climate change and its impacts."

[0113] (2) Regarding step S101 of the data augmentation method (FIG. 11), the original sequence section X in which the child state does not change much 1:t+N However, the original sequence section X 1:t+N may be selected. In that case, the time t+1 at which the change occurs is specified manually (the state transition presence / absence discriminator 51 described later may be used), and the modulation and synthesis are performed so that no change occurs in the synthesized sequence. The subsequent processing is the same. In this case, the child state of the original sequence changes at time t+1 accompanied by a sign of a child state change, and the child state of the synthesized sequence remains the same even after time t+1 despite the sign of a state change, and since the contents of the original sequence and the synthesized sequence that were the premise of the example EX8 in FIG. 12 are interchanged, it is sufficient to interchange the negative example cases Ca and Cd with each other and also interchange the positive example cases Cc and Cb with each other in the case classification table of the example EX8 in FIG. 12. In other words, the method of determining the positive example and negative example is the same as that described in FIG. 12 on the premise that the child state of the original sequence does not change much, but only the meanings are interchanged and interpreted as described above.

[0114] Similarly, in the first embodiment, by using the state transition existence discriminator 51, the original sequence section X 1:t+N However, if we exclude this assumption and select the original sequence section X where the child state changes, regardless of whether the child state changes or not, 1:t+N On the premise that a search result that satisfies the condition is obtained, positive examples and negative examples can be set by the same process as in the first embodiment. That is, in the original sequence section X 1:t+N In the case of a transition from a child state α to another child state β, the entire original sequence X (original sequence interval X 1:t+N ⊂X), such a transition from α to β is included in the original sequence interval X 1:t+N By obtaining the other location as a search result, it is possible to set positive examples and negative examples in exactly the same manner as in the first embodiment. Note that the positive example in this case corresponds to a posture that has transitioned to another child state β.

[0115] However, even if it is not possible to prepare original learning data in which the child state "α → β" transition occurs two or more times as described above (or if only a small number of such data can be prepared), it is possible to learn only the cases in which a single child state continues (or to mainly learn such cases) using the first embodiment, and to allow the system to sufficiently learn cases involving child state transitions using the third embodiment.

[0116] (3) In addition, in the learning of the third embodiment using the data augmentation method, the final frame timing t' may be fixed and used, but by learning for various input ranges while changing t', it is possible to learn expressions that perform more correct state transition judgment. In other words, by simultaneously learning negative examples of child state transitions and data augmentation (data padding) that becomes correct examples of child state transitions, it is possible to learn expressions that capture more correct child state transitions and transition timings. By using t' while changing it, it is also possible to effectively use synthetic data. By learning the final frame timing t' including cases of both t'≦t (before the child state transitions) and t'≧t+1 (after the child state transitions), it becomes easier to learn the transition boundaries.

[0117] (4) The methods shown in each embodiment have been described using child state transitions, but similar data augmentation and negative example selection / creation methods can also be used for state transitions such as changing from “walking” to “running.”

[0118] In addition, in each embodiment, the input is time-series posture information, but even if there is only one input frame, the method of each embodiment can be applied as long as the movement is input based on the posture movement vector information from the previous frame.

[0119] (5) About "State Transition Detector 51" In addition, the positive examples in the third embodiment are obtained by dividing the input sequence S as a part of the original sequence X, not as a part of the composite sequence S in FIG. tm:t' ), a binary label can be added to the sequence of interest, with the child state transition occurring between times t and t+1 when t'≦t, and the binary label indicating "no sign of change" when t'≧t+1, and "sign of change included" when t'≧t+1. (Note that the binary label is added to the input sequence S tm:t' Instead of assigning each time step to each frame of the input sequence S tm:t' The time t and t+1 at which the child state transitions occur are to be given manually in advance.) This data can be used for supervised learning to be used as learning data for a "state transition presence / absence discriminator 51" that discriminates whether or not an input sequence contains a sign of change (transition). In other words, when the input sequence is judged by this discriminator 51 to have a sign of change, this means that the input sequence does not correspond to a single child state, but is judged to have a change in the child state midway. This discriminator 51 may be configured as any deep learning network or the like that performs binary classification.

[0120] In the above, it is permitted to use an original sequence X in which a child state transition can occur between times t and t+1. However, instead of and / or in addition to this original sequence X, a composite sequence S may be used.

[0121] In addition, when changing from one action (state / substate) to another action (state / substate), a blend ratio may be assigned and the system may learn to infer the blend ratio.

[0122] (6) The positive examples and negative examples described in each embodiment can be used not only for contrast learning purposes, but also for learning the action discriminator 52 of the predicted output, and can be used for analysis purposes similar to the above-mentioned "state transition presence / absence discriminator 51". For example, in both the first and third embodiments, the positive examples can be used as possible outputs and the negative examples as less likely outputs for learning the action discriminator. In other words, the positive and negative examples are used for learning the action discriminator 52 of possible future postures. At this time, the first input is the first half sequence of the sequence of interest, the second input is the candidate value of the posture after Δt of the first half sequence, and the output is learned as the action discrimination result (the discrimination result that the candidate value of the posture is actually possible / impossible, or information on the likelihood of the posture candidate value, such as a defined probability). This action discriminator 52 may be configured as any deep learning network that performs binary classification, etc. Δt may be added to the third input as an option.

[0123] (7) In addition, the positive and negative examples obtained in each embodiment may be used as a prediction of various futures when Δt is relatively large (Δt≧1 second). In the first embodiment, both the positive and negative examples can be regarded as future postures that are actually possible. In the third embodiment, they can be regarded as being synthesized, so that future postures with high diversity can be created. By presenting the positive and negative examples as examples of future postures by the presentation device 53, it is possible to use them as an example of use to help CG animation creators select future actions. In the negative examples of each embodiment, it is considered that the diversity of postures and the diversity in the time direction (diversity of the timing of starting the motion transition, diversity of speed) will appear as separate negative examples. It is also good to calculate the similarity between negative examples and group negative examples with similar postures. In addition, if it is not desired to consider the diversity in the speed direction, it is good to group similar negative examples while performing time shifting or speed change as appropriate. In the case of grouping, it is good to present one posture in the group, and it is good to select one with low similarity to the positive example. If it is possible to present a sequence, it is good to present the sequence.

[0124] 14 shows functional block configurations (only input / output relationships) of the state transition presence / absence discriminator 51, the action discriminator 52, and the presentation device 53 with respect to the above (5) to (7). Note that the presentation device 53 performs a process similar to the process of deriving positive examples and negative examples from the original sequence (using a synthesized sequence, etc. as appropriate) in the learning process of each embodiment, and a process of consolidating the above negative examples, and displays the positive and negative examples to provide a user interface to CG animation creators, etc., and can also be used for purposes other than the learning process.

[0125] (8) FIG. 15 is a diagram showing an example of a hardware configuration in a general computer device 70. The motion analysis device 40 and each device in FIG. 14 can be realized as one or more computer devices 70 having such a configuration. When the motion analysis device 40 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, a keyboard, a touch panel, or the like, and a bus BS for transmitting and receiving data between them.

[0126] Each functional unit of the motion analysis device 40, each step in each flowchart of Figures 7, 8, 9, 11, 13, etc. (excluding the case where the original learning data is prepared manually in step S1 of Figure 7 instead of automatically acquiring it by accessing a pre-constructed database), and each device of Figure 14 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit or each step from a ROM 74 and executes it. Note that both the CPU 71 and the GPU 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the GPU 72. [Explanation of symbols]

[0127] 40...motion analysis device, 10...feature extraction unit, 20...generation unit, 30...sample setting unit, 31...learning feature extraction unit, 32...contrast learning unit, 33...task learning unit, 34...update unit, 35...extension unit f...encoder, g...decoder, ψ...first extractor, φ...second extractor

Claims

1. A learning method using a plurality of sequences that are posture time-series information as learning data, comprising the steps of: applying an encoder to the sequence to convert it into a latent space, applying a decoder to the latent space to predict future values ​​of the sequence, and calculating a task error based on the difference between the future values ​​and a ground truth for the sequence; applying a first extractor to the latent space to convert it into a query, determining positive examples and negative examples from the sequence, applying a second extractor to the positive examples and negative examples to convert them into keys, and calculating a contrast error in contrast learning using the query and the key; learning parameters of the encoder, the decoder, the first extractor and the second extractor using the task error and the control error; determining positive examples and negative examples from the sequences includes: in each sequence, setting a sequence fragment that is a part of the sequence, searching within the sequence for a similar sequence fragment that is similar to the sequence fragment, determining a posture that is located a predetermined width in the future from the sequence fragment as a positive example, and determining, among the postures that are located the predetermined width in the future from the searched similar sequence fragment, those that are dissimilar to the positive example as negative examples.

2. The learning method according to claim 1, characterized in that predicting future values ​​of the sequence by applying the encoder and the decoder includes setting, for each sequence, a sequence piece that is a part of the sequence, and predicting an attitude located a predetermined width in the future from the sequence piece.

3. Determining positive and negative examples from the sequence further comprises:

3. The learning method according to claim 2, further comprising: in predicting a future value of the sequence by applying the encoder, comparing a result of predicting a posture located in the future by the predetermined width from the sequence piece with a correct answer; and if the prediction result is determined to be dissimilar to the correct answer, setting the prediction result determined to be dissimilar as a negative example.

4. The learning method according to any one of claims 1 to 3, wherein the encoder, the decoder, the first extractor and the second extractor are configured as a deep learning network.

5. A learning method using a plurality of sequences that are posture time-series information as learning data, comprising the steps of: applying an encoder to the sequence to convert it into a latent space, applying a decoder to the latent space to predict future values ​​of the sequence, and calculating a task error based on the difference between the future values ​​and a ground truth for the sequence; applying a first extractor to the latent space to convert it into a query, determining positive examples and negative examples from the sequence, applying a second extractor to the positive examples and negative examples to convert them into keys, and calculating a contrast error in contrast learning using the query and the key; learning parameters of the encoder, the decoder, the first extractor and the second extractor using the task error and the control error; To expand the learning data, For at least one sequence in the learning data, a first sequence piece (X 1:t ) and finds a second sequence fragment (X t+1:t＋N ) and The first sequence piece (X 1:t ), followed by the second sequence piece (X t+1:t＋N ) modulated the operation of the third sequence piece (S t+1:t＋Δτ ) to obtain training data by extending the one sequence; To determine positive and negative examples from the expanded learning data, The first sequence piece (X 1:t ) and the third sequence piece (S t+1:t＋Δτ ) is connected to the sequence, the fourth sequence piece (S tｍ: t' ) and The fourth sequence piece (S tｍ: t' The last time (t') of the first sequence piece (X 1:t ) is outside the range, The fourth sequence piece (S tｍ: t' ) and the second sequence piece (X t+1:t＋N ) as negative examples (Cd).

6. In order to determine positive and negative examples from the expanded training data, The fourth sequence piece (S tｍ: t' The last time (t') of the first sequence piece (X 1:t ) range, The fourth sequence piece (S tｍ: t' ) and is positioned a predetermined width (Δt) in the future from the third sequence piece (S t+1:t＋Δτ ) are taken as negative examples (Ca), Or The fourth sequence piece (S tｍ: t' ) and the second sequence piece (X t+1:t＋N ) is taken as a positive example (Cb), or The fourth sequence piece (S tｍ: t' The last time (t') of the first sequence piece (X 1:t ) is outside the range, The fourth sequence piece (S tｍ: t' ) and is positioned a predetermined width (Δt) in the future from the third sequence piece (S t+1:t＋Δτ 6. The learning method according to claim 5, wherein postures belonging to the first group are regarded as positive examples (Cc).

7. The learning method according to any one of claims 1 to 4, further comprising: In order to expand the learning data used, searching for a first sequence fragment in one sequence that is determined to have a small state change, and setting a second sequence fragment as a portion of the sequence that continues to the first sequence fragment; and obtaining data extending one sequence by connecting a third sequence fragment in which the operation in the second sequence fragment is modulated after the first sequence fragment.

8. The method of claim 7 , wherein modulating the movement includes stretching in space and / or time.

Images