How to learn

Abstract

To provide a prediction apparatus that obtains prediction parameters for enabling an appropriate avatar drawing in a remote communication by the avatar drawing, by applying a prediction that, even though delays or the like occurs in various ways, appropriately reduces this effect to time series of avatar drawing parameters.SOLUTION: A prediction apparatus that receives avatar drawing parameters acquired for a remote user whose avatar is drawn at each time point via a communication network and predicts the parameters at a current time point includes a measurement unit 5 that acquires a delay between an acquisition time point associated with received latest parameters and the current time points, and a prediction unit 6 that predicts the parameters at the current time point from a history of the received parameters up to the latest parameters, a time change in the history, and the delay.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a prediction device for avatar drawing parameters suitable for remote communication involving delays, etc., and a learning method and program for the prediction process. [Background technology]

[0002] In a system that realizes remote communication via an avatar by estimating a user's movements and applying them to the avatar, smooth communication can be achieved if delays caused by transmission, etc. can be reduced.

[0003] Examples of conventional techniques for achieving the above include those disclosed in Patent Documents 1 and 2, which disclose the following methods. Patent Document 1 discloses a method for estimating the future movements of two users in VR (augmented reality) and controlling the fast-forwarding and delaying of video to synchronize the timing at which avatars' gazes meet when they look at each other. Patent Document 2 discloses a method for suppressing transmission delays by transmitting reference posture identification data that can identify information about the posture of the viewing user with a small amount of data, rather than posture feature values ​​that require a large amount of data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-009295 A [Patent Document 2] Patent No. 6731532 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-described conventional techniques do not necessarily realize sufficiently smooth remote communication in the face of various variable delay conditions.

[0006] In Patent Document 1, the gazes of the avatars match, but the user whose video is fast-forwarded does not yet see the other avatar, and conversely, the user whose video is delayed must wait for the movement to be reflected, which causes a problem of impeding smooth communication. In addition, the amount of delay in transmission and reception fluctuates depending on the network conditions, making it difficult to predict the timing of matching. Patent Document 2 attempts to suppress transmission delays by reducing the amount of transmission, but the time required for posture recognition and transmission is unavoidable, and there is a problem of delays corresponding to this time.

[0007] In view of the problems with the conventional techniques described above, a first object of the present invention is to provide a prediction device or a program for causing a computer to function as said device, which obtains predicted parameters that enable appropriate avatar drawing by applying predictions that appropriately reduce the effects of various delays and the like caused by transmission and / or drawing to a time series of avatar drawing parameters. A second object of the present invention is to provide a learning method for effectively learning a deep learning network that can be used in this prediction device. [Means for solving the problem]

[0008] In order to achieve the first object, a first feature of the present invention is a prediction device that receives avatar drawing parameters acquired at each time for a user for whom an avatar is to be drawn, and predicts the parameters at a current time, comprising: a measurement unit that acquires a delay between a time associated with the latest received parameters and the current time; and a prediction unit that predicts the parameters at the current time from a history of received parameters up to the latest parameters, changes over time in the history, and the delay.

[0009] In order to achieve the second object, the prediction device further includes a prediction unit that predicts parameters at the current time by using a deep learning network, and obtains a predicted value of a time change of the parameter at the current time from a history of the received parameters and a time change of the history by using the deep learning network, and calculates an amount of change of the parameter that occurs during the delay from the predicted value of the time change and the delay, and predicts the parameter at the current time by adding the amount of change to the latest received parameter. The present invention relates to a learning method for learning parameters of a deep learning network used by a prediction unit in the prediction device. A second feature of this learning method is that a parameter history, a time derivative of the history, and a delay are given in advance as correct answer data for learning, the parameter history, the time derivative, and the delay are input to a deep learning network in the middle of learning, and a parameter value and its time derivative at a time in the future that is the delay away from the history are output, and learning is performed by using the deviation from the correct answer data as a cost, and in evaluating the cost, when it is determined that the absolute value of the correct answer value of the time derivative is large, the cost of the parameter value and its time derivative at a time in the future that is the delay away from the history is weighted more heavily than when this determination is not made. Effect of the Invention

[0010] According to the first feature, the drawing parameters at the current time are predicted from the history and changes of the latest drawing parameters that have been received and are available for reference, and from the actual delay. Since the prediction is made while taking into account possible delays, it is possible to appropriately predict the drawing parameters at the current time that have not yet been received. According to the second feature, since learning is performed with a cost that is heavily weighted when the absolute value of the correct answer value of the time derivative of the parameter is large, it is possible to learn to more accurately predict behaviors that have a large visual impact, such as the opening and closing of the eyes and mouth. [Brief description of the drawings]

[0011] [Figure 1]FIG. 1 is a configuration diagram of a prediction system according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram of remote communication realized by a prediction system according to an embodiment. [Diagram 3] FIG. 2 is a functional block diagram of a first terminal and a second terminal constituting a prediction system according to an embodiment. [Figure 4] FIG. 13 is a diagram showing a schematic example of how delay varies; [Diagram 5] FIG. 13 is a schematic graph showing a case where a predicted value is obtained from a differential. [Figure 6] 1 is a configuration example of a deep learning network of a prediction unit according to an embodiment. [Figure 7] FIG. 4 is a diagram showing a modification of the configuration of FIG. 3 with respect to the implementation of the prediction system. [Figure 8] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1 is a configuration diagram of a prediction system 100 according to an embodiment, the prediction system 100 includes a first terminal 10 used by a first user U1 and a second terminal 20 used by a second user U2, and the first terminal 10 and the second terminal 20 are capable of communicating with each other via a network NW such as the Internet. Both the first terminal 10 and the second terminal 20 can be configured as general computer terminals such as smartphones, tablets, desktop or laptop computers, etc.

[0013] Figure 2 is a schematic diagram of remote communication realized by the prediction system 100 according to an embodiment. The first user U1 makes a greeting voice to the user U2 who is the partner of the remote communication at the current time t, and the voice packet is transmitted to the side of the second user U2 at high speed, and the greeting voice is played back in real time to the second user U2 at a time that can be regarded as approximately the current time t. On the other hand, for the first user U1, in order to draw the avatar A1 on the side of the second user U2, drawing parameters reflecting expressions and the like are acquired, transmitted to the side of the second user U2, and the avatar A1 is displayed in real time to the second user U2 by performing avatar drawing.

[0014] Here, the transmission of the drawing parameters and the avatar drawing involve a delay and require more time than the voice transmission / playback. Therefore, when the second user U2 plays back the voice of the first user at the current time t, only the drawing parameters up to the past time t - d (d > 0 and t - d < t) are obtained. That is, there is a delay of magnitude d, and moreover, this delay generally varies with time, becoming d = d(t). According to the prediction system 100 of the present embodiment, by predicting the drawing parameters of the first user U1 at the current time t using the drawing parameters that have a delay d and are only obtained up to the past time t - d and performing drawing, the avatar A1 can be drawn in real time in synchronization with the voice of the first user U1, and smooth remote communication can be realized.

[0015] Note that in FIG. 2, only the case where the first user U1 acquires the expression parameters and the avatar A1 is drawn and displayed to the second user U2 on the other side is shown. However, by swapping the roles, it is similarly possible to perform predictive drawing on the first user U1 for the second avatar A2 (not shown) of the second user U2 while preventing the delay effect and display it in real time. Thereby, the first user U1 and the second user U2 can perform two-way real-time remote communication via their respective first avatar A1 and second avatar A2.

[0016] However, since similar processing is possible by swapping roles as described above, the following will explain unidirectional processing when realizing two-way remote communication, where a first user U1 using the first terminal 10 is on the “self” side and is the target whose drawing parameters are acquired and whose avatar A1 is drawn, and a second user U2 using the second terminal 20 is on the “other” side and is the target whose drawn avatar A1 is displayed remotely.

[0017] 3 is a functional block diagram of a first terminal 10 and a second terminal 20 constituting a prediction system 100 according to an embodiment. As shown in the figure, the first terminal 10 includes an imaging unit 1, a recognition unit 2, a sound recording unit 3, and an analysis unit 4, and the second terminal 20 includes a measurement unit 5, a prediction unit 6, a drawing unit 7, and a presentation unit 8. Note that a prediction device 9 can be realized as a configuration in which any part of the prediction system 100 including the prediction unit 6 is extracted.

[0018] In the prediction system 100 of the present embodiment, the first terminal 10 extracts parameters A(t) for drawing an avatar A1 of a first user U1 and transmits them to a second terminal 20. The second terminal 20 then draws the avatar A1 of the first user U1 using the parameters A(t) and displays it for a second user U2. This process is repeated for each real-time time t=1, 2, .... However, as shown in FIG. 2, when drawing an avatar at a current time t on the second terminal 20, the parameters A(t) at the current time t are not received and cannot be referred to due to the existence of a delay d>0, and only parameters A(td), A(td-1), A(td-2), ... up to the past time td can be referred to. Therefore, by applying prediction, a predicted value A'(t) of the parameters A(t) at the current time t is used for drawing.

[0019] Hereinafter, the details of the processing contents of each functional block in Fig. 3 will be described as details of the common processing contents that are repeatedly executed for each real-time time t=1, 2, .... Note that the times t=1, 2, ... will be described as integers in chronological order, like frame numbers in the case of video. The first terminal 10 and the second terminal 20 are assumed to have their clocks (timekeeping functions managed as system clocks in a computer OS (operating system) or the like) synchronized in advance using an existing method such as a network time protocol, and can perform processing for each time t under a common clock.

[0020] The imaging unit 1 can be configured as hardware such as a camera, and obtains a captured image P(t) at a current time t by capturing an image of a first user U1, and outputs the captured image P(t) to the recognition unit 2.

[0021] The recognition unit 2 recognizes the first user U1 who is the user captured in the captured image P(t), calculates parameters A(t) for controlling the avatar (i.e., parameters A(t) required for drawing the avatar) as being associated with the current time t of the captured image P(t), and outputs the parameters A(t) to the measurement unit 5 and the drawing unit 7. This output is performed by transmitting the parameters A(t) from the first terminal 10 to the second terminal 20 via the network NW, and is subject to a delay.

[0022] An example of the control parameter A(t) may be a blend shape parameter that expresses the user's facial expression by weighting a plurality of mesh models. In another embodiment, a parameter that expresses the coordinates of each joint of the user may be used. Any type of parameter that enables avatar drawing may be used, and any existing method may be used as a method for extracting the parameter by recognition processing of the captured image P(t).

[0023] The recording unit 3 is composed of a microphone as hardware, records the speech of the first user U1, acquires voice data S(t) at the current time t, outputs it to the analysis unit 4, and also outputs it to the measurement unit 5 and presentation unit 8 of the second terminal 20 by transmitting it as a voice packet via the network NW.

[0024] The analysis unit 4, which replaces the recognition unit 2 in another embodiment, calculates the avatar control parameter A(t). For example, the analysis unit 4 uses the technique of the following non-patent document 1 to analyze the speech pattern (facial expression, etc.) of the first user U1 through waveform analysis of the voice S(t) to obtain the parameter A(t). The parameter A(t) is then transmitted to the second terminal 20 via the network NW, and output to the measurement unit 5 and the prediction unit 6. [Non-patent document 1] TIAN, Guanzhong; YUAN, Yi; LIU, Yong. Audio2face: Generating speech / face animation from single audio with attention-based bidirectional lstm networks. In: 2019 IEEE international conference on Multimedia & Expo Workshops (ICMEW). IEEE, 2019. p. 366-371.

[0025] The parameter A(t) for drawing the avatar may be acquired by any method other than the image analysis and audio analysis methods described above, for example, a motion capture sensor, etc. A combination of multiple methods may also be used.

[0026] The measurement unit 5, under the time synchronization between the terminals as the pre-processing described above, refers to the time td recorded as a timestamp in the latest avatar control parameter A(td) among those that have been received up to the current time t, measures the difference d between the current time t and the latest received timestamp td as the actual value of the delay d required for communication between the terminals and each process, and outputs it to the prediction unit 6.

[0027] Alternatively, the measurement unit 5 may measure the time required for the prediction unit 6 and the rendering unit 7 to complete processing after the avatar control parameter A(td) is obtained at each current time t, and estimate the required time Δd by estimating this time Δd and estimating "d+Δd" as the delay. (Note that this required time Δd can have any length in real time independent of the interval of the current time t=1, 2, ....) Note that if the relationship "Δd<<d" always holds, where the required time Δd is sufficiently small relative to the measured delay value d, based on the actual results of the communication band of the network NW and the calculation specifications of the second terminal 20, the estimation of the required time Δd may be omitted and the measured delay value d may be used. In the following explanation, the delay value obtained by the measurement unit 5 is assumed to be d.

[0028] The prediction unit 6 performs the following operations in order: (1) referencing the history, (2) calculating the derivative of the history, (3) predicting the derivative value at the current time t using a deep learning network, and (4) calculating the predicted value A'(t) of the avatar control parameter A(t) at the current time t. The prediction unit 6 then outputs the calculated parameter predicted value A'(t) to the rendering unit 7.

[0029] (1) Refer to the following parameter history H(td) of a certain length up to the latest avatar control parameter A(td) obtained from the first terminal 10 with a delay d up to the current time t. H(td)={A(te), A(t-e+1),…,A(td-1), A(td)} where e>d, e-d+1 is the length (number of data points) of this parameter history H(td).

[0030] (2) Calculate the time derivative history H1(td) from the parameter history H(td) as follows: (D(x) is the time derivative of the control parameter A(x) at the same time x (x=te, ..., td).) H1(td)={D(te), D(t-e+1),…,D(td-1), D(td)}

[0031] (3) The parameter history H(td) and its derivative H1(td) as well as the delay value d obtained by the measurement unit 5 are input to a pre-trained deep learning network N as follows, and a predicted value D'(t) of the time derivative D(t) of the avatar control parameter A(t) at the current time t is output. D'(t)=N(H(td), H1(td), d)

[0032] (4) Using the latest avatar control parameter A(td), delay d, and the predicted differential value D'(t) at the current time t, obtain the predicted value A'(t) of the avatar control parameter A(t) at the current time t as follows: A'(t)=A(td)+d*D'(t)

[0033] The above method of the prediction unit 6 is shown in schematic form with reference to Figures 4 and 5. Figure 4 shows a schematic example of the manner in which delay d varies as a premise, and in the same way that the latest parameter obtained at the current time t is A(td) and involves delay d, the latest parameters obtained at past times t1, t2, t3, ... (where t>t1>t2>t3>...) are A(t1-d1), A(t2-d2), A(t3-d3), ..., etc., and these delays d1, d2, d3, ... generally exhibit varying behavior.

[0034] As in the example of Figure 4, the latest parameters sequentially received by the second terminal 20 do not have constant time information such as the above-mentioned delays d1, d2, and d3 due to differences in imaging timing and variations in the load of recognition processing, so the prediction unit 6 of this embodiment calculates the derivative values ​​of the parameters and infers future derivative values ​​using the network N that has been trained including the derivative values.

[0035] When the prediction unit 6 calculates the derivative in the above (3), the difference between consecutive parameters may be obtained by dividing it by the difference in time using the forward difference or backward difference method. For example, in the case of backward difference, if the parameters A(t - n) and A(t - m) (n < m) at times t - n and t - m are used, the derivative D(t - n) at time t - n may be obtained by the following equation. Note that other methods such as central difference may also be used, but in the case of backward difference, the derivative can be calculated for the latest received parameter A(t - d) (even if the next parameter A(t - d + 1) has not been received). D(t - n) = {A(t - m) - A(t - n)} / (m - n)

[0036] FIG. 5 shows a schematic graph in which the prediction unit 6 estimates the predicted value D'(t) of the derivative D(t) by a network to obtain the predicted value A'(t) of the parameter A(t) from the above derivative and the like.

[0037] The rendering unit 7 takes the predicted control parameter A'(t) obtained by the prediction unit 6 as an input, and in accordance with the predicted parameter A'(t), renders the avatar A1 of the first user U1 to obtain the rendering result G(t) at the current time t, and outputs it to the presentation unit 8. The CG model for rendering the avatar A1 may be prepared in advance and deformed into an expression and / or posture according to the parameter A'(t) and then rendered by an existing method. When rendering the avatar A1 in a three-dimensional space for applications such as augmented reality display, after giving the information on the position and orientation of the virtual camera viewpoint, it may be rendered by an existing three-dimensional computer graphics method. It is expected that the predicted parameter A'(t) predicted according to the parameter A(t - d) with the delay d received by the second terminal 20 approximates the parameter A(t) actually acquired at the first terminal 10 at the current time t, leading to concealing the transmission delay and processing delay and achieving the effect of realizing smooth communication.

[0038] The presentation unit 8 receives the drawing result G(t) as an input and displays the drawing result G(t) to the second user U2. The drawing result G(t) (the drawing result G'(t) of the predicted parameter A'(t)) is displayed in real time for each time t, thereby displaying an avatar image as a predicted result of the behavior of the first user U1 at the current time t. The presentation unit 8 can be a display or projector mounted on the terminal. The presentation unit 8 can also play back the voice S(t) at the current time t obtained from the recording unit 3 through a speaker, thereby conveying the spoken content of the first user U1 to the second user U2.

[0039] As described above, according to the embodiments of the present invention, in drawing an avatar, the influence of transmission delays and the like can be suppressed by predicting the movement of a user to be drawn based on future differential information inferred by a prediction method including differential information of past parameters and drawing the avatar. Various supplementary examples, additional examples, alternative examples, etc. related to the embodiments of the present invention will be described below.

[0040] (1) According to an embodiment of the present invention, it is possible to realize realistic remote communication by drawing avatars. This makes it possible to hold remote conferences without necessarily requiring users to actually travel to remote locations, and it is possible to reduce carbon dioxide emissions by saving the energy resources required for user travel, which makes it possible to contribute to Goal 13 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "take urgent action to combat climate change and its impacts."

[0041] (2) FIG. 6 shows an example of the configuration of a deep learning network N(k) of the prediction unit 6 according to an embodiment. This network N(k) predicts N+2 time t n ,t n-1 ,…,t n-N-1 The kth element parameter history w over k (t n ),w k (t n-1 ),…,w k (t n-N-1 ) is input, and m-times into the future, t n+mParameter w k (t n+m ) predicted value w' k (t n+m ), and this figure shows an example of the details of each of the processes (1) to (4) described above. It should be noted that there are a total of K control parameters for the avatar, and the network N(k) in FIG. 6 predicts the kth parameter (k=1, 2, ..., K). (That is, for each element of the avatar's control parameters, the structure of network N(k) is the same, but the parameters to be learned are different.)

[0042] In the deep learning network N(k), the differential calculation process described above is performed on the upstream side, and then the history and its differential are input to the first LSTM (long-short-term memory) network L1. The output of this first LSTM network L1 is further input to the second LTSM network L2. The output of this second LSTM network L2 is further input to the fully connected layer (Dense) network DS. The output of this fully connected layer network DS is used as the prediction time t n+m Parameter differential prediction value Δw' at k (t n+m ) is obtained.

[0043] In addition, the differential calculation process is performed using the input history w k (t n ),w k (t n-1 ),…,w k (t n-N-1 ) for the i-th adjacent time pair W(i) = {w k (t n-i ),w k (t n-i-1 )}(i=0,1,...,N), the difference process S(i) and the division process D(i) by the adjacent time difference are calculated as follows, and the input history w k (t n ),w k (t n-1 ),… ,w k (t n-N+1 ),w k (t n-N) and its derivative Δw k (t n ),Δw k (t n-1 ),… ,Δw k (t n-N+1 ),Δw k (t n-N ) is input to the first LSTM. Δw k (t n-i )={w k (t n-i )-w k (t n-i-1 )} / (t n-i -t n-i-1 )

[0044] The output of the fully connected network DS is Δw' k (t n+m ), the delay amount is "t n+m -t n ” and the latest time t n Parameter w k (t n ) and the addition process AD to obtain the predicted value w' k (t n+m ) can be output as follows. (Note that this corresponds to the above-mentioned "A'(t) = A(td) + d * D'(t)" expressed in time notation as shown in Figure 6 for the k-th component of the parameter, and "t = t n+m ", td=t n " so that the time notations correspond to each other.) w' k (t n+m )=w k (t n )+(t n+m -t n )*Δw' k (t n+m )

[0045] (3) The method for learning the parameters of the deep learning network constituting the prediction unit 6, an example of which is shown in FIG. 6, may be as follows.

[0046] That is, a neural network can be trained by preparing a large number of parameter time series {A(t)|t=1,2,...} for avatar control as training data, deriving a differential time series {D(t)|t=1,2,...} from the past part of the time series, and deriving the differential of the future part from the parameter time series including the differential time series. That is, a delay d is given for training, and similarly to inference, the differential prediction value D'(t) and prediction value A'(t) at future time t are obtained from the parameter time series up to time td and its derivative in the network in the middle of training (Fig. 6), and the deviation from the correct answers D(t) and A(t) is used as a cost to update the network parameters (parameters of the first LTSM network L1, the second LTSM network L2, and the fully connected network DS) until convergence is determined.

[0047] At this time, it is desirable to weight the samples used for learning with consideration of their visual importance. For example, in the case of using blend shape parameters, the degree to which the eyes and mouth are open or closed has a large visual impact, so by using variable weights that increase the weight for samples whose absolute value of the derivative of the parameter is equal to or greater than a certain value, the effect of improving the prediction accuracy of the movement can be obtained. For example, the loss function L during learning can be set as the mean squared logarithmic error as shown in the following equation, and learning is performed so that A'(t) and D'(t) that minimize the loss function L can be output.

[0048]

number

[0049] Here, A k (t) and D k (t) is the k-th element of the parameter time series and its derivative in the training dataset, respectively, A k '(t) and D k'(t) is the predicted value for each (the output of the network during training using the same process as inference), K is the number of parameters (the number of elements in the blend shape, etc.), and w k represents the weight for each sample (each element of the parameter), and λ is a preset constant.

[0050] That is, when the variable weight is used, for example, the weight w k (Weights w are determined for each time t to be predicted and parameter type k = 1, 2, ..., K. k ) for the correct absolute value of the differential of the type k |D k The weights may be determined according to whether |(t)| is large or not, as follows: TH>0 is the threshold value for judgment. w k =Wa if |D k (t)|>TH w k =Wb if |D k (t)|≦TH Wa>Wb>0

[0051] For the loss function L, the (non-differentiated) parameter A k (t) Term L A and the differentiated parameter D k (t) Term L D However, evaluation is based on only one of them (L = L A Or L=L D By using both, more accurate learning can be achieved. A and L D Although we evaluate the sum of a total of K parameters for N(k), it is also possible to evaluate it for a single k (k = 1, 2, ..., K) and train only the corresponding network N(k) individually.

[0052] (4) In the prediction unit 6, the network in Fig. 6 predicts only up to the first derivative as "A'(t) = A(td) + d * D'(t)", but since the purpose is to accurately estimate the change behavior during the delay d, it is also possible to predict using the Taylor expansion formula with up to the second or higher derivative as input. For example, it is also possible to predict as follows using up to the third derivative as the network input. A'(t)=A(td)+d*D'(t)+d 2 *D''(t) / 2+d 3 *D''(t) / 6

[0053] (5) The delay d evaluated by measurement by the measurement unit 6 may be evaluated by defining an appropriate predetermined processing timing (or processing completion timing) as the current time t according to the implementation method of the prediction system 100 and the past results of the computational resources and network delays, obtaining the latest possible timestamp td of the parameter A(td) reflected in the result of the current time t, and evaluating it as the difference d between them. If the computational resources, etc. may vary and the processing time may vary, a variable delay d may be estimated in anticipation of such variations.

[0054] For example, the current time t may be the time when drawing is performed (drawing is completed) by drawing unit 7, or the time when the drawing result is displayed by presentation unit 8.

[0055] For example, the delay d in the description of the configuration in FIG. 3 above is a delay based on the assumption that the delay in transmitting the avatar control parameter A(t) from the first terminal 10 to the second terminal 20 is the largest bottleneck element compared to various other processing delays and transmission delays, as explained in FIG. 2 etc., and that the drawing by the drawing unit 7, the display on the presentation unit 8, and audio-related transmission and reception, etc. are completed in an instantaneous time that is negligible compared to this delay d, and are therefore completed almost simultaneously at the current time t, enabling the audio S(t) and the drawing result G(t) to be synchronized.

[0056] If this premise is not taken into account, the current time t managed by a clock may be used simply without considering synchronization with the voice S(t) in the configuration of Fig. 3. Also, if the voice S(t) involves a non-negligible delay, the current time t may be defined to reflect this delay.

[0057] Also, for example, as a modified example of the implementation of the prediction system 100 shown in Fig. 3, it is possible to adopt the configuration shown in Fig. 7. In the prediction system 100 in Fig. 7, the second terminal 20 has a so-called thin client configuration and includes only the measurement unit 5 and the presentation unit 8, while the other units are included in the first terminal 10, so that it is possible to display the avatar drawing result on the second terminal 20 by the same process as in Fig. 3.

[0058] In the configuration of Figure 7, the delay d in the measurement unit 5 reflects the time required for the drawing process and the transmission process of the drawing result, assuming that after the parameter A(td) is obtained, the drawing process by the drawing unit 7 and the drawing result G(t) are transmitted and displayed as the current time t on the second terminal 20.

[0059] (6) FIG. 8 is a diagram showing an example of a hardware configuration of a general computer device 70. Each of the first terminal 10 and the second terminal 20 (and the prediction device 9 including at least the prediction unit 6 in the prediction system 100) constituting the prediction system 100 can be realized as one or more computer devices 70 having such a configuration. When each terminal or device is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, a keyboard, a touch panel, or the like, a camera 78, a microphone 79, and a speaker 80, and a bus BS for transmitting and receiving data between them.

[0060] Each functional unit of the prediction system 100 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when a display-related process is performed, a display 76 also operates in conjunction with the GPU 72, and when a communication-related process related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the GPU 72. The avatar display function in the presentation unit 8 may be realized by the display 76, and the imaging unit 1 may be realized as a camera 78. The speech of the first user U1 when performing avatar communication may be recorded by a microphone 79 and the speech may be output to the second user U2 on the other side via a speaker 80. [Explanation of symbols]

[0061] 100... prediction system, 10... first terminal, 20... second terminal, 9... prediction device 1...imaging unit, 2...recognition unit, 3...recording unit, 4...analysis unit, 5...measurement unit, 6...prediction unit, 7...drawing unit, 8...presentation unit

Claims

[Claim 1] A learning method for learning parameters of a deep learning network used by a prediction unit included in a prediction device that receives avatar drawing parameters acquired at each time for a user for whom an avatar is to be drawn, and predicts the parameters at a current time, the method comprising: The prediction device comprises: a measurement unit that acquires a delay between an acquisition time associated with the latest parameter that has been received and the current time; a prediction unit that predicts a parameter at the current time based on a history of received parameters up to the latest parameter, a time change of the history, and the delay; The prediction unit predicts parameters at the current time by using a deep learning network; The prediction unit obtains a predicted value of a time change of a parameter at the current time from a history of the received parameters and a time change of the history by using the deep learning network, the prediction unit calculates an amount of change in the parameter that occurs during the delay from the predicted value of the time change and the delay, and predicts the parameter at the current time by adding the amount of change to the latest parameter that has been received; In the learning method, As the correct answer data for learning, a parameter history, a time derivative of the history, and a delay are given in advance, The parameter history, time differential, and delay are input to a deep learning network in the middle of learning, and the parameter value and its time differential at one time in the future that is the delay away from the history are output, and learning is performed by setting the deviation from the correct answer data as a cost; A learning method characterized in that, in evaluating the cost, the cost of a parameter value and its time derivative at a time in the future that is the delay away from the history is calculated by weighting it more heavily when it is determined that the absolute value of the correct value of the time derivative is greater than a judgment threshold value than when this judgment is not made.

Images