Prediction device and program

Abstract

To provide a prediction device for parameters for avatar control acquired at each clock time, which can reduce the influence of delays under the presence of delays, for example, as what is suitable for remote communication.SOLUTION: A prediction device receives parameters for avatar control obtained for a user at each clock time and predicts the parameters at a future clock time. In order to predict parameters of a second clock time located more ahead than a first clock time as the future clock time from received parameters up to the first clock time, the prediction device divides the interval from the first clock time to the second clock time into two or more intervals, performs prediction for each interval, and combines (third procedure) results of prediction for each interval (first procedure and second procedure).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a device and a program for predicting parameters for controlling an avatar. [Background technology]

[0002] In a system that estimates a user's movements and applies them to an avatar, if transmission delays can be reduced, smooth communication will be possible. Examples of prior art that achieves this include those disclosed in Patent Documents 1 and 2, which disclose the following methods.

[0003] Patent Document 1 discloses a method for estimating the future movements of two users in VR (virtual reality) and fast-forwarding or delaying the video to synchronize the timing at which the avatars' gazes meet when they look at each other. Patent Document 2 discloses a method for reducing transmission delays by transmitting reference posture identification data that can identify information about the posture of the viewing user with a small amount of data, rather than posture feature data that requires a large amount of data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-009295 [Patent Document 2] Patent No. 6731532 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-described conventional techniques do not necessarily contribute to the realization of sufficiently smooth remote communication, etc., in the presence of transmission delays and / or delays associated with the completion of other processes.

[0006] In Patent Document 1, the gazes of the avatars are aligned, but the user whose video is fast-forwarded does not yet see the other user's avatar, and conversely, the user whose video is delayed is forced to wait for their movements to be reflected, resulting in the problem of inability to communicate smoothly. Furthermore, the amount of delay in transmission and reception fluctuates depending on the network conditions, making it difficult to predict the timing for alignment. Patent Document 2 attempts to suppress transmission delays by reducing the amount of transmission, but the time required for posture recognition and transmission inevitably exists, resulting in a problem of delays corresponding to these times.

[0007] In view of the problems with the conventional technology described above, an object of the present invention is to provide a device and a program for predicting parameters for controlling an avatar acquired at each time, which are suitable for, for example, remote communication and can reduce the effect of delay when a delay exists. [Means for solving the problem]

[0008] To achieve the above object, the present invention provides a prediction device that receives avatar control parameters acquired for a user at each time and predicts the parameters at a future time, and in order to predict parameters at a second time that is further in the future than the first time from the received parameters up to a first time, the device divides the period from the first time to the second time into two or more sections and combines the results of predictions for each section.The present invention also provides a program that causes a computer to function as the prediction device. [Effects of the Invention]

[0009] According to the present invention, by dividing the period from the first time to the second time, which is the subject of parameter prediction, into two or more periods and then combining the results of predictions made for each period, it is possible to obtain predicted results for avatar control parameters that reduce the effect of delays even when there is a delay. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram of a prediction system according to an embodiment. [Figure 2] FIG. 1 is a schematic diagram of remote communication realized by a prediction system according to an embodiment. [Figure 3] FIG. 1 is a schematic diagram of predicted delay. [Figure 4] FIG. 2 is a functional block diagram illustrating a first terminal and a second terminal. [Figure 5] FIG. 2 is a schematic diagram of a first embodiment of a prediction unit. [Figure 6] FIG. 10 is a schematic diagram of a second embodiment of the prediction unit. [Figure 7] FIG. 10 is a schematic diagram of a fourth embodiment of the prediction unit. [Figure 8] 1 is a configuration example of a deep learning network for the prediction unit. [Figure 9] FIG. 1 is a diagram illustrating an example of a hardware configuration of a general computer. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 is a configuration diagram of a prediction system 100 according to one embodiment. The prediction system 100 includes a first terminal 10 used by a first user U1 and a second terminal 20 used by a second user U2, and the first terminal 10 and the second terminal 20 are capable of communicating with each other via a network NW such as the Internet. Both the first terminal 10 and the second terminal 20 can be configured as general computer terminals such as smartphones, tablets, desktop or laptop computers, etc.

[0012] FIG. 2 is a schematic diagram of remote communication realized by a prediction system 100 according to an embodiment. At the current time t, the first user U1 makes a greeting voice to the user U2 who is the partner of the remote communication. The voice packet is transmitted to the side of the second user U2 at high speed, and the greeting voice is reproduced in real time for the second user U2 at a time that can be regarded as approximately the current time t. On the other hand, for the first user U1, in order to draw the avatar A1 on the side of the second user U2, drawing parameters reflecting expressions and the like are acquired, transmitted to the side of the second user U2, and by performing avatar drawing, a real-time display of the avatar A1 is performed for the second user U2.

[0013] Here, the transmission of the drawing parameters and the avatar drawing are accompanied by a delay and require more time than the voice transmission and reproduction. Therefore, when reproducing the voice of the first user at the current time t on the side of the second user U2, only the drawing parameters up to the past time t - d (where d > 0 and t - d < t) are obtained. That is, there is a delay of magnitude d, and moreover, this delay generally varies with time, becoming d = d(t). According to the prediction system 100 of this embodiment, by predicting the drawing parameters of the first user U1 at the current time t using the drawing parameters that have a delay and are only obtained up to the past time t - d, and performing drawing, the avatar A1 can be drawn in real time in synchronization with the voice of the first user U1, and smooth remote communication can be realized.

[0014] Here, for the delay d(t), in addition to the influence of the transmission delay of the above-described drawing parameters, there is also a processing delay in the prediction process itself when the prediction system 100 predicts the drawing parameters at the current time (the current time as the future seen from the past time) using the latest received drawing parameters with a delay. Therefore, this processing delay also needs to be considered.

[0015] 3 (a diagram schematically illustrating a prediction delay), considering the time when the prediction process is completed using the drawing parameters up to the first past time t-n1, even while the prediction process is in progress, drawing parameters for times t-n1+1, t-n1+2, ..., later than the past time t-n1 are continuously received, so that at the time when the prediction process is completed, additional drawing parameters up to the second past time t-n2 (n1>n2>0), which is further in the future than the first past time t-n1, have also been received. Therefore, depending on the situation, it is thought that a more appropriate drawing result can be obtained by performing drawing taking into account the drawing parameters up to the second past time t-n2 that have been obtained up to the time when the prediction process is completed, rather than simply using the drawing parameters predicted from the drawing parameters up to the first past time t-n1.

[0016] According to the prediction system 100 of the third embodiment (to be described later), as described above, it is possible to perform adaptive prediction that takes into account not only the results of simple prediction using only the drawing parameters up to the first past time t-n1, but also the drawing parameters up to the second past time t-n2 that have been additionally received at the time the prediction process is completed, thereby achieving more appropriate prediction than when only simple prediction is used. This will be described in more detail later.

[0017] 2 only shows a case in which facial expression parameters are acquired for a first user U1, and the avatar A1 is drawn and displayed for a second user U2 on the other side. However, by switching roles, it is possible to predictively draw and display a second avatar A2 (not shown) of the second user U2 in real time for the first user U1, while preventing the effects of delay. This allows the first user U1 and the second user U2 to engage in two-way remote communication in real time via their own first avatar A1 and second avatar A2. By adding additional terminal configurations, two-way remote communication between three or more people is also possible.

[0018] However, since similar processing is possible by switching roles as described above, the following will explain the unidirectional processing when realizing two-way remote communication in the case of two people (the same applies to unidirectional processing in the case of three or more people), where the first user U1 using the first terminal 10 is on the "self" side and is the target whose drawing parameters are acquired and the avatar A1 is drawn, and the second user U2 using the second terminal 20 is on the "other" side and is the target whose drawn avatar A1 is displayed remotely.

[0019] Fig. 4 is a functional block diagram showing the first terminal 10 and the second terminal 20, which are components of the prediction system 100. The terminals 10 and 20 each include an imaging unit 11, a recognition unit 12, a prediction unit 21, a drawing unit 22, and a presentation unit 23. In Fig. 4, the first terminal 10 includes the imaging unit 11 and the recognition unit 12, and the second terminal includes the prediction unit 21, the drawing unit 22, and the presentation unit 23, but the combination can be set arbitrarily depending on the intended use, etc.

[0020] The second terminal 20 includes at least the prediction unit 21 and constitutes the prediction device 20 according to each embodiment of the present invention, but will be referred to as the second terminal 20 in the following description.

[0021] In the processing procedure, first, an image is captured by the imaging unit 11 to obtain a captured image. The imaging unit 11 outputs the captured image to the recognition unit 12 as imaging information (a captured image linked to the time of capture). This image capture time represents the time of capture as a timestamp. This image capture time is also used as a timestamp in the control parameters described below. (Note that the clocks (timekeeping functions) managed by the first terminal 10 and the second terminal 20 are set in advance to a common clock so that the timestamp can be used appropriately.)

[0022] The recognition unit 12 recognizes the user captured in the image information, calculates parameters for controlling the avatar, and outputs the recognized control parameters to the prediction unit 21. The recognition unit 12 uses the recognized control parameters as recognition information (the control parameters and the image capture times of the captured images used to calculate the control parameters).

[0023] An example of the control parameters is a blend shape parameter that represents the user's facial expression by weighting multiple mesh models. In another embodiment, a parameter that represents the coordinates of each joint of the user can be used. In other words, the control parameters are extracted from the imaging information, such as the facial expression and / or pose of the user (first user U1 using the first terminal 10) at the time of imaging, and are used as drawing parameters in the subsequent drawing unit 22 for drawing an avatar that reflects the facial expression and / or pose at the time of imaging. (Note that, as will be described later, in practice, the control parameters (recognition information) obtained as actual measured values by the recognition unit 12 are not directly used, but the drawing unit 22 performs drawing using the control parameters predicted by the prediction unit 21 using these actual measured values.)

[0024] The prediction unit 21 predicts future recognition information from the transition of past recognition information. Any method such as deep learning can be used as the prediction method. As an example of a prediction method, a recurrent neural network can be used, which learns fluctuations in control parameters in advance. Generally, the longer the future time span to be predicted, the more difficult the prediction becomes, so to improve prediction accuracy, it is desirable to predict a shorter future time span. Furthermore, rather than predicting x seconds into the future, it is expected that the overall prediction accuracy can be improved by learning recognition information at multiple points in time shorter than x seconds in the future so as to minimize errors using individual training data.

[0025] First Embodiment Therefore, the prediction unit 21 according to the first embodiment decomposes and predicts the future that is originally desired to be predicted. For example, by predicting and combining recognition information and a change amount for a future that is shorter than the future that is originally desired to be predicted, the recognition information of the future that is originally desired to be predicted is predicted. Specifically, as shown in FIG. 5 (schematic diagram of the first embodiment), at time t (here, time t is not the current time, but the latest time among the actually measured recognition information that can be referred to in the prediction process), recognition information R' at time t+x (here, the time that is originally desired to be predicted is time t+x) that is x seconds ahead is calculated. t+x When predicting xm seconds ahead, the recognition information R' t+x-m and the change D' between x seconds ahead and xm seconds ahead t+x-m,t+x After predicting the recognition information R' t+x is calculated using the following formula (0 <m<x) R' t+x = R' t+x-m + D' t+x-m,t+x …(1)

[0026] That is, in the first embodiment, the recognition information R' is generated by the following first to third steps shown in FIG. t+x can be obtained. (Step 1) History R of actual recognition information up to time t k From (k=t, t-1, t-2, ...), the predicted value of the subsequent recognition information at time k=t+1, t+2, ..., t+xm is R' k Find (k=t+1,t+2,…,t+xm). (Step 2) History R k (k=t,t-1,t-2,…) and predicted value R' k (k=t+1, t+2, ..., t+xm) (here, only some of the newer time points may be used instead of all of these history and predicted values) and the latest predicted value of the recognition information xm seconds ahead, R' t+x-m The change in the recognition information D' from the time t to the time t+x that you want to predict t+x-m,t+x Predict. (Third step) Recognition information R' at time t+x that is originally to be predicted using equation (1) t+x Ask for.

[0027] Thus, in the first embodiment, instead of directly predicting the future x seconds ahead which is the future to be originally predicted, a short future x - m seconds ahead before that is predicted, and by applying the change amount prediction between the short future and the future to be originally predicted, the effect of improving the prediction accuracy can be obtained. At this time, if the parameter m, which is the parameter for setting how much short future before the future to be originally predicted, is set large, the future time of the recognition information to be predicted becomes short, but the future time of the change amount to be predicted becomes long. If m is set small, the opposite relationship holds. Therefore, when the prediction accuracy in the prediction of the recognition information and the change amount is equal, it is desirable to set the prediction time to be the same (m = x / 2). When the prediction accuracies of both are different, the one with higher prediction accuracy can be predicted for a longer time and the one with lower prediction accuracy can be predicted for a shorter time. Since the level of prediction accuracy and robustness can be judged during each learning, the parameter m can be set in advance as a relative value with respect to x (a value within the range of 0 < m < x) according to the judgment.

[0028] <Second Embodiment> Alternatively, as described above with reference to FIG. 3, since the processing of the prediction unit 21 takes a certain amount of time, taking advantage of the fact that recognition information continues to be input while the prediction is being made, in the prediction unit 21 according to the second embodiment, the recognition information of the short future predicted in the first embodiment is substituted with the latest recognition information input to the second terminal 20. Specifically, if the prediction processing time is n, by setting x - m (the time width for future prediction) in the description of the first embodiment to x - m = n, the change amount D' t+n between the latest recognition information R t+n,t+x and the future x seconds ahead is predicted, and then the recognition information R' t+x can also be obtained by the following formula (2). R' t+x = R t+n + D' t+n,t+x …(2)

[0029] The effect of reducing the prediction error can be obtained by limiting the prediction target to the change amount. FIG. 6 is a schematic diagram of the second embodiment. The recognition information R tis received by the second terminal 20, and time n has passed since the prediction process at the second terminal 20. Recognition information continues to be received at times t+1, t+2, ..., t+n-1, t+n during the prediction process, and at the time the prediction process is completed, the recognition information R t+n is received by the second terminal 20. In the prediction process, the amount of change D' between the future time t+x that is originally predicted and the time t+n at which the prediction process is expected to be completed is calculated. t+n,t+x The recognition information R' at time t+x is predicted. t+x According to equation (2), R t+n + D' t+n,t+x It is predicted as:

[0030] Regarding the prediction processing in the first and second embodiments, it is considered that the rate-limiting step that accounts for the main processing load is the processing using a deep learning network or the like, and the addition processing of equations (1), (2), etc. is not considered to be the rate-limiting step. In the first embodiment, there are two processes using a deep learning network or the like, the first and second procedures described above, while in the second embodiment, the change amount D' t+n,t+x Since only one deep learning network or the like is required to predict the above, the second embodiment is expected to be faster than the first embodiment.

[0031] <Third embodiment> Alternatively, in the prediction unit 21 according to the third embodiment, which corresponds to the weighted combination method of the first and second embodiments, the error (corresponding to an evaluation of prediction accuracy) between the recognition information predicted for the short future and the actually measured recognition information may be biased and reflected in the predicted value. Specifically, the recognition information R' for x m seconds ahead may be t+x-m When predicting the actual value R t+n The error is calculated using the weighting factors w1 and w2 using the following equation (3). R' t+x = R t+n +w1*R t +w2*R' t+x-m +D' t+x-m,t+x …(3)

[0032] For example, recognition information R' t+x-m The actual measured value R t+nWhen linearly dividing the time t+n, the error (R t+n -((R' t+x-m -R t )*n / (xm)+R t )) occurs, so at least this error must be reduced to R' t+x-m When this is reflected in the equation (3), the final result is the above equation. In this case, the weighting coefficients are w2=n / (xm), w1=-(1-w2). Note that weights w1 and w2 (which can be negative numbers) can be used according to the method of error evaluation (linear / nonlinear or interpolation / extrapolation), such as a linear method without internal division or a method that reflects the error in external division. In other words, the term "R" in equation (3) t+n +w1*R t +w2*R' t+x-m " is the actual measured value R t+n and predicted value R' t+x-m From this, the predicted value R' is calculated using a preset error model (in the linear case, this error model is expressed by weights w1 and w2). t+x-m The true value at time t+xm (actual measurement value R t+x-m ) to estimate the predicted value R' t+x Regarding the true value (actual measurement value R t+x ) (the predicted value R' is approximated by applying a certain bias) t+x (reflected in R t ,R t+n ,R' t+x-m ) can be used.

[0033] Here, the actual measured value R t+n In the formula (3), n is R' t+x-m and D' t+x-m,t+x This is the time required to calculate the actual measurement value R t+n To obtain the predicted value R', external division (extrapolation) is performed from t+x-m When evaluating the error of (predicted value R' t+x-mWhen performing internal division in the past direction from this point to evaluate the error), and when the measured value R is obtained because x - m < n t+n Interpolate (extrapolate) in the direction of the past time from this point, (or, since x - m < n, using the measured value R that has already been received t+x-m as it is to), evaluate the error of the predicted value R' t+x-m When evaluating the error of (the predicted value R' t+x-m when performing extrapolation in the future direction from this point to evaluate the error), either of these is possible.)

[0034] <Fourth Embodiment> In the prediction unit 21 according to the fourth embodiment, which uses another method of decomposing the future that is originally desired to be predicted, the future that is originally desired to be predicted is decomposed into one or more intervals, and the amount of change in each interval is predicted and synthesized to predict the recognition information of the future that is originally desired to be predicted. Specifically, as also shown in FIG. 7 (schematic diagram of the fourth embodiment), when predicting the recognition information R' at time t + x, x seconds ahead at time t (the time t of the latest measured recognition information referred to in the prediction process), t+x when predicting, (First interval) the amount of change D' between time t and m1 seconds ahead t,t+m1 and, (Second interval) the amount of change D' between m1 seconds ahead and m2 seconds ahead t+m1,t+m2 and, (Third interval) the amount of change D' between m2 seconds ahead and x seconds ahead t+m2,t+x and, after predicting, the recognition information R' t+x is obtained by the following formula (4). (0 < m1 < m2 < x) R' t+x =R t +D' t,t+m1 +D' t+m1,t+m2 +D' t+m2,t+x …(4)

[0035] In the fourth embodiment, by predicting the shorter future of m1 seconds ahead and m2 seconds ahead rather than directly predicting x seconds ahead, an effect of improving the prediction accuracy can be obtained. In the above description, an example is given for three intervals divided by m1 and m2, but more intervals or any number of intervals of one or more can also be used.

[0036] In the fourth embodiment, processing using a deep learning network or the like is performed for the number of divided intervals for calculating the amount of change. By combining the first embodiment with the fourth embodiment, the amount of change calculated in the second procedure in the first embodiment may be calculated by dividing it into two or more intervals.

[0037] Each embodiment of the prediction unit 21 has been described above, so we will return to the description of each unit in FIG.

[0038] The drawing unit 22 receives the predicted information as input and draws an avatar according to the predicted information. Each embodiment of the prediction unit 21 can obtain predicted information predicted according to the recognition information received by the second terminal 20 as information that approximates the recognition information at the first terminal 10 at the current time, thereby concealing transmission delays and processing delays and achieving smooth communication.

[0039] The presentation unit 23 receives drawing information as input and presents the drawing information to the user. The presentation unit 23 can be a display or projector mounted on the terminal.

[0040] As described above, according to each embodiment of the present invention, by predicting future movements based on future recognition information and change amounts inferred from past recognition information, it is possible to conceal transmission delays and processing delays and realize smooth communication. Various supplementary examples, alternative examples, additional examples, etc. will be described below.

[0041] <1> According to an embodiment of the present invention, realistic remote communication can be realized by drawing avatars. This enables remote conferences and other events to be held without necessarily requiring users to travel to remote locations, and reduces carbon dioxide emissions by saving the energy resources required for user travel. This can contribute to Goal 13 of the United Nations' Sustainable Development Goals (SDGs), which states, "Take urgent action to combat climate change and its impacts."

[0042] <2> 8 shows an example of the configuration of a deep learning network N(k) for carrying out the first step of the first embodiment in the prediction unit 21 according to one embodiment. This network N(k) is a network that predicts N+2 times t n ,t n-1 ,…,t n-N-1 The kth element parameter history w over k (t n ),w k (t n-1 ),…,w k (t n-N-1 ) is input, and m future time t n+m The parameter w k (t n+m ) predicted value w' k (t n+m ), and this shows an example of the details of each of the processes (1) to (4) below. It should be noted that there are a total of K control parameters for the avatar, and the network N(k) in Fig. 8 is responsible for making predictions about the kth (k=1, 2, ..., K) parameter. (That is, for each element of the avatar's control parameters, the structure of network N(k) is the same, but the parameters to be learned are different.)

[0043] (1) Refer to the following parameter history H(td) of a certain length up to the latest avatar control parameter A(td) obtained from the first terminal 10 with a delay d up to the current time t. H(td)={A(te), A(t-e+1),…,A(td-1), A(td)} where e>d, e-d+1 is the length (number of data points) of this parameter history H(td).

[0044] (2) The time derivative history H1(td) is calculated from the parameter history H(td) as follows: (D(x) is the time derivative of the control parameter A(x) at the same time x (x=te, ..., td).) H1(td)={D(te), D(t-e+1),…,D(td-1), D(td)}

[0045] (3) By inputting the parameter history H(td), its derivative H1(td), and the delay value d into a pre-trained deep learning network N as follows, the predicted value D'(t) of the time derivative D(t) of the avatar control parameter A(t) at the current time t is output. D'(t)=N(H(td), H1(td), d)

[0046] (4) Using the latest avatar control parameter A(td), delay d, and the predicted differential value D'(t) at the current time t, the predicted value A'(t) of the avatar control parameter A(t) at the current time t is obtained as follows: A'(t)=A(td)+d*D'(t)

[0047] In the deep learning network N(k), after performing the differential calculation process described above in the previous stage, the history and its differential are input to the first LSTM (long-short-term memory) network L1, the output of this first LSTM network L1 is further input to the second LSTM network L2, the output of this second LSTM network L2 is further input to the fully connected layer (Dense) network DS, and the output of this fully connected layer network DS is used as the prediction time t n+m Parameter differential prediction value Δw' at k (t n+m ) is obtained.

[0048] The differential calculation process is performed using the input history w k (t n ),w k (t n-1 ),…,w k (t n-N-1 ) for the i-th adjacent time pair W(i) = {w k (t n-i ),w k (t n-i-1 )}(i=0,1,...,N), the input history w is calculated as follows by the difference process S(i) and the division process D(i) by the adjacent time difference. k (t n ),w k (t n-1 ),… ,w k(t n-N+1 ),w k (t n-N ) and its derivative Δw k (t n ),Δw k (t n-1 ),… ,Δw k (t n-N+1 ),Δw k (t n-N ) is input to the first LSTM. Δw k (t n-i )={w k (t n-i )-w k (t n-i-1 )} / (t n-i -t n-i-1 )

[0049] Δw' is the output of the fully connected network DS k (t n+m ) for the delay amount "t n+m -t n ” and the latest time t n The parameter w k (t n ) and add the predicted value w' k (t n+m ) can be output as follows. (Note that this corresponds to the above-mentioned "A'(t) = A(td) + d * D'(t)" expressed in time notation as shown in Figure 8 for the k-th component of the parameter, and "t = t n+m ", "td=t n " so that the time notations correspond to each other.) w' k (t n+m )=w k (t n )+(t n+m -t n )*Δw' k (t n+m )

[0050] In the above description, in the overall configuration of FIG. 8, the prediction unit 21 calculates, for example, the recognition information R′ in the formula (1) of the first embodiment. t+x As a result, the change D' in the same equation (1) can be calculated. t+x-m,t+x The structure for predicting Δw' is the output of the fully connected (Dense) network DS. k (t n+m ) can be used.

[0051] That is, the prediction unit 21 calculates the recognition information R' t+x When obtaining predicted values such as the above, the entire configuration of FIG. 8 (hereinafter referred to as the "prediction value network") can be used, and the prediction unit 21 calculates the change amount D' t+x-m,t+x To obtain a variation such as

[0000] , it is sufficient to use a configuration (referred to as a "variation network") in which the final multiplication process ML and addition process AD are removed from the overall configuration of Fig. 8. The predicted value network and the variation network thus have most of the same network structure, but they may be configured with different parameters by performing learning separately.

[0052] <3> The first terminal 10 acquires the avatar control parameters A(t) of the user whose avatar is to be drawn at each time t. Instead of the acquisition method using image analysis by the imaging unit 11 and the recognition unit 12 as shown in FIG. 4, the parameters A(t) may be acquired by audio analysis by the recording unit 13 and the analysis unit 14 (not shown).

[0053] The recording unit 13 is configured as hardware such as a microphone, and records the speech of the first user U1, who is a user, to obtain voice data S(t) at the current time t, and outputs the data to the analysis unit .

[0054] The analysis unit 14, which replaces the recognition unit 2 in another embodiment, calculates the avatar control parameters A(t), and for example, acquires the parameters A(t) by analyzing the speech pattern (facial expression, etc.) of the first user U1 through waveform analysis of the voice S(t) using the technique of the following non-patent document 1, and outputs the parameters A(t) to the prediction unit 21 by transmitting the parameters A(t) to the second terminal 20 via the network NW. [Non-Patent Document 1] TIAN, Guanzhong; YUAN, Yi; LIU, Yong. Audio2face: Generating speech / face animation from single audio with attention-based bidirectional lstm networks. In: 2019 IEEE international conference on Multimedia & Expo Workshops (ICMEW). IEEE, 2019. p. 366-371.

[0055] Note that the parameter A(t) for avatar drawing may be acquired by any method other than the image analysis and audio analysis methods described above, such as by using a motion capture sensor, or by combining multiple methods.

[0056] <4> FIG. 9 is a diagram showing an example of the hardware configuration of a general computer device 70. Each of the first terminal 10 and the second terminal 20 (and the prediction device 20 including at least the prediction unit 21 in the prediction system 100) constituting the prediction system 100 can be realized as one or more computer devices 70 having such a configuration. When each terminal or device is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the CPU 71's execution instructions in place of or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 78, a microphone 79, and a speaker 80, and a bus BS for transmitting and receiving data among them.

[0057] Each functional unit of the prediction system 100 can be realized by a CPU 71 and / or a GPU 72 that loads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are types of computing devices (processors). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and the GPU 72. The avatar display function of the presentation unit 23 may be realized by the display 76, the imaging unit 11 may be realized as a camera 78, and the recording unit 13 may be realized as a microphone 79. When avatar communication is performed, the speech of the first user U1 may be recorded by the microphone 79 and output to the second user U2 on the other side via a speaker 80. [Explanation of symbols]

[0058] 100...prediction system, 10...first terminal, 20...second terminal (prediction device) 11...imaging unit, 12...recognition unit, 21...prediction unit, 22...drawing unit, 23...presentation unit

Claims

1. A prediction device receives parameters for controlling an avatar, the parameters being acquired by actually measuring the parameters at each time point, and predicts the parameters corresponding to the actually measured values at a future time point, the prediction device comprising: In order to predict parameters of a second time that is located in the future than the first time from the parameters up to the first time that have already been received, as the future time, the section from the first time to the second time is divided into two or more sections, and the results of predictions made for each section are combined; Dividing the period from the first time to the second time into a first half period and a second half period, predicting parameters for all times up to the end time of the first half section using the parameters up to the first time point that have already been received; predicting a parameter change amount in the latter half interval using the received parameters up to the first time point and the parameters for all times predicted in the former half interval; the parameter change amount is reflected in the parameter predicted for the last time, and A prediction device characterized by setting the length of the first half section and the length of the second half section in advance so that the higher the prediction accuracy, the longer the lengths, and the lower the prediction accuracy, based on results of a pre-evaluation of the prediction accuracy by deep learning for the first half section and the prediction accuracy by deep learning for the second half section.

2. A prediction device receives parameters for controlling an avatar, the parameters being acquired by actually measuring the parameters at each time point, and predicts the parameters corresponding to the actually measured values at a future time point, the prediction device comprising: In order to predict parameters of a second time that is located in the future than the first time from the parameters up to the first time that have already been received, as the future time, the section from the first time to the second time is divided into two or more sections, and the results of predictions made for each section are combined; Dividing the period from the first time to the second time into a first half period and a second half period, For the first half section, predicting parameters at the final time of the first half section; For the latter half section, a parameter change amount in the latter half section is predicted; the parameter change amount is reflected in the parameter predicted for the last time, and evaluating the prediction accuracy of the parameter at the final time from the parameters already received at a third time between the first time and the second time, the third time being a time when prediction of the parameter at the final time of the first half interval and prediction of the parameter change amount in the second half interval are completed; The predicted value of the parameter at the last time (R' t+x-m ) and the parameter (R t+n ) and parameters received before the third time (R t ,) and the predicted value of the parameter at the last time (R' t+x-m ) to evaluate the prediction accuracy by evaluating the error of The parameter predicted for the last time (R' t+x-m ), the parameter change amount (D' t+x-m,t+x ) and the evaluated error are reflected by the following equation (3) using weighting coefficients w1 and w2, and the predicted value of the parameter at the future time (R' t+x- ) is obtained. R' t+x- = R t+n +w1* R t +w2*R' t+x-m + D' t+x-m,t+x …(3)

3. The prediction device according to claim 1 , wherein the latter half section is divided into two or more sections, and the parameter change amount is predicted for each section.

4. The prediction device according to claim 2 , wherein the latter half section is divided into two or more sections, and the parameter change amount is predicted for each section.

5. A program that causes a computer to function as the prediction device according to any one of claims 1 to 4.

Images