Apparatus, method and program for estimating dependent variable

Abstract

According to one embodiment of the present invention, in a learning mode, a plurality of pieces of learning multivariate time-series data that are respectively generated from a plurality of different sources and correlated with a target variable are acquired. A plurality of pieces of paired data that can be objectively compared are acquired, the plurality of pieces of paired data being produced on the basis of the acquired plurality of pieces of learning multivariate time-series data. Rank learning is carried out using the acquired plurality of pieces of paired data as teacher labels, and a prescribed target function is optimized in the time-series direction, whereby model parameters are learned. In an operation mode, unknown multivariate time-series data is acquired and inputted to a learning model, and a target variable corresponding to the unknown multivariate time-series data is estimated by using the learned model parameters in the learning model.

Description

[Technical field]

[0001] One aspect of the present invention relates to a dependent variable estimation device, method, and program that uses a machine learning model to estimate, as a dependent variable, an element that is difficult to absolutely evaluate, such as human emotions or feelings, from multivariate time-series data. [Background technology]

[0002] There are cases where we want to estimate a target variable that is difficult to evaluate absolutely using multivariate time series data. For example, this would be the case when we want to estimate a person's feelings and emotions while watching a movie from biometric data. Estimation of feelings and emotions can be defined as a learning problem for a model that inputs multivariate time series data, i.e. biometric data, as learning data and outputs target variables that are difficult to evaluate absolutely, such as excitement and emotion.

[0003] In general, training a model in machine learning requires teacher labels as training data. When estimating a target variable that is difficult to evaluate absolutely, subjective evaluation labels obtained from questionnaire responses from subjects are often used as teacher labels.

[0004] However, there are two issues to be considered when estimating objective variables that are difficult to evaluate absolutely using subjective evaluation labels. The first issue is that subjective evaluations are generally unreliable, making it difficult to compare with others. For example, when evaluating the degree of excitement on a multi-level scale from 1 to 9 points, it is expected that there will be people (hereinafter referred to as A) who evaluate a non-excited state as 5 points, and people (hereinafter referred to as B) who evaluate it as 1 point. In this case, it is not appropriate to treat the 5 points judged by A and the 5 points judged by B as being on the same level. In addition, since these objective variables are difficult to evaluate absolutely, it is not appropriate to judge that B was 5 times more excited than A based on the evaluation scores of A's 1 point and B's 5 points. However, in terms of machine learning evaluation values, 5 points and 5 points are equivalent, and 1 point and 5 points are 5 times the evaluation value. Although a difference of 4 points as in this example is extreme, a difference of 1 or 2 points may occur frequently.

[0005] The second issue is that teacher labels become sparse. When conducting subjective evaluation, the timing of evaluation is often a time when viewing is not disturbed, such as after viewing of content. In this case, the number of teacher labels is one for each sample, which is assigned after viewing. In other words, even though the measured biometric data is acquired densely as time-series data, the teacher labels become sparse in the time series. This makes it impossible to obtain teacher labels during content viewing. This makes it difficult to estimate objective variables that change finely, such as transitions in excitement level.

[0006] Meanwhile, conventional methods for estimating the degree of comfort / discomfort and excitement / calmness of a subject while watching video or music from biological data include the methods introduced in, for example, Non-Patent Document 1 or Non-Patent Document 2. These methods estimate a target variable by extracting features from electrocardiogram (ECG), electroencephalography (EEG), or the like, and inputting them into a machine learning model. However, all of these methods use the results of subjective evaluation as the teacher labels used in the machine learning, and the two problems mentioned above remain unsolved.

[0007] In response to this, there is a method of estimating a target variable using rank learning of paired data as a method of absorbing the variation in evaluation points in subjective evaluation. For example, Non-Patent Document 3 or Non-Patent Document 4 describes a method of estimating the pleasant / unpleasant and excited / relaxed level of a subject listening to a speech using rank learning, in which audio features are input to a rank learning machine called RankSVM, and paired data obtained from the subject's subjectivity is used as teacher labels to estimate pleasant / unpleasant levels, etc.

[0008] These methods alleviate the first problem mentioned above by treating subjective evaluations relatively as a comparison of paired data, rather than as absolute values. However, the fundamental point remains the same: they use subjective evaluation labels. Therefore, the problem of different evaluation criteria for each subject and the second problem mentioned above, that teacher labels become sparse over time, remain unsolved. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu. “Identifying stable patterns over time for emotion recognition from eeg.” IEEE Transactions on Affective Computing, Vol.10, No.3, pp. 417-429, 2019. [Non-Patent Document 2] Mimma Nardelli, Gaetano Valenza, Alberto Greco, Antonio Lanata, and EnzoPasquale Scilingo. “Recognizing emotions induced by affective sounds through heart rate variability.” IEEE Transactions on Affective Computing, Vol.6, No.4, pp. 385-394, 2015. [Non-Patent Document 3] Srinivas Parthasarathy, Reza Lotfian, and Carlos Busso. “Ranking emotional attributes with deep neural networks.” In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4995-4999, 2017. [Non-Patent Document 4] Jianyu Fan, K.Tatar, Miles Thorogood, and P.Pasquier. “Ranking-based emotion recognition for experimental music.” In ISMIR, 2017. Summary of the Invention [Problem to be solved by the invention]

[0010] As described above, all conventional methods use subjective evaluation labels to estimate objective variables, which are difficult to evaluate absolutely. This has the problem that the evaluation criteria differ for each subject, and that the objective variables are sparse over time.

[0011] The present invention has been made in light of the above-mentioned circumstances, and aims to provide a technique that further reduces the variation in the evaluation score criteria and enables precise estimation of the dependent variable even when the teacher label is coarse. [Means for solving the problem]

[0012] In order to solve the above problem, one aspect of a dependent variable estimation device or method according to the present invention estimates a dependent variable using a learning model that receives multivariate time series data as input and outputs a dependent variable that is difficult to evaluate absolutely, by acquiring, in a learning mode, a plurality of learning multivariate time series data that are generated from a plurality of different sources and are correlated with the dependent variable, and acquiring a plurality of paired data that can be objectively compared and are generated based on the acquired plurality of learning multivariate time series data. Then, rank learning is performed using the acquired paired data as teacher labels to optimize a predetermined objective function in the time series direction, thereby learning parameters of the learning model. Then, in an operation mode, when unknown multivariate time series data is input, the input unknown multivariate time series data is converted into the learning data. Model Enter this learning Model The learned parameters are used to estimate a response variable corresponding to the unknown multivariate time series data. Furthermore, in one aspect of the objective variable estimation device or method according to the present invention, the learning model is defined by a first function that obtains a current latent variable from past latent variables and the current multivariate time-series data, and a second function that estimates the current objective variable from the current latent variable, and the objective function is expressed by an arbitrary function that minimizes an output value when the order relationship of the pair data is maintained, and a function that slows down the change in the output value in the time series direction and limits the value of the parameter of the learning model.

[0013] According to one aspect of the present invention, the objective variable is estimated using trained model data that has been rank trained using multiple paired data that can be objectively compared as teacher labels, making it possible to reduce the variation in the evaluation point standard that occurs when estimation is performed using model data trained with subjective evaluation labels. In addition, when learning the model parameters, a process is added to optimize the objective function in the time series direction. Therefore, even if the teacher labels are coarse in the time axis direction, the objective variable can be output densely, making it possible to obtain an objective variable with high continuity in the time axis direction. Effect of the Invention

[0014] That is, according to one aspect of the present invention, it is possible to provide a technology that further reduces the variation in the evaluation score criteria and enables a detailed estimation of the dependent variable even if the teacher label is sparse. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a system including a server device having the functions of a response variable estimating device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a hardware configuration of the server device shown in FIG. [Diagram 3] FIG. 3 is a block diagram showing a software configuration of the server device shown in FIG. [Figure 4] FIG. 4 is a flowchart showing the process and contents of the learning mode executed by the server device shown in FIG. [Diagram 5] FIG. 5 is a flowchart showing the procedure and contents of a process in the operation mode executed by the server device shown in FIG. [Figure 6] FIG. 6 is a diagram illustrating an example of multivariate time series data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] [One embodiment] (Configuration example) (1) System FIG. 1 is a diagram showing an example of the overall configuration of a system including a server device SV having the functions of a response variable estimating device according to an embodiment of the present invention.

[0018] In one embodiment, the system targets an arbitrary user among a plurality of users US1 to USn who are watching content such as a movie or music. The users US1 to USn each have a biosensor SC1 to SCn. The biosensors SC1 to SCn are capable of data communication with a server device SV via a network NW. Note that MT indicates a management terminal such as a personal computer used by a system administrator or the like.

[0019] The biosensors SC1 to SCn are, for example, wearable terminals, and measure biodata such as heartbeats of the users US1 to USn. The biodata is multivariate time-series data that changes over time, and the biosensors SC1 to SCn transmit the multivariate time-series data to the server device SV from a communication interface unit incorporating the data. The biosensors SC1 to SCn may be dedicated devices having only a function of measuring biodata, and in this case, the biodata measured by the biosensors SC1 to SCn is transmitted to the server device SV via a mobile information terminal such as a smartphone. In addition to the heartbeat, the biodata measured may be blood pressure, respiration, sweating, tears, and other biodata that reflect human emotions.

[0020] Examples of the network NW that may be used include, but are not limited to, wireless networks such as Bluetooth (registered trademark) and WiFi (registered trademark), wired LAN (Local Area Network), public wireless networks adopting standards such as 4G or 5G, and public optical communication networks.

[0021] (2) Server device SV 2 and 3 are block diagrams showing the hardware and software configurations of the server device SV, respectively.

[0022] The server device SV is composed of a server computer arranged on the Web or the cloud, for example, and includes a control unit 1 using a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, a communication interface (hereinafter, the interface will be referred to as I / F) unit 4, and an input / output I / F unit 5 are connected to the control unit 1 via a bus 6.

[0023] The server device SV may be, for example, one accommodated in a local network of a company or organization, and a general-purpose personal computer may be used instead of the server computer.

[0024] The communication I / F unit 4 transmits and receives data to and from each of the biosensors SC1 to SCn using a communication protocol defined by the network NW under the control of the control unit 1. If the users US1 to USn have mobile terminals such as smartphones, the communication I / F unit 4 may receive the biodata measured by the biosensors SC1 to SCn via the mobile terminals.

[0025] The management terminal MT is connected to the input / output I / F unit 5. The management terminal MT is, for example, a general-purpose personal computer, and has an input device 7 and a display device 8. The input / output I / F 5 receives input data from the input device 7, and outputs display data generated by the control unit 1 to the display device 8 for display.

[0026] The program storage unit 2 is configured, for example, by combining a non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) as a storage medium that can be written to and read at any time, and a non-volatile memory such as a ROM (Read Only Memory), and stores various programs necessary for executing various control processes according to one embodiment of the present invention, in addition to middleware such as an OS (Operating System).

[0027] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an HDD or SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium, and is provided with a multivariate time series data storage unit 31, a pair data storage unit 32, a setting parameter storage unit 33, a model parameter storage unit 34, and a target variable storage unit 35 as storage areas necessary for implementing one embodiment of the present invention.

[0028] The multivariate time series data storage unit 31 is used to store multivariate time series data (biometric data in this example) acquired from the biometric sensors SC1 to SCn in association with identification information (hereinafter referred to as user ID) of the transmitting biometric sensors SC1 to SCn or users US1 to USn.

[0029] The paired data storage unit 32 is used to store a plurality of paired data that can be objectively compared and that are extracted from the plurality of multivariate time series data stored in the multivariate time series data storage unit 31. The paired data includes a pair of data sampled at two different times of any one multivariate time series data, and a pair of data sampled at any time of any two different multivariate time series data.

[0030] The setting parameter storage unit 33 is used to store setting parameters of the learning model inputted by the system administrator on the management terminal MT, for example, in the learning mode.

[0031] The model parameter storage unit 34 is used to store the results of learning by the control unit 1 for the above-mentioned setting parameters as learned model parameters in the learning mode.

[0032] In the operation mode, the objective variable memory unit 35 is used to store objective variables estimated by the control unit 1 using the learned model parameters in the learning model for unknown multivariate time series data transmitted from the biosensors SC1 to SCn.

[0033] The control unit 1 includes, as processing functions according to an embodiment of the present invention, a multivariate time series data acquisition processing unit 11, a paired data acquisition processing unit 12, a setting parameter acquisition processing unit 13, a model parameter learning processing unit 14, a response variable estimation processing unit 15, and a response variable output processing unit 16. Each of these processing units 11 to 16 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2.

[0034] The multivariate time series data acquisition processing unit 11 receives the multivariate time series data transmitted from the biosensors SC1 to SCn via the communication I / F unit 4, and performs a process of storing each received multivariate time series data in the multivariate time series data storage unit 31 in association with the user ID of the sender.

[0035] The paired data acquisition processing unit 12 extracts a plurality of pairs of data at any different time from each of the plurality of multivariate time series data stored in the multivariate time series data storage unit 31, and extracts a plurality of pairs of data at any time for a combination of any two time series data from the plurality of multivariate time series data. That is, the paired data acquisition processing unit 12 extracts a plurality of paired data that can be objectively compared. Then, the paired data acquisition processing unit 12 performs a process of storing the extracted plurality of paired data that can be objectively compared in the paired data storage unit 32 in association with the time data on the multivariate time series data.

[0036] In the learning mode, the setting parameter acquisition processing unit 13 imports the setting parameters of the learning model entered by the system administrator via the management terminal MT via the input / output I / F unit 5, and performs a process of storing the imported setting parameters in the setting parameter memory unit 33.

[0037] The model parameter learning processing unit 14 selectively reads a plurality of paired data that can be objectively compared from the paired data storage unit 32, and performs rank learning on the setting parameters using the read paired data as teacher labels. In the rank learning, a process is performed to optimize the objective function in the time series direction. Then, the model parameter learning processing unit 14 stores the results obtained by the learning process in the model parameter storage unit 34 as learned model parameters. An example of the model parameter learning process will be described in the operation example.

[0038] In the operation mode, when unknown multivariate time series data transmitted from the biosensors SC1 to SCn of the users US1 to USn to be observed is received by the multivariate time series data acquisition processing unit 11, the dependent variable estimation processing unit 15 inputs the received unknown multivariate time series data into a learning model. Then, in the learning model, the dependent variable estimation processing unit 15 estimates a dependent variable corresponding to the unknown multivariate time series data using the learned model parameters stored in the model parameter storage unit 34, and stores the estimated dependent variable in the dependent variable storage unit 35 in association with a user ID.

[0039] The objective variable output processing unit 16 reads out the estimated objective variables from the objective variable storage unit 35, and generates display data for the read objective variables. Then, the generated display data is output to the management terminal MT via the input / output I / F unit 5, and a process of displaying the display data on the display device 8 of the management terminal MT is performed.

[0040] (Example of operation) Next, an example of the operation of the server device SV configured as above will be described. 4 and 5 are flow charts showing an example of the procedure and contents of the process executed by the control unit 1 of the server device SV.

[0041] (1) Processing operations in learning mode When the learning mode is set, the control unit 1 of the server device SV executes a learning process for the model parameters to be set in the learning model according to the processing procedure shown in FIG.

[0042] (1-1) Acquisition of multivariate time series data In the standby state, the control unit 1 of the server device SV monitors data input in step S11. In this state, suppose that biometric data is transmitted from the biometric sensors SC1 to SCn of the users US1 to USn to be observed, respectively, while the users are watching a movie or music content.

[0043] Then, under the control of the multivariate time series data acquisition processing unit 11, the control unit 1 of the server device SV receives each of the above biometric data via the communication I / F unit 4 in step S12, and stores each of the above received biometric data in the multivariate time series data storage unit 31 as multivariate time series data for learning.

[0044] The learning multivariate time series data is not limited to the data transmitted in real time from the biosensors SC1 to SCn of the observation target as described above, but may be, for example, biodata collected in the past of a plurality of observation target users during viewing and stored in an external database server or the like, acquired from the database server. The learning multivariate time series data may be created by simulating the biodata.

[0045] (1-2) Acquiring pair data When the multivariate time series data for a predetermined unit time is accumulated in the multivariate time series data storage unit 31, the control unit 1 of the server device SV then reads any one piece of multivariate time series data from the multivariate time series data storage unit 31 under the control of the paired data acquisition processing unit 12 in step S12. Then, data at any two different times from the read multivariate time series data is extracted, and the extracted data pair is stored in the paired data storage unit 32. At the same time, the paired data acquisition processing unit 12 reads any two pieces of multivariate time series data from the multivariate time series data storage unit 31. Then, data at any time from the read two multivariate time series data is extracted, and the extracted data pair is stored in the paired data storage unit 32.

[0046] That is, the paired data acquisition processing unit 12 extracts a plurality of paired data that can be objectively compared from an arbitrary plurality of multivariate time-series data, and stores the paired data in the paired data storage unit 32 .

[0047] In addition, in the case where a plurality of paired data for objective comparison created based on a plurality of past learning multivariate time series data has already been prepared in the external device as teacher labels, the server device SV may acquire the plurality of paired data for teacher labels from the external device and store them in the paired data storage unit 32.

[0048] (1-3) Acquiring configuration parameters The system administrator inputs the setting parameters of the learning model in the management terminal MT. In response to this, the control unit 1 of the server device SV, under the control of the setting parameter acquisition processing unit 13, imports the setting parameters input in the management terminal MT via the input / output I / F unit 5 in step S13, and stores the imported setting parameters in the setting parameter storage unit 33.

[0049] (1-4) Learning model parameters When the acquisition of the setting parameters is completed, the control unit 1 of the server device SV, in step S14, selectively reads a plurality of paired data that can be objectively compared from the paired data storage unit 32 under the control of the model parameter learning processing unit 14. Then, rank learning is performed on the setting parameters using the read paired data as teacher labels.

[0050] Moreover, the model parameter learning processing unit 14 performs a process of optimizing the objective function in the time series direction in the rank learning. Then, the model parameter learning processing unit 14 stores the results obtained by the learning process in the model parameter storage unit 34 as learned model parameters.

[0051] (2) Detailed operation in learning mode Next, the series of processes in the learning mode described above will be explained in more detail.

[0052] (2-1) Input data Observable multivariate time series data x t = {x1, . . . , x T}, and let X be a set of M people collected at a given sampling interval T. X = {x m t | m = 1, . . . , M} Figure 6 shows an example of the change in the feature quantity x of this multivariate time series data along the time axis.

[0053] In addition, let D, which shows the relationship between pair data in rank learning, be D = {v k > u k | k = 1, . . . , K} However, v k ∈ X, u k ∈ X and v k > u k u k From v k is preferred.

[0054] When defining the paired data set D, the number of combinations for extracting any paired data from the multivariate time series data set X is very large, 1 / 2MT(MT-1), and it is not realistic to cover all paired data. Therefore, in this example, we assume that the paired data set D is a sparse teacher label extracted from a portion of the multivariate time series data set X, that is, K ≪ 1 / 2MT(MT-1).

[0055] (2-2) Input model Any model that outputs continuous estimates in the time series direction can be used as the input model. For example, for a given time t, the latent variable z m t-1 and the multivariate time series data x at the current time t m t Using this, the latent variable z at the current time t m t The function g that obtains the latent variable z at the current time t m t to the objective variable y m t The model f consists of the following: f = g ○ h z m t = g(z m t-1 , x m t ) y m t = h(z m t ) + ε y Assume the following.

[0056] Any function can be used for the function g. For example, using a state space model, g (z m t-1 ,x m t ) = z m t-1 + α m t xm t +β Here, α m t = α m t-1 + η α,t β m t = β m t-1 + η β,t η α,t ~ N(0, σ 2 α,t ) η β,t ~ N(0, σ 2 β,t ) It is.

[0057] Also, like perceptrons g (z m t-1 , x m t ) = w z z m t-1 + w x x m t +b In addition, deep learning techniques such as RNN (Recurrent Neural Network) architectures such as LSTM (Long Short Term Memory) and multivariate time series data such as image sequences can be used. g :R H×W×C → R d It is also anticipated that a Convolutional Neural Network (CNN) such as the following will be used.

[0058] On the other hand, the function h can be a function in which the objective variable y follows a certain probability distribution, or an arbitrary function that serves to restrict the value range of the objective variable y.

[0059] (2-3) Output The output of the above model f is a value correlated with the objective variable y according to the relationship of the paired data set D. Note that due to the nature of rank learning, this output value does not emphasize the accuracy of the score, but rather emphasizes maintaining the order relationship of the samples.

[0060] (2-4) Objective function The parameters of the above model are estimated by optimizing the objective function based on rank learning using a set of paired data D. However, because the set of paired data D is sparse, the estimated values ​​may only fit locally and may not be continuous along the time axis. Therefore, in order to make the change along the time axis gentler, a regularization term Ω that limits the parameter θ of the model f and its product with the weight λ are added to the objective function. Taking this into account, the objective function L(θ) is L(θ) = Φ(d k ) + λΩ(θ) d k = f(v k ) - f(u k ) It is expressed as follows.

[0061] In this objective function L(θ), any function that minimizes the output while preserving the order of paired data can be used for Φ. For example, similar to RankNet, we can use a sigmoid function and an entropy loss function to Φ(d k ) = -log(e dk / 1+e dk ) It may be defined as:

[0062] For more information on RankNet, please refer to the following literature: Christopher Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory Hullender. “Learning to rank using gradient descent”. pp. 89-96, 01 2005. is described in detail in.

[0063] Also, using a Gaussian distribution,

number

number

[0064] For more information on the Gaussian distribution, see the following literature: "Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. pp. 137-144, 08 2005. is described in detail in.

[0065] On the other hand, in the objective function L(θ), the regularization term Ω can also be set arbitrarily. For example,

number

[0066] (2-5) Optimization of the objective function Any optimization method, such as the gradient method, can be applied to optimize the objective function L(θ). When using the gradient method, the kth optimization step is θ k+1 ← θ k -γ k ∇ θ L(θ) We can simply repeat the process of updating the parameter θ according to the following equation: k denotes the learning rate parameter. The gradient of the objective function L(θ) ∇ θ L(θ) may be a function derived by calculation, or may be a function calculated numerically.

[0067] In addition, σ in the above objective function L(θ) 2 ε ,σ 2 α,t ,σ 2 β,t denotes the variance of the noise, and these are the weights λ of the regularization term Ω, the learning rate parameter γ k Together with the above, it is preset as a configuration parameter.

[0068] (3) Processing in operational mode When the learning of the model parameters in the above learning mode is completed and the operation mode to be applied to actual operation is set, the control unit 1 of the server device SV executes a process of estimating a target variable indicating the emotions, feelings, etc. of a user US1 to USn who is actually watching a movie or music content from the biometric data of the user, according to the processing procedure shown in Figure 5.

[0069] (3-1) Acquiring unknown multivariate time series data In the standby state, the control unit 1 of the server device SV monitors input of data in step S21. In this state, when biometric data is transmitted from the biometric sensors SC1 to SCn of the users US1 to USn who are enjoying video or music content, the control unit 1 of the server device SV receives each of the biometric data via the communication I / F unit 4 in step S22 under the control of the multivariate time series data acquisition processing unit 11. Then, the control unit 1 stores each of the received biometric data, that is, unknown multivariate time series data, in the multivariate time series data storage unit 31 in association with the user ID of the transmission source.

[0070] (3-2) Estimation of the objective variable When the unknown multivariate time series data is acquired, the control unit 1 of the server device SV, under the control of the objective variable estimation processing unit 15, first reads the unknown multivariate time series data from the multivariate time series data storage unit 31 and inputs it to the learning model in step S23. Next, the objective variable estimation processing unit 15 estimates an objective variable corresponding to the unknown multivariate time series data in the learning model using the learned model parameters stored in the model parameter storage unit 34. Then, the objective variable estimation processing unit 15 stores the estimated objective variable in the objective variable storage unit 35 in association with the user ID.

[0071] At this time, as described above in the learning mode, the trained model parameters are trained by using paired data that allows objective comparison obtained from the multivariate time series data for training as teacher labels, and further optimizing the objective function in the time series direction to perform rank training. Therefore, the objective variables output from the learning model have a unified evaluation point standard by using objective evaluation labels, and have values ​​that have continuity in the time series direction.

[0072] Thereafter, the objective variable estimation processing unit 15 similarly repeatedly executes the process of estimating objective variables for the biometric data of users US1 to USn acquired during observation, for example at regular time intervals, and storing the results in the objective variable storage unit 35.

[0073] (3-3) Output of estimation results The control unit 1 of the server device SV monitors the input of an output request of the estimation result in step S24 while executing the estimation process of the above-mentioned objective variable. In this state, for example, it is assumed that a content distributor inputs an output request of the estimation result in the input device 7 of the management terminal MT in order to analyze the user's reaction to the distributed content.

[0074] Then, in step S25, the control unit 1 of the server device SV, under the control of the objective variable output processing unit 16, reads out the stored objective variables from the objective variable storage unit 35, and generates display data for the read objective variables. Then, the generated display data is output to the management terminal MT via the input / output I / F unit 5, and is displayed on the display device 8.

[0075] As a result, for example, for each user viewing content, the display device 8 of the management terminal MT displays the objective estimation results of the emotions or feelings of the user continuously at regular time intervals, associated with the user ID of the target user or the corresponding user name.

[0076] The estimation result can also be displayed on the requesting terminal by, for example, a user sending an output request from his / her own mobile terminal such as a smartphone to the server device SV.

[0077] (Action and effect) As described above, in one embodiment, in the learning mode, a plurality of learning multivariate time series data are acquired, a plurality of paired data that can be objectively compared are extracted from the multivariate time series data, and the extracted paired data are used as teacher labels to perform rank learning and optimize the objective function in the time series direction, thereby learning the setting parameters of the learning model and generating learned model parameters.Then, in the operation mode, when biometric data of a user during viewing is acquired, the biometric data is input to the learning model as unknown multivariate time series data, and in this learning model, the learned model parameters are used to estimate objective variables representing the user's feelings and emotions corresponding to the biometric data, and display data representing the estimated results is output.

[0078] Therefore, since the objective variable is estimated using trained model data that has been rank-trained using paired data that allows for objective comparison as teacher labels, it is possible to reduce the variation in the evaluation point standard that occurs when estimation is performed using model data trained with subjective evaluation labels. In addition, when learning the model parameters, a process is added to optimize the objective function in the time series direction. Therefore, even if the teacher labels are coarse in the time axis direction, the objective variable can be output densely, and it is possible to obtain an estimation result of the objective variable that has high continuity in the time axis direction.

[0079] [Other embodiments] (1) In the above embodiment, the observation target is a user who is watching a movie or listening to music, and the user's feelings and emotions are estimated based on the user's biometric data. However, the present invention is not limited to this, and can also be applied to a case where the observation target is an animal or machine other than a human, and the animal's feelings and emotions or a malfunction and signs of the machine are estimated based on the measurement data of the behavior or operation of the animal.

[0080] (2) In the above embodiment, the gradient method is used as an optimization method for the objective function, but any other method such as a stochastic gradient method or Adam can be used. Similarly, any time series data estimation model or regularization term for the objective function can be used.

[0081] (3) In the above embodiment, the server device SV is provided with the functions of the objective variable estimation device according to the present invention. However, the present invention is not limited to this. For example, the functions of the objective variable estimation device according to the present invention may be provided in the management terminal MT or a mobile terminal owned by a user, or the functions of the objective variable estimation device according to the present invention may be distributed among the server device and the terminal. Furthermore, the functions of the objective variable estimation device according to the present invention may be downloaded from a higher-level system such as a cloud to a local server such as an edge server when necessary, and installed therein for use.

[0082] (4) In addition, the functional configuration, processing procedure and processing content, type and configuration of the learning model, applications, etc. of the dependent variable estimation device can be modified and implemented in various ways without departing from the spirit and scope of the present invention.

[0083] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted.

[0084] In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0085] SV: Server device US1~USn…User SC1~SCn: Biometric sensors NW…Network TM: Management terminal 1...Control unit 2...Program memory section 3. Data storage section 4. Communication I / F section 5...Input / output interface 6…Bus 11...Multivariate time series data acquisition processing section 12...Pair data acquisition processing unit 13...Setting parameter acquisition processing unit 14...Model parameter learning processing unit 15...Objective variable estimation processing unit 16...Objective variable output processing unit 31...Multivariate time series data storage unit 32...Pair data storage section 33...Setting parameter storage section 34…Model parameter storage section 35…Objective variable storage section

Claims

1. A response variable estimation device using a learning model that receives multivariate time series data as input and outputs a response variable that is difficult to evaluate absolutely, comprising: a time series data acquisition processing unit that acquires a plurality of learning multivariate time series data generated from a plurality of different generation sources and correlated with the objective variable; a paired data acquisition processing unit that acquires a plurality of paired data that can be objectively compared and is generated based on the acquired plurality of learning multivariate time-series data; a parameter learning processing unit that sets parameters for the learning model, and performs rank learning using the acquired plurality of pair data as teacher labels to optimize a predetermined objective function in a time series direction, thereby learning the parameters; a response variable estimation processing unit that, when unknown multivariate time series data is input, inputs the input unknown multivariate time series data into the learning model and estimates the response variable corresponding to the unknown multivariate time series data by using the parameters that have been learned by the learning model; Equipped with the learning model is defined by a first function that obtains a current latent variable from a past latent variable and the current multivariate time series data, and a second function that estimates the current objective variable from the current latent variable; The objective function is composed of an arbitrary function that minimizes an output value when the order relationship of the pair data is maintained, and a function that slows down the change in the time series direction of the output value and limits the value of the parameter of the learning model. Objective variable estimator.

2. 2. The response variable estimation device according to claim 1, wherein the optimization process of the objective function uses one of a gradient method, a stochastic gradient method, and Adam as an optimization algorithm.

3. A method for estimating a response variable executed by an estimation device using a learning model that receives multivariate time series data as input and outputs a response variable that is difficult to evaluate absolutely, comprising the steps of: acquiring a plurality of pieces of multivariate time series data for learning, the data being generated from a plurality of different sources and correlated with the objective variable; A step of acquiring a plurality of paired data that can be objectively compared and is generated based on the acquired plurality of learning multivariate time-series data; A step of setting parameters for the learning model, performing rank learning using the acquired plurality of pair data as teacher labels, and optimizing a predetermined objective function in a time series direction, thereby learning the parameters; a step of inputting unknown multivariate time series data into the learning model when the unknown multivariate time series data is input, and estimating the objective variable corresponding to the unknown multivariate time series data using the parameters already learned by the learning model; Equipped with the learning model is defined by a first function that obtains a current latent variable from a past latent variable and the current multivariate time series data, and a second function that estimates the current objective variable from the current latent variable; The objective function is composed of an arbitrary function that minimizes an output value when the order relationship of the pair data is maintained, and a function that slows down the change in the time series direction of the output value and limits the value of the parameter of the learning model. Method of estimating the objective variable.

4. 3. A program for causing a processor included in the response variable estimation device to execute processing of each of the processing units included in the response variable estimation device according to claim 1 or 2.

Images