Biometric data processing device and biometric data processing method

Abstract

To enable reports about intervention measures and data collection at appropriate timing such as when the user is in a subjective state different from usual.SOLUTION: A physiological measurement data processing device includes: a reception unit 51 for receiving, from a user terminal of a user, physiological measurement data of a physiological state of the user detected by a sensor; a deviation assessment unit 54 for assessing a deviation of physiological measurement data received by the reception unit with respect to past physiological measurement data of the user; a subjective score assessment 53 for estimating a subjective score indicating a body and mind subjective state of the user on the basis of the physiological measurement data received by the reception unit; an uncertainty evaluation unit 55 for assessing an uncertainty of the subjective score estimated by the subjective score assessment unit; and a discrepancy assessment 56 for assessing degree of a discrepancy indicating a discrepancy to a usual state of the user on the basis of the deviation assessed by the deviation assessment unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a biometric data processing device that is used to determine when a user is in a subjective state different from normal based on a biometric condition measured from the user, and Biometric data processing method Regarding. [Background technology]

[0002] Patent Document 1 discloses a home medical support system. A wearable device attached to the living body of the person to be supported collects biological information from the living body of the person to be supported, and the collected biological information is wirelessly transmitted by a receiver, which is a home device, and the biological information is analyzed by an information analysis unit of the receiver. If the information analysis unit determines that there is an abnormality in the biological information, the biological information is transmitted to a medical institution even if the person to be supported does not press a doctor call button. Here, examples of abnormalities in the biological information include pulse rate, blood pressure, and oxygen saturation information exceeding their respective predetermined thresholds.

[0003] Meanwhile, research is being conducted on technology that estimates the mental and physical health state of a user, such as the emotions, fatigue, and mood, based on data measured in daily life. Using this technology, it will be possible to determine whether a user is in a mental state different from normal from normal based on biometric data, and to implement intervention measures such as providing mental care based on the results of the assessment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-122434 A Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 performs intervention based on bioinformation that can be measured by a wearable device. In contrast, when considering a service that estimates the subjective state of the mind and body, among the mental and physical health conditions, based on bioinformation and performs intervention measures for individuals, the relationship between the subjective state of the mind and body and the biological state is complex, so it is difficult to perform intervention measures at the appropriate time by simply determining the condition of the bioinformation. It is desirable to collect data on the actual state of the bioinformation in the past in the case of the subjective state of the mind and body of the user, and to construct a model between the bioinformation and the subjective state of the mind and body by machine learning. For this purpose, it is necessary to collect bioinformation when the user is in various subjective states of the mind and body as learning data.

[0006] However, it is difficult to obtain high-quality learning data for such matters related to the user's subjective opinion. For example, even if the subjective state of the mind and body is inquired about periodically or randomly using a questionnaire method (Experience Sampling Method, ESM or Ecological Momentary Assessment, EMA), it is not possible to guarantee that the subjective state of the mind and body of the user has changed at the time of the questionnaire, and there is a risk of missing data when the user is experiencing strong emotions, which occur infrequently. If the subjective state of the mind and body can be recorded at the time when the user is aware of the change, it will be possible to capture large changes in the subjective state of the mind and body, but it will be impossible to capture unconscious changes in the subjective state of the mind and body.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to enable notifications for intervention measures and data collection to be implemented at appropriate timing, such as when the user is in a subjective state that is different from usual. [Means for solving the problem]

[0008] A biomeasurement data processing device according to one embodiment of the present invention includes a reception unit that receives biomeasurement data detected by a sensor from a user terminal of the user, the biomeasurement data being detected by the sensor, a deviation evaluation unit that evaluates the degree of deviation of the biomeasurement data received by the reception unit from the user's past biomeasurement data, a subjective value evaluation unit that estimates a subjective value indicating the user's mental and physical subjective state based on the biomeasurement data received by the reception unit, an uncertainty evaluation unit that evaluates the uncertainty of the subjective value estimated by the subjective value evaluation unit, and an uncertainty evaluation unit that evaluates the uncertainty of the subjective value estimated by the subjective value evaluation unit and the user's usual state based on the degree of deviation evaluated by the deviation evaluation unit and the uncertainty evaluated by the uncertainty evaluation unit. Mind-body subjectivity and a deviation evaluation unit that evaluates a deviation indicating the degree of deviation from the state. Effect of the Invention

[0009] It will be possible to take measures such as intervention measures and notifications for collecting correct data while taking into account the user's subjective state.

[0010] The details of at least one implementation of the subject matter disclosed herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosed subject matter will become apparent from the following disclosure, drawings, and claims. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of a main configuration of a biomeasurement data processing system. [Diagram 2] 13 is a flowchart showing an example of a receiving process from a user terminal. [Diagram 3] 13 is a flowchart showing an example of a learning process of a deviation evaluation model in subjective value estimation. [Figure 4] 13 is a flowchart showing an example of a deviation evaluation process in subjective value estimation. [Figure 5A] 13 is a flowchart illustrating an example of a notification determination process based on a deviation degree. [Figure 5B] 13 is a flowchart illustrating an example of a process for outputting a notification of an event determined to be notifiable based on a deviation degree. [Figure 6] 13 is a flowchart illustrating an example of a receiving process of a response result in response to a notification output. [Figure 7A] 13 is an example of a notification output screen displayed on a user terminal. [Figure 7B] 13 is an example of a notification output screen displayed on a user terminal. [Figure 8] 13 is an example of a notification output screen displayed on an administrator terminal. [Figure 9A] 13 is an example of a display screen of a subjective value estimation result displayed on a user terminal. [Figure 9B] 13 is an example of a display screen of a subjective value estimation result displayed on a user terminal. [Figure 9C] 13 is an example of a display screen of a subjective value estimation result displayed on a user terminal. [Figure 10A] FIG. 2 is a diagram illustrating an example of a data structure of user data. [Figure 10B] FIG. 4 is a diagram illustrating an example of a data structure of subjective value estimation data. [Figure 10C] FIG. 13 is a diagram illustrating an example of a data structure of deviation data. [Figure 10D] FIG. 11 is a diagram illustrating an example of a data structure of determination result data. [Figure 10E] FIG. 13 is a diagram illustrating an example of a data structure of response result data. [Figure 10F] FIG. 4 is a diagram illustrating an example of a data structure of user characteristic data. [Figure 10G] FIG. 13 is a diagram illustrating an example of a data structure of subjective answer value data. [Figure 11A] 13 is an example of a screen displayed on a user terminal to notify the user of the need for intervention based on a subjective value. [Figure 11B] 13 is an example of a screen displayed on a user terminal to notify the user of the need for intervention based on a subjective value. [Figure 11C] 13 is an example of a screen displayed on a user terminal to notify the user of the need for intervention based on a subjective value. [Figure 12A] 13 is a flowchart illustrating an example of a learning process of a similar user classification. [Figure 12B] 13 is a flowchart illustrating an example of a similar user classification process. [Figure 13A] 13 is an example of an output screen displayed on a user terminal. [Figure 13B] 13 is an example of an output screen displayed on a user terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. EXAMPLES

[0013] 1 is a block diagram showing an example of the main configuration of a biometric data processing system. The biometric data processing system of this embodiment includes one or more user terminals 7, one or more administrator terminals 8, and a biometric data processing device 1 that processes data received from the user terminals 7 via a network 9.

[0014] The user terminal 7 includes a biometric sensor 11 that detects the biometric condition of the user, a biometric device 12 that controls the biometric sensor 11, an input / output device 13, a communication device 14, and a notification / notification device 15.

[0015] The biometric sensor 11 includes a heart rate sensor 21 that detects the user's heart rate interval (RR Interval: RRI), an electrodermal activity sensor 22 that detects the user's sweat rate, and an acceleration sensor 23 that detects the user's movement. The heart rate sensor 21 may be a sensor that detects the heart rate based on an electrocardiogram, a pulse wave, a pressure change, a heart sound, or the like. The biometric sensor 11 is not limited to the above example, and other sensors that detect body temperature, blinking, eye movement, electromyogram, brain waves, or the like may be adopted. The biometric sensor 11 may be a wearable device that the user can wear, or a system built into a smartphone that the user can carry around.

[0016] The biomeasurement device 12 controls the biomeasurement sensor 11 and performs calculation processing and compression processing on the biological condition measured by the biomeasurement sensor 11 as necessary to generate biomeasurement data 82.

[0017] The input / output device 13 displays information on the screen of the user terminal 7 and receives correct subjective value data 91, which is the correct data on the subjective mental and physical state that is the subject of estimation based on biometric information.

[0018] The notification device 15 notifies the user in response to the notification output process from the biomeasurement data processing device 1. For example, in addition to the screen display on the user terminal 7 performed by the input / output device 13, the notification output to the user may be performed using vibration or sound.

[0019] In this example, the user terminal 7 includes the biometric sensor 11 and the biometric device 12, but they do not necessarily have to be configured as one device (hardware). For example, the input / output device 13 and the communication device 14 of the user terminal 7 may be configured as a smartphone, and the biometric device 12, the biometric sensor 11, and the notification device 15 may be configured as a smartwatch, and the user terminal 7 may be configured by regarding the smartphone and the smartwatch as an integrated system.

[0020] The manager terminal 8 includes an input / output device 31, a communication device 32, and a notification / alarm device 33. The manager here is not limited to a specific relationship such as a superior to subordinates, but broadly refers to a person who has a role of managing and supervising users. The notification / alarm device 33 notifies the manager in response to notification output processing from the biometric data processing device 1. For example, in addition to the screen display on the manager terminal 8 performed by the input / output device 31, notification output to the manager may be performed using vibration or sound.

[0021] The biomeasurement data processing device 1 is a computer including a processor 2, a memory 3, a storage device 4, an input / output device 5, and a communication device 6. The memory 3 loads each of the functional units, namely, a reception unit 51, a preprocessing unit 52, a subjective value evaluation unit 53, a deviation evaluation unit 54, an uncertainty evaluation unit 55, a deviation evaluation unit 56, a notification determination unit 57, a notification output unit 58, a result display unit 59, and a similar user classification unit 60, as a program. Each program is executed by the processor 2. The details of each functional unit will be described later.

[0022] The processor 2 operates as a functional unit that provides a predetermined function by executing processing according to the program of each functional unit. For example, the processor 2 functions as a deviation evaluation unit 56 by executing a deviation evaluation program. The same applies to other programs. Furthermore, the processor 2 also operates as a functional unit that provides each function of multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units.

[0023] The storage device 4 stores data used by each of the above-mentioned functional units. The storage device 4 stores user data 81, biometric data 82, preprocessed data 83, subjective value estimation data 84, deviation data 85, uncertainty data 86, deviation data 87, judgment result data 88, correspondence result data 89, user characteristic data 90, subjective value correct answer data 91, data preprocessing model 92, subjective value estimation model 93, deviation evaluation model 94, uncertainty evaluation model 95, deviation evaluation model 96, judgment criterion model 97, and similar user classification model 98. Details of these data and models will be described later.

[0024] The input / output device 5 includes input devices such as a mouse, a keyboard, a touch panel, or a microphone, and output devices such as a display, a speaker, etc. The communication device 6 communicates with a user terminal 7 and an administrator terminal 8 via a network 9.

[0025] In the following embodiment, a case is illustrated in which the user always wears or carries the user terminal 7 in daily life and the biometric sensor 11 is in operation, but the present invention is not limited to this example. For example, the biometric sensor 11 may be operated using the user terminal 7 several times a day, such as when waking up or going to bed.

[0026] FIG. 2 is a flow chart showing an example of a reception process from the user terminal 7, which is performed by the biometric data processing device 1.

[0027] The accepting unit 51 of the biometric data processing device 1 starts data reception S21 when a connection with the user terminal 7 is established via the network 9. When data reception S21 starts, the biometric data processing device 1 receives bioinformation measured by the biometric sensor 11 of the user terminal 7 and digitized by the biometric device 12, and stores it as biometric data 82. Data reception continues until the connection with the user terminal 7 is cut off. In the following, an example will be given of a case where heartbeat interval data, electrodermal activity data, and acceleration data are measured and stored as the biometric data 82.

[0028] Furthermore, when correct answer data regarding the subjective physical and mental state to be estimated based on biometric information is input into the input / output device 13 of the user terminal 7, the biometric data processing device 1 receives the also input correct answer data and stores it as subjective value correct answer data 91.

[0029] Here, the subjective mental and physical state to be estimated by the system is assumed to be a natural emotion in daily life. In this case, it is considered that the natural emotion is measured as an emotion dimension consisting of arousal and valence as the subjective value correct answer data 91. In this case, it is possible to measure the natural emotion using an Affective Scale that represents and measures the emotion dimension using pictograms at both ends of a Visual Analogue Scale (VAS), which is a visual scale, or a Self-Assessment Manikin (SAM) that measures using multi-level pictograms.

[0030] In addition to measuring emotion dimensions, experienced emotions can also be measured using the Positive and Negative Affect Schedule (PANAS), which measures discrete emotions such as happiness based on the degree to which they correspond to adjectives that describe the emotion.

[0031] In this example, the biometric data 82 is continuously received, but it does not necessarily have to be continuously received. For example, the user terminal 7 may receive biometric data 82 summarized for a certain period of time, such as 2 minutes or 30 minutes. Also, a connection may be established and the data receiving process S21 may be performed only when the user terminal 7 has performed a transmission process to the biometric data processing device 1.

[0032] 3 is a flowchart showing an example of a learning process of a deviation evaluation model 96 in subjective value estimation, which is performed by the biomeasurement data processing device 1. First, the reception unit 51 of the biomeasurement data processing device 1 performs a data reading process S31 for reading the subjective value correct answer data 91 used for learning, the biomeasurement data 82, and the user data 81 storing information on the user from whom the biomeasurement data 82 was acquired. In the following, unless otherwise specified, an example is shown in which one set of biomeasurement data 82 is made up of heartbeat interval data, electrodermal activity data, and acceleration data for 30 minutes, and one set of subjective value correct answer data 91 is the strength of valence and arousal for 30 minutes, and the learning process of the deviation evaluation model 96 is performed by using a plurality of such sets.

[0033] In the data preprocessing learning process S32, the preprocessing unit 52 first performs a data preprocessing learning process using the loaded user data 81 and biometric data 82. In the data preprocessing learning process, a data preprocessing model 92 is learned, which performs correction processing for individual differences contained in the biometric data 82, feature extraction processing from the biometric data 82, feature compression processing of features extracted from the biometric data 82, etc. These processes may be explicitly executed sequentially as a pipeline by the data preprocessing model 92 that groups together multiple processes (subtasks), or may be processed end-to-end (all at once without dividing into subtasks) using the data preprocessing model 92 learned from the data.

[0034] For example, the series of data preprocessing may be configured as a correction process for individual differences, in which data normalization processing is performed based on a series of biometric data 82 previously measured from the user, and the feature extraction processing may be configured as a feature compression processing by principal component analysis using the biometric data 82 normalized for each user as input. In this case, the normalization parameters for each user, the number of principal components in the principal component analysis, and the obtained eigenvectors can be learned as a learning process for data preprocessing to create a data preprocessing model 92. Of this data preprocessing model 92, the normalization parameters for each user used in the correction process for individual differences can be treated as a physical difference correction model for individual differences, and the portion of the data preprocessing model 92 excluding the physical difference correction model (normalization parameters) becomes a data preprocessing model that can be commonly used by multiple users, in which individual differences due to physical characteristics between individuals have been removed, thereby improving versatility.

[0035] In the case of end-to-end processing, a process of removing measurement noise from biomeasurement data 82 may be set as data preprocessing, and a latent vector of a Variational Auto Encoder (VAE), which is a deep generative model inputting the biomeasurement data 82 from which noise has been removed, may be used as feature extraction and feature compression processing. In this case, the VAE model is trained from the biomeasurement data 82 by unsupervised learning.

[0036] For the purpose of correcting individual differences and removing noise, normalization processing of the signal scale to be used and noise removal processing may be performed. For example, the normalization processing may be min-max normalization, which normalizes the maximum and minimum values ​​for each user or for each measurement date, for each measurement date of a user, or among all users, z-score normalization, which normalizes the average and standard deviation of the signal, or quantile normalization, which normalizes the signal intensity distribution using a quantile. In addition, if it is known that the signal intensity varies due to aging, etc., deviation value processing, which normalizes the signal intensity for each age group in consideration of age information of the user data 81, may be performed. Furthermore, in the noise removal processing, clipping or winsorization processing, which removes abnormal values ​​of the signal and places them within a certain range, moving average processing, which suppresses and smoothes sudden fluctuations at one time, zeroth order differentiation processing using a Savitzky-Golay filter, etc. may be performed.

[0037] In the feature extraction process from the biomeasurement data 82, feature extraction process may be performed according to the biosignal of the biomeasurement data 82 to be used. For example, for the heartbeat interval data acquired by the heartbeat sensor 21, the average heart rate, the low frequency component (LF) or high frequency component (HF) obtained by frequency domain analysis and known to mainly reflect sympathetic nerve activity or parasympathetic nerve activity, respectively, the SDNN, RMSSD, and NN50 used in the time domain analysis, the feature using the Lorenz plot used in the nonlinear domain analysis, the feature obtained by the detrended fluctuation analysis, the feature obtained by the complex demodulation method, etc. may be used. In addition, in the case of the electrodermal activity data acquired by the electrodermal activity sensor 22, the skin conductance level (SCL) or the skin conductance response (SCR) may be used. In addition, in the case of triaxial acceleration data obtained from the acceleration sensor 23, the acceleration norm or the number of zero crossings, which is the number of times that a signal processed by a band-pass filter for the acceleration norm passes through a threshold of ±0.01 G when the gravitational acceleration is 1 G, may be used.

[0038] Furthermore, in addition to the biomeasurement data 82, the subjective value correct answer data 91 may also be subjected to a predetermined data pre-processing. For example, when estimating the strength of valence and arousal, pre-processing may be performed according to the type of subjective value estimation model 93 used in the subjective value estimation model learning process described below. For example, if valence and arousal are measured on a five-level Likert scale from 1 to 5, the scale may be converted to a range from -1 to 1 when learning a regression model. In addition, in the case of a classification model, 1 and 2 may be binarized as negative example 0, and 4 and 5 as positive example 1, except for the intermediate value 3.

[0039] The feature compression process may use a known algorithm, such as the above-mentioned principal component analysis, an autoencoder (AE), or a uniform manifold approximation and projection (UMAP).

[0040] In addition, since the learning process of the data preprocessing is typically based on unsupervised learning, the learning process may be performed using biometric data 82 that does not correspond to the subjective value correct answer data 91. Since the cost of acquiring the subjective value correct answer data 91 is high, by configuring in this way, the data preprocessing model 92 can be learned from a larger number of biometric data 82, and an effect is obtained that it is possible to learn the data preprocessing model 92 that can generate preprocessed data that can express a wider variety of states as features.

[0041] When the learning process of the data preprocessing is completed as described above, the preprocessing unit 52 performs data preprocessing on the biomeasurement data 82 using the learned data preprocessing model 92 to generate preprocessed data 83.

[0042] In the subjective value estimation model learning process S33, the subjective value evaluation unit 53 learns the subjective value estimation model 93 using the subjective value correct answer data 91 and the preprocessed data 83. For example, when natural emotions in daily life are to be estimated, a set of subjective value correct answer data 91 is used to learn an emotion dimension estimation model that estimates the intensities of valence and arousal for 30 minutes by supervised learning. The subjective value estimation model 93 can be configured using a known algorithm. For example, a logistic regression model, a decision tree, a Random Forest, a Support Vector Machine, a neural network, a Bayesian neural network, a deep learning model, or the like can be used as the machine learning algorithm. As for the algorithm, a classification algorithm or a regression algorithm can be used depending on the subjective value to be estimated. For example, a regression algorithm can be used when estimating the intensities of valence and arousal from -1 to 1, and a classification algorithm can be used instead of a regression algorithm when estimating the high and low intensities of valence and arousal.

[0043] In this flowchart, the learning process of the data preprocessing and the learning process of the subjective value estimation model are described as different processes, but a part of both processes may be configured as one. For example, in the learning process of the data preprocessing, a data preprocessing model 92 including only a noise removal process for preprocessing the measurement noise of the biomeasurement data 82 may be configured as a subjective value estimation model 93 that performs feature extraction processing, feature compression processing, and subjective value estimation processing end-to-end. In this case, it may be configured in the form of a deep learning model consisting of a fully connected layer and a long short-term memory, graph neural network, convolutional neural network, graph neural network, self attention, etc. that are good at handling time-series data.

[0044] In addition, the subjective value correct answer data 91, which is the estimation target of the subjective value estimation model 93, is often data that is subjectively annotated by the user himself, especially when the subjective value correct answer data 91 is obtained in daily life, and therefore the correct answer data itself may be unreliable. In consideration of learning the subjective value estimation model 93 from such unreliable correct answer labels, a model configuration suitable for this may be used. For example, the subjective value correct answer data 91 may be weighted and learned according to the reliability of the answer of each user. In addition, a known algorithm may be used as a model configuration for a correct answer label with low reliability or certainty. For example, although a correct answer label is assigned to the entire biomeasurement data 82 for the last 30 minutes, Multiple Instance Learning may be used in consideration of the fact that it is not possible to determine exactly which point in time the correct answer label is. In the case where the correct answer label is not determined as a single label, but the reliability of the distribution position or order itself is relatively high, a Bayesian deep learning model or Label Distribution Learning may be used.

[0045] Furthermore, the subjective value estimation model 93 may also be configured as a model that takes into account individual differences, as in the data pre-processing model 92. It is considered that the subjective value correct data 91, which is the correct label learned by the subjective value estimation model 93, contains differences in cognition, such as the subjective cognition method for a certain event and the answer tendency that occurs when verbalizing the recognized content. Therefore, when the data pre-processing model 92 corrects for physical differences among the individual differences, the model may be configured to include a cognition difference correction process that corrects such cognition differences for the correct label. In this case, it is possible to perform subjective value estimation that is compatible with the user's cognition tendency, and the effect of improving compatibility with the user and the user's acceptability of the subjective value estimation result is obtained.

[0046] The cognitive difference correction process may be performed by adding a dedicated process to the subjective value estimation model 93, or by configuring the subjective value estimation model 93 itself as an end-to-end process. For example, when a dedicated process is added, the answer style of the user's subjective value correct data 91 may be regarded as the user's cognitive tendency, and a subjective value estimation that applies to all users may be performed using a classification model or a regression model, and then a process may be added to perform a correction on the estimation result taking into account the central response tendency or extreme response tendency, which is the answer style, to obtain the subjective value estimation data 84. In addition, when configuring it as an end-to-end process, the subjective value estimation model 93 may be configured as a neural network, and a layer close to the final layer may be divided for each user and configured by multi-task learning that divides for each user, so that estimation that learns the cognitive tendency according to the user can be realized, or a subjective value estimation model 93 common to users may be created and configured to be adapted to a specific user by fine tuning for each user.

[0047] When the learning process of the subjective rating estimation model 93 is completed as described above, the subjective rating assessment unit 53 generates the subjective rating estimation data 84 from the preprocessed data 83 using the learned subjective rating estimation model 93 .

[0048] In the deviation evaluation model learning process S34, the deviation evaluation unit 54 learns a deviation evaluation model 94 from the biomeasurement data 82 and the preprocessed data 83. The deviation evaluation model 94 evaluates the degree to which the input biomeasurement data 82 deviates from normal data as deviation. The deviation evaluation model 94 is typically an unsupervised model using a known anomaly detection algorithm, and can be configured as a known statistical model or machine learning model. As the statistical model, for example, a statistical model that outputs a z-score using the average value or standard deviation for the data distribution of the preprocessed data 83 obtained by the data preprocessing model 92, a statistical model that outputs a deviation value, a statistical model that outputs a quantile on the data distribution, etc. can be used. In addition, as the machine learning model, for example, a machine learning model that outputs a distance based on a cluster center of the data distribution estimated nonparametrically from the preprocessed data 83 can be used.

[0049] For example, when the data preprocessing model 92 is configured hierarchically as a body difference correction model for removing individual differences due to the physical characteristics of individuals and a feature extraction model such as a variational auto encoder that can be commonly used by multiple users from which individual differences due to the physical characteristics of individuals have been removed, it is possible to calculate estimated biometric data common to individuals obtained by inputting the preprocessed data 83 into the decoder of the feature extraction model, and estimated biometric data for each individual obtained by inverse conversion using the body difference correction model from the estimated biometric data common to individuals. In this case, the deviation evaluation model 94 can be trained as a model for calculating a reconstruction error indicating the degree to which the estimated biometric data common to individuals has been reconstructed from the biometric data from which the individual differences due to the physical characteristics of individuals have been removed. The deviation evaluation model 94 can also be trained as a model for calculating a reconstruction error indicating the degree to which the estimated biometric data for each individual has been reconstructed from the biometric data 82. Furthermore, the deviation evaluation model 94 can be configured as a model for evaluating the reconstruction errors of both processes together and calculating a weighted value of the reconstruction error. In such a configuration, the degree of deviation will be small if the data is close to the biomeasurement data 82 measured under normal circumstances, and will be large if there is an abnormality in the biostatus or a problem with the measurement conditions for the biostatus, making it possible to evaluate the deviation of the input biomeasurement data 82 itself from normal times as the degree of deviation.

[0050] When the learning process of the deviation evaluation model 94 is completed as described above, the deviation evaluation unit 54 generates deviation data 85 from the biomeasurement data 82 and the preprocessed data 83 using the learned deviation evaluation model 94.

[0051] In the uncertainty assessment model learning process S35, the uncertainty assessment unit 55 learns the uncertainty assessment model 95 from the preprocessed data 83 and the subjective value estimation data 84. Typically, the subjective value estimation model 93 may be used as the uncertainty assessment model 95. In this case, perturbation is added to the preprocessed data 83 used to input the subjective value estimation model 93 and calculate the subjective value estimation data 84, and the degree of variation of the subjective value estimation data 84 is evaluated as the uncertainty, so that it can be used as the uncertainty assessment model 95. In addition, when the subjective value estimation model 93 is configured by a Bayesian deep learning model or label distribution learning that takes into account the uncertainty of the input data and the model, the subjective value estimation data 84 is obtained as a distribution with a spread indicating the certainty of the estimation, so that the uncertainty assessment model 95 can be learned as a model that evaluates the spread of the subjective value estimation data 84. As described above, by calculating the subjective value estimation data 84 from the biometric data 82, it is possible to evaluate the degree to which the subjective state itself corresponding to the estimated input is uncertain and different from normal times, in addition to the degree of abnormality of the biometric data 82.

[0052] When the learning process of the uncertainty assessment model 95 is completed as described above, the uncertainty assessment unit 55 uses the learned uncertainty assessment model 95 to generate uncertainty data 86 from the preprocessed data 83 and the subjective value estimation data 84.

[0053] In the deviation evaluation model learning process S36, the deviation evaluation unit 56 learns a deviation evaluation model 96 from the deviation data 85 and the uncertainty data 86. The deviation evaluation model 96 is typically an unsupervised model using a known anomaly detection algorithm that evaluates the degree of anomaly using the deviation data 85 and the uncertainty data 86 as input, and can be configured as a known statistical model or machine learning model.

[0054] Also, the deviation evaluation model 96 may be configured to evaluate the belief of how much importance is attached to the deviation data 85 and the uncertainty data 86 as deviations, without being based on learning. For example, when the difference between the input biometric data 82 itself and normal times is emphasized, the deviation data 85 may be multiplied by 0.9 as a coefficient A, the uncertainty data 86 may be multiplied by 0.1 as a coefficient B, and the weighted sum may be calculated to evaluate the deviation. When the degree of uncertainty of the subjective state itself corresponding to the estimated input is emphasized, the deviation data 85 may be multiplied by 0.1 as a coefficient A, the uncertainty data 86 may be multiplied by 0.9 as a coefficient B, and the weighted sum may be calculated to evaluate the deviation. In this case, the deviation is calculated as coefficient A × deviation + coefficient B × uncertainty, and the set of coefficient A and coefficient B is stored as the deviation evaluation model 96.

[0055] This makes it possible to calculate indices for determining the timing of implementing intervention measures at an appropriate time, taking into account both the degree of difference between the biological state itself and the normal state, and the degree to which the subjective state of the mind and body itself is uncertain and differs from normal conditions, in relation to the complex relationship between the biological state and the subjective state of the mind and body, and for determining the timing of notification of data collection for an estimation model of the subjective state.

[0056] When the learning process of the deviation evaluation model 96 is completed as described above, the deviation evaluation unit 56 generates deviation data 87 from the deviation data 85 and the uncertainty data 86 using the learned deviation evaluation model 96 .

[0057] The series of learning processes shown in Fig. 3 is executed at least once before the process of evaluating the degree of deviation regarding the subjective value estimation shown in Fig. 4 described later. In addition, this process is executed at regular intervals as the subjective value correct answer data 91 and the biometric data 82 increase, and each of the models, the data preprocessing model 92, the subjective value estimation model 93, the deviation evaluation model 94, the uncertainty evaluation model 95, and the deviation evaluation model 96, can be re-learned. As a result, each model with even higher evaluation accuracy can be generated.

[0058] In addition, although this example discloses a case where one model each of the data preprocessing model 92, the subjective value estimation model 93, the deviation evaluation model 94, the uncertainty evaluation model 95, and the deviation evaluation model 96 is used, it is typically desirable to prepare a number of each model and use them in combination. For example, as the deviation evaluation model 96, a number of pairs of coefficients A and B may be prepared so that a number of deviations with different beliefs regarding how much importance should be attached to the deviation data 85 and the uncertainty data 86 as deviations can be evaluated. As a result, when making a notification decision based on the deviation, which will be described later, it becomes possible to realize a notification decision from various viewpoints.

[0059] 4 is a flowchart showing an example of a deviation evaluation process in subjective value estimation performed by the biomeasurement data processing device 1. First, the reception unit 51 of the biomeasurement data processing device 1 performs a data reading process S41 to read the biomeasurement data 82 and the user data 81 storing information on the user from whom the biomeasurement data 82 was acquired.

[0060] Next, the preprocessing unit 52 reads out the data preprocessing model 92 , performs data preprocessing S 42 on the user data 81 and the biometric data 82 using the read out data preprocessing model 92 , and outputs the preprocessed data 83 .

[0061] In the subjective value estimation process S43, the subjective value evaluation unit 53 reads out the subjective value estimation model 93, and generates the subjective value estimation data 84 from the read-in subjective value estimation model 93 and the preprocessed data 83. In the deviation evaluation process S44, the deviation evaluation unit 54 reads out the deviation evaluation model 94, and generates the deviation data 85 from the read-in deviation evaluation model 94, the biomeasurement data 82, and the preprocessed data 83. In addition, in the uncertainty evaluation process S45, the uncertainty evaluation unit 55 reads out the uncertainty evaluation model 95, and generates the uncertainty data 86 from the read-in uncertainty evaluation model 95, the preprocessed data 83, and the subjective value estimation data 84. Note that, here, an example in which the subjective value estimation process S43, the uncertainty evaluation process S45, and the deviation evaluation process S44 are performed in parallel has been shown, but this is not necessarily limited to this configuration. For example, the subjective value estimation process S43, the deviation evaluation process S44, and the uncertainty evaluation process S45 may be performed sequentially. In the deviation evaluation process S46, the deviation evaluation unit 56 reads out the deviation evaluation model 96, and generates deviation data 87 from the read deviation evaluation model 96, the deviation data 85, and the uncertainty data 86.

[0062] As described above, deviation data 87, which is a deviation indicating the state of the user, can be generated based on deviation data 85 indicating the deviation of the biometric data 82 itself from normal times and uncertainty data 86 indicating the uncertainty of the subjective value estimation data 84 estimated based on the biometric data 82. This has the effect of providing a measure for determining whether the user is in a subjective state different from normal.

[0063] 5A is a flowchart showing an example of a notification determination process based on the deviation degree, which is performed by the notification determination unit 57 of the biomeasurement data processing device 1. In a data reading process S51, the notification determination unit 57 reads the user data 81 and the deviation degree data 87. Thereafter, while any deviation degree data 87 has an undetermined determination flag, the notification determination process S52 is performed.

[0064] In the notification determination process S52, a notification determination is made based on the determination reference model 97 for data for which the determination done flag included in the deviation data 87 has not been completed, and the determination result is output to the determination result data 88. In addition, for deviation data 87 for which the notification determination has been completed, the determination done flag is updated to completed.

[0065] In this example, one type of judgment criterion model 97 is used, but multiple judgment criterion models 97 may be prepared. For example, by preparing multiple judgment criterion models 97 suitable for the deviation evaluation model 96 for deviation data 87 calculated by different deviation evaluation models 96, it is possible to perform an appropriate notification judgment process taking into account the calculation criteria of the deviation. Also, multiple judgment criterion models 97 may be prepared for deviation data 87 calculated from a single deviation evaluation model 96. For example, by providing a first judgment criterion model for judging whether the notification criteria for notifying the user himself / herself are exceeded and a second judgment criterion model for judging whether the notification criteria for notifying the administrator who manages the users are exceeded, it becomes possible to perform notification judgment with different criteria depending on the notification target, and it becomes possible to judge the situation in which notification should be performed depending on the action to be taken by the notification recipient. Furthermore, by using multiple determination criterion models 97 according to the characteristics of the user data 81, notification determination can be made taking into account information not included in the biometric data 82 or the subjective value estimation data 84, such as age.

[0066] 5B is a flowchart showing an example of a notification output process based on the determination result data 88, which is performed by the notification output unit 58 of the biomeasurement data processing device 1. The notification output unit 58 reads the determination result data 88 in a data reading process S53. Thereafter, while there is unnotified data whose notification status is uncompleted among the determination result data 88, the notification output unit 58 performs a notification process to the target.

[0067] First, if unnotified data exists, in the detail reading process S54, details of data related to the unnotified judgment result data 88 are read. In this example, in the detail reading process S54, biometric data 82, preprocessed data 83, and subjective value estimation data 84 are read. By reading these related data, it is possible to implement notification taking into account the information of the related data.

[0068] Thereafter, a notification process S55 is performed for the unnotified data of the judgment result data 88. In the notification process S55, the input / output device 5 notifies the notification target via the network 9. For example, if the notification target is a user, a notification process S55 is performed for the user terminal 7 corresponding to the user ID of the judgment result data 88, and the notification notification device 15 of the user terminal 7 issues a notification to the user. At this time, if a smartphone is used as the user terminal 7, the notification can be made in the form of a push notification of the smartphone or a message to a notification application for the user. Also, if the notification target is an administrator, a notification process S55 is performed for the administrator terminal 8 corresponding to the administrator who manages the users corresponding to the user IDs of the judgment result data 88, and the notification notification device 33 of the administrator terminal 8 issues a notification to the administrator. At this time, if a PC is used as the administrator terminal 8, the notification can be made in the form of output of an alarm sound to the PC, an email notification to the PC, or a notification display to an administrator application. Also, for the judgment result data 88 for which the notification process has been completed, the notification status flag is updated to completed.

[0069] As described above, the degree of discrepancy can be used to determine whether the user is in a subjective state that is different from usual, and notifications can be sent to the target at appropriate times for intervention measures and data collection based on the subjective state.

[0070] FIG. 6 is a flowchart showing an example of a receiving process of a response result according to a notification output, which is performed by the biometric data processing device 1. The reception unit 51 performs a response result receiving process S61 for receiving a response from the target terminal for which the notification process has been performed. In the following, as an example of receiving a response from the user terminal 7, a case where a user is notified of a timing when the user is assumed to be in a subjective emotional state different from normal based on a deviation degree is shown. In this case, in the response result receiving process S61, correct answer data representing a subjective emotional state generated based on the user's response to the user terminal 7 is received as the response result data 89. In addition, when the response result data 89 is received, the response status flag of the judgment result data 88 is updated to done.

[0071] As described above, it is possible to obtain the result of the action taken by the subject who received the notification in response to the notification issued at an appropriate timing for intervention measures or data collection, such as when the user is in a subjective state different from usual. For example, in the case of this embodiment, if the subjective value correct data 91 is obtained as correct data of the emotion corresponding to the moment of the notification regarding the timing when the subjective state of a different emotion is assumed, it is possible to efficiently obtain correct data regarding emotions that rarely occur in daily life. In addition, in the case of Example 2 described later, in response to the notification regarding the timing when the subjective state of a mental and physical health, such as fatigue or mood different from usual, a flag value indicating whether the user confirmed the notification, the content of the action taken, and the actual subjective mental and physical health state at that time are obtained, so that it is possible to obtain the response result data 89 to be used for verifying the accuracy of the subjective value estimation and evaluating the intervention effect.

[0072] 7A and 7B are diagrams showing an example of a notification output screen that the result display unit 59 outputs to the display of the input / output device 13 of the user terminal 7. In this example, an example is shown in which the notification process is performed in the form of a message to a notification application for the user.

[0073] A display screen 1000 shown in FIG. 7A is an example of a display screen based on a deviation threshold. If the present invention is not implemented, a notification application, such as Signal #1 1001, notifies a user irregularly (experience sampling based on time information) and attempts to collect correct answer data for emotions corresponding to a certain moment. However, since emotions such as strong joy or sadness do not occur frequently in daily life, simple irregular notifications would take a long time to collect correct answer data, or a specified period would end without acquiring a sufficient number of correct answer data.

[0074] In contrast, by using the deviation degree in this embodiment, the timing when the user is in a subjective state (emotional state) different from usual can be notified to the user terminal 7 in the form of Event #1 1002, and the input of the emotional state can be prompted. If a response based on this notification is received as the response result data 89 in the reception process of the response result, the subjective value correct answer data 91 corresponding to a rare emotional state can be obtained more efficiently than when the subjective value correct answer data 91, which is the emotion correct answer data based on an irregular notification such as Signal #1 1001, is obtained. In addition, by performing a notification at this time that also displays the deviation degree 1003 at the time of notification compared to the usual deviation degree, the user can obtain material for introspecting the notification content when generating the response result data 89.

[0075] A display screen 1010 shown in Fig. 7B is an example of a display screen that displays the subjective value estimation result when the subjective value estimation model 93 is created for each user. In this case, abstract information such as the degree of deviation is not presented to the user, but rather, the subjective value emotional state estimated based on the biometric data 82 is displayed as Event #1 1012 after indicating the possible range of deviation from the user's normal state. For example, the subjective state at that time can be displayed with a distribution range on a two-dimensional graph 1013 of arousal and valence based on Russell's circumplex model.

[0076] As described above, the user refers to information about what the user's condition was like at a certain moment and provides feedback on the response result data 89, which enables the biometric data processing device 1 to collect subjective value correct answer data 91.

[0077] 8 is a diagram showing an example of a notification output screen that the result display unit 59 outputs to the display of the input / output device of the administrator terminal 8. In this example, an example of a dashboard 2000 is shown which displays a list of notification statuses for a series of users managed by the administrator.

[0078] The dashboard 2000 is a display screen for displaying and analyzing notifications notified to a group of users on a certain date 2001. The upper left table 2002 displays a list of notifications and the user's response to the notifications. When a specific notification 2003 is selected from this table, a plurality of deviation degree judgment results corresponding to the subjective value estimation data 84 of the notification target are displayed in the lower left table 2004. When a row 2005 in which the judgment result is judged to be a notification target is selected, a contact button 2006 that can contact the notification target user in relation to the data of the notification target and a button that displays the action that the administrator has taken or is expected to take in response to the judgment result are displayed. Specifically, an unconfirmed button 2007 indicating that the content has not been confirmed, an unhandled button 2008 indicating that the content has been confirmed but no action has been taken for the user, an action in progress button 2009 indicating that action is being taken for the user, and a handled button 2010 indicating that the action for the user has been completed can be displayed. The current status is displayed by the buttons 2007 to 2010, and the administrator can update the status by pressing the corresponding button each time a response is taken.

[0079] The right side of the dashboard 2000 displays details of the data corresponding to the selected row 2005 in the bottom left table 2004 .

[0080] In the upper right area 2020, the deviation status of the subjective value estimation at a certain point in time corresponding to the selected row 2005 is shown by multiple means. For example, the first continuous emotion intensity 2021 shows a one-dimensional confidence interval when an interval estimation of valence or arousal is performed in the subjective value estimation. The second continuous emotion intensity 2022 is an example of a similar confidence interval displayed as a two-dimensional graph. The multiple emotion certainty 2023 is an example of a probability value display of the degree of each basic emotion when multiple basic emotions are estimated in addition to valence and arousal. Even in this case, the administrator can check the degree of emotion fluctuation and the uncertainty of the estimation from the minimum and maximum values ​​of the estimation probability of each basic emotion. As a result, the administrator can visually determine how the user's subjective state deviates from normal times at a certain point in time.

[0081] The lower right area 2030 shows an example of a deviation state of the subjective value estimation by multiple means, taking into account the estimation process on the same measurement date before a certain time corresponding to one line 2005 of the selected line. For example, in the first emotion transition 2031, the time series change and the subjective estimation data that is originally predicted are displayed one-dimensionally for the one-dimensional emotion estimation value in the first continuous emotion intensity 2021. Furthermore, in the second emotion transition 2032, the trajectory of the transition process of the subjective value estimation data 84 in the second continuous emotion intensity 2022 is shown on a two-dimensional graph. As described above, the deviation from normal times can be visually grasped, taking into account the state within the day.

[0082] 9A to 9C are diagrams showing an example of a display screen of the subjective value estimation result displayed on the user terminal 7. As in the example of FIG. 8 showing the dashboard for the administrator, the same contents as the deviation situation of the subjective value estimation at a certain time point (for example, the first continuous emotion intensity 2021, the second continuous emotion intensity 2022, and the multiple emotion certainty 2023) can be displayed for the user. In particular, when multiple types of subjective value estimation models 93 are prepared, such as for each user, it is easy to understand how much the situation of the subjective value estimation data 84 at a certain time point differs from that of general others by using the subjective value estimation data 84 estimated based on the subjective value estimation model 93 for the user himself / herself and the subjective value estimation data 84 estimated by the subjective value estimation models 93 for multiple people other than the user himself / herself. For example, the display screen 1020 (FIG. 9A) shows the first continuous emotion intensity, the display screen 1030 (FIG. 9B) shows the second continuous emotion intensity, and the display screen 1040 (FIG. 9C) shows the multiple emotion certainty degree, allowing comparison between the user's subjective value estimation data and the subjective value estimation data of general others. This allows the user to understand that the estimation results are suited to the user himself / herself, and is expected to improve the acceptability of the estimation contents of the subjective value estimation data 84.

[0083] Next, a characteristic structure of each piece of data used in the biomeasurement data processing device 1 will be described.

[0084] 10A is a diagram showing an example of the data structure of user data 81 held in the biometric data processing device 1. The user data 81 typically stores a user ID 100, update date 101, age 102, gender 103, affiliation 104, occupation 105, etc. In addition, when performing similar user classification processing described later as a third embodiment, for example, a similar classification 106 indicating a similar user category and the number of days of measurement up to now 107 serving as an index for determining whether or not to apply the similar user classification processing may be stored.

[0085] FIG. 10B is a diagram showing an example of the data structure of the subjective value estimation data 84 held in the biometric data processing device 1. The subjective value estimation data 84 typically stores a user ID 110, a date and time 111, and an estimated subjective value. For example, in the case of the subjective value estimation data 84 corresponding to the first continuous emotion intensity in FIG. 9A, the quantiles of the values ​​normalized to the Valence intensity value from −1 to 1 may be stored. In this case, the median of the Valence estimation value is stored in V_500 112, and the 25% quantile, 75% quantile, 2.5% quantile, and 97.5% quantile are stored in V_250 113, V_750 114, V_025 115, and V_975 116, respectively. Note that when subjective values ​​are estimated using a plurality of subjective value estimation models, subjective values ​​estimated using the same data may differ (first and second rows in FIG. 10B).

[0086] 10C is a diagram showing an example of a data structure of deviation data 87 held in the biometric data processing device 1. The deviation data 87 typically stores a user ID 120, date and time 121, deviation evaluation model 122 indicating model information used to calculate the deviation, deviation 124, uncertainty 125, deviation 126, and determined flag 127 storing a determined flag. In addition, for example, when performing a similar user classification process described later as a third embodiment, a similarity classification 123 indicating whether or not a similar user classification was used in the deviation calculation may be stored.

[0087] 10D is a diagram showing an example of a data structure of the determination result data 88 held in the biomeasurement data processing device 1. The determination result data 88 typically stores a user ID 130, date and time 131, a deviation evaluation model 132, a determination criterion model 133 that is the criterion for determining whether or not to notify, a determination 134 that stores the determination result, a notification status 135, an event ID 136 for uniquely identifying the determination result notification event, and a response status 137 to the notification.

[0088] 10E is a diagram showing an example of a data structure of response result data 89 held in the biometric data processing device 1. The response result data 89 typically stores a user ID 140, an event ID 141, a notification confirmation status 142, a response status to the notification 143, and an answer date and time 144. When collecting correct answer data related to emotions, an answer V 145 that stores correct answer data on valence and an answer A 146 that stores correct answer data on arousal level may be stored as responses from the user.

[0089] 10F is a diagram showing an example of the data structure of user characteristic data 90 held in the biomeasurement data processing device 1. The user characteristic data 90 is used in Example 3 described later. The user characteristic data 90 typically stores a user ID 150, date and time 151, and a characteristic name 152. Furthermore, when the Big-Five Personality is measured as a characteristic by the NEO-PI-R or NEO-FFI, the scores of the five types of characteristics corresponding to the Big-five personality may be stored in characteristic 1 153 to characteristic 5 157.

[0090] 10G is a diagram showing an example of the data structure of the subjective value correct answer data 91 held in the biometric data processing device 1. The subjective value correct answer data 91 typically stores a user ID 160, a date and time 161 indicating the date and time when the user answered, a start time 162 and an end time 163 of a period during which the user judged the subjective mental and physical state, etc. In addition, when collecting correct answer data regarding emotions, an answer V 164 that stores correct answer data of emotional valence as a response from the user may be stored.

[0091] As described above, the biometric data processing device 1 of this embodiment includes the receiving unit 51 that receives the biometric data 82 of the user, the deviation evaluation unit 54 that evaluates the degree of deviation from past biometric data 82 based on the biometric data 82, the uncertainty evaluation unit 55 that evaluates the uncertainty when a subjective value indicating the subjectivity of the user is estimated based on the biometric data 82, and the deviation evaluation unit 56 that evaluates the degree of deviation indicating the degree of deviation from the user's usual state based on the deviation and the uncertainty. Furthermore, the notification determination unit 57 performs notification determination, and when the deviation data 87 exceeds the deviation threshold, a notification is output to the target.

[0092] As a result, the biomeasurement data processing device 1 of this embodiment evaluates and outputs the degree of deviation indicating the user's condition based on the uncertainty of the subjective value estimated based on the biomeasurement data 82, in addition to the degree of deviation of the biomeasurement data 82 itself, thereby obtaining the effect that the degree of deviation can be used to take measures such as intervention measures or notifications for collecting correct data while taking into account the subjective condition. EXAMPLES

[0093] Except for the differences described below, each unit of the biomeasurement data processing device 1 of the second embodiment has the same function as each unit in the first embodiment denoted by the same reference numerals, and therefore a duplicated description will be omitted.

[0094] In the second embodiment, the subjective mood related to one's own health, such as QoL and wellbeing in daily life, is used as the mental and physical subjective state to be estimated, and an intervention measure is implemented at an appropriate timing when the user is in a different subjective state than usual or is expected to be in a different subjective state in the future. If the natural feelings in daily life estimated in the first embodiment are taken as a cutout of the mental and physical subjective state at a certain point in time, the subjective mood estimated in the second embodiment is, so to speak, a higher-level mental and physical subjective state that appears as an accumulation or fluctuation of those. Therefore, in the second embodiment, the subjective value estimation model 93 is configured in multiple stages by a first subjective value estimation submodel and a second subjective value estimation submodel, the first subjective value estimation submodel is used to obtain the first subjective value estimation data as a latent variable from the biomeasurement data 82, and the second subjective value estimation submodel is used to obtain the final second subjective value estimation data using the first subjective value estimation data as a latent variable, and the finally obtained second subjective value estimation data is used as the subjective value estimation data 84, and a notification determination is made based on the degree of deviation. The second subjective value estimation data may be estimated by a plurality of types of first subjective value estimation data estimated by a plurality of first subjective value estimation sub-models. In this way, in the second embodiment, the subjective value estimation model 93 is multi-staged to estimate a higher-level mind-body subjective state, and based on the judgment result, it is possible to encourage the user.

[0095] 11A to 11C are diagrams showing an example of a display screen of the subjective value estimation result displayed on the user terminal 7. In this embodiment, when it is estimated or predicted that the user will have a subjective state different from usual, such as when a mood drop is predicted, an appropriate intervention notification based on the prediction is performed on the user. In the subjective value estimation process, an example is shown below in which biomeasurement data 82 is time-divided, an emotional state is estimated from each piece of time-divided biomeasurement data 82, and a mood is estimated from the fluctuation of the emotional state.

[0096] For example, the mood prediction index display screen 1100 shown in FIG. 11A is an example showing the basis of mood estimation for a user when natural emotions in daily life are used as explanatory variables (first subjective value estimation data) for estimating future moods. Natural emotions in daily life can be estimated by the first subjective value estimation sub-model, similar to the subjective value estimation model shown in the first embodiment. In this example, first, each emotional state estimated from each time-divided biometric data 82 is plotted on a two-dimensional graph by points 1101. The occurrence frequency of the estimated emotional state in each quadrant is shown in three stages in a radar chart 1102 superimposed on this two-dimensional plot. The occurrence frequency distribution of the estimated emotional state represents the diversity of emotional experiences (emotion diversity) in a certain time period, and the mood is estimated based on this occurrence frequency distribution.

[0097] In this example, the point cloud and the radar chart are biased toward the first quadrant, suggesting a low diversity of emotional experiences. When such a situation is detected, the emotional ups and downs seem to be monotonous, and examples of measures to improve the situation are notified as advice 1103. Furthermore, a dealt with button 1104 indicating that the user who received the notification has taken some action in response to the notification, an unaddressed button 1105 indicating that no action has been taken, and a comment field 1106 for sending the details of the action to the biomeasurement data processing device 1 as action result data 89 when action has been taken may be provided.

[0098] The first mood prediction display screen 1200 shown in FIG. 11B is an example of a display screen that estimates the future mood of the user using the subjective value estimation data, and notifies the user when a decline in the mood is predicted. In this case, in addition to an estimated sequence 1201 of the mood transition so far, a future change prediction 1203 and a mood change prediction 1204 when a certain intervention measure is taken are displayed. In addition, advice 1205 to be implemented to prevent the mood from declining is notified to the user as a notification. Furthermore, a dealt with button 1206 indicating that the user who received the notification has taken some action in response to the notification, an unaddressed button 1207 indicating that no action has been taken, and a comment field 1208 for sending the action content to the biomeasurement data processing device 1 as action result data 89 when action has been taken may be provided.

[0099] The second mood prediction display screen 1300 shown in Fig. 11C shows an example of a display screen displayed to the user when the user's future mood estimated using the subjective value estimation data is estimated to be about the same as normal. Even if no deviation from normal is detected, the user can check the estimation results on the screen to confirm the estimated current and future subjective states. In this case, since the user's condition is still good as normal, the comment field 1301 displays a message recommending that the user maintain the current condition.

[0100] In this way, the biomeasurement data processing device 1 of this embodiment has the effect of being able to take measures such as intervention measures and notifications for collecting correct data while taking into account the subjective states, even in cases where the final subjective state changes differently from normal due to changes in one or more subjective states. EXAMPLES

[0101] Except for the differences described below, each unit of the biomeasurement data processing device 1 of the third embodiment has the same function as each unit with the same reference numerals in the first or second embodiment, and therefore a duplicated description will be omitted.

[0102] In the first and second embodiments, the models used in the data preprocessing S42, the subjective value estimation process S43, the deviation evaluation process S44, the uncertainty evaluation process S45, and the deviation evaluation process S46 are either models for each user learned from the data of the user himself / herself, or models that can be commonly used by a plurality of users are learned after removing individual differences for each user in the data preprocessing S42 and the subjective value estimation process S43. However, if a model for each user is re-learned every time a new user is added, the cost of collecting data for the new users is high, and it takes a lot of time to collect a sufficient amount of biometric data 82 for learning, resulting in opportunity loss. In the third embodiment, a similar user classification process is performed to enable new users to use the system as soon as possible.

[0103] FIG. 12A is a flowchart showing an example of a learning process of similar user classification performed by the similar user classifying unit 60 of the biometric data processing device 1. First, the similar user classifying unit 60 reads the user characteristic data 90 (see FIG. 10F) and the user data 81 (see FIG. 10A) in the data reading process S71. The biometric data 82 may also be read in at the same time. The biometric data 82 read in at this time may be a small amount of data compared to the amount required to re-learn each model used in the data pre-processing S42, the subjective value estimation process S43, the deviation evaluation process S44, the uncertainty evaluation process S45, and the deviation evaluation process S46 for the user. That is, even when the biometric data 82 is read in the data reading process S71, it is not necessary to measure a large amount of biometric data 82 required for re-learning in advance from a new user, and the effect of reducing the measurement cost can be obtained.

[0104] Next, a similar user classification model learning process S72 is performed using the read data. In the similar user classification model learning process S72, a similar user classification model 98 for classifying or characterizing subjects is generated based on information on users with low measurement costs. For example, it is known that the answer tendency of the user's subjective value correct answer data 91 is influenced by the user's own personality and temperament. Therefore, as the user characteristic data 90 representing the user's characteristics, for example, personality and temperament information based on Big Five Personality, job group information, answer tendency to psychological scales such as central response tendency and extreme response tendency, and motivation for intervention can be used. In addition, user attribute information such as gender and age stored in the user data 81 may be used. In addition, biometric data 82 such as average heart rate and blood pressure information assumed to be related to the user's health consciousness, which can be characterized from a small amount of biometric data 82, may be used.

[0105] A known algorithm can be used to train the similar user classification model 98. For example, a hard clustering algorithm using the k-means method or the spectral clustering method may be used as a machine learning algorithm for classifying data without a teacher to train the similar user classification model 98 that classifies users with similar characteristics into several similar user groups. In addition, a soft clustering algorithm such as a Gaussian mixture model or a Dirichlet mixture process may be used to train the similar user classification model 98 that estimates the degree of closeness to a representative user group. After that, for each of the obtained similar user groups, models to be used in the data preprocessing S42, the subjective value estimation process S43, the deviation evaluation process S44, the uncertainty evaluation process S45, and the deviation evaluation process S46 are trained according to the flowchart of FIG. 3.

[0106] Fig. 12B is a flow chart showing an example of a similar user classification process performed by the similar user classification unit 60 of the biometric data processing device 1. The similar user classification unit 60 reads user characteristic data 90 (see Fig. 10F) and user data 81 (see Fig. 10A) in a data reading process S73. It may also read biometric data 82. Thereafter, in a similar user classification process S74, a similar user classification model 98 is read, and the read data is used to classify a new user into a similar user group, and information on the similar user group is stored in the user data 81. Since the measurement cost of each data used in the similar user classification process is low, the similar user classification process S74 can be executed early even for a new user.

[0107] Then, using the obtained information on the similar user groups, a series of evaluation processes related to subjective value estimation can be carried out in accordance with the flowchart of Figure 4, utilizing the models used in the data pre-processing S42, the subjective value estimation process S43, the deviation evaluation process S44, the uncertainty evaluation process S45, and the deviation evaluation process S46 learned for each similar user group.

[0108] As a result, even for new users, it is possible to detect early on when a new user is in a subjective state that is different from usual and to notify the user at an appropriate time, without waiting for the amount of biomeasurement data 82 required for relearning the entire model to be accumulated for that user.

[0109] 13A and 13B are diagrams showing an example of an output screen displayed on the user terminal 7. The display screen 1600 when the subjective value estimation result shown in FIG. 13A is displayed can notify that the subjective state is different from usual for the similar user group (Event #1 1601), instead of the display screen 1010 of FIG. 7B displaying the message (Event #1 1012) that the subjective state is different from usual for the user. This allows the function to be provided to new users early, and the user who is notified can understand that the estimation may contain errors because the result is not individualized only for the user. In addition, the individual difference comparison display screen 1700 shown in FIG. 11B allows the difference in characteristics for each similar user group to be confirmed. For example, when subjective answer tendencies are used as one index for classification in the classification of similar user groups, the difference in subjective answer tendencies for each similar user group may be displayed. This allows the user to judge what his or her characteristics are like in comparison with other users and similar user groups.

[0110] As described above, in the third embodiment, in order to enable each model in the first and second embodiments to be used quickly for new users, the users are classified into similar user groups using the user characteristic data 90, which requires short measurement costs and a short required time, and each model is trained for each similar user group. This provides an effect that it is possible to take measures such as intervention measures and notifications for collecting correct data quickly for new users while taking into account their subjective state.

[0111] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0112] The above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The above-mentioned configurations, functions, etc. may also be realized in software by the processor 2 interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a storage device such as the memory 3, a hard disk drive, or an SSD (Solid State Drive), or in a computer-readable non-transitory data storage medium such as an IC card, an SD card, or a DVD.

[0113] In addition, the drawings show control lines and information lines that are considered necessary for explaining the embodiments, but do not necessarily show all of the control lines and information lines included in an actual product to which the present invention is applied. In reality, it may be considered that almost all components are connected to each other. [Explanation of symbols]

[0114] 1: Biometric data processing device, 2: Processor, 3: Memory, 4: Storage device, 5: Input / output device, 6: Communication device, 7: User terminal, 8: Administrator terminal, 9: Network, 11: Biometric sensor, 12: Biometric device, 13: Input / output device, 14: Communication device, 15: Notification / alarm device, 21: Heart rate sensor, 22: Electrodermal activity sensor, 23: Acceleration sensor, 31: Input / output device, 32: Communication device, 33: Notification / alarm device, 51: Reception unit, 52: Preprocessing unit, 53: Subjective value evaluation unit, 54: Deviation evaluation unit, 55: Uncertainty evaluation unit, 56: Deviation evaluation unit, 57: Notification determination unit, 58: Notification output unit, 59: Result display unit, 60: Similar user classification unit, 81: User data, 82: Biometric data, 83: Preprocessed data data, 84: subjective value estimation data, 85: deviation data, 86: uncertainty data, 87: deviation data, 88: judgment result data, 89: correspondence result data, 90: user characteristic data, 91: subjective value correct answer data, 92: data preprocessing model, 93: subjective value estimation model, 94: deviation evaluation model, 95: uncertainty evaluation model, 96: deviation evaluation model, 97: judgment standard model, 98: similar user classification model, 100: user ID, 101: update date, 102: age, 103: gender, 104: affiliation, 105: occupation, 106: similar classification, 107: number of measurement days, 110: user ID, 111: date and time, 112: median, 113: 25% quantile, 114: 75% quantile, 115: 2.5% quantile, 116: 97.5% quantile, 120: user ID, 121: date and time, 122: deviation evaluation model, 123: similar classification, 124: deviation, 125: uncertainty, 126: deviation, 127: judged flag, 130: user ID, 131: date and time, 132: deviation evaluation model, 133: judgement standard model, 134: judgement, 135: notification status, 136: event ID, 137: response status, 140: user ID ,141: Event ID, 142: Confirmation status, 143: Response status 143, 144: Answer date and time, 145: Answer V, 146: Answer A, 150: User ID, 151: Date and time, 152: Characteristic name, 153: Characteristic 1, 154: Characteristic 2, 155: Characteristic 3, 156: Characteristic 4, 157: Characteristic 5, 160: User ID, 161: Date and time, 162: Start time, 163: End time, 164: Answer V, 1000, 1010, 1020, 1030, 1040: display screen, 1001, 1011: Signal #1, 1002, 1012: Event #1, 1003: deviation, 1013: two-dimensional graph, 1100: mood prediction index display screen, 1101: point, 1102: radar chart, 1103: advice, 1104: dealt with button, 1105: undealt with button, 1106: comment field, 1200: first mood prediction display screen, 1201: estimated sequence, 1203, 1204: change prediction, 1205: advice, 1206: dealt with button, 1207: undealt with button, 1208: comment field, 1300: second mood prediction display screen, 1301: comment field, 1600: display screen, 1601: Event #1, 1700: Individual difference comparison display screen, 2000: Dashboard, 2001: Date, 2002: Upper left table, 2003: Notification, 2004: Lower left table, 2005: Row, 2006: Contact button, 2007: Unconfirmed button, 2008: Unaddressed button, 2009: Addressing button, 2010: Addressed button, 2020: Upper right area, 2021: First continuous emotion intensity, 2022: Second continuous emotion intensity, 2023: Multiple emotion certainty, 2030: Lower right area, 2031: First emotion transition, 2032: Second emotion transition. .

Claims

1. a reception unit that receives biometric data obtained by detecting a biometric condition of a user using a sensor from a user terminal of the user; a deviation evaluation unit that evaluates a degree of deviation of the biometric data accepted by the acceptance unit with respect to past biometric data of the user; a subjective value evaluation unit that estimates a subjective value indicating a mental and physical subjective state of the user based on the biometric data received by the reception unit; an uncertainty assessment unit for assessing the uncertainty of the subjective value estimated by the subjective value assessment unit; a deviation degree evaluation unit that evaluates a deviation degree indicating a degree of deviation from the user's usual subjective mental and physical state based on the deviation degree evaluated by the deviation degree evaluation unit and the uncertainty evaluated by the uncertainty evaluation unit; A biometric data processing device comprising:

2. In claim 1, A biomeasurement data processing device comprising: a notification determination unit that makes a notification determination based on the degree of deviation evaluated by the degree of deviation evaluation unit.

3. In claim 2, The subjective value evaluation unit estimates the subjective value of the user using a subjective value estimation model, A biomeasurement data processing device characterized in that the subjective value estimation model is trained using a pair of subjective value correct answer data entered by the user into the user terminal regarding his / her mental and physical subjective state, and preprocessed data obtained by subjecting the user's biomeasurement data during the period corresponding to the subjective value correct answer data to a predetermined preprocessing.

4. In claim 3, The biomeasurement data processing device is characterized in that the notification determination unit determines whether or not to send a notification to the user terminal of the user to prompt the user to input their mental and physical subjective state based on the deviation evaluated by the deviation evaluation unit.

5. In claim 4, This biomeasurement data processing device is characterized in that, in the notification, it has a result display unit that displays the deviation evaluated by the deviation evaluation unit or the subjective value estimated by the subjective value evaluation unit on the user terminal of the user.

6. In claim 3, The predetermined pre-processing includes a correction process for removing individual differences caused by physical characteristics, and a feature extraction process from the biometric data of the user after the correction process, 2. The biometric data processing device according to claim 1, wherein the subjective value estimation model is a model that receives as input a feature quantity obtained by the feature extraction process.

7. In claim 6, The subjective value estimation model is trained using subjective value correct answer data that has been subjected to a cognitive difference correction process based on the user's cognitive difference and that is input into the user terminal by the user regarding his / her mental and physical subjective state.

8. In claim 3, the subjective value estimation model includes a first subjective value estimation sub-model and a second subjective value estimation sub-model; the first subjective value estimation sub-model estimates a first subjective value indicating a mental and physical subjective state of the user based on the biomeasurement data received by the reception unit; the second subjective value estimation sub-model estimates a second subjective value indicating a mental and physical subjective state of the user based on the first subjective value; The biometric data processing device according to claim 1, wherein the subjective value evaluation unit outputs the second subjective value as an estimated subjective value.

9. In claim 8, The biomeasurement data processing device is characterized in that the notification determination unit determines whether or not to send a notification to the user terminal of the user providing advice to improve the subjective mental and physical state based on the deviation evaluated by the deviation evaluation unit.

10. In claim 3, a similar user classification unit for estimating a group of similar users similar to the user; the deviation evaluation unit evaluates a degree of deviation of the biometric data accepted by the acceptance unit with respect to past biometric data of a group of similar users similar to the estimated user; the subjective score estimation model is trained using a pair of the subjective score correct answer data and the preprocessed data of the similar user group; The biomeasurement data processing device is characterized in that the deviation evaluation unit evaluates a deviation indicating the degree of deviation from the usual mental and physical subjective state of the similar user group based on the deviation evaluated by the deviation evaluation unit and the uncertainty evaluated by the uncertainty evaluation unit.

11. A biomeasurement data processing method for estimating a mental and physical subjective state of a user using a biomeasurement data processing device including a receiving unit, a deviation evaluation unit, a subjective value evaluation unit, an uncertainty evaluation unit, and a deviation evaluation unit, comprising: The receiving unit receives biometric data obtained by detecting a biometric condition of the user using a sensor from a user terminal of the user, the deviation evaluation unit evaluates a degree of deviation of the biometric data accepted by the acceptance unit with respect to past biometric data of the user; The subjective value evaluation unit estimates a subjective value indicating a mental and physical subjective state of the user based on the biometric data accepted by the acceptance unit, the uncertainty assessment unit assesses the uncertainty of the subjective value estimated by the subjective value assessment unit; A biomeasurement data processing method characterized in that the deviation evaluation unit evaluates a deviation indicating the degree of deviation from the user's usual mental and physical subjective state based on the deviation evaluated by the deviation evaluation unit and the uncertainty evaluated by the uncertainty evaluation unit.

12. In claim 11, The biometric data processing device includes a notification determination unit, The biomeasurement data processing method, wherein the notification determination unit makes a notification determination based on the degree of deviation evaluated by the degree of deviation evaluation unit.

13. In claim 12, The subjective value evaluation unit estimates the subjective value of the user using a subjective value estimation model, A biomeasurement data processing method characterized in that the subjective value estimation model is trained using a pair of subjective value correct answer data entered by the user into the user terminal regarding his / her mental and physical subjective state, and preprocessed data obtained by subjecting the user's biomeasurement data during the period corresponding to the subjective value correct answer data to a predetermined preprocessing.

14. In claim 13, A biomeasurement data processing method characterized in that the notification determination unit determines whether to send a notification to the user to prompt the user to input their mental and physical subjective state to the user terminal based on the deviation evaluated by the deviation evaluation unit.

15. In claim 13, the subjective value estimation model includes a first subjective value estimation sub-model and a second subjective value estimation sub-model; the first subjective value estimation sub-model estimates a first subjective value indicating a mental and physical subjective state of the user based on the biomeasurement data received by the reception unit; the second subjective value estimation sub-model estimates a second subjective value indicating a mental and physical subjective state of the user based on the first subjective value; The biometric data processing method according to claim 1, wherein the subjective value evaluation section outputs the second subjective value as an estimated subjective value.

16. In claim 15, A biomeasurement data processing method characterized in that the notification determination unit determines whether or not to send a notification to the user terminal of the user providing advice to improve the subjective mental and physical state based on the deviation evaluated by the deviation evaluation unit.

Images