Situation estimation device, situation estimation system, situation estimation method using the same, and situation estimation program

Abstract

To increase estimation precision of a user's situation using the user's biological data.SOLUTION: A situation estimation system 1 includes: a wearable device 5 for acquiring biological data of a user 3; and a situation estimation device 8 for estimating a situation of the user 3 by using a specific learned model individually provided to the user 3 on the basis of the biological data. The situation estimation device 8 includes: a model selection section 61 for selecting a specific learned model from among multiple previously learned candidate models; a model re-learning section 83 for executing re-learning of a specific learned model; a model updating section 63 for updating a specific learned model on the basis of a result of re-learning executed by the model re-learning section 83; and a situation estimation section 64 for estimating a situation of the user 3 on the basis of the most recent biological data by using a specific updated learned model. The model re-learning section 83 is constituted so as to execute re-learning by using biological data acquired when a prescribed situation occurs to the user 3 as learning data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a situation estimation device, a server device, a situation estimation system, and a situation estimation method that estimate a user's situation based on biometric data of the user acquired by a wearable device. [Background technology]

[0002] Conventionally, technologies have been developed that acquire biometric data (heart rate, pulse rate, body temperature, etc.) of workers working in factories or outdoors, and estimate the deterioration of the worker's physical condition, etc., based on the biometric data (see Patent Documents 1 and 2). These conventional technologies can ensure the safety of workers by early detection of abnormalities in the worker and issuing a warning to the worker.

[0003] On the other hand, in the above-mentioned conventional technology, changes in biometric data are judged based on predetermined criteria (e.g., past normal values), but changes in biometric data judged based on simple criteria may not allow for an accurate estimation of the worker's physical condition.

[0004] In response to this, a technique is known in which an estimation function table is used to estimate the worker's physical condition from biometric data, and the estimation function table is appropriately updated based on the estimation result of the worker's physical condition and the worker's subjective symptoms (see Patent Document 3). According to this conventional technique, the worker's physical condition is estimated using an appropriately updated estimation function table, so that if the accuracy of the estimation function table is improved, the worker's physical condition can be estimated more accurately. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-154920 A [Patent Document 2] JP 2010-218126 A [Patent Document 3] JP 2017-173899 A Summary of the Invention [Problem to be solved by the invention]

[0006] In recent years, AI (artificial intelligence) technology has been developing rapidly, and it is believed that AI technologies such as machine learning can also be applied to estimating the worker's condition (including the worker's physical condition, posture, movement state, etc.) as described in the above-mentioned Patent Documents 1-3.

[0007] However, in AI technology, generating a trained model for accurately estimating the status of a worker usually requires a large amount of training data on many workers, which increases the time and cost required for training.In addition, because the characteristics of human movements vary greatly from person to person, it may be difficult to improve the accuracy of estimating the status of a worker with a general-purpose trained model, even if a large amount of training data is used.

[0008] Therefore, the main objective of the present disclosure is to provide a situation estimation device, a server device, a situation estimation system, and a situation estimation method that can improve the accuracy of estimation when estimating a user's situation based on the user's biometric data acquired by a wearable device. [Means for solving the problem]

[0009] The situation estimation system disclosed herein is a situation estimation system that estimates a user's situation, and includes a wearable device that is worn by the user and acquires biometric data of the user, and a situation estimation device that estimates the user's situation using a specific trained model individually provided to the user based on the biometric data, wherein the situation estimation device has a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models, a model re-learning unit that performs re-learning of the specific trained model, a model updating unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit, and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model updating unit, and is configured such that when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as learning data.

[0010] In addition, the situation estimation device disclosed herein is a situation estimation device that estimates the situation of a user using a specific trained model provided individually to the user based on the user's biometric data acquired by a wearable device, and has a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models, a model re-learning unit that performs re-learning of the specific trained model, a model updating unit that updates the specific trained model based on the result of the re-learning performed by the model re-learning unit, and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model updating unit, and is configured such that when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as learning data.

[0011] In addition, the server device of the present disclosure is a server device that constitutes a situation estimation device in the situation estimation system, and is configured to have a model updating unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit, and a situation estimation unit that uses the specific trained model updated by the model updating unit to estimate the user's situation based on the latest biometric data.

[0012] In addition, the server device of the present disclosure is a server device that re-learns a specific trained model that is provided individually to a user based on the user's biometric data acquired by a wearable device, and has a candidate model memory unit that stores a plurality of pre-trained candidate models, a model selection unit that selects the specific trained model from the candidate models, and a model re-learning unit that performs re-learning of the specific trained model, and is configured such that when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as learning data.

[0013] In addition, the situation estimation method disclosed herein is a situation estimation method executed on at least one computer to estimate a user's situation using a specific trained model provided individually to the user based on the user's biometric data acquired by a wearable device, and is configured to select the specific trained model from a plurality of pre-trained candidate models, perform re-learning of the specific trained model, update the specific trained model based on the result of the re-learning, and estimate the user's situation based on the latest biometric data using the updated specific trained model, and in the re-learning of the specific trained model, when a specified situation occurs for the user, perform the re-learning using the biometric data acquired in the specified situation as learning data. Effect of the Invention

[0014] According to the present disclosure, when a user's condition is estimated based on biometric data of the user acquired by a wearable device, it is possible to improve the accuracy of the estimation. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram showing the overall configuration of a situation estimation system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a functional block diagram showing the configuration of the user device shown in FIG. 1; [Diagram 3] FIG. 2 is a functional block diagram showing the configuration of the situation estimation device shown in FIG. 1. [Figure 4] FIG. 4 is an explanatory diagram of a candidate model stored in the candidate model storage unit shown in FIG. 3. [Diagram 5] An explanatory diagram of a trained model set stored in the specific model storage unit shown in FIG. 3. [Figure 6] Flow diagram showing authentication processing when putting on and taking off a wearable device [Figure 7] An explanatory diagram showing the flow of the initial setup process [Figure 8] A diagram showing the process flow for retraining a specific trained model [Figure 9] FIG. 13 is an explanatory diagram showing an example of information related to biometric data; [Figure 10] FIG. 13 is an explanatory diagram showing an example of an inquiry screen displayed on a user terminal. [Figure 11] FIG. 4 is a diagram showing a modification of the functional block diagram shown in FIG. 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] A first invention made to solve the above problem is a situation estimation system that estimates a user's situation, comprising: a wearable device that is worn by a user and acquires biometric data of the user; and a situation estimation device that estimates the user's situation using a specific trained model individually provided to the user based on the biometric data, wherein the situation estimation device has a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models, a model re-learning unit that performs re-learning of the specific trained model, a model updating unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit, and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model updating unit, and wherein when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as learning data.

[0017] According to this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

[0018] In addition, the second invention is configured such that the plurality of candidate models include a plurality of trained models generated using biometric data previously acquired from a plurality of other users as individual training data.

[0019] This makes it possible to reduce the time and cost required to obtain a specific trained model to be provided to a target user by utilizing multiple trained models generated individually using the biometric data of multiple other users.

[0020] In addition, the third invention is configured such that the specified situation includes at least one of the user's walking, the user's body unsteadiness, the user's fall, the user's stumbling, and a specified posture of the user.

[0021] With this, re-learning is performed using biometric data acquired in specified situations (the user walking, the user's body unsteadiness, the user falling, the user stumbling, or the user's specified posture) whose characteristics vary greatly from person to person as learning data, making it possible to more reliably improve the accuracy of estimating the target user's situation.

[0022] In addition, a fourth invention is configured such that the model selection unit selects the specific trained model based on an estimation result of the user's movement by each of the candidate models based on the biometric data acquired when the user performs a predetermined movement.

[0023] According to this, a specific trained model is selected from the estimation results of the user's movements by each candidate model based on biometric data when the user performs a specified movement, thereby maintaining good estimation accuracy of the user's situation.

[0024] In addition, a fifth invention is configured such that the wearable device further has an audio output unit that outputs a guide voice to guide the user to perform the specified action, and the situation estimation device further has an audio control unit that causes the audio output unit to output the guide voice.

[0025] This makes it possible to more reliably acquire biometric data when the user is made to perform a predetermined action.

[0026] In a sixth aspect of the present invention, the predetermined motion includes a walking motion of the user.

[0027] According to this, by having the user perform walking movements suitable for selecting a specific trained model, it is possible to more appropriately select a specific trained model.

[0028] In addition, a seventh invention is configured such that the wearable device further has a voice output unit that outputs a query voice to query the user as to whether or not the specified situation has occurred, and a voice input unit that inputs a response voice of the user responding to the query voice, and the situation estimation device further has a voice control unit that causes the voice output unit to output the query voice, and whether or not the specified situation has occurred for the user is determined based on the response voice.

[0029] This makes it possible to more reliably acquire biometric data when a specified situation occurs for the user, based on the user's response voice to the inquiry voice.

[0030] In addition, an eighth invention is a configuration further comprising a user terminal used by the user, the user terminal further comprising a display unit that displays a query screen to query the user as to whether or not the specified situation has occurred, and an input unit that accepts input operations by the user in response to the query screen, and the situation estimation device further comprising a display control unit that causes the query screen to be displayed on the display unit.

[0031] This makes it possible to grasp in more detail the specific situation that the user has encountered based on the user's input operation on the inquiry screen, thereby improving the accuracy of the biometric data used for relearning.

[0032] In a ninth aspect of the present invention, the wearable device is an earphone-type device that is worn on the user's ear.

[0033] This makes it possible to reduce the influence on biometric data caused by differences in the movement characteristics of each body part among individuals.

[0034] In addition, a tenth invention is a situation estimation device that estimates the situation of a user using a specific trained model provided individually to the user based on the user's biometric data acquired by a wearable device, and includes a model selection unit that selects the specific trained model from a plurality of pre-trained candidate models, a model re-learning unit that performs re-learning of the specific trained model, a model updating unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit, and a situation estimation unit that estimates the user's situation based on the latest biometric data using the specific trained model updated by the model updating unit, and is configured such that when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as learning data.

[0035] According to this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

[0036] In addition, an eleventh invention is a server device constituting a situation estimation device in the situation estimation system, comprising: a model updating unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit; and a situation estimation unit that uses the specific trained model updated by the model updating unit to estimate the user's situation based on the latest biometric data.

[0037] This makes it possible to execute the process of estimating a user's situation using relatively few computational resources when estimating the user's situation based on the user's biometric data acquired by a wearable device.

[0038] In addition, a twelfth invention is a server device that re-learns a specific trained model that is provided individually to a user based on the user's biometric data acquired by a wearable device, and has a candidate model memory unit that stores a plurality of pre-trained candidate models, a model selection unit that selects the specific trained model from the candidate models, and a model re-learning unit that performs re-learning of the specific trained model, and is configured such that when a specified situation occurs for the user, the model re-learning unit performs the re-learning using the biometric data acquired in the specified situation as learning data.

[0039] According to this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

[0040] In addition, the 13th invention is a situation estimation method executed on at least one computer to estimate a user's situation using a specific trained model provided individually to the user based on the user's biometric data acquired by a wearable device, comprising the steps of: selecting the specific trained model from a plurality of pre-trained candidate models; performing re-learning of the specific trained model; updating the specific trained model based on the result of the re-learning; and using the updated specific trained model to estimate the user's situation based on the latest biometric data, and in re-learning the specific trained model, when a specified situation occurs for the user, performing the re-learning using the biometric data acquired in the specified situation as learning data.

[0041] According to this, when estimating a user's situation based on the user's biometric data acquired by a wearable device, a specific trained model for estimating the target user's situation is selected from multiple pre-trained candidate models, and the specific trained model is re-trained using the user's biometric data in a specified situation as training data, thereby making it possible to improve the accuracy of estimating the user's situation.

[0042] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0043] Fig. 1 is an overall configuration diagram of a situation estimation system 1 according to this embodiment. As shown in Fig. 1, the situation estimation system 1 includes a wearable device 5 and a user terminal 7 as a user-side device 4 worn or carried by a user 3. The situation estimation system 1 also includes a user management server 9 and a learning server 11 as a situation estimation device 8 that executes a process of estimating the situation of the user 3 (including physical condition, posture, motion state, etc.).

[0044] The user 3 is, for example, a worker working in a factory or outdoors, and is the target of situation estimation by the situation estimation device 8. However, the user 3 is not limited to a worker, and may be any person who needs to confirm (estimate) a situation. In this embodiment, for convenience, only one user 3 will be described, but in practice, multiple users similar to the user 3 will be the target of situation estimation. When there are multiple users, each user wears or carries a wearable device 5 and a user terminal 7.

[0045] The wearable device 5 is an earphone-type device worn on the ear of the user 3. The wearable device 5 has a function of acquiring biometric data of the user 3. The acquired biometric data includes data related to the body of the user 3, such as the heart rate, pulse, and body temperature of the user 3, as well as data related to the behavior of the user 3, such as the movement, position, and posture of the user 3. The wearable device 5 is not necessarily limited to an earphone-type device, and may be, for example, a glasses-type or wristwatch-type device. However, by using an earphone-type device as the wearable device 5 (i.e., acquiring biometric data from the head of the user 3), there is an advantage that the influence on the acquired biometric data can be suppressed even if the characteristics of the movements of the body parts other than the head (e.g., the arms) differ greatly from person to person.

[0046] The user terminal 7 is an information device with a wireless communication function, such as a smartphone or tablet terminal, carried by the user 3. The user terminal 7 can communicate with the wearable device 5 by short-range wireless communication conforming to the Bluetooth (registered trademark) standard, for example. The user terminal 7 is used to supplement functions (such as long-range communication function and display function) that are lacking in the wearable device 5 having a relatively simple configuration. Therefore, if the wearable device 5 can perform the same functions as the user terminal 7, the user terminal 7 can be omitted.

[0047] The user management server 9 manages the activities of the user 3 and the safety of the user 3 during those activities. To ensure the safety of the user 3, the user management server 9 estimates the situation of the user 3 using a trained model (hereinafter referred to as a specific trained model) that is individually provided to the user 3 based on the biometric data of the user 3. As such a specific trained model, a neural network that learns by deep learning or a statistical machine learning model such as an SVM (support vector machine) can be used.

[0048] The user management server 9 is placed in the same LAN (Local Area Network) so as to be able to communicate with the user terminal 7. This allows the user management server 9 to sequentially acquire the biometric data of the user 3 acquired by the wearable device 5 via the user terminal 7. However, the user management server 9 may be communicatively connected to the user terminal 7 via a wider network (for example, a network 13 described later).

[0049] The learning server 11 generates multiple pre-trained candidate models that are candidates for a specific trained model used by the user management server 9. Such candidate models include multiple trained models (hereinafter referred to as individual trained models) that are generated using biometric data previously acquired from multiple other users as individual training data. An individual trained model is a trained model specialized for estimating the situation of a specific user (i.e., a user from whom the biometric data used as training data was acquired).

[0050] Furthermore, when a specified situation occurs for the user 3, the learning server 11 executes re-learning of the specific trained model by using the biometric data acquired in the specified situation as learning data. Such specified situations include at least one of the walking of the user 3, the unsteadiness of the body of the user 3, the fall of the user 3, the stumbling of the user 3, and a predetermined posture of the user 3.

[0051] The learning server 11 can communicate with the user management server 9 via a network 13 such as the Internet. However, the learning server 11 may be communicatively connected to the user management server 9 via a LAN or a dedicated line.

[0052] FIG. 2 is a functional block diagram showing the configuration of the user side device 4 shown in FIG.

[0053] The wearable device 5 includes a proximity sensor 21, a pulse wave sensor 22, a motion sensor 23, and a GPS sensor 24 as sensors for acquiring biometric data of the user 3. These sensors 21-24 may be publicly known sensors. The proximity sensor 21 detects the state of proximity to the body of the user 3. The pulse wave sensor 22 detects the pulse wave of the user 3. The motion sensor 23 includes a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor, and detects the movement and posture of the user 3. The GPS sensor 24 detects the position of the user 3 based on a signal from a satellite. Note that, when the user 3 is active indoors, the wearable device 5 may detect the position of the user 3 by including a beacon signal receiving unit that receives a signal from a beacon transmitter placed at an appropriate location. When the user terminal 7 includes a GPS sensor or a beacon signal receiving unit, the position of the user 3 may be detected using the GPS sensor or the beacon signal receiving unit of the user terminal 7.

[0054] The wearable device 5 also includes a speaker (audio output unit) 27 that outputs audio, and a microphone (audio input unit) 28 that inputs the voice of the user 3 and sounds around the user.

[0055] In the wearable device 5, the communication unit 31 executes communication with another device (here, the user terminal 7) in accordance with a known wireless communication standard. The communication unit 31 has known hardware (not shown) such as an antenna, a modem, and a wireless communication circuit for communicating with other devices. However, the wearable device 5 may be connected to the user terminal 7 so as to enable wired communication according to the USB standard or the like. As will be described in detail later, the wearable device 5 transmits biometric data detected by the sensors 21-24, voice input to the microphone 28, and the like to the user terminal 7. These data are further transmitted (transferred) from the user terminal 7 to the user management server 9.

[0056] Furthermore, in the wearable device 5, the control unit 33 comprehensively controls each unit of the wearable device 5. The control unit 33 has a user authentication unit 34 that performs a process of authenticating the user 3 who wears the wearable device 5. The control unit 33 includes one or more processors as hardware. The functions of the control unit 33, including the processing of the user authentication unit 34, are realized by the processor executing a program stored in a memory (not shown).

[0057] The user terminal 7 includes a display unit 41 that displays information including characters and figures to the user 3, and an input unit 42 that accepts input operations of data and information by the user 3. The display unit 41 and the input unit 42 are realized by, for example, a known liquid crystal panel and touch panel.

[0058] In the user terminal 7, the communication unit 45 communicates with other devices (here, the wearable device 5 and the user management server 9) in accordance with a known wireless communication standard. The communication unit 45 has known hardware (not shown) such as an antenna, a modem, and a wireless communication circuit for communicating with other devices. As will be described in detail later, the user terminal 7 transmits (transfers) a control command (audio output command) from the user management server 9 to the wearable device 5 in order to output a predetermined sound from the speaker 27.

[0059] In addition, in the user terminal 7, the control unit 47 comprehensively controls each unit of the wearable device 5. Furthermore, the control unit 47 executes a process of estimating the situation of the user 3 in cooperation with the user management server 9 using a pre-installed user management application 49. The control unit 47 includes one or more processors as hardware. The functions of the control unit 47, including the user management application 49, are realized by the processor executing a program stored in a memory (not shown).

[0060] Fig. 3 is a functional block diagram showing the configuration of the situation estimation device 8 shown in Fig. 1. Fig. 4 is an explanatory diagram of a candidate model stored in the candidate model storage unit 55 shown in Fig. 3. Fig. 5 is an explanatory diagram of a trained model set stored in the specific model storage unit 57 shown in Fig. 3. Fig. 11 shows a modified example of the functional block diagram shown in Fig. 3.

[0061] The user management server 9 is a server device having a known hardware configuration. The user management server 9 includes a communication unit 51, a control unit 53, a candidate model storage unit 55, a specific model storage unit 57, and a biological data storage unit 59.

[0062] The communication unit 51 communicates with other devices (here, the user terminal 7 and the learning server 11) in accordance with a known communication standard. The communication unit 51 has known hardware (not shown) such as an antenna, a modem, and a wireless communication circuit for communicating with other devices. As will be described in detail later, the user management server 9 transmits to the learning server 11 learning data for generating an individual trained model and learning data for re-learning a specific trained model.

[0063] The control unit 53 comprehensively controls each unit of the user management server 9. The control unit 53 further includes a model selection unit 61, a learning data extraction unit 62, a model update unit 63, a situation estimation unit 64, a voice control unit 65, and a display control unit 66.

[0064] The model selection unit 61 selects a specific trained model to be individually provided to the user 3 from among multiple candidate models prepared in advance. The candidate model storage unit 55 stores multiple candidate models in advance (see pre-trained models (1), (2), (3), etc. in FIG. 4). The multiple candidate models are composed of individual trained models.

[0065] The specific trained models selected by the model selection unit 61 are associated with a specific user (here, user 3) and sequentially stored in the specific model storage unit 57. Usually, each specific trained model is used only for estimating the situation of the associated single user.

[0066] However, multiple specific trained models may be selected for one user 3. In that case, for example, as shown in Fig. 5, a set of trained models is stored in the specific model storage unit 57 as specific trained models for estimating the situation of, for example, user A. The set of trained models includes, for example, a trained model 67 including the walking detection pattern of user A as a parameter, a trained model 68 including the body sway detection pattern of user A as a parameter, and a trained model 69 including the fall detection pattern of user A as a parameter (the same applies to other users B, C, etc.).

[0067] The biometric data of the user 3 acquired by the wearable device 5 is sequentially stored in the biometric data storage unit 59. The learning data extraction unit 62 extracts a part of the biometric data of the user 3 stored in the biometric data storage unit 59 as learning data for re-learning a specific trained model. More specifically, when the learning data extraction unit 62 determines that a specified situation (such as the user 3's body unsteadiness) has occurred in the user 3, it extracts a part of the stored biometric data acquired in the specified situation as learning data. In addition, the learning data extraction unit 62 can also extract a part of the biometric data of the user 3 as learning data for generating an individual trained model in the learning server 11.

[0068] The model update unit 63 appropriately updates the specific trained model stored in the specific model storage unit 57 based on the result of re-learning of the specific trained model executed by the learning server 11.

[0069] The situation estimation unit 64 uses the specific trained model to estimate the situation of the user 3 based on biometric data obtained in real time from the user 3 (i.e., the latest biometric data). When the specific trained model is updated by the model update unit 63, the situation estimation unit 64 estimates the situation of the user 3 using the updated specific trained model. However, when starting to newly estimate the user's situation, the situation estimation unit 64 estimates the situation of the user 3 using the candidate model selected by the model selection unit 61 (i.e., a specific trained model that has not been re-trained).

[0070] The voice control unit 65 transmits a voice output command to the user terminal 7 to output a predetermined voice from the speaker 27 of the wearable device 5. The voice output from the speaker 27 includes a guide voice for making the user 3 execute a predetermined action when the model selection unit 61 selects a specific trained model. The voice output from the speaker 27 also includes an inquiry voice for inquiring of the user 3 as to whether or not a specified situation (such as unsteadiness of the user 3's body) has occurred.

[0071] The display control unit 66 transmits a screen display command to the user terminal 7 to display a predetermined screen on the display unit 41 of the user terminal 7. Such a screen includes an inquiry screen for the user management server 9 to inquire about the situation of the user 3. If the wearable device 5 has a display unit (such as a liquid crystal display or a projector), the display control unit 66 may display a predetermined screen on the display unit of the wearable device 5 instead of the display unit 41 of the user terminal 7 (or together with the display unit 41 of the user terminal 7).

[0072] The learning server 11 is a server device having a known hardware configuration. The user management server 9 includes a communication unit 71, a control unit 73, a pre-trained model storage unit 75, a re-trained model storage unit 77, and a learning data storage unit 79. The learning server 11 appropriately provides the user management server 9 with candidate models used by the user management server 9 and specific re-trained trained models.

[0073] The communication unit 71 executes communication with another device (here, the user management server 9) in accordance with a known communication standard. The communication unit 71, like the communication unit 51 of the user management server 9, has known hardware.

[0074] The control unit 73 comprehensively controls each unit of the learning server 11. The control unit 73 further includes a model generation unit 81, a model re-learning unit 83, and a model distribution unit 85.

[0075] The model generation unit 81 generates a plurality of individual trained models by using biometric data acquired from a plurality of users other than the user 3 as individual training data. The trained models generated by the model generation unit 81 are sequentially stored in the pre-trained model storage unit 75. However, the pre-trained model storage unit 75 may include a general-purpose trained model. The general-purpose trained model may be generated, for example, by using a plurality of biometric data corresponding to a specified situation, which is collected in advance as sample data according to the purpose, as training data.

[0076] The model re-learning unit 83 executes re-learning of the corresponding specific trained model based on the training data extracted by the user management server 9 (training data extraction unit 62) and transmitted to the training server 11. The re-learned models are sequentially stored in the re-learned model storage unit 77. The re-learned model storage unit 77 may store the specific trained model to be re-learned (i.e., before re-learning). The model re-learning unit 83 can execute re-learning of the same specific trained model (i.e., associated with the same user) every time new training data is acquired. As another variation, the training data received by the training server 11 can be stored in the training data storage unit 79 in the training server 11 and used as appropriate as data for re-learning executed by the model re-learning unit 83. The timing of transmission of the learning data extracted by the learning data extraction unit 62 may be manually input / instructed by the user at any timing using an application (not shown) of the user management server, or the learning data extracted by the learning data extraction unit 62 may be temporarily stored in a recording area of ​​the user management server 9, and the temporarily stored learning data may be automatically transmitted at a predetermined interval.

[0077] The model distribution unit 85 appropriately transmits the initial trained models (individual trained models, general-purpose trained models) stored in the pre-trained model storage unit 75 to the user management server 9 as candidate models. In addition, when re-learning of a specific trained model is completed, the model distribution unit 85 appropriately transmits the re-trained specific trained model to the user management server 9 as a fixed trained model for update.

[0078] In the situation estimation system 1, when the user management server 9 has sufficient computational resources, it can be configured to have at least a part of the functions of the learning server 11. Conversely, when the user management server 9 has insufficient computational resources, it can be configured to have a part of the functions of the learning server 11. For example, as shown in FIG. 11, instead of the user management server 9, the learning server 11 may have the functions of a candidate model storage unit 55 and a model selection unit 61. In some cases, the situation estimation device 8 may be realized as a single device having the functions of both the user management server 9 and the learning server 11. Furthermore, the situation estimation device 8 can be configured to further include other devices (such as a server or a PC) that can execute some of the functions of the user management server 9 and the learning server 11 in addition to the user management server 9 and the learning server 11. The situation estimation device 8 is not limited to a server device, and can be configured by at least one computer having similar functions. It is also possible to improve security against cyber attacks on the network by configuring the processing of the situation estimation unit 64 and the model re-learning unit 83 to be secret sharing processing by multiple servers. These functional layouts can be appropriately designed and implemented depending on the resources (CPU power, memory, storage capacity, etc.) of the user management server 9 and the learning server 11, the network communication speed and environment between the user management server 9 and the learning server 11, and the security level required for the situation assessment system.

[0079] FIG. 6 is a flow diagram showing authentication processing when wearable device 5 is put on and taken off.

[0080] The wearable device 5 acquires biometric data only of the specific user with which it is associated. Therefore, the wearable device 5 performs authentication processing for the wearer when the wearable device 5 is worn and removed, as shown in FIG.

[0081] First, when the user 3 puts on the wearable device 5 when starting an activity, the wearable device 5 detects by the proximity sensor 21 that the wearable device 5 is put on by the user 3 (ST101: Yes).

[0082] Next, the wearable device 5 makes an authentication request to the user 3 (ST102). In the authentication request, for example, a voice is output from the speaker 27 to prompt the user 3 to speak. Then, the user authentication unit 34 executes a voiceprint authentication process on the user's voice input to the microphone 28 in response to the voice.

[0083] Thereafter, when the user 3 is successfully authenticated (ST103: Yes), the wearable device 5 enters a continuous authentication state (ST104).

[0084] At the same time, the wearable device 5 transmits an authentication completion notification to the user terminal 7 to inform the user 3 that the user 3 has been successfully authenticated (ST105). The user terminal 7 that has received the authentication completion notification displays an authenticated screen on the display unit 41 indicating that the user 3 has been successfully authenticated. Note that the wearable device 5 may output a sound from the speaker 27 to inform the user 3 that the user 3 has been successfully authenticated.

[0085] Thereafter, when the user 3 has finished his / her activity and removes the wearable device 5, the wearable device 5 detects through the proximity sensor 21 that the wearable device 5 has been removed from the user 3 (ST106: Yes). This causes the wearable device 5 to enter a de-authentication state (ST107) and stops acquiring biometric data from the user 3.

[0086] In addition, in the above step ST101, the wearable device 5 may use the motion sensor 23 instead of (or together with) the proximity sensor 21 to detect that the wearable device 5 is worn by the user 3. In addition, in the above step ST102, the wearable device 5 may authenticate the user 3 using a method other than voiceprint authentication (for example, a biometric authentication process based on the characteristics of the movement of the user 3 detected by the motion sensor 23, or a keyword authentication process by voice utterance and voice recognition using a prearranged password, etc.).

[0087] 7 is an explanatory diagram showing the flow of an initial setup process when the user 3 starts using the situation estimation system 1. The initial setup process is a process for selecting a candidate model that is most suitable for the user 3 from multiple candidate models as a specific trained model.

[0088] The user management server 9 starts an initial setup process (1001) in response to a setup request from the user terminal 7 (user management application 49) based on an input operation by the user 3. Then, the user management server 9 transmits an action execution command to the wearable device 5 via the user terminal 7 to cause the user 3 to execute a predetermined action (1003).

[0089] The wearable device 5, which has received the action execution command, outputs a guidance voice from the speaker 27 to make the user 3 execute a predetermined action (1005). Data of the guidance voice is stored in the wearable device 5 in advance, or is included in the action execution command transmitted from the user management server 9 to the wearable device 5. The guidance voice includes, for example, voices such as "Turn right and walk five steps," and "Go up the stairs." As a result, the user 3 executes a predetermined action (for example, turn right and walk five steps) in accordance with the guidance voice.

[0090] The wearable device 5 acquires biometric data of the user 3 performing the predetermined action (1007). Furthermore, the wearable device 5 transmits the acquired biometric data to the user management server 9 via the user terminal 7 (1009). In this way, the user management server 9 can more reliably acquire biometric data when the user 3 is made to perform the predetermined action. Note that by including a characteristic action (e.g., a jumping action) at the start and end of the predetermined action by the action execution command, the user management server 9 can easily recognize the start and end of the predetermined action in the biometric data acquired as time-series data.

[0091] When the user management server 9 acquires the biometric data of the user 3 from the wearable device 5 when the user 3 performs a predetermined motion, the user management server 9 executes a process of selecting a specific trained model based on the biometric data (1011). More specifically, the model selection unit 61 uses a plurality of candidate models stored in the candidate model storage unit 55 to calculate the estimation accuracy (%) of the user 3 performing a predetermined motion (e.g., a walking motion of turning right and walking five steps) from the biometric data of the user 3, and selects the candidate model with the highest estimation accuracy as the specific trained model.

[0092] Thereafter, the user management server 9 notifies the user terminal 7 of an initial setup completion notification including information on the selected specific trained model (1013). In the user terminal 7, the information on the selected specific trained model is registered by the user management application 49 (1015). Note that the information on the selected specific trained model is not registered in the user terminal 7, and the user management server 9 manages the association between the information on the selected specific trained model and the ID of the user 3 (or the ID of the user terminal 7), making it possible to apply the selected specific trained model to the user 3.

[0093] Fig. 8 is an explanatory diagram showing a process flow of re-learning a specific trained model in the situation estimation system 1. Fig. 9 is an explanatory diagram showing an example of information related to biometric data. Fig. 10 is an explanatory diagram showing an example of an inquiry screen displayed on the user terminal 7.

[0094] 8, the wearable device 5, after completing the initial setup, continuously acquires the biometric data of the user 3 during his / her activity at a predetermined period (which may be the sampling period of the sensors such as the pulse wave sensor 22 and the motion sensor 23, or may be a period in which the sampling period is appropriately thinned to reduce the battery power of the wearable device) (2001). The acquired biometric data is transmitted from the wearable device 5 to the user management server 9 via the user terminal 7 (2003). The user management server 9 sequentially stores information related to the biometric data received from the wearable device 5 in the biometric data storage unit 59 (2005).

[0095] The information on the biometric data stored in the biometric data storage unit 59 includes, for example, as shown in Fig. 9, "sensor data" which is a sensor detection value as biometric data, "location information" which indicates the location of the user 3 when the data was detected, as well as related information such as a "user ID" for identifying the user 3, the "date" and "time" when the sensor data was detected, and a "prescribed situation detection result" which indicates that a prescribed situation has occurred with respect to the sensor data. The sensor data may include not only data detected at a prescribed time, but also data detected during a prescribed period (e.g., waveform data). The "prescribed situation detection result" is not included in the initial biometric data information stored in the biometric data storage unit 59, and is added as appropriate when a prescribed situation occurs.

[0096] The user management server 9 estimates the status of the user 3 using the specific trained model based on the biometric data received from the wearable device 5 (2007). If the user management server 9 determines as a result of the estimation that the status of the user 3 does not correspond to the specified status (or that it is unlikely to correspond), it waits for the next reception of biometric data from the wearable device 5.

[0097] On the other hand, when the user management server 9 determines from the estimation result of the user 3's situation that a specified situation (such as the user 3's body unsteadiness) has occurred, it transmits a voice output command to the wearable device 5 via the user terminal 7 (2009).

[0098] The wearable device 5, which has received the voice output command, outputs a voice inquiry about the situation of the user 3 from the speaker 27 (2011). The voice inquiry includes, for example, voice inquiring about the situation of the user 3, such as "Are you OK?" or "Did something happen?". The voice inquiry may also include advice about managing the user 3's physical condition, such as "It's hot, so please take a break and drink some fluids."

[0099] When the user 3 responds to the voice inquiry by speaking, the response voice of the user 3 is input to the microphone 28, and the response voice data is transmitted to the user management server 9 via the user terminal 7 (2013). The voice inquiry includes, for example, voices such as "I feel a bit dizzy" and "I'm fine."

[0100] The control unit 53 of the user management server 9 receives the data of the response voice and estimates the content of the response voice by voice recognition processing (2015). Next, when the user management server 9 determines from the estimation result that the specified situation has occurred for the user 3, it transmits a screen display command to display a query screen on the display unit 41 of the user terminal 7 (2017). At this time, the user management server 9 adds the result of estimating the content of the response voice (for example, that the user 3 has become unsteady) to the information on the corresponding biometric data (see FIG. 9) as a specified situation detection result. In this way, the user management server 9 can more reliably acquire biometric data when the specified situation occurs for the user 3, based on the response voice of the user 3 to the query voice.

[0101] Based on a screen display command from the user management server 9, the user terminal 7 displays an inquiry screen 91 for checking the status of the user 3 in more detail (2019). The inquiry screen displays confirmation items related to the motion (here, the body sway) estimated based on the response voice of the user 3, as shown in FIG. 10, for example. Here, as confirmation items related to the body sway of the user 3, options such as swaying during "normal walking", swaying during "hurry walking", swaying while climbing "stairs", "falling" after swaying, swaying due to "stumbling", and "other" items are displayed. The result of the input operation (e.g., selection of an option) of the user 3 on the inquiry screen 91 is transmitted to the user management server 9 as response data (2021).

[0102] The user management server 9, which has received the response data from the user terminal 7, determines the situation of the user 3 based on the response data, and adds the situation of the user 3 (for example, that the user 3 tripped while climbing stairs) to the corresponding biometric data information (see FIG. 9) as a specified situation detection result (2023). In this way, the user management server 9 can grasp the specified situation that occurred to the user 3 in more detail based on the input operation of the user 3 on the inquiry screen, thereby improving the accuracy of the biometric data used for relearning.

[0103] The user management server 9 can add a specified situation detection result of the corresponding biometric data information (see FIG. 9 ) based on at least one of the response voice from the wearable device 5 and the response data from the user terminal 7. In other words, at least one of the process for acquiring the response voice from the wearable device 5 and the process for acquiring the response data from the user terminal 7 may be omitted.

[0104] The user management server 9 appropriately extracts the biometric data with the specified situation detection result as learning data and transmits it to the learning server 11 (2025). The learning server 11 re-learns the specific trained model based on the received learning data (2027). At this time, the specified situation detection result is used as teacher data.

[0105] The learning server 11 transmits the specific trained model for which re-learning has been completed to the user management server 9 as information regarding the result of the re-learning (2029). The user management server 9 updates the specific trained model currently being used with the re-learned specific trained model (2031).

[0106] In addition, when the specific trained model includes multiple trained models (see FIG. 5), the re-learning of the specific trained model may be performed for each trained model. In addition, the re-learning in the training server 11 may be performed every time training data is received from the user management server 9. Alternatively, the frequency of occurrence of a case where the estimation result of the user 3's situation by the specific trained model applied to the user 3 differs from the specified situation detection result (the user's actual situation determined by the response data of the user 3) (estimation error) is measured, and the re-learning process is performed when the occurrence frequency exceeds a predetermined criterion, thereby eliminating the execution of unnecessary re-learning processes and performing re-learning only when the accuracy of the situation estimation is deteriorated. In this case, the same effect can be obtained by determining the estimation error on the user management server 9 side, transmitting training data to the training server 11 when the occurrence frequency exceeds a predetermined criterion, and performing re-learning on the training server 11 side every time training data is received.

[0107] In this way, the situation estimation system 1 selects a specific trained model for estimating the situation of the target user 3 from multiple pre-trained candidate models, and re-trains the specific trained model using the biometric data of the user 3 in a specified situation as learning data, thereby making it possible to improve the accuracy of estimating the situation of the user 3.

[0108] In addition, the situation estimation system 1 can reduce the time and cost required to obtain a specific trained model to be provided to the target user 3 by using multiple trained models generated individually using the biometric data of multiple other users as candidate models.

[0109] While the present disclosure has been described above based on specific embodiments, these embodiments are merely examples, and the present disclosure is not limited to these embodiments.

[0110] For example, the situation estimation system 1 can be configured to omit the user management server 9 and have the user terminal 7 function as the user management server 9. Furthermore, the situation estimation system 1 can not only manage the safety of the active user 3, but also manage the work of the user 3 based on, for example, an estimation of the walking status of the user 3.

[0111] The components of the situation estimation device, server device, situation estimation system, and situation estimation method according to the present disclosure shown in the above embodiments are not necessarily all essential, and can be selected as appropriate at least as long as they do not deviate from the scope of the present invention. [Explanation of symbols]

[0112] 1: Situation estimation system 3: User 4: User side device 5: Wearable devices 7: User terminal 8: Situation estimation device 9: User management server 11: Learning server 13: Network 21: Proximity sensor 22: Pulse wave sensor 23: Motion sensor 24: GPS sensor 27: Speaker 28: Microphone 31: Communications Department 33: Control unit 34: User authentication section 41: Display section 42: Input section 45: Communications Department 47: Control unit 49: User management application 51: Communications Department 53: Control unit 55: Candidate model storage unit 57: Specific model memory section 59: Biometric data storage unit 61: Model selection section 62: Learning data extraction unit 63: Model Update Section 64: Situation Estimation Department 65: Audio control section 66: Display control unit 71: Communications Department 73: Control unit 75: Pre-trained model memory 77: Retrained model memory section 79: Learning data storage unit 81: Model generation section 83: Model retraining section 85: Model Distribution Department 91: Inquiry screen

Claims

1. A situation estimation system for estimating a user's situation, comprising: A device for acquiring biometric information of the user; a situation estimation device that estimates a situation of the user by using a specific trained model that is individually provided to the user based on the biometric information, The situation estimation device includes: A model selection unit that selects the specific trained model from a plurality of pre-trained candidate models; A model re-learning unit that performs re-learning of the specific trained model; A model update unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit; A situation estimation unit that estimates a situation of the user based on the latest biometric information by using the specific trained model updated by the model update unit. Situation estimation system.

2. The device comprises: At least one of an earphone-type wearable device, a glasses-type wearable device, and a wristwatch-type wearable device. The situation estimation system according to claim 1 .

3. The device is worn or carried by the user. The situation estimation system according to claim 1 .

4. the model re-learning unit, when a specified situation occurs for the user, executes the re-learning by using the biometric information acquired in the specified situation as learning data. The situation estimation system according to claim 1 .

5. The device comprises: a display unit that displays an inquiry screen for inquiring of the user as to whether or not a specified situation has occurred; an input unit that accepts an input operation by the user in response to the inquiry screen; Further comprising: The situation estimation device includes: Further comprising a display control unit that causes the display unit to display the inquiry screen. The situation estimation system according to claim 1 .

6. a display unit that displays an inquiry screen for inquiring of the user as to whether or not a specified situation has occurred; an input unit that accepts an input operation by the user in response to the inquiry screen; A display device having The situation estimation system according to claim 1 .

7. A situation estimation method for a situation estimation system that estimates a user's situation, comprising: The situation estimation system includes a device and a situation estimation apparatus, The device acquires biometric information of the user, the situation estimation device estimates the user's situation based on the biometric information by using a specific trained model individually provided to the user; Selecting the specific trained model from among a plurality of pre-trained candidate models; Retraining the specific trained model; updating the specific trained model based on the results of the re-training performed; Using the updated specific trained model, a state of the user is estimated based on the latest biometric information. How to estimate the situation.

8. A situation estimation device that estimates a situation of each user using a specific trained model individually provided to the user based on biometric information of the user, A model selection unit that selects the specific trained model from a plurality of pre-trained candidate models; A model re-learning unit that performs re-learning of the specific trained model; A model update unit that updates the specific trained model based on a result of the re-learning performed by the model re-learning unit; a situation estimation unit that estimates a situation of the user based on the latest biometric information by using the specific trained model updated by the model update unit; having Situation estimation device.

9. A situation estimation method for estimating a situation of a user by a situation estimation device, the situation estimation device estimating a situation of the user by using a specific trained model individually provided to the user based on biometric information of the user, Selecting the specific trained model from among a plurality of pre-trained candidate models; Retraining the specific trained model; updating the specific trained model based on the results of the re-training; Using the updated specific trained model, a state of the user is estimated based on the latest biometric information. How to estimate the situation.

10. A situation estimation program that causes a computer to execute information processing to estimate a situation of each user using a specific trained model individually provided to the user based on biometric information of the user, The information processing includes: The computer, Selecting the specific trained model from among a plurality of pre-trained candidate models; Retraining the specific trained model; updating the specific trained model based on the results of the re-training; A situation estimation program comprising a step of estimating the user's situation based on the latest biometric information using the updated specific trained model.

Images