Vital information management device, vital information management system, method for managing vital information

Abstract

To efficiently connect the estimated value of vital information and a user to each other and determine whether the vital information of a target person is to be measured by a medical apparatus.SOLUTION: A video of the face of a user is taken by a built-in camera (S1) and the vital data of the user is estimated by RPPG (S2). For example, it is estimated that the heart beat is 70 / min and the blood pressure is 163 / 98 mmHg. When the estimated vital data is off standard values, a measurement using a medical apparatus is promoted (S3). For example, the message that "the blood pressure is off the standard values. Do you want to make a measurement again by a medical apparatus?" is displayed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a vital information management device, a vital information management system, and a vital information management method that can efficiently link estimated values ​​of vital information with a user and recommend whether or not the subject's vital information should be measured by a medical device. [Background technology]

[0002] Conventionally, a technology called remote photoplethysmography (hereinafter referred to as "RPPG") is known that estimates heart rate by analyzing changes in skin color caused by blood flow. For example, Patent Document 1 discloses a technology that, upon receiving a sequence of measured values ​​of the luminance of various areas of a person's skin, determines the frequency coefficient of a photoplethysmography waveform corresponding to the measured luminance.

[0003] By using such RPPG technology, it is possible to estimate vital information such as pulse rate, blood oxygen level (hereinafter referred to as "SpO2"), blood pressure, and body temperature using a person's facial image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2020-527088 Summary of the Invention [Problem to be solved by the invention]

[0005] However, for example, in a care setting where there are many people requiring care, if vital sign information is obtained using an RPPG and this vital sign information must be linked to the people requiring care manually, there is a risk that someone else's vital sign information will be linked to the person requiring care.

[0006] In addition, because the blood pressure obtained by RPPG is not actually measured using medical equipment, there are cases where vital information should be measured using medical equipment, but it is difficult to determine whether or not vital information should be measured using medical equipment in a nursing care setting.

[0007] The present invention has been made to solve the problems of the above-mentioned conventional technology, and aims to provide a vital information management device, a vital information management system, and a vital information management method that can efficiently link estimated values ​​of vital information with users and recommend whether or not a subject's vital information should be measured with a medical device. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention provides a vital information management device having an imaging unit that captures a facial image of a user and an operation reception unit that receives a predetermined operation to obtain vital information, characterized in that the device is equipped with an estimation means that estimates vital information based on the facial image captured by the imaging unit, a face authentication means that performs face authentication of the facial image captured by the imaging unit, and a memory means that, when the predetermined operation is received by the operation reception unit, stores the vital information estimated by the estimation means in correspondence with identification information of the user authenticated by the face authentication means.

[0009] Furthermore, in the above invention, the present invention is characterized in that the estimation means estimates vital information including pulse, blood oxygen concentration, blood pressure and / or body temperature from a heart rate obtained by applying remote photoplethysmography to the face image.

[0010] Further, in the above-mentioned invention, the present invention is characterized in that the estimation means estimates the vital information using a vital information estimation model trained using heart rate and ground truth data obtained by applying remote photoplethysmography to a plurality of face images as training data.

[0011] In addition, the present invention is characterized in that, in the above invention, it further comprises a first display control means for controlling the display of recommendation information recommending measurement of the user's vital information using a specified medical device on a specified display unit based on the vital information estimated by the estimation means.

[0012] In addition, the present invention is characterized in that, in the above invention, it further comprises a health degree estimation means for estimating a health degree using a health state estimation model trained using a plurality of facial images and correct answer data as training data, and further comprises a second display control means for controlling the display of recommendation information recommending measurement of the user's vital information using a specified medical device on a specified display unit based on the vital information estimated by the estimation means and the health degree estimated by the health degree estimation means.

[0013] In addition, the present invention, in the above invention, further comprises a health degree estimation means for estimating a health degree using a health state estimation model trained using a plurality of facial images, voice data, and correct answer data as training data, and a voice collection means for collecting voice information of the user, and a third display control means for controlling the display of recommendation information recommending measurement of the vital information of the user using a specified medical device on a specified display unit based on the vital information estimated by the estimation means and the health degree estimated by the health degree estimation means.

[0014] The present invention also provides a vital information management system having a terminal device equipped with an imaging unit that captures a facial image of a user and an operation receiving unit that receives a predetermined operation to obtain vital information, and a server device capable of communicating with the terminal device, wherein the server device comprises an estimation means for estimating vital information based on the facial image captured by the imaging unit, a face authentication means for performing face authentication of the facial image captured by the imaging unit, and a storage means for storing the vital information estimated by the estimation means in correspondence with identification information of the user authenticated by the face authentication means when a notification is received from the terminal device that the predetermined operation has been accepted.

[0015] The present invention also provides a vital information management method executed by a vital information management device having an imaging unit that captures a facial image of a user and an operation reception unit that receives a predetermined operation to obtain vital information, and is characterized in including an estimation process of estimating vital information based on the facial image captured by the imaging unit, a face authentication process that performs face authentication on the facial image captured by the imaging unit, and a storage process that, when the predetermined operation is received by the operation reception unit, stores the vital information estimated by the estimation process in a predetermined storage unit in association with identification information of the user authenticated in the face authentication process. Effect of the Invention

[0016] According to the present invention, it is possible to efficiently link estimated values ​​of vital information with users, and to recommend whether or not the subject's vital information should be measured with a medical device. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an overview of a vital information management device according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram showing the configuration of the vital information management device shown in FIG. [Diagram 3] FIG. 3 is a diagram showing an example of the authentication data, the standard data, and the vital data shown in FIG. [Figure 4] FIG. 4 is an explanatory diagram of an overview of the photoplethysmography method according to the first embodiment. [Diagram 5] FIG. 5 is a diagram (part 1) showing an example of a display in the vital sign information management device according to the first embodiment. [Figure 6] FIG. 6 is a diagram (part 2) showing an example of a display in the vital sign information management device according to the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating a processing procedure in the vital information management device according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an overview of a vital information management device according to the second embodiment. [Figure 9] FIG. 9 is a functional block diagram showing the configuration of the vital information management device shown in FIG. [Figure 10] FIG. 10 is a flowchart showing a processing procedure for generating a health state estimation model according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an overview of a vital sign information management system according to the third embodiment. [Figure 12] FIG. 12 is a diagram illustrating an overview of a vital sign information management system according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] [Embodiment 1] Hereinafter, a vital information management device, a vital information management system, and a vital information management method according to the first embodiment will be described in detail with reference to the drawings.

[0019] <Outline of vital information management device 10 according to embodiment 1> First, an overview of the vital information management device 10 according to the present embodiment 1 will be described. FIG. 1 is an explanatory diagram for explaining the overview of the vital information management device 10 according to the present embodiment 1. The vital information management device 10 according to the present embodiment 1 captures a moving image of the face of a user with a built-in camera (S1), and estimates the vital data of the user by RPPG (S2). For example, it estimates that the heart rate is 70 beats / min and the blood pressure is 163 / 98 mmHg.

[0020] If the estimated vital data is outside the standard values, the system prompts the user to measure it using a medical device (S3). For example, the system displays a message saying, "Your blood pressure is outside the standard values. Would you like to measure it again using a medical device?"

[0021] In this way, the vital information management device of this embodiment 1 is configured to prompt measurement with a medical device if the vital data estimated using a captured video of the face deviates from the standard value, so that the estimated vital information can be efficiently linked to the user and a recommendation can be made as to whether or not the subject's vital information should be measured with a medical device.

[0022] <Configuration of vital information management device 10> Next, a description will be given of the configuration of the vital information management device 10. Fig. 2 is a functional block diagram showing the configuration of the vital information management device 10 shown in Fig. 1. As shown in Fig. 2, the vital information management device 10 has a display operation unit 11, a camera 12, a speaker 13, a communication unit 14, a storage unit 15, and a control unit 16.

[0023] The display operation unit 11 is an input / output device such as a touch panel display. The camera 12 is an input device for taking a video of the user's face. The speaker 13 is an output device for generating a sound to alert the user. The communication unit 14 is an interface unit for data communication with the Internet.

[0024] The storage unit 15 is a storage device formed of a non-volatile memory or the like, and stores authentication data 15a, standard data 15b, face video data 15c, and vital data 15d.

[0025] Authentication data 15a is data used for authenticating a user. Standard data 15b is data indicating standard values ​​of vital data. Facial video data 15c is video data obtained by capturing a face of the user. Vital data 15d is data indicating vital data estimated using facial video data 15c.

[0026] The control unit 16 is a control unit that performs overall control of the vital information management device 10, and has an authentication registration unit 16a, a setting unit 16b, an authentication unit 16c, a face moving image acquisition unit 16d, a vital data estimation unit 16e, a determination unit 16f, and a vital data acquisition unit 16g. In practice, these programs are loaded into a CPU (Central Processing Unit) and executed, causing the authentication registration unit 16a, the setting unit 16b, the authentication unit 16c, the face moving image acquisition unit 16d, the vital data estimation unit 16e, the determination unit 16f, and the vital data acquisition unit 16g to execute processes corresponding thereto.

[0027] The authentication registration unit 16a is a processing unit that manages the authentication data 15a. When the authentication registration unit 16a receives a face image as authentication data from the camera 12 and subsequently receives name data from the display operation unit 11, the authentication registration unit 16a generates a unique identification ID and stores the identification ID in the authentication data 15a in association with each of the received data.

[0028] The setting unit 16b is a processing unit that manages the standard data 15b. When the setting unit 16b receives standard value data of the vital data from the display operation unit 11, the setting unit 16b stores the received data in the standard data 15b.

[0029] The authentication unit 16c is a processing unit that performs authentication processing. When the authentication unit 16c receives an authentication operation from the display operation unit 11, the authentication unit 16c takes a picture of the face of the user with the camera 12 and performs authentication using the captured image data and the authentication data 15a. Then, if the authentication is successful, the authentication unit 16c specifies an identification ID corresponding to the authenticated image data from the authentication data 15a and displays an RPPG reception screen on the display operation unit 11.

[0030] The face moving image acquiring unit 16d is a processing unit that acquires a face moving image and manages the face moving image data 15c. When the face moving image acquiring unit 16d accepts an RPPG measurement operation on the RPPG acceptance screen displayed on the display operation unit 11, the face moving image acquiring unit 16d captures a moving image of the user's face with the camera 12 and stores this moving image in the face moving image data 15c. During the face moving image capture, a face moving image capture screen is displayed on the display operation unit 11.

[0031] The vital data estimation unit 16e is a processing unit that estimates vital data and manages the vital data 15d. When the facial video data 15c is generated by the facial video acquisition unit 16d, the vital data estimation unit 16e estimates vital data by RPPG using the facial video data 15c. Then, the estimated data is displayed on the vital data display screen, and the data is stored in the vital data 15d in association with the identification ID specified by the authentication unit 16c. At this time, the measurement category is stored as "RPPG". The functional unit that authenticates the user may be configured on the cloud.

[0032] The determination unit 16f is a processing unit that determines whether or not the vital data 15d is included in the standard value. When the vital data 15d is generated by the vital data estimation unit 16e, the determination unit 16f compares the generated data with the standard data 15b. If the generated data is not included in the standard data 15b, the determination unit 16f displays a message on the vital data display screen encouraging measurement with a medical device.

[0033] The vital data acquiring unit 16g is a processing unit that accepts input of vital data. When the vital data acquiring unit 16g accepts measurement by a medical device on the vital data display screen displayed on the display operation unit 11, it displays a vital data input screen. Then, when the vital data input screen accepts input of vital data, it associates the accepted data with the identification ID specified by the authentication unit 16c and stores it in the vital data 15d. At this time, the measurement category is stored as "medical device".

[0034] Next, a specific example of data stored in the storage unit 15 of the vital information management device 10 shown in Fig. 2 will be described. Fig. 3 is a diagram showing an example of the authentication data 15a, the standard data 15b, and the vital data 15d shown in Fig. 2.

[0035] The authentication data 15a shown in FIG. 3(a) associates the identification ID "SD0001" with a state in which the name is "Patent Taro" and the registered image is "SD0001003.jpg", and associates the identification ID "SD0015" with a state in which the name is "Invention Hanako" and the registered image is "SD0015002.jpg".

[0036] Standard data 15b shown in FIG. 3(b) indicates a state in which the heart rate is "65 to 75 beats per minute," the SpO2 is "96 to 99%," the blood pressure is "139 mmHg or less / 89 mmHg or less," and the body temperature is "36.9 degrees or less."

[0037] Vital data 15d shown in Figure 3 (c) corresponds to the date and time "2023 / 08 / 06 8:10", the identification ID is "SD0015", the measurement category is "RPPG", the heart rate is "75" beats per minute, the SpO2 is "96", the blood pressure is "130 / 80" mmHg, and the body temperature is "36.2" degrees.

[0038] In addition, vital data 15d corresponds to the date and time "2023 / 08 / 06 8:15" with an identification ID of "SD0001", a measurement category of "RPPG", a heart rate of "70" beats per minute, an SpO2 of "98", a blood pressure of "145 / 95" mmHg, and a body temperature of "36.6" degrees.

[0039] In addition, vital data 15d associates the date and time "2023 / 08 / 06 8:30" with the following conditions: identification ID is "SD0001", measurement category is "medical equipment", heart rate is "-", SpO2 is "-", blood pressure is "138 / 89" mmHg, and body temperature is "-".

[0040] <Outline of the photoplethysmography method according to the first embodiment> Next, there will be described an overview of the photoplethysmography method used when estimating vital data in this embodiment 1. Fig. 4 is an explanatory diagram of the overview of the photoplethysmography method according to this embodiment 1.

[0041] As shown in Figure 4, photoplethysmography (PPG) is a method of obtaining in vivo information such as heart rate and blood flow using light. Light from a light source is specularly reflected by the surface of the skin, which is captured by a camera. The light that enters the skin is reflected in various ways and emerges outside the skin as diffuse reflection, which is also captured by the camera. Vital data is estimated by measuring the changes in light and brightness of the specular and diffuse reflections from the skin captured by the camera.

[0042] In this embodiment 1, a non-contact photoplethysmography (RPPG) is used in which the light source and camera are several tens of centimeters away from the subject to be photographed (skin), but this is a method that is similar in principle to a contact photoplethysmography (PPG) in which the light source and camera are close to the subject to be photographed (skin).

[0043] <Example of display in vital information management device 10 according to embodiment 1> Next, a description will be given of an example of a display in the vital sign information management device 10 according to the present embodiment 1. Fig. 5 and Fig. 6 are diagrams showing an example of a display in the vital sign information management device 10 according to the present embodiment 1.

[0044] As shown in Figure 5, when taking a video of the face to estimate vital data, a screen is displayed during the video recording of the face, displaying precautions such as the distance from the camera. Specifically, the screen displays, "Please remove your mask, stand 20-5 cm away from the device, and record so that your entire face fits within the measurement frame. Try to remain as still as possible for about 10 seconds while recording."

[0045] During the shooting, voice alerts are also given to the user. Specifically, voices such as "x seconds left," "It's almost over," and "Please do not move" are announced through the speaker 13.

[0046] As shown in Fig. 6, the vital data display screen displays the heart rate, SpO2, blood pressure, and body temperature as the estimated vital data. For example, "heart rate 70 beats / min", "SpO2 98%, "blood pressure 163 / 98mmHg", and "body temperature 36.2 degrees" are displayed.

[0047] If the estimated vital data is outside the standard range, a message will be displayed urging the user to measure with a medical device. For example, the message will say, "Your blood pressure is outside the standard range. Would you like to measure again with a medical device?"

[0048] <Processing Procedure of Vital Information Management Device 10 According to the First Embodiment> Next, a description will be given of the processing procedure in the vital sign information management device 10 according to the present embodiment 1. Fig. 7 is a flowchart showing the processing procedure in the vital sign information management device 10 according to the present embodiment 1.

[0049] If the face authentication process is successful (step S101; Yes), the vital information management device 10 captures a moving image of the face. If the face moving image is acquired from the camera 12 (step S102; Yes), the vital information management device 10 estimates vital data using the acquired face moving image (step S103).

[0050] If the estimated vital data is within the standard values ​​(step S104; Yes), the process ends. If the estimated vital data is out of the standard values ​​(step S104; No), a message is displayed prompting the user to take a measurement using a medical device (step S105).

[0051] If the measurement is not to be performed using a medical device (step S106; No), the process ends. If the measurement is to be performed using a medical device (step S106; Yes), the measurement result is accepted and stored (step S107), and the process ends.

[0052] As described above, the vital information management device of this embodiment 1 is configured to prompt measurement with a medical device if vital data estimated using a captured video of a face deviates from a standard value. This makes it possible to efficiently link the estimated vital information with the user and to recommend whether or not the subject's vital information should be measured with a medical device.

[0053] In the above embodiment 1, the configuration in which the vital data measured by the medical device is manually input to the vital information management device has been described, but the present invention is not limited to this. It is also possible to configure the medical device and the vital information management device to be communicatively connected by short-range wireless communication such as Bluetooth (registered trademark), so that the vital data measured by the medical device is automatically transmitted to the vital information management device.

[0054] In the above-mentioned embodiment 1, the configuration for estimating vital data using a captured video of a face has been described, but the present invention is not limited to this. It is also possible to accumulate vital data estimated using a captured video of a face, and estimate vital information using a vital information estimation model trained using the accumulated vital data as training data.

[0055] [Embodiment 2] In the above embodiment 1, a configuration for estimating vital data using a video of a captured face has been described, but the present invention is not limited to this. A video of a captured face is stored, and a health condition estimation model trained with the stored video data as training data is used to estimate a health condition, and the estimation result can be used to recommend whether or not the subject's vital information should be measured with a medical device. In the present embodiment 2, a vital information management device is described that stores a video of a captured face, estimates a health condition using a health condition estimation model trained with the stored video data as training data, and uses the estimation result to recommend whether or not the subject's vital information should be measured with a medical device.

[0056] <Outline of vital information management device 20 according to embodiment 2> An overview of the vital sign information management device 20 according to the second embodiment will be described below. Fig. 8 is an explanatory diagram of the overview of the vital sign information management device 20 according to the second embodiment.

[0057] 8, in the vital information management device 20 according to the second embodiment, a facial video of the user is captured (S11). Then, the captured facial video and a health condition estimation model are used to estimate the health condition of the user (S12). The health condition estimation model is generated by accumulating facial video of the user captured in advance and training using the accumulated video data as training data.

[0058] If the estimated health condition is not within the predetermined health condition, the user is prompted to take a measurement using a medical device (S13). For example, a message such as "Your health condition does not seem to be excellent. Would you like to take another measurement using a medical device?" is displayed.

[0059] In this way, the vital information management device 20 of this embodiment 2 accumulates video footage of a captured face, estimates a health condition using a health condition estimation model trained using this accumulated video data as training data, and can use the estimation results to recommend whether or not the subject's vital information should be measured by medical equipment.

[0060] <Configuration of vital information management device 20> Next, a description will be given of the configuration of vital information management device 20. Fig. 9 is a functional block diagram showing the configuration of vital information management device 20 shown in Fig. 8. As shown in Fig. 9, vital information management device 20 has a display operation unit 11, a camera 12, a speaker 13, a communication unit 14, a storage unit 25, and a control unit 26. It should be noted that a description of functional units similar to those of the vital information management device 10 shown in FIG. 2 will be omitted.

[0061] The storage unit 25 is a storage device consisting of a non-volatile memory or the like, and stores authentication data 15a, standard data 15b, face video data 15c, vital data 15d, and training data 25a. The training data 25a is data in which face video data 15c is accumulated, and is used for training the health condition estimation model.

[0062] The control unit 26 is a control unit that performs overall control of the vital information management device 20, and has an authentication registration unit 16a, a setting unit 16b, an authentication unit 16c, a face moving image acquisition unit 16d, a vital data estimation unit 16e, a training data collection unit 26a, a model training unit 26b, a health condition estimation unit 26c, a judgment unit 26d, and a vital data acquisition unit 16g. In practice, by loading these programs into a CPU and executing them, the authentication registration unit 16a, the setting unit 16b, the authentication unit 16c, the face moving image acquisition unit 16d, the vital data estimation unit 16e, the training data collection unit 26a, the model training unit 26b, the health condition estimation unit 26c, the judgment unit 26d, and the vital data acquisition unit 16g are caused to execute processes corresponding thereto.

[0063] The training data collection unit 26a is a processing unit that manages the training data 25a. When the facial moving image data 15c is updated, the training data collection unit 26a adds the facial moving image data 15c to the training data 25a to update it.

[0064] Model training unit 26b is a processing unit that generates a health state estimation model. When model training unit 26b receives a training start operation from display operation unit 11, model training unit 26b performs training using training data 25a and generates a health state estimation model.

[0065] The health condition estimation unit 26c is a processing unit that estimates the health condition using a health condition estimation model. When the face moving image data 15c is updated, the health condition estimation unit 26c estimates the health condition using the face moving image data 15c and the health condition estimation model. Then, an icon indicating the estimated health condition is displayed on the display operation unit 11. The health condition estimation model estimates the health condition, for example, into categories of healthy, fairly healthy, somewhat unhealthy, and unhealthy.

[0066] The determination unit 26d is a processing unit that determines whether the estimated vital data and health condition are within a predetermined range. When the vital data estimation unit 16e generates vital data 15d, the determination unit 26d compares this generated data with the standard data 15b. If the generated data is not included in the standard data 15b, the determination unit 26d displays on the vital data display screen a message encouraging the user to take a measurement with a medical device. If the health condition estimated by the health condition estimation model is not a predetermined health condition (e.g., healthy or fairly healthy), the determination unit 26d displays on the vital data display screen a message encouraging the user to take a measurement with a medical device.

[0067] <Processing procedure for generating a health state estimation model according to the second embodiment> Next, a description will be given of a processing procedure for generating a health state estimation model according to the present embodiment 2. Fig. 10 is a flowchart showing the processing procedure for generating a health state estimation model according to the present embodiment 2.

[0068] When the vital information management device 20 acquires face moving image data (step S201), the vital information management device 20 adds the acquired face moving image data to the training data 25a to update the training data (step S202).

[0069] If training is not to be performed (step S203; No), the process proceeds to step S201. If training is to be performed (step S203; Yes), training is performed using training data 25a, a health state estimation model is generated (step S204), and the process ends.

[0070] As described above, the vital information management device of this embodiment 2 accumulates video footage of a face, estimates a health condition using a health condition estimation model trained using the accumulated video data as training data, and can use the estimation results to recommend whether or not the subject's vital information should be measured by medical equipment.

[0071] In the above-mentioned second embodiment, a configuration has been described in which captured face videos are accumulated as training data and a health condition estimation model is generated using this training data, but the present invention is not limited to this. It is also possible to configure the configuration so that captured face videos and voice data are accumulated as training data and a health condition estimation model is generated using this training data.

[0072] [Embodiment 3] In the above embodiment 1, a configuration has been described in which vital data is estimated using a captured video of a face in a vital information management device such as a mobile terminal, but the present invention is not limited to this. It is also possible to configure a cloud or a server on a network to estimate vital data using a captured video of a face, and to display this vital data in the vital information management device. In the present embodiment 3, a vital information management system will be described in which vital data is estimated using a captured video of a face in a cloud or a server on a network, and this vital data is displayed in the vital information management device.

[0073] <Outline of the vital information management system according to the third embodiment> An outline of the vital information management system according to the third embodiment will be described below. Fig. 11 is an explanatory diagram of the outline of the vital information management system according to the third embodiment.

[0074] 11, in the vital information management system according to the third embodiment, a facial moving image of a user is captured by the vital information management device 100 (S21). Then, the captured facial moving image is transmitted to the server 200 (S22).

[0075] Server 200 estimates vital data using the received facial video (S23) and transmits this vital data to vital information management device 100 (S24). Vital information management device 100 displays the received vital data (S25). At this time, if the vital data deviates from the standard values, it prompts the user to measure it using a medical device.

[0076] Once measurements have been made using medical equipment, the measurement results are input to the vital information management device 100. At this time, in order to record comments from the person making the measurements or the subject, text, photographs, audio (the actual voices of the person making the measurements or the subject), etc. may be stored and accumulated in the vital information management device 100.

[0077] In this way, in the vital information management system of this embodiment 3, vital data can be estimated using captured video of a face on a server in the cloud or on a network, and this vital data can be displayed on a vital information management device.

[0078] Here, the description has focused on the estimation of vital information, but the server 200 can also be configured to estimate vital information based on a facial video, and perform facial authentication of the facial image contained in the facial video, and when notified by the vital information management device 100 that a specified operation has been accepted, store the estimated vital information in a memory unit in association with the identification information of the user whose face has been authenticated.

[0079] [Embodiment 4] In the above-mentioned second embodiment, a configuration was described in which a vital information management device such as a mobile terminal stores a captured video of a face, and estimates a health condition using a health condition estimation model trained with the stored video data as training data, but the present invention is not limited to this. A cloud or a server on a network can be configured to estimate a health condition using a health condition estimation model trained with the stored video data of a face as training data, and the estimated health condition information can be displayed on the vital information management device. In the fourth embodiment, a vital information management system is described in which a cloud or a server on a network estimates a health condition using a health condition estimation model trained with the stored video data of a face as training data, and the estimated health condition information is displayed on the vital information management device.

[0080] <Outline of the vital information management system according to the fourth embodiment> An outline of the vital information management system according to the fourth embodiment will be described below. Fig. 12 is an explanatory diagram of the outline of the vital information management system according to the fourth embodiment.

[0081] 12, in the vital information management system according to the fourth embodiment, a facial moving image of a user is captured by the vital information management device 300 (S31). Then, the captured facial moving image is transmitted to the server 400 (S32).

[0082] The server 400 estimates the health condition using the received facial video (S33) and transmits this health condition information to the vital information management device 300 (S34). The vital information management device 300 displays the received health condition information (S35). At this time, if the estimated health condition is not within a predetermined health condition, the server 400 prompts the user to take a measurement using a medical device.

[0083] Once measurements have been made using medical equipment, the measurement results are input to the vital information management device 300. At this time, in order to record comments from the person making the measurements or the subject, text, photographs, audio (the actual voices of the person making the measurements or the subject), etc. may be stored and accumulated in the vital information management device 300 or server 400.

[0084] In this way, in the vital information management system of this embodiment 4, a health condition estimation model trained on a server on the cloud or network using accumulated facial video data as training data is used to estimate a health condition, and this estimated health condition information can be displayed on the vital information management device.

[0085] Here, the description has focused on estimating health condition information, however, server 400 can also be configured to estimate health condition information based on facial video, and perform facial authentication of the facial image contained in the facial video, and when it receives a notification from vital information management device 300 that a specified operation has been accepted, store the estimated health condition information in the memory unit in association with the identification information of the user whose face has been authenticated.

[0086] Note that the configurations illustrated in the above embodiments are merely functional schematics and do not necessarily have to be physically configured as illustrated. In other words, the form of distribution and integration of each device is not limited to that illustrated, and all or part of the devices can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. [Industrial Applicability]

[0087] The vital information management device, vital information management system, and vital information management method of the present invention efficiently link estimated vital information values ​​with users, and are suitable for cases where a subject's vital information should or should not be measured using medical equipment. [Explanation of symbols]

[0088] 10. Vital Information Management Device 11 Display operation section 12 Camera 13 Speaker 14 Communications Department 15 Storage section 15a Authentication Data 15b Standard Data 15c Face video data 15d Vital Data 16 Control section 16a Certification Registration Department 16b Setting section 16c Authentication Section 16d Face video acquisition section 16e Vital Data Estimation Unit 16f Judgment section 16g Vital data acquisition unit 20 Vital Information Management Device 25 Memory section 25a Training data 26 Control Unit 26a Training Data Collection Section 26b Model Training Department 26c Health Status Estimation Section 26d Judgment section 100 Vital Information Management Device 200 Servers 300 Vital Information Management Device 400 Servers

Claims

1. A vital information management device having an imaging unit that captures a face image of a user and an operation receiving unit that receives a predetermined operation for acquiring vital information, an estimation means for estimating vital information based on a face image captured by the imaging unit; A face authentication means for performing face authentication of a face image captured by the imaging unit; a storage means for storing, when the operation reception unit receives the predetermined operation, the vital sign information estimated by the estimation means in association with identification information of the user authenticated by the face authentication means; A vital information management device comprising:

2. The estimation means includes: The vital information management device according to claim 1, characterized in that vital information including pulse, blood oxygen concentration, blood pressure and / or body temperature is estimated from a heart rate obtained by applying remote photoplethysmography to the face image.

3. The estimation means includes: The vital information management device according to claim 2, characterized in that the vital information is estimated using a vital information estimation model trained using heart rate and ground truth data obtained by applying remote photoplethysmography to a plurality of face images as training data.

4. A vital information management device as described in any one of claims 1 to 3, further comprising a first display control means for controlling the display of recommendation information on a specified display unit, the recommendation information recommending measurement of the user's vital information using a specified medical device, based on the vital information estimated by the estimation means.

5. The method further includes a health level estimation means for estimating a health level using a health state estimation model trained using a plurality of face images and correct answer data as training data, The vital information management device of any one of claims 1 to 3, further comprising a second display control means for controlling the display of recommendation information on a specified display unit, the recommendation information recommending measurement of the user's vital information using a specified medical device, based on the vital information estimated by the estimation means and the health degree estimated by the health degree estimation means.

6. a health level estimation means for estimating a health level using a health state estimation model trained using a plurality of face images, voice data, and correct answer data as training data; A voice collecting means for collecting voice information of the user; Further equipped with The vital information management device of any one of claims 1 to 3, further comprising a third display control means for controlling the display of recommendation information recommending measurement of the user's vital information using a specified medical device on a specified display unit, based on the vital information estimated by the estimation means and the health degree estimated by the health degree estimation means.

7. A vital information management system including a terminal device including an imaging unit for capturing a face image of a user and an operation receiving unit for receiving a predetermined operation for acquiring vital information, and a server device capable of communicating with the terminal device, The server device includes: an estimation means for estimating vital information based on a face image captured by the imaging unit; A face authentication means for performing face authentication of a face image captured by the imaging unit; a storage means for storing, when receiving a notification from the terminal device that the predetermined operation has been accepted, the vital sign information estimated by the estimation means in association with identification information of the user authenticated by the face authentication means; A vital information management system comprising:

8. A vital information management method executed by a vital information management device having an imaging unit that captures a face image of a user and an operation receiving unit that receives a predetermined operation for acquiring vital information, an estimation step of estimating vital information based on a face image captured by the imaging unit; a face authentication process for performing face authentication on a face image captured by the imaging unit; a storage step of storing, in a predetermined storage unit, the vital sign information estimated in the estimation step in association with identification information of the user authenticated in the face authentication step when the predetermined operation is received by the operation reception unit; A vital information management method comprising:

Images