Bather monitoring system

Abstract

To provide a bather monitoring system that can automatically assess user's bathing comfort.SOLUTION: A bather monitoring system includes: image acquisition means for acquiring image information including user's face while bathing in a bathtub; and determination means for determining state information indicating user's bathing state based on the image information acquired by the image acquisition means. The image acquisition means acquires the image information including the face and the bathtub. The determination means extracts distance information regarding the distance between the face and a water surface of the bathtub based on the image information acquired by the image acquisition means, and determines the state information based on the extracted distance information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a bather monitoring system. [Background technology]

[0002] Bathing has traditionally been considered an important lifestyle activity, as it relieves physical and mental stress. For this reason, there is a demand for technology to make bathing safer, such as that shown in Patent Document 1.

[0003] Patent Document 1 discloses a heartbeat classification device that estimates a classification indicating the classification of each heartbeat in an electrocardiogram signal based on the electrocardiogram signal, and that includes: an extraction means that extracts a partial signal including a part of the electrocardiogram signal from the electrocardiogram signal; a probability calculation means that calculates the probability between the partial signal extracted by the extraction means and a reference partial signal including a peak of the electrocardiogram signal; a separation means that separates an electrocardiogram beat signal indicating one cycle of the electrocardiogram signal based on the partial signal whose probability calculated by the probability calculation means is equal to or greater than a threshold; and a classification estimation means that estimates the classification based on the electrocardiogram beat signal separated by the separation means. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2024-022122 Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 1 does not anticipate determining the state of a user from an image of the user taking a bath. As a result, the technology does not take into account information obtained from the image, which poses a problem in that it is not possible to determine the state of a user taking a bath with high accuracy.

[0006] The present invention was devised in consideration of the above-mentioned problems, and its purpose is to provide a bather monitoring system that can accurately determine the condition of a user while bathing. [Means for solving the problem]

[0007] The bather monitoring system of the first invention is characterized by comprising an image acquisition means for acquiring image information including the face of a user bathing in a bathtub, and a determination means for determining status information indicating the user's bathing status based on the image information acquired by the image acquisition means.

[0008] The bather monitoring system of the second invention is characterized in that, in the first invention, the image acquisition means acquires image information including the face and the surface of the water in the bathtub, and the determination means extracts distance information regarding the distance between the face and the surface of the water in the bathtub based on the image information acquired by the image acquisition means, and determines the status information based on the extracted distance information.

[0009] The bather monitoring system of the third invention is characterized in that, in the first invention, the image acquisition means acquires image information including the temperature distribution of the nostrils on the face, and the judgment means judges the status information based on the image information acquired by the image acquisition means.

[0010] The bather monitoring system of the fourth invention is characterized in that, in the first invention, the judgment means refers to a state model generated using multiple pieces of learning data, with input data based on previously acquired learning image information and learning state information linked to the input data as a pair of learning data, and judges the state information based on the image information acquired by the image acquisition means.

[0011] A bather monitoring system according to a fifth aspect of the present invention is the bather monitoring system according to the first aspect of the present invention, characterized in that the determining means determines the state information including comfort information indicating the comfort of the user.

[0012] The bather monitoring system according to the sixth invention is characterized in that, in any of the first to fifth inventions, it further comprises a drainage means for draining the bathtub based on the status information determined by the determination means.

[0013] The bather monitoring system of the seventh invention, in the first invention, further comprises an electrocardiogram acquisition means for acquiring the user's electrocardiogram signal, a first calculation means for calculating first comfort information indicating the user's comfort based on the electrocardiogram signal acquired by the electrocardiogram acquisition means, an HRV calculation means for calculating HRV information indicating heart rate variability based on the electrocardiogram signal acquired by the electrocardiogram acquisition means, a second calculation means for calculating second comfort information based on the HRV information calculated by the HRV calculation means, and a third calculation means for calculating third comfort information based on the first comfort information calculated by the first calculation means and the second comfort information calculated by the second calculation means, and is characterized in that the determination means determines the status information based on the image information acquired by the image acquisition means and the third comfort information calculated by the third calculation means.

[0014] The bather monitoring system of the eighth invention is characterized in that, in the first invention, the bather monitoring system further comprises a judgment means for referring to a treatment model generated using a plurality of learning data, with input data based on previously acquired learning status information and learning treatment information linked to the input data, indicating a method of treatment for the status information, as a pair of learning data, and determining treatment information for the status information determined by the judgment means. [Effects of the Invention]

[0015] According to the first to eighth aspects of the present invention, the bather monitoring system determines status information indicating the bathing status of the user based on image information. This makes it possible to take into account information obtained from the image, thereby enabling the status of the user while bathing to be determined with high accuracy.

[0016] In particular, according to the second aspect of the present invention, the bather monitoring system includes a determining means that extracts distance information from image information and determines status information based on the extracted distance information. This allows the distance between the user and the bath surface to be extracted from the image information, and the bathing status of the user to be determined based on information such as whether the user's face is above the water surface. This allows the bather's status to be determined with higher accuracy.

[0017] In particular, according to the third aspect of the present invention, the bather monitoring system determines the condition information based on image information including the temperature distribution of the nostrils on the face. This makes it possible to determine the condition information based on, for example, the temperature distribution of the nostrils on the face that is correlated with the condition information. This allows for more accurate determination of the user's condition while bathing.

[0018] In particular, according to the fourth aspect of the present invention, the bather monitoring system refers to a neural network model and determines status information based on image information. This allows for more appropriate status information determination by referring to a status model trained using training data. This allows for more accurate determination of the user's status while bathing.

[0019] In particular, according to the fifth aspect of the present invention, the bather monitoring system determines status information including comfort information indicating the user's comfort, thereby making it possible to determine the user's comfort based on image information.

[0020] In particular, according to the sixth aspect of the present invention, the bather monitoring system drains the bathtub based on the status information, thereby enabling automatic drainage according to the user's status.

[0021] In particular, according to the seventh aspect of the present invention, the bather monitoring system determines the condition information based on the image information and the third comfort information. This makes it possible to determine the user's condition taking into account the image and the user's electrocardiogram signal. This allows for more accurate determination of the user's condition while bathing.

[0022] In particular, according to the eighth aspect of the present invention, the bather monitoring system determines treatment information for the status information, thereby enabling the bather to determine with high accuracy the appropriate treatment for the user. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a schematic diagram showing an example of a bather monitoring system according to this embodiment. [Figure 2] FIG. 2 is a schematic diagram showing an example of a bathtub in this embodiment. [Figure 3] FIG. 3 is a schematic diagram showing an example of the configuration of the determination device according to this embodiment. [Figure 4] FIG. 4 is a schematic diagram showing an example of the function of the determination device in this embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of calculating comfort information of the determination device according to this embodiment. [Figure 6] 6(a) is a schematic diagram showing an example of an electrocardiogram signal, and FIG. 6(b) is a schematic diagram showing an example of a comfort index. [Figure 7] FIG. 7 is a schematic diagram showing an example of an electrocardiogram division signal. [Figure 8] FIG. 8 is a schematic diagram showing an example of a neural network model. [Figure 9] FIG. 9 is a schematic diagram showing an example of the third comfort index. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of the bather monitoring system in this embodiment. [Figure 11] FIG. 11 is a schematic diagram showing an example of the temperature distribution of the nostrils on the user's face. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of a bather monitoring system according to a first embodiment of the present invention will be described below with reference to the drawings.

[0025] FIG. 1 is a schematic diagram showing an example of a bather monitoring system 100 according to the first embodiment.

[0026] The bather monitoring system 100 is used to determine the comfort and condition of a user U who is taking a bath.

[0027] Bather monitoring system 100 includes, for example, a determination device 1, a terminal 2, a server 3, and a bathroom 5, as shown in Figure 1. Determination device 1, terminal 2, server 3, and bathroom 5 may be connected directly or via a communication network 4.

[0028] Alternatively, determination device 1 may be directly connected to bathroom 5, and determination device 1, terminal 2, and server 3 may be connected via communication network 4.

[0029] The terminal 2 is owned, for example, by a family member or caregiver of the user U whose comfort and condition are determined by the bather monitoring system 100, and is connected to the determination device 1 via a communication network 4. The terminal 2 may be, for example, an electronic device such as a computer or a tablet terminal. The terminal 2 may have at least some of the functions of the determination device 1, for example.

[0030] The server 3 is connected to the determination device 1, etc., via, for example, a communication network 4. The server 3 records various past data, etc., and various data is transmitted from the determination device 1 and the bathroom 5 as needed. The server 3 may, for example, have at least some of the functions of the determination device 1, or may perform at least some of the processing in place of the determination device 1. The server 3 may also transmit various information to the determination device 1, etc., as needed. The server 3 may also be an edge computing device housed in a single housing together with the determination device 1 installed in the bathtub 5.

[0031] The communication network 4 is, for example, the Internet network to which the determination device 1 is connected via a communication circuit. The communication network 4 may be configured as a so-called optical fiber communication network. Furthermore, the communication network 4 may be realized by known communication technologies such as a wired communication network or a wireless communication network. Furthermore, the communication network 4 may be a LAN (Local Area Network).

[0032] Bathroom 5 is an enclosed space where user U takes a bath. As shown in FIG. 2, bathroom 5 is installed within bathroom 5 and includes a bathtub 51 having a water surface 58, a sensor 53 provided within bathroom 5, a monitor 54, a ventilation fan 56, a camera 57, a drainage device 52 provided in bathtub 51, and electrodes 55. Bathroom 5 may also be equipped with two or more electrodes 55. These components may be portable devices. These components may also be wireless devices equipped with a battery. These components are not limited to being automated, and may also be manually operated devices. These components may also be highly compatible general-purpose devices.

[0033] Bathroom 5 may also be equipped with an air conditioner, lighting equipment, audio equipment, a fragrance generator, etc. (not shown). Sensor 53, ventilation fan 56, drainage device 52, monitor 54, electrode 55, and camera 57 are connected to determination device 1. Sensor 53, ventilation fan 56, drainage device 52, monitor 54, electrode 55, and camera 57 may communicate with each other via a wireless or wired connection or communication network 4.

[0034] Drainage device 52 is a device for draining water from bathtub 51 in response to a command from determination device 1. Drainage device 52 may be, for example, a device with any drainage function that can be retrofitted, but is not limited to this and any device may be used. Drainage device 52 may be, for example, a device that drains water from bathtub 51 by opening a water faucet in bathtub 51 using a spring or the like in response to a command from determination device 1.

[0035] Sensor 53 is a sensor that measures environmental information indicating one or more of the temperature, humidity, air pressure, illuminance, sound level (music, etc.), and scent in bathroom 5, and various sensors such as a temperature sensor, humidity sensor, and air pressure sensor are used. Sensor 53 transmits the environmental information to various devices. In such cases, sensor 53 may transmit the information directly to the various devices, or may transmit the information via communication network 4, for example.

[0036] The monitor 54 is a monitor for presenting treatment information indicating a treatment method for the user U in response to a command from the determination device 1.

[0037] The electrodes 55 are electrodes for measuring the electrocardiogram signal of the user U. The electrodes 55 measure the electrocardiogram signal of the user U by measuring the potential difference at multiple locations near the hands, feet, chest, etc. of the user U. There may be multiple electrodes 55, and the electrodes 55 may be embedded in the bathtub 51. Alternatively, the electrodes 55 may be attached to the wall of the bathtub 51. The user U can measure the electrocardiogram signal without touching the electrodes 55.

[0038] Camera 57 is any imaging device that captures an image or video including face 59 of user U bathing in bathtub 51. Camera 57 may capture an image including face 59 and bathtub 51. Camera 57 may also be a thermography camera that measures the temperature distribution of the imaged object. Camera 57 transmits image information related to the captured image to various devices. In such cases, camera 57 may transmit information directly to various devices, or may transmit the information via communication network 4, for example.

[0039] An electronic device such as a mini PC or a single-board computer may be used as the determination device 1, but is not limited to these and may also be a laptop PC or a desktop PC. As shown in Fig. 3, the determination device 1 includes a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage unit 104, and I / Fs 105 to 107. The components 101 to 107 are connected via an internal bus 110.

[0040] The CPU 101 controls the entire determination device 1. The ROM 102 stores operation code for the CPU 101. The RAM 103 is a working area used when the CPU 101 is operating. The storage unit 104 records various information such as a database and learning target data. As the storage unit 104, for example, a data storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) is used. Note that the determination device 1 may also include a GPU (Graphics Processing Unit), not shown, for example.

[0041] The I / F 105 is an interface for transmitting and receiving various types of information to and from the terminal 2, the server 3, a website, etc., as needed, via the communication network 4. The I / F 106 is an interface for transmitting and receiving information to and from the input unit 108. For example, a keyboard is used as the input unit 108, and a user of the determination device 1 inputs various types of information, control commands for the determination device 1, etc., via the input unit 108. The I / F 107 is an interface for transmitting and receiving various types of information to and from the display unit 109. The display unit 109 displays various types of information, content, etc., stored in the storage unit 104. A display is used as the display unit 109, and in the case of a touch panel type, for example, it is provided integrally with the input unit 108. Furthermore, a speaker may be used as the display unit 109.

[0042] 4 is a block diagram showing an example of the functions of the determination device 1. The determination device 1 includes an environment receiving unit 11 connected to a sensor 53 and a camera 57, a control unit 12 connected to the environment receiving unit 11, an estimation unit 13 connected to the control unit 12, a user recording unit 14 connected to the estimation unit 13, an electrocardiogram receiving unit 15 connected to the electrodes 55 and the control unit 12, a preprocessing unit 16 connected to the electrocardiogram receiving unit 15, a first calculation unit 17 and an HRV calculation unit 19 connected to the preprocessing unit 16, a second calculation unit 20 connected to the HRV calculation unit 19, a third calculation unit 18 connected to the first calculation unit 17 and the second calculation unit 20, a state determination unit 21 connected to the user recording unit 14 and the third calculation unit 18, a treatment determination unit 22 connected to the state determination unit 21, and a notification recording unit 23 and a treatment control unit 24 connected to the treatment determination unit 22. Each function shown in FIG. 4 is realized by the CPU 101 using the RAM 103 as a work area to execute a program recorded in the storage unit 104 or the like, and may be controlled by, for example, artificial intelligence or the like.

[0043] The environment receiving unit 11 acquires various data such as environmental information, vital data of the user U, image information, etc. The environment receiving unit 11 may acquire, for example, vital data input to the estimation unit 13, as well as environmental information, image information, etc. from the bathroom 5, for example, via the communication network 4. The environment receiving unit 11 outputs the acquired information to the control unit 12.

[0044] If necessary, the control unit 12 may output a control signal for starting processing of the electrocardiogram signal to the electrocardiogram receiving unit 15. The control unit 12 also outputs environmental information and the like to the estimation unit 13.

[0045] Various types of information are input to the estimation unit 13. The estimation unit 13 acquires, for example, vital data and environmental information. The estimation unit 13 outputs the various types of information to the user recording unit 14.

[0046] The user recording unit 14 records environmental information and vital data. The user recording unit 14 may also record various information such as comfort information and electrocardiogram signals output from the preprocessing unit 16, the first calculation unit 17, the second calculation unit 20, the third calculation unit 18, etc. The user recording unit 14 may also output image information transmitted by the camera 57 to the third calculation unit as needed. The user recording unit 14 also outputs various information to the state determination unit 21 as needed.

[0047] The electrocardiogram receiving unit 15 receives an electrocardiogram signal from the electrodes 55. The electrocardiogram receiving unit 15 outputs the received electrocardiogram signal to the preprocessing unit 16 in response to a control signal from the control unit 12.

[0048] The preprocessing unit 16 divides the electrocardiogram signal into electrocardiogram division signals, and outputs the processed electrocardiogram division signals to the first calculation unit 17 and the HRV calculation unit 19.

[0049] The first calculator 17 uses the electrocardiogram signal as input and outputs the calculated first comfort information to the third calculator 18. The first calculator 17 calculates the first comfort information by referring to a neural network model generated using a plurality of pieces of training data, where the input data is a pair of training data based on the training electrocardiogram signal acquired in advance and training comfort information associated with the input data.

[0050] The HRV calculation unit 19 calculates HRV information indicating heart rate variability based on the electrocardiogram signal. The HRV calculation unit 19 outputs the calculated HRV information to the second calculation unit 20.

[0051] The second calculation unit 20 calculates the second comfort information based on the HRV information and outputs the calculated second comfort information to the third calculation unit 18.

[0052] The third calculation unit 18 calculates third comfort information based on the first comfort information and the second comfort information, and outputs the calculated third comfort information to the state determination unit 21.

[0053] The state determination unit 21 determines the state information based on various types of information. The state determination unit 21 determines the state information based on, for example, image information. The state determination unit 21 determines the state information by referring to a state database that records state information indicating the state of the user U in relation to the previously acquired comfort information and bathing information, for example, based on the third comfort information and bathing information that is a combination of various types of information. The state determination unit 21 outputs the state information to the treatment determination unit 22.

[0054] The treatment determination unit 22 uses previously acquired status information and treatment information indicating a treatment method for the status information as a pair of determination learning data, and determines the treatment information for the status information by referring to a treatment information database in the notification recording unit 23 in which multiple determination learning data are recorded. The treatment determination unit 22 outputs the treatment information to the treatment control unit 24.

[0055] The treatment control unit 24 performs each treatment according to the treatment information. For example, the treatment control unit 24 drains water from the bathtub 51 provided in the bathroom 5 using the drainage device 52 based on the treatment information.

[0056] Next, an example of the operation of the bather monitoring system 100 in this embodiment will be described.

[0057] The bather monitoring system 100 is executed, for example, via a program installed in the determination device 1. That is, the state of the user U is managed through the program installed in the determination device 1.

[0058] 5 is a flowchart showing an example of the operation of calculating comfort information by the determination device 1 in this embodiment. The determination device 1 calculates comfort information through the steps shown in FIG.

[0059] First, in step S110, the determination device 1 acquires various pieces of information. In step S110, for example, the environment receiving unit 11 acquires environmental information measured by the sensor 53. Also in step S110, for example, the environment receiving unit 11 acquires image information acquired from the camera 57. Also in step S110, for example, the electrocardiogram receiving unit 15 acquires the electrocardiogram signal of the user U measured by the electrodes 55. In such a case, the electrodes 55 may measure the electrocardiogram signal for the entire bathing time or a predetermined time period. The electrodes 55 may measure the electrocardiogram signal of the user U for, for example, 10 minutes. Alternatively, the timing of the user U's entry and exit from the bath may be automatically detected, and the electrocardiogram signal of the user U may be measured for the entire bathing time, from the time the user U enters and leaves the bath. The bathroom 5 transmits the measured information to the determination device 1 via the communication network 4.

[0060] An electrocardiographic signal is a signal that indicates changes in cardiac potential (in the order of mV) over time (s), as shown in Figure 6(a), and includes P waves that indicate atrial excitation, QRS waves that indicate ventricular excitation, and T waves that indicate ventricular de-excitation. One cycle of an electrocardiographic signal is the period until these specific signals are observed again, and indicates, for example, the period from one P wave to the next. One cycle of an electrocardiographic signal may also be an electrocardiographic signal divided into fixed time intervals.

[0061] Furthermore, in step S110, the electrocardiogram receiving unit 15 may output an electrocardiogram signal to the pre-processing unit 16 based on the environmental information or image information acquired in step S110. For example, the environmental information or image information acquired by the environmental receiving unit 11 may be output to the control unit 12, the control unit 12 may output a control signal to the electrocardiogram receiving unit 15 in accordance with the environmental information or image information, and the electrocardiogram receiving unit 15 may output an electrocardiogram signal to the pre-processing unit 16 in accordance with the control signal. This allows comfort to be evaluated in accordance with the environmental information or image information, making it possible to perform treatment at an appropriate time and improving the safety of the user U.

[0062] Next, in step S120, the preprocessing unit 16 separates the electrocardiographic signal acquired in step S110 into electrocardiographic segment signals. The electrocardiographic segment signals are, for example, electrocardiographic signals for a specific period of T seconds. The electrocardiographic segment signals include at least one cycle of the electrocardiographic signal and include critical points of the electrocardiographic signal. The critical points of the electrocardiographic signal are, for example, QRS waves, but are not limited thereto and may also be peaks, critical points, or extreme values ​​indicating the maximum or minimum values ​​of other signals. The critical points of the electrocardiographic signal may also be determined by the magnitude of the absolute value relative to a threshold value.

[0063] In step S120, the preprocessing unit 16 separates the electrocardiographic signal acquired in step S110 into electrocardiographic segment signals as shown in FIG. 7. For example, the preprocessing unit 16 may separate the electrocardiographic signal to acquire first electrocardiographic segment signals separated for each T1 epoch and second electrocardiographic segment signals separated from the same electrocardiographic signal for each T2 epoch different from the T1 epoch. In such a case, for example, the T2 epoch may be twice as long as the T1 epoch. Furthermore, the T1 epoch may be, for example, a 30-second electrocardiographic signal, and the T2 epoch may be, for example, a 60-second electrocardiographic signal. Furthermore, the preprocessing unit 16 may divide the electrocardiographic signal into first to fifteenth electrocardiographic segment signals as shown in FIG. 7, and acquire the second electrocardiographic segment signal, which is a combination of the first and second electrocardiographic segment signals, and the second electrocardiographic segment signal as the first electrocardiographic segment signal, as the Nth data set. Furthermore, the preprocessing unit 16 may acquire, for example, a second electrocardiographic partial signal obtained by combining the second and third electrocardiographic partial signals of the electrocardiographic signal and the third electrocardiographic partial signal as a first electrocardiographic partial signal, as the (N+1)th data set, and similarly acquire a plurality of data sets such as (N+2), (N+3), etc. The preprocessing unit 16 outputs the first electrocardiographic partial signal to the first calculation unit 17, and outputs the second electrocardiographic partial signal to the HRV calculation unit 19.

[0064] In step S120, the preprocessing unit 16 may convert the electrocardiographic division signals into a scalogram image. The scalogram image is an image showing a scalogram of the electrocardiographic division signals.

[0065] Furthermore, the preprocessing unit 16 may remove noise from the electrocardiogram signal acquired in step S110 using an appropriate filter or the like.

[0066] Next, in step S130, the HRV calculation unit 19 calculates HRV information indicating heart rate variability based on the second electrocardiographic division signals separated in step S120. The HRV information is information about HRV (Heart Rate Variability), such as information about changes in RRI (RR Interval, or R-R interval), the duration of each beat. The HRV information is also information about the time domain, frequency domain, nonlinear domain, etc. of HRV. The HRV information may also be fluctuation values ​​of each element, such as the average value of RRI, SDNN (standard deviation of NN intervals), LF (low frequency spectral power), HF (high frequency spectral power), LF / HF, SD (standard deviation)², ApEn (approximate entropy), SampEn (sample entropy), and TotalPower. Here, SDNN indicates the standard deviation of RRI, LF indicates the power of the low-frequency components (0.04-0.15 Hz) when HRV is frequency-analyzed using fast Fourier transform, etc., and HF indicates the power of the high-frequency components (0.15-0.4 Hz) when HRV is frequency-analyzed. LF / HF is the value obtained by dividing LF by HF. SD2 indicates an index of the RRI fluctuation range. SampEn indicates an evaluation of the similarity of patterns of subsequences (samples) taken from time-series data. SampEn calculates the probability that a subsequence (pattern) of length m starting from a certain data point is the same up to the next data point, and is an evaluation based on entropy calculated using this. ApEn, like SampEn, indicates an evaluation of the similarity of patterns of subsequences. When comparing whether subsequences are similar, ApEn determines an acceptable range using a certain threshold (r) and calculates an evaluation based on the degree of agreement within that range. TotalPower indicates the sum of the RRI frequency range. The HRV calculation unit 19 may calculate HRV information based on, for example, a change in the time length of the RRI included in the second electrocardiographic split signals separated in step S120. The HRV calculation unit 19 outputs the calculated HRV information to the second calculation unit 20.

[0067] Next, in step S140, the first calculator 17 outputs first comfort information indicating the comfort of the user U based on the first electrocardiogram division signal separated in step S120. The comfort information is information indicating the comfort of the user U. The comfort information may be, for example, a comfort index indicating the comfort of the user U. The comfort information may also be information regarding a change in the comfort index over time, as shown in FIG. 6(b). The first comfort information is, for example, comfort information calculated based on the first electrocardiogram division information. The first comfort information is, for example, subjective comfort information calculated from the electrocardiogram signal using a neural network.

[0068] In step S140, the first calculation unit 17 may refer to a neural network model generated using a plurality of pieces of training data, where the input data is based on the training electrocardiogram signal acquired in advance and the training comfort information linked to the input data, and calculate the first comfort information using the first electrocardiogram division signal separated in step S120 as an input.

[0069] The neural network model may include a trained model generated by machine learning using a plurality of training data. The trained model may be, for example, a neural network model such as a convolutional neural network (CNN) or a support vector machine (SVM). Deep learning, for example, may be used as the machine learning model. FIG. 8 is a schematic diagram showing an example of training data. For example, as shown in FIG. 8, the training data associates training electrocardiographic signals with comfort information. The training electrocardiographic information is electrocardiographic information or electrocardiographic segmentation information acquired in advance as training data for the neural network. The training electrocardiographic information may be an electrocardiographic signal having the same duration as the first electrocardiographic segmentation signal. Furthermore, the comfort information used as training data may be, for example, training comfort information input and acquired in advance by the bather, etc., together with the training electrocardiographic information.

[0070] The neural network model learns associations, for example, between training electrocardiogram signals or training electrocardiogram division signals (input data) and comfort information (output data). The association indicates the degree of connection between the input data and the output data, and it can be determined that the higher the association, the stronger the connection between the data. The association may be expressed, for example, as a percentage or in several stages.

[0071] For example, the association is constructed by the degree of connection between many-to-many information (plurality of input data, paired with plural output data). The association is appropriately updated during the machine learning process and indicates, for example, a function (classifier) ​​optimized based on plural input data and plural output data. Note that the association may have, for example, plural association degrees indicating the degree of connection between each data. For example, when the database is constructed using a neural network, the association degree can correspond to a weight variable.

[0072] For this reason, the determination device 1 selects output data suitable for input data by using, for example, correlations that take into account all of the results of determination by the classifiers. This makes it possible to quantitatively select output data suitable for input data not only when the input data is identical to or similar to the learning input data, but also when the input data is dissimilar.

[0073] The correlation may indicate, for example, the degree of connection between a plurality of output data and a plurality of input data. In this case, by using the correlation, the degree of relationship between a plurality of input data ("learning electrocardiogram signal A" to "learning electrocardiogram signal C") can be linked and recorded for each of the plurality of output data ("comfort information A" to "comfort information C"). Therefore, for example, through the correlation, a plurality of output data can be linked to one input data with a specific degree of correlation. This makes it possible to obtain comfort information for the learning electrocardiogram signal.

[0074] The correlation has, for example, multiple correlations linking each output data with each input data. The correlation is expressed in three or more levels, such as a percentage, a 10-point scale, or a 5-point scale, and is expressed, for example, by the characteristics of the line (e.g., thickness, etc.). For example, "learning electrocardiogram signal A" included in the input data shows a correlation AA of "73%" with "comfort information A" included in the output data, a correlation AB of "12%" with "comfort information B" included in the output data, and a correlation AC of "15%" with "comfort information C" included in the output data. In other words, the "correlation" indicates the degree of connection between each data, and for example, the higher the correlation, the stronger the connection between each data.

[0075] Furthermore, the neural network model may be configured to perform machine learning by providing at least one or more hidden layers between the input data and the output data. The above-mentioned correlation is set in either or both of the input data and the hidden layer data, and this serves as a weighting for each data, and output selection is based on this. If this correlation exceeds a certain threshold, that output may be selected. Furthermore, if the comfort information is a continuous value indicating, for example, a time change in a comfort index, the neural network model may use a regression model.

[0076] In step S140, the first calculator 17 uses the above-described neural network model to input the first electrocardiogram division signals separated in step S120 and outputs first comfort information.

[0077] The first calculator 17 may also calculate the first comfort information based on a scalogram image of the first electrocardiogram division signals. In this case, the first calculator 17 may calculate the first comfort information by using the scalogram image as an input, with reference to a neural network model generated using a plurality of pieces of training data, where the input data is based on a training scalogram image acquired in advance and the training comfort information associated with the input data is used as a pair of training data.

[0078] Next, in step S150, the second calculation unit 20 calculates second comfort information based on the HRV information calculated in step S130. The second comfort information is comfort information calculated based on the second electrocardiogram divided signal. The second comfort information is objective comfort information calculated by a method different from that of the first comfort information, but based on the same electrocardiogram signal. The second calculation unit 20 calculates the difference Δ of each element using, for example, equation (1). In this case, meanNNn indicates the average value of the 0th to nth healthy RRI intervals. If the difference Δ is less than 0, it is set to 1, and otherwise it is set to 0. In step S150, the second calculation unit 20 calculates the difference Δ of the other elements based on the HRV information calculated in step S130, and determines whether to set it to 1 or 0 depending on the difference Δ. The second calculation unit 20 calculates the difference Δ for each element such as the RRI average value, SDNN, LF, HF, LF / HF, SD2, ApEn, SampEn, and TotalPower based on the HRV information calculated in step S130, and calculates the second comfort information using equation (2) according to the proportion of each element for which the difference Δ is 1. The second calculation unit 20 outputs the calculated second comfort information to the third calculation unit 18.

[0079]

number

number

[0080] Next, in step S160, the third calculation unit 18 calculates third comfort information based on the first comfort information calculated in step S140 and the second comfort information calculated in step S150. The third comfort information is comfort information calculated based on the first comfort information and the second comfort information. The third calculation unit 18 calculates the third comfort information, for example, based on a weighted average of the first comfort information calculated in step S140 and the second comfort information calculated in step S150. For example, as shown in FIG. 9, the third calculation unit 18 may calculate the weighted average of the first comfort information calculated in step S140 and the second comfort information calculated in step S150, and use an approximation curve of the calculated values ​​as the third comfort information.

[0081] By performing the above steps, the operation of calculating comfort information of the bather monitoring system 100 is completed. The determination device 1 calculates the first comfort information based on the electrocardiogram signal, thereby making it possible to automatically evaluate the comfort of the user U while bathing.

[0082] Furthermore, the determination device 1 refers to a neural network model and calculates the first comfort information using the electrocardiogram signal as input. This makes it possible to more appropriately evaluate the comfort of the user U by using a neural network.

[0083] The determination device 1 also calculates the third comfort information based on the first comfort information and the second comfort information. This makes it possible to calculate the third comfort information using the first comfort information and the second comfort information calculated using different methods. This makes it possible to evaluate the comfort of the user U by combining subjective sensations and objective evaluations.

[0084] The determination device 1 also calculates the third comfort information based on a weighted average of the first comfort information and the second comfort information. Alternatively, the determination device 1 may calculate the third comfort information in real time based on the first comfort information and the second comfort information using linear regression analysis. This allows the time change and trend of the user U's comfort to be calculated, for example, by calculating an approximation curve of the weighted average of the first comfort information and the second comfort information as the third comfort information. This allows the user U's comfort to be evaluated with higher accuracy.

[0085] Furthermore, the determination device 1 calculates the first comfort information based on the ECG signal acquired in the T1 epoch, and calculates the second comfort information based on the ECG signal acquired in the T2 epoch, which is different from the T1 epoch. This allows the determination device 1 to handle ECG signals of more appropriate durations depending on the method for calculating comfort information, for example, by calculating the first comfort information based on the ECG signal acquired over a 30-second period and the second comfort information based on the ECG signal acquired over a 60-second period.

[0086] Next, the bather monitoring system 100 manages the condition of the user U based on the calculated comfort information. Fig. 10 is a flowchart showing an example of the operation of managing the user U's condition.

[0087] First, in step S210, the environment receiving unit 11 acquires various data, similar to step S110. In this case, the environment receiving unit 11 acquires environmental information about the bathroom 5 measured by the sensor 53, vital data of the user U, etc. Also, in step S210, the environment receiving unit 11 acquires image information about the image measured by the sensor 53. The environment receiving unit 11 outputs the acquired information to the control unit 12, and the control unit 12 outputs the information to the estimation unit 13.

[0088] The image information is information about an image or video including the face 59 of the user U bathing in the bathtub 51. The image information may be an image including the face 59 and the bathtub 51. The image information may also be information about an image showing the temperature distribution of the imaged object.

[0089] Next, in step S220, the estimation unit 13 acquires vital data. The vital data is information indicating the health condition of the user U. The vital data may be information such as the gender, age, height, body temperature BT before and after bathing, blood pressure BP, weight BW, pulse rate PR, etc. of the user U. The vital data may also be information on an ID such as an identification number assigned to the user U.

[0090] Furthermore, in step S220, estimation unit 13 may refer to a health database and acquire vital data based on the environmental information acquired in step S210. The health database is a database that records environmental information and vital data corresponding to the environmental information. The health database is a database that records electrocardiographic signals and vital data corresponding to the electrocardiographic signals. The health database may record a health model generated using multiple pieces of training data, with input data being environmental information and output data being vital data, and with a pair of input data and output data being training data. In such a case, this health model differs from the above-described neural network model in that the input data is environmental information and the output data is vital data.

[0091] Next, in step S230, the state determination unit 21 determines the state information of the user U based on the third comfort information calculated in step S160 and the bathing information that combines environmental information and vital data, by referring to a state database that records state information indicating the state of the user U in relation to the previously acquired comfort information and bathing information. Alternatively, the state determination unit 21 may determine the state information based only on the comfort information, without using the bathing information. Alternatively, in step S230, the state determination unit 21 determines the state information of the user U based on the third comfort information calculated in step S160 and the bathing information that combines environmental information and vital data, by referring to a state database that records the user U's past comfort information in relation to the previously acquired bathing information. In this case, the state determination unit 21 may compare the past comfort information linked to the bathing information with the third comfort information and determine the state information based on the comparison result.

[0092] The state database is a database that records state information for comfort information and bathing information. The state database may also be a database that records past comfort information for bathing information. The state database may record a state model generated using multiple pieces of training data, with comfort information and bathing information as input data, state information as output data, and a pair of input data and output data as training data. In such a case, this state model differs from the above-mentioned neural network model in that the input data is comfort information and bathing information, and the output data is state information.

[0093] The condition information is data indicating the condition of the user U, and indicates, for example, conditions such as "normal," "abnormal," "heat stroke," "myocardial infarction," "syncope," "comfortable," "uncomfortable," and "disturbed breathing." The condition information may also be a comfort index. The condition information may also be information indicating, for example, whether the user U is drowning.

[0094] Furthermore, in step S230, the state determination unit 21 may refer to the state database and determine the state information of the user U based on the change over time in the third comfort information calculated in step S160. In this case, the state determination unit 21 determines the state information using information on the change over time in the comfort information as input. In this case, for example, the state determination unit 21 may output state information of "abnormal" when the amount of change in chronologically continuous comfort information is greater than a reference value.

[0095] Furthermore, in step S230, the state determination unit 21 may determine state information of the user U based on the image information acquired in step S210. For example, the state determination unit 21 may determine the state information based on the image information by referring to a state model generated using a plurality of pieces of learning data, with the input data being information including image information, the output data being state information, and a pair of input data and output data being learning data. The state determination unit 21 may determine the state information based on, for example, image information and comfort information. For example, the state determination unit 21 may determine the state information based on the image information and comfort information by referring to a state model generated using a plurality of pieces of learning data, with the input data being image information and comfort information, the output data being state information, and a pair of input data and output data being learning data.

[0096] For example, the state determination unit 21 may extract distance information regarding the distance h between the user U's face 59 and the water surface 58 in the bathtub 51 based on an image including the user U's face 59 and the water surface 58 included in the image information, and determine the state information based on the extracted distance information. In such a case, the state determination unit 21 may recognize the face 59 and the water surface 58 from the image information using, for example, any image recognition technology, and extract the distance h from the recognized positions of the face 59 and the water surface 58. The state determination unit 21 may also extract distance information indicating whether the face 59 is above or in contact with the water surface 58. The state determination unit 21 may also extract distance information based on image information indicating the temperature distribution between the face 59 and the water surface 58 in the bathtub 51. The state determination unit 21 may, for example, refer to a relationship table indicating the relationship between distance information and state information, and determine the state information based on the extracted distance information. Furthermore, the state determination unit 21 may refer to a state model generated using multiple pieces of learning data, for example, with the input data being information including distance information, the output data being state information, and a pair of input data and output data being learning data, and determine the state information based on the distance information.

[0097] The distance information is information relating to the distance h between the face 59 and the hot water surface 58. The distance information may also be information indicating whether or not the face 59 is above the hot water surface 58. The distance information may also be information indicating whether or not the face 59 is in contact with the hot water surface 58. The distance information may also be information indicating whether or not a part of the face 59, such as the nose, is above the hot water surface 58.

[0098] Furthermore, in step S230, the state determination unit 21 may determine the state information based on image information regarding an image including the temperature distribution of the nostrils F of the face 59 as shown in FIG. 11 , acquired in step S210. In this case, the state determination unit 21 determines the state information by referring to a temperature distribution database in which previously acquired images including the temperature distribution of the nostrils F of the face 59 are associated with and recorded. The temperature distribution database is a database in which image information regarding an image including the temperature distribution of the nostrils F of the face 59 is associated with and recorded as state information. The temperature distribution database may also be a database in which past state information for image information is recorded. The temperature distribution database may record a temperature distribution model generated using multiple pieces of training data, with input data being image information and output data being state information, and with a pair of input data and output data being training data. In this case, this temperature distribution model differs from the above-described neural network model in that the input data is image information and the output data is state information. Furthermore, the state determination unit 21 may determine the state information based on image information relating to a plurality of frames of images including the temperature distribution of the nostrils F of the face 59 acquired in step S210. The state determination unit 21 may determine the state information based on, for example, a video consisting of a plurality of frames of images that are consecutive in time and include the temperature distribution of the nostrils F of the face 59. This makes it possible to infer the state of breathing and the like from the movement of the nostrils, and thus makes it possible to determine the state information with higher accuracy.

[0099] Next, in step S240, the treatment determination unit 22 determines treatment information for the status information. The treatment information is data indicating a method of treatment for the status information, such as "drain the bathtub 51," "call an ambulance," or "notify family members."

[0100] The treatment determination unit 22 uses, as a pair of judgment learning data, pre-acquired status information and treatment information indicating a method of treatment for the status information, which are recorded in the notification recording unit 23, for example, and refers to a treatment information database in which multiple judgment learning data are recorded to determine treatment information for the status information determined in step S230. Furthermore, the treatment information may be determined further according to attribute information of the user U included in the user information.

[0101] The treatment information database is a database that records treatment information for status information. The treatment information database may also be a database that records treatment information for status information and bathing information. The treatment information database may record a treatment model generated using multiple pieces of judgment learning data, with input data as status information, output data as treatment information, and a pair of input data and output data as judgment learning data. In such a case, this treatment model differs from the neural network model described above in that input data is status information and output data is treatment information. The treatment information database may also record literature on comfort.

[0102] Next, in step S250, the treatment control unit 24 transmits the treatment information determined in step S240. In this case, the output unit 15 transmits the treatment information to, for example, the terminal 2 or the monitor 54, and causes the treatment information to be presented to the family member, caregiver, or user U of the user U. This makes it possible to present an appropriate treatment method to the family member, caregiver, or user U of the user U.

[0103] Next, in step S260, drainage device 52 drains bathtub 51 in accordance with the treatment information transmitted in step S250. In this case, drainage device 52 may be set in advance to drain or not drain for each treatment information. Also, ventilation fan 56 may ventilate bathroom 5 in accordance with the treatment information. In this case, whether ventilation or not is performed may be set in advance for each treatment information.

[0104] Next, in step S270, the environment receiving unit 11 acquires result data indicating the result of the treatment for the treatment information determined in step S240. The result data is data indicating the result of the treatment given to the user U, and includes data such as "no problem" or "problem". The user recording unit 14 may record the result data transmitted from the terminal 2, for example.

[0105] Next, in step S280, the user recording unit 14, in accordance with the result data acquired in step S270, records the status information determined in step S230 and the treatment information determined in step S240 as a pair of judgment learning data in the treatment information database. For example, if the result data is "no problem," the user recording unit 14 may record the judgment learning data in the treatment information database, but if the result data is "problem," the user recording unit 14 may not record the judgment learning data in the treatment information database. Furthermore, for example, if the result data is "no problem," the correlation between the status information and the treatment information may be set to be high. This allows for feedback of the results, thereby further improving accuracy.

[0106] By performing the above-described steps, the bather monitoring system 100 completes its operation of managing the state of the user U. This makes it possible to determine the state of the user U with high accuracy.

[0107] An example of a bather monitoring system according to a second embodiment of the present invention will be described below. The bather monitoring system according to the second embodiment differs from the first embodiment in that, in step S160, comfort information is calculated based on image information, and, in step S230, condition information is determined based on the comfort information calculated in step S160. Further, explanations similar to those of the first embodiment will be omitted.

[0108] An example of the operation of the determination device 1 in the second embodiment will be described below. In the second embodiment, steps S110 to S150 may be omitted.

[0109] In step S160, the third calculator 18 calculates the comfort information based on the image information acquired in step S210. In this case, the third calculator 18 may calculate the comfort information based on the image information acquired in step S210 and output from the user recording unit 14.

[0110] In step S160, the third calculator 18 may calculate the comfort information based on the image information acquired in step S210. For example, the third calculator 18 may extract distance information regarding the distance h between the user U's face 59 and the bath water surface 58 based on an image including the user U's face 59 and the bath water surface 58 included in the image information, and calculate the comfort information based on the extracted distance information. In this case, the third calculator 18 may recognize the face 59 and the bath water surface 58 from the image information using, for example, any image recognition technology, and extract the distance h from the recognized positions of the face 59 and the bath water surface 58. The third calculator 18 may also extract distance information indicating whether the face 59 is above or in contact with the bath water surface 58. The third calculator 18 may determine the comfort information based on the extracted distance information, for example, by referring to a relationship table showing the relationship between distance information and comfort information.

[0111] Furthermore, in step S160, the third calculation unit 18 may calculate comfort information based on image information about an image including the temperature distribution of the nostrils F of the face 59, as shown in FIG. 9, acquired in step S210. In this case, the third calculation unit 18 determines comfort information by referring to a temperature distribution database in which previously acquired images including the temperature distribution of the nostrils F of the face 59 are associated with comfort information and recorded. The temperature distribution database is a database in which image information about images including the temperature distribution of the nostrils F of the face 59 is associated with comfort information and recorded. The temperature distribution database may also be a database in which past comfort information for image information is recorded. The temperature distribution database may record a temperature distribution model generated using multiple sets of training data, with input data being image information and output data being comfort information, with each pair of input data and output data being training data. In this case, this temperature distribution model differs from the above-described neural network model in that the input data is image information and the output data is comfort information.

[0112] In step S230, the determination device 1 in the second embodiment determines the state information based on the comfort information calculated in step S160. In this case, the state determination unit 21 may determine the state information based on the comfort information, similar to the method of determining the state information based on the third comfort information in the first embodiment.

[0113] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. Such novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the inventions described in the claims and their equivalents. [Explanation of symbols]

[0114] 1: Judgment device 2: Terminal 3: Server 4: Communication network 5:Bathroom 10: Housing 11: Environmental receiver 12: Control section 13: Estimation part 14: User record section 15: ECG receiver 16: Preprocessing section 17: First calculation unit 18: Third calculation unit 19:HRV calculation section 20: Second calculation unit 21: Status determination unit 22: Treatment decision section 23: Notification recording unit 24: Treatment control unit 51: Bathtub 52: Drainage device 53: Sensor 54: Monitor 55: Electrode 56: Ventilation fan 57: Camera 58: Water surface 59: Face 100: Bather monitoring system 101: CPU 102:ROM 103:RAM 104: Preservation Department 105: Interface 106: Interface 107: Interface 108: Input section 109: Display section 110: Internal bus

Claims

1. image acquisition means for acquiring image information including the face of a user bathing in a bathtub; and determining means for determining state information indicating the bathing state of the user based on the image information acquired by the image acquisition means. A bather monitoring system characterized by the above.

2. the image acquisition means acquires the image information including the face and the water surface of the bathtub; The determining means extracts distance information relating to the distance between the face and the surface of the bathtub water based on the image information acquired by the image acquiring means, and determines the state information based on the extracted distance information. The bather monitoring system according to claim 1,

3. the image acquisition means acquires the image information including a temperature distribution of the nostrils of the face; The determining means determines the state information based on the image information acquired by the image acquiring means. The bather monitoring system according to claim 1,

4. The determination means determines the state information based on the image information for learning acquired by the image acquisition means by referring to a state model generated using a plurality of pieces of learning data, the pair of learning data being input data based on image information for learning acquired in advance and learning state information linked to the input data. The bather monitoring system according to claim 1,

5. The determining means determines the state information including comfort information indicating the comfort of the user. The bather monitoring system according to claim 1,

6. Further, a draining means for draining the bathtub based on the state information determined by the determining means is provided. The bather monitoring system according to any one of claims 1 to 5, characterized in that:

7. an electrocardiogram acquiring means for acquiring an electrocardiogram signal of a user; a first calculation means for calculating first comfort information indicating a comfort level of the user based on the electrocardiogram signal acquired by the electrocardiogram acquisition means; an HRV calculation means for calculating HRV information indicating heart rate variability based on the electrocardiogram signal acquired by the electrocardiogram acquisition means; second calculation means for calculating second comfort information based on the HRV information calculated by the HRV calculation means; a third calculation means for calculating third comfort information based on the first comfort information calculated by the first calculation means and the second comfort information calculated by the second calculation means, The determining means determines the state information based on the image information acquired by the image acquiring means and the third comfort information calculated by the third calculating means. The bather monitoring system according to claim 1,

8. The system further includes a determination unit that determines the treatment information for the status information determined by the determination unit by referring to a treatment model generated using a plurality of pieces of training data, the treatment information being a pair of input data based on previously acquired training status information and training treatment information associated with the input data and indicating a treatment method for the status information. The bather monitoring system according to claim 1,

Images