Monitoring system

Abstract

To provide a monitoring system capable of improving detection accuracy of a change in a mental and physical state of a user and achieving early detection.SOLUTION: A monitoring system includes a data acquisition unit that acquires inference data including one or both of a performance history of electrical devices 200 in a house and an operation history to the electrical devices 200, an inference unit that outputs a behavior pattern of a user 10 from the inference data acquired by the data acquisition unit using a first learned model for inferring the behavior pattern of the user 10 in the house from the inference data, a determination unit that determines whether there is a change in a mental and physical state of the user 10 on the basis of a temporal change in the behavior pattern of the user 10 output by the inference unit, and an output unit that outputs a determination result by the determination unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a monitoring system. [Background technology]

[0002] There is known a device that has an acquisition unit that acquires the operation history of a household electrical appliance, counts the number of incorrect operations based on the operation history over a period of time, such as one month, and determines that the operator of the household electrical appliance may have dementia if the number of incorrect operations reaches a certain number (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-104289 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a monitoring system such as that shown in Patent Document 1, incorrect operation of electrical devices can occur not only due to dementia but also due to other mental and physical conditions such as a decline in energy, excessive stress, and mental fatigue. Therefore, a user who is notified of the possibility of dementia may recognize the diagnosis as incorrect and may not realize that they are feeling unwell. Furthermore, because the number of incorrect operation counts is based on a relatively long period, such as one month, it may be difficult to detect changes in the user's mental and physical condition early.

[0005] The present disclosure has been made to solve such problems, and its purpose is to provide a monitoring system that can improve the detection accuracy of changes in the user's mental and physical state and enable early detection. [Means for solving the problem]

[0006] The monitoring system according to the present disclosure includes a data acquisition unit that acquires inference data including one or both of the operation history of electrical devices in a house and the operation history of the electrical devices; an inference unit that outputs the behavior pattern of the person in the house from the inference data acquired by the data acquisition unit using a first trained model for inferring the behavior pattern of the person in the house from the inference data; a judgment unit that judges whether or not there has been a change in the physical and mental state of the person based on the change over time in the behavior pattern of the person in the house output by the inference unit; and an output unit that outputs the judgment result by the judgment unit. [Effects of the Invention]

[0007] The monitoring system according to the present disclosure has the effect of improving the accuracy of detecting changes in the user's mental and physical state and enabling early detection of such changes. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing the configuration of a monitoring system according to a first embodiment. [Figure 2] 1 is a block diagram showing the configuration of a monitoring system according to a first embodiment. [Figure 3] 4 is a flow diagram showing an example of the operation of the watching system according to the first embodiment. [Figure 4] 1 is a block diagram showing a configuration of a learning device according to a first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a neural network in the learning device according to the first embodiment. [Figure 6] 4 is a flowchart showing an example of the operation of the learning device according to the first embodiment. FIG. [Figure 7] FIG. 10 is a block diagram showing the configuration of a modified example of the watching system according to the first embodiment. [Figure 8] 1 is a diagram illustrating an example of a configuration for realizing the functions of a mental and physical state estimation device, a speech recognition and utterance device, and a learning device according to a first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiments for implementing a monitoring system according to the present disclosure will be described with reference to the accompanying drawings. In each drawing, identical or corresponding parts are designated by the same reference numerals, and redundant explanations will be appropriately simplified or omitted. For convenience, the following description may express the positional relationship of each structure based on the illustrated state. Note that the present disclosure is not limited to the following embodiments, and any combination of the embodiments, any modification of any component of each embodiment, or any omission of any component of each embodiment are possible within the scope of the present disclosure.

[0010] Embodiment 1 A first embodiment of the present disclosure will be described with reference to Figs. 1 to 8. Figs. 1 and 2 are block diagrams showing the configuration of a monitoring system. Fig. 3 is a flow diagram showing an example of the operation of the monitoring system. Fig. 4 is a block diagram showing the configuration of a learning device. Fig. 5 is a diagram showing an example of a neural network in the learning device. Fig. 6 is a flow diagram showing an example of the operation of the learning device. Fig. 7 is a block diagram showing the configuration of a modified example of the monitoring system. Fig. 8 is a diagram showing an example of a configuration for realizing the functions of a mental and physical state estimation device, a voice recognition and speech utterance device, and a learning device.

[0011] The monitoring system according to this embodiment is a system that monitors a user 10 in a house using information collected mainly from electrical appliances 200 in the house. As shown in FIG. 1 , the monitoring system according to this embodiment includes a mental and physical state estimation device 100. As part of monitoring of the user 10 by the monitoring system, the mental and physical state estimation device 100 estimates changes in the mental and physical state of the user 10 based on the information collected from the electrical appliances 200.

[0012] The mental and physical state estimation device 100 is connected to each of the electrical appliances 200 in the house so that they can communicate with each other. Communication between the mental and physical state estimation device 100 and the electrical appliances 200 may be wireless or wired. Furthermore, the mental and physical state estimation device 100 and the electrical appliances 200 may communicate with each other via another system, such as a Home Energy Management System (HEMS).

[0013] The electrical devices 200 in the house may specifically include, for example, various home appliances such as those listed below. · Televisions and other audio-visual appliances · Information appliances such as PCs, tablets, video game consoles, etc. ·Household appliances such as washing machines, dryers, and vacuum cleaners · Kitchen appliances such as refrigerators, rice cookers, microwave ovens, oven ranges, induction cooking heaters, automatic cookers, dishwashers, etc. Seasonal appliances such as air conditioners, electric fans, air purifiers, dehumidifiers, humidifiers, electric heaters, electric carpets, bathroom ventilation / drying / heating units, etc. · Lighting fixtures, ventilation fans, heated toilet seats, heat pump water heaters, and other home appliances

[0014] Each of the electric devices 200 is provided with an electric device storage unit 210. The electric device storage unit 210 stores electric device history information of the electric device 200 as needed. This electric device history information includes one or both of the operation history and the manipulation history of the electric device 200. The operation history of the electric device 200 includes, for example, information on power on / off, start / end of operation, details of operation, various settings, and the date and time. The manipulation history of the electric device 200 includes, for example, details of operations performed on switches, buttons, etc. on an operation panel, remote control, etc., and information on the date and time. It is desirable that the operation history and the manipulation history of the electric device 200 be recorded in as much detail as possible, for example, in units of seconds.

[0015] As shown in FIG. 1, the mental and physical state estimation device 100 includes an inference data acquisition unit 110, a behavior pattern estimation unit 121, and a learned model storage unit 300. Inference data is input to the mental and physical state estimation device 100. The inference data acquisition unit 110 acquires the inference data input to the mental and physical state estimation device 100. The inference data input to the mental and physical state estimation device 100 is device history information of the electric device 200. In other words, the inference data input to the mental and physical state estimation device 100 includes one or both of the operation history of the electric device 200 in the house and the operation history of the electric device 200.

[0016] As described above, each electric device 200 and the mental and physical state estimation device 100 are communicatively connected. Each electric device 200 transmits device history information stored in the device storage unit 210 of the electric device 200 to the mental and physical state estimation device 100. As described above, the device history information stored in the device storage unit 210 includes one or both of the operation history of the electric device 200 and the operation history of the electric device 200. The inference data acquisition unit 110 of the mental and physical state estimation device 100 acquires the device history information transmitted from each electric device 200 as inference data.

[0017] The trained model storage unit 300 stores a first trained model. The first trained model is used to infer, from inference data, the behavioral patterns of a user 10 who is a person inside a house. The first trained model stored in the trained model storage unit 300 is generated, for example, by a learning device 500 described below. The trained model storage unit 300 may be provided in a server device or the like that is capable of communicating with the mental and physical state estimation device 100.

[0018] A behavioral pattern of user 10 is a type of behavior that repeatedly appears with a certain frequency or more in the daily life and activities of user 10. The behavioral pattern of user 10 may include a pattern of user 10's behavior that is repeated regularly, such as daily, weekly, or monthly, as well as a pattern observed in user 10's behavior that is irregular. Examples of regularly repeated patterns of user 10's behavior include housework, cooking, bathing, watching television, operating an air conditioner, and the like, which are performed at approximately the same time each day. A regularly repeated pattern of user 10's behavior may include elements such as the content of the behavior (housework, etc.), the start and end times of the behavior, the duration of the behavior, the operation of electrical appliance 200 performed during the behavior, and the time required for the operation. Furthermore, even in irregular behavior of user 10, for example, if the same electrical appliance 200 is operated each time during the behavior, a pattern may be observed in the operation of this electrical appliance 200. In such cases, the operation of electrical appliance 200 performed during the behavior and the time required for the operation are elements of user 10's behavioral pattern.

[0019] The behavior pattern estimation unit 121 of the mental and physical state estimation device 100 is an inference unit that infers a behavior pattern of a user 10, who is a person in a house, from the inference data acquired by the inference data acquisition unit 110, i.e., one or both of the operation history of the electric appliances 200 in the house and the operation history of the electric appliances 200, using the first learned model stored in the learned model storage unit 300. The behavior pattern estimation unit 121 inputs the input data acquired by the inference data acquisition unit 110 into the first learned model, thereby being able to output the behavior pattern of the user 10 inferred from the input data. In this way, the behavior pattern estimation unit 121 outputs the behavior pattern of the user 10 from the input data acquired by the inference data acquisition unit 110, using the first learned model for inferring the behavior pattern of the user 10 from the input data.

[0020] The mental and physical state estimation device 100 further includes a decline in energy determination unit 131, a decline in cognitive function determination unit 132, and a mental fatigue determination unit 133. The decline in energy determination unit 131, the decline in cognitive function determination unit 132, and the mental fatigue determination unit 133 are examples of determination units that determine whether or not there has been a change in the mental and physical state of the user 10, based on a change over time in the behavior pattern of the user 10 output by the behavior pattern estimation unit 121, which is an inference unit.

[0021] In the present disclosure, when the energy decline determination unit 131, the cognitive decline determination unit 132, and the mental fatigue determination unit 133 are collectively referred to without distinction, they are referred to as the "mental and physical state change determination unit" or simply as the "determination unit." In the illustrated example, the mental and physical state estimation device 100 includes three mental and physical state change determination units, namely the energy decline determination unit 131, the cognitive decline determination unit 132, and the mental fatigue determination unit 133, but it is not necessary to include all three. It is sufficient for the mental and physical state estimation device 100 to include at least one of the energy decline determination unit 131, the cognitive decline determination unit 132, and the mental fatigue determination unit 133 as the mental and physical state change determination unit.

[0022] These determination units determine whether or not there is a change in the mental and physical state of the user 10, particularly whether or not there is a deterioration or decline in the mental and physical state of the user 10. More specifically, the decline in energy determination unit 131 determines whether or not there is a decline in the mental and physical state of the user 10, as the presence or absence of a change in the mental and physical state of the user 10. The decline in cognitive function determination unit 132 determines whether or not there is a decline in the cognitive function of the user 10, as the presence or absence of a change in the mental and physical state of the user 10. The mental fatigue determination unit 133 determines whether or not there is mental fatigue of the user 10, as the presence or absence of a change in the mental and physical state of the user 10.

[0023] The mental and physical state change determination unit may compare a change over time in an index value that quantifies the behavior pattern of user 10 output by behavior pattern estimation unit 121 with a preset reference value to determine whether or not there is a change in the mental and physical state of user 10. Specific examples of the index value that quantifies the behavior pattern of user 10 include the number of times, frequency, and time required for a specific behavior, the usage time (operating operation time) of each electrical appliance 200, the number of times, frequency, and time required for a series of operations on electrical appliance 200, etc. The mental and physical state change determination unit calculates these index values ​​for the behavior pattern of user 10 output by behavior pattern estimation unit 121. Then, for example, if the calculated index value fluctuates by more than a reference value, it determines that there is a change in the mental and physical state of user 10. The reference value for this determination is set in advance for each behavior content, type of electrical appliance 200, and index value.

[0024] Next, a specific example will be described in which each determination unit determines whether or not there is a change in the mental and physical state of the user 10. The energy fading determination unit 131 determines that the user 10 has a decline in energy when the behavior pattern of the user 10 shows a decrease in the amount of activity due to the user 10's energy fading. The energy fading determination unit 131 determines whether or not the user 10 has a decline in energy based on whether or not there is a tendency for the user 10 to decrease in activity, such as becoming lazy and doing housework less frequently, or not using complex functions of home appliances, in the change over time in the behavior pattern of the user 10.

[0025] Specifically, for example, the energy fading determination unit 131 determines whether the user 10 has lost his energy based on whether the following conditions are met: If at least one of the following conditions is met, it may be determined that the user 10 has lost his energy, or these conditions may be combined to make a comprehensive determination.

[0026] -Decreased frequency of operating home appliances (for example, reduced use of induction cooking heaters or no longer using them (less frequent cooking)) - Not making full use of the functions of home appliances (for example, when operating a rice cooker, you no longer use the functions you used to use (such as detailed settings for the finished product))

[0027] The cognitive function decline determination unit 132 determines that the user 10 has a decline in cognitive function when there are changes in the user's 10 behavioral patterns due to a decline in cognitive function, such as incorrect operation of home appliances, housework taking longer than usual (operations become slower), inability to perform difficult operations (inability to use complex functions), a decrease in frequency of operations (no longer wanting to do housework because it cannot be done efficiently), or an increase in frequency of operations (forgetting to do it and repeating the same action).

[0028] Specifically, for example, the cognitive function decline determination unit 132 determines whether or not the user 10 has a cognitive function decline based on whether or not the following conditions are met: If at least one of the following conditions is met, it may be determined that the user 10 has a cognitive function decline, or these conditions may be combined to make a comprehensive determination.

[0029] - Reduced accuracy in operating home appliances (for example, setting the air conditioner to heating even though it is summer, or starting an induction cooktop even though there is no pot on it) - Reduction in the speed of operating home appliances (for example, when repeatedly pressing the same button, such as setting the heating intensity of an induction cooking heater, the time interval between button presses becomes longer) - Not making full use of the functions of home appliances (for example, when operating a rice cooker, you no longer use the functions you used to use (such as detailed settings for the finished product)) Increase or decrease in frequency of operation (for example, opening and closing the refrigerator significantly more or less frequently, etc.)

[0030] The mental fatigue determination unit 133 determines that the user 10 is suffering from mental fatigue when behavioral changes such as incorrect operation of home appliances, taking longer to do housework than usual (movements become slower), being unable to perform difficult operations (being unable to use complex functions), and a decrease in the frequency of operations (not wanting to do housework because it is not possible to do it efficiently) are observed in the behavioral patterns of the user 10 due to reduced concentration and distraction caused by mental fatigue. Although the items are the same as those of the cognitive decline described above, the degree is different (e.g., the incidence of incorrect operations is higher in the case of cognitive decline) and / or there is no increase in the frequency of operations due to forgetting actions that the user has performed, it can be distinguished from the cognitive decline.

[0031] Specifically, for example, the cognitive function decline determination unit 132 determines whether or not the user 10 has cognitive function decline based on whether or not the following conditions are met. Note that if at least one of the following conditions is met, it may be determined that the user 10 has cognitive function decline, or a comprehensive determination may be made by combining these conditions. However, as mentioned above, from the perspective of distinguishing from cognitive function decline, it is desirable to use different determination criteria from those for cognitive function decline, or to combine multiple conditions and make a comprehensive determination.

[0032] - Reduced accuracy in operating home appliances (for example, setting the air conditioner to heating even though it is summer, or starting an induction cooktop even though there is no pot on it) - Reduction in the speed of operating home appliances (for example, when repeatedly pressing the same button, such as setting the heating intensity of an induction cooking heater, the time interval between button presses becomes longer) - Not making full use of the functions of home appliances (for example, when operating a rice cooker, you no longer use the functions you used to use (such as detailed settings for the finished product)) -Decreased frequency of operation (for example, opening and closing the refrigerator has become significantly less frequent)

[0033] In addition, the mental and physical state change determination unit may use a second trained model for inferring whether or not there has been a change in the mental and physical state of a person from the time-varying change in the behavior pattern of the user 10 within the house, and determine whether or not there has been a change in the mental and physical state of the user 10 from the time-varying change in the behavior pattern of the user 10 output by the behavior pattern estimation unit 121.

[0034] In this case, a second trained model is further stored in the trained model storage unit 300. The second trained model is used to infer whether or not there has been a change in the mental and physical state of the user 10 from a change over time in the behavior pattern of the user 10. The second trained model stored in the trained model storage unit 300 is generated, for example, by a learning device 500, which will be described later.

[0035] The mental and physical state change determination unit of the mental and physical state estimation device 100 uses the second learned model stored in the learned model storage unit 300 to infer whether or not there has been a change in the mental and physical state of the user 10 from the time change in the behavior pattern of the user 10 output by the behavior pattern estimation unit 121. The mental and physical state change determination unit inputs the time series data of the behavior pattern of the user 10 output by the behavior pattern estimation unit 121 into the second learned model, thereby being able to output whether or not there has been a change in the mental and physical state of the user 10 inferred from the input data. In this way, the mental and physical state change determination unit uses the second learned model for inferring whether or not there has been a change in the mental and physical state of the user 10 from the input data, and outputs whether or not there has been a change in the mental and physical state of the user 10 from the time series data of the behavior pattern of the user 10 output by the behavior pattern estimation unit 121.

[0036] The mental and physical state change determination unit determines, by inference using the second trained model, whether or not there is a change in the mental and physical state of the user 10, particularly whether or not there is a deterioration or decline in the mental and physical state of the user 10. That is, by inference using the second trained model, the mental and physical state change determination unit determines, by inference using the second trained model, at least one of whether or not there is a decline in the energy of the user 10, whether or not there is a decline in the cognitive function of the user 10, and whether or not there is mental fatigue of the user 10.

[0037] The mental and physical state estimation device 100 further includes an output unit 140. The output unit 140 outputs the determination results of the mental and physical state change determination units, i.e., the energy decline determination unit 131, the cognitive function decline determination unit 132, and the mental fatigue determination unit 133, to the outside of the mental and physical state estimation device 100. In the example shown in the figure, the output unit 140 transmits the determination results of the mental and physical state change determination units to the mobile terminal 20 and the electrical device 200. The mobile terminal 20 is, for example, a smartphone, tablet terminal, smartwatch, or the like carried by the user 10. The user 10 can check the determination results of the mental and physical state change determination units using the mobile terminal 20 or the electrical device 200.

[0038] When the mental and physical state change determination unit determines that the mental and physical state of user 10 has deteriorated or deteriorated, the user 10 may be notified of this by one or both of the mobile terminal 20 and the electrical device 200. In this case, when the mental and physical state change determination unit determines that the mental and physical state of user 10 has deteriorated or deteriorated, the output unit 140 may transmit a notification signal to one or both of the mobile terminal 20 and the electrical device 200. When one or both of the mobile terminal 20 and the electrical device 200 receives a notification signal from the mental and physical state estimation device 100, they notify the user 10. Furthermore, the mental and physical state estimation device 100 may be provided with a notification means.

[0039] As shown in FIG. 2 , the determination result of the mental and physical state change determination unit may be notified to a third party 30 other than the user 10, i.e., a person in the house. In this case, the output unit 140 outputs a signal for notifying the third party 30 other than the user 10 of the determination result by the mental and physical state change determination unit. If the user 10 to be monitored is elderly, examples of the third party 30 include relatives such as the user's child 31, medical professionals 32, care managers 33, and welfare officers 34. The output unit 140 transmits the determination result by the mental and physical state change determination unit to a mobile device 20 or the like possessed by these third parties 30. The output unit 140 may also transmit the determination result by the mental and physical state change determination unit to a server or the like and store the determination result by the mental and physical state change determination unit in the server or the like. In this case, the third party 30 can confirm the determination result by the mental and physical state change determination unit by accessing the server or the like from the mobile device 20 or the like possessed by the third party 30.

[0040] The output unit 140 may change the output destination of the determination result depending on the determination result by the mental and physical state change determination unit. In this case, for example, depending on whether the mental and physical state change of the user 10 is determined by the energy decline determination unit 131 to be a decline in the user 10's energy, the cognitive function decline determination unit 132 to be a decline in the user 10's cognitive function, or the mental fatigue determination unit 133 to be mentally fatigued, the output unit 140 selects or changes the output destination of the determination result from among the child 31, medical personnel 32, care manager 33, and welfare officer 34. In this way, if there is a significant change in the user 10's behavioral pattern and the symptoms are potentially serious, it is possible to ensure that measures are taken by notifying people other than the user 10, such as family members. Furthermore, even if the user 10 himself / herself does not recognize the illness despite being notified and does not take any measures, a third party 30 can take action.

[0041] Next, an example of the operation of a monitoring system including the mental and physical state estimation device 100 configured as described above will be described with reference to the flowchart in Fig. 3. First, in step S11, the inference data acquisition unit 110 of the mental and physical state estimation device 100 acquires, from the electric device 200, input data that is device history information including one or both of the operation history of the electric device 200 and the operation history of the electric device 200. In the following step S12, the behavior pattern estimation unit 121 of the mental and physical state estimation device 100 inputs the input data acquired in step S11 to a first trained model. In further following step S13, the behavior pattern estimation unit 121 outputs data on the behavior pattern of the user 10, which is the inference result obtained by inputting the input data to the first trained model in step S12.

[0042] The data output from the behavior pattern estimation unit 121 in step S13 is input to the mental and physical state change determination units, i.e., the energy decline determination unit 131, the cognitive function decline determination unit 132, and the mental fatigue determination unit 133. After step S13, the mental and physical state estimation device 100 performs the processing of step S14. In step S14, the mental and physical state change determination unit determines whether or not the mental and physical state of user 10 has deteriorated or declined, based on the behavior pattern data of user 10 input in step S13. Then, the output unit 140 outputs and notifies the result of the determination made by the mental and physical state change determination unit as to whether or not the mental and physical state of user 10 has deteriorated or declined. When the processing of step S14 is completed, the series of operations ends.

[0043] According to the monitoring system configured as described above, it is possible to determine whether or not there has been a change in the mental and physical state of user 10 based on changes in the behavioral patterns of user 10 over time, and it is possible to improve the detection accuracy and detect changes in the mental and physical state of user 10 due to not only dementia but also loss of energy, excessive stress, mental fatigue, etc., at an early stage.

[0044] The monitoring system according to this embodiment may further include a learning device 500 as shown in Fig. 4. The learning device 500 according to this embodiment learns the behavioral patterns of the user 10 described above. Next, the configuration of the learning device 500 according to this embodiment will be described with reference to Fig. 4. As shown in the figure, the learning device 500 includes a learning data acquisition unit 510 and a model generation unit 520.

[0045] The learning data acquisition unit 510 acquires learning data. The learning data includes the above-mentioned device history information of the electric device 200 and the above-mentioned behavioral patterns of the user 10. As described above, the device history information of the electric device 200 includes one or both of an operation history of the electric device 200 and an operation history for the electric device 200. The learning data is data in which the device history information of the electric device 200 and the behavioral patterns of the user 10 are associated with each other.

[0046] The model generation unit 520 learns the behavioral patterns of the user 10 from the above-mentioned learning data created based on a combination of the device history information of the electric device 200 and the behavioral patterns of the user 10. That is, the model generation unit 520 uses the learning data acquired by the learning data acquisition unit 510 to generate a first learned model that infers the behavioral patterns of the user 10 from the device history information of the electric device 200.

[0047] The learning algorithm used by the model generation unit 520 may be a known algorithm such as supervised learning or semi-supervised learning. As an example, a case where a neural network is applied will be described. The model generation unit 520 learns the behavioral patterns of the user 10 by so-called supervised learning, for example, according to a neural network model. Here, supervised learning refers to a technique in which a set of input and result (label) data is provided to the learning device 500, whereby the device learns the features of the learning data and infers the result from the input.

[0048] A neural network is composed of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer or two or more layers. For example, in a three-layer neural network as shown in Figure 5, when multiple inputs are input to the input layer (X1-X3), the values ​​are multiplied by weight W1 (w11-w16) and input to the intermediate layer (Y1-Y2), and the result is further multiplied by weight W2 (w21-w26) and output from the output layer (Z1-Z3). This output result varies depending on the values ​​of weights W1 and W2.

[0049] In the present disclosure, the neural network learns the behavioral pattern of user 10 by so-called supervised learning based on the above-mentioned learning data created based on a combination of the device history information of electric device 200 acquired by learning data acquisition unit 510 and the behavioral pattern of user 10. That is, the neural network learns by inputting the device history information of electric device 200 to the input layer and adjusting weights W1 and W2 so that the result output from the output layer is close to the behavioral pattern of user 10.

[0050] The model generation unit 520 generates and outputs a first trained model by performing the above-described learning. The trained model storage unit 300 stores the first trained model output from the model generation unit 520. As described above, the trained model storage unit 300 may be provided in, for example, a server device or the like that is capable of communicating with the mental and physical state estimation device 100, or may be provided in the mental and physical state estimation device 100 itself. The trained model storage unit 300 may also be provided in the learning device 500. Note that the learning algorithm used in the learning device 500 may be deep learning, which learns to extract feature quantities themselves, or other known methods.

[0051] Next, an example of the operation of learning device 500 configured as described above will be described with reference to the flow diagram of Fig. 6. First, in step S21, learning data acquisition unit 510 acquires learning data. Note that, although the device history information of electric device 200 and the behavioral pattern data of user 10 included in the learning data are acquired simultaneously, it is sufficient that these data are input in an associated manner, and the device history information of electric device 200 and the behavioral pattern data of user 10 may be acquired at different times.

[0052] After step S21, the learning device 500 then performs the process of step S22. In step S22, the model generation unit 520 uses the learning data acquired in step S21 to learn the behavioral pattern of the user 10 through so-called supervised learning, and generates a first trained model. In the following step S23, the trained model storage unit 300 stores the first trained model generated in step S22. When the process of step S23 is completed, the series of operations ends.

[0053] The model generation unit 520 may acquire and use a trained model from an external source. The model generation unit 520 may acquire training data from multiple electrical appliances 200 used in the same area, or may learn the behavioral patterns of the user 10 using training data collected from multiple electrical appliances 200 operating independently in different areas. It is also possible to add or remove electrical appliances 200 that collect training data from the target during the training process. Furthermore, the learning device 500 that has learned the behavioral patterns of the user 10 with respect to a certain electrical appliance 200 may be applied to another electrical appliance 200, and the behavioral patterns of the user 10 with respect to the other electrical appliance 200 may be re-learned and updated.

[0054] The learning device 500 may learn whether the mental or physical condition of the user 10 has deteriorated or deteriorated. In this case, the learning data acquisition unit 510 acquires learning data. The learning data includes time-series data of the behavioral patterns of the user 10 and whether the mental or physical condition of the user 10 has deteriorated or deteriorated. The learning data is data in which the time-series data of the behavioral patterns of the user 10, the behavioral patterns of the user 10, and whether the mental or physical condition of the user 10 has deteriorated or deteriorated are associated with each other.

[0055] The model generation unit 520 learns the behavioral patterns of the user 10, indicating whether or not the mental or physical condition of the user 10 has deteriorated or deteriorated, from the above-mentioned learning data created based on a combination of the time-series data of the behavioral patterns of the user 10, the behavioral patterns of the user 10, and whether or not the mental or physical condition of the user 10 has deteriorated or deteriorated. That is, the model generation unit 520 uses the learning data acquired by the learning data acquisition unit 510 to generate a second trained model that infers whether or not the mental or physical condition of the user 10 has deteriorated or deteriorated from the time-series data of the behavioral patterns of the user 10. In this case, the learning algorithm used by the model generation unit 520 may be a known algorithm such as supervised learning or semi-supervised learning.

[0056] In the monitoring system according to this embodiment, the mental and physical state change determination unit may determine a change in the mental and physical state of user 10 in stages. For example, the mental and physical state change determination unit may determine whether there is a sign of deterioration or decline in the mental and physical state of user 10. For example, as described above, when determining whether the mental and physical state change is due to a comparison between a time change in an index value that quantifies the behavior pattern of user 10 output by behavior pattern estimation unit 121 and a reference value, a first reference value and a second reference value that is smaller than the first reference value are set as the determination reference values, and if the time change in the index value is equal to or greater than the first reference value, it is determined that there is a deterioration or decline in the mental and physical state of user 10, and if the time change in the index value is less than the first reference value but equal to or greater than the second reference value, it is determined that there is a sign of deterioration or decline in the mental and physical state of user 10. In addition, when using a second trained model for inferring whether or not there has been a change in the mental and physical state of a person from changes over time in the person's behavior patterns within a house, and inferring whether or not there has been a change in the mental and physical state of the user 10 from changes over time in the behavior patterns of the user 10 output by the behavior pattern estimation unit 121, a trained model that has undergone machine learning may also be used to make a judgment about signs of changes in the mental and physical state of the user 10.

[0057] Furthermore, in the monitoring system according to this embodiment, a method for operating the electrical device 200 may be provided to the user 10 based on the behavioral pattern of the user 10 output by the behavioral pattern estimation unit 121. For example, if the behavioral pattern of the user 10 includes a function of the electrical device 200 that is not used or that is used infrequently, and there is a possibility that the electrical device 200 is not being used effectively, the operation method for using the function of the electrical device 200 is provided to the user 10. In addition, if the behavioral pattern of the user 10 includes a delay in operation due to an incorrect operation of the electrical device 200, the correct operation method is provided to the user 10. The operation method can be provided to the user 10, for example, as follows. That is, information for providing the operation method is transmitted from the output unit 140 to the mobile terminal 20 and the electrical device 200 of the user 10, and the operation method is notified to the user 10 from the mobile terminal 20 and the electrical device 200.

[0058] When the operation method of electric device 200 is guided to user 10 in this manner, mental and physical state change determination unit may determine whether or not there is a change in the mental and physical state of user 10 based on the operation of electric device 200 when the operation method of electric device 200 is guided. For example, cognitive function decline determination unit 132 may determine whether or not cognitive function decline of user 10 has occurred based on whether or not user 10 was able to operate electric device 200 correctly following the guidance when the operation method of electric device 200 is guided, how long it took for the operation, and the like.

[0059] Next, a modified example of the watching system according to this embodiment will be described with reference to Fig. 7. Note that Fig. 7 does not show the output unit 140 included in the mental and physical state estimation device 100. In this modified example, the watching system further includes a voice recognition and speech utterance device 400. As shown in Fig. 7, the voice recognition and speech utterance device 400 includes a voice analysis unit 410, a user utterance content estimation unit 420, a response content generation unit 430, and a voice output unit 440. The voice recognition and speech utterance device 400 also includes a microphone and a speaker, which are not shown.

[0060] Speech uttered by a user 10 inside the house is input to the microphone of the speech recognition and speech utterance device 400. The speech analysis unit 410 analyzes the speech of the user 10 input from the microphone. The user utterance content estimation unit 420 estimates the content of the speech of the user 10 based on the speech analysis result of the speech analysis unit 410. The answer content generation unit 430 generates an answer to the speech of the user 10 based on the content of the speech of the user 10 estimated by the user utterance content estimation unit 420. The audio output unit 440 generates audio of the answer content generated by the answer content generation unit 430 and outputs it from a speaker. In this way, the speech recognition and speech utterance device 400 is able to have an audio conversation with the user 10. In addition, known methods can be used for analyzing the voice of user 10 by the voice analysis unit 410, estimating the voice of user 10 by the user utterance content estimation unit 420, generating the answer content to the voice of user 10 by the answer content generation unit 430, and synthesizing and outputting the voice by the voice output unit 440.

[0061] In the monitoring system according to this modification, the mental and physical state estimation device 100 is communicatively connected to the voice recognition and utterance device 400. Communication between the mental and physical state estimation device 100 and the voice recognition and utterance device 400 may be wireless or wired. Furthermore, the mental and physical state estimation device 100 and the voice recognition and utterance device 400 may communicate via another system such as a HEMS.

[0062] The speech recognition and utterance device 400 transmits information about the content of the speech of the user 10 estimated by the user utterance content estimation unit 420 to the mental and physical state estimation device 100. The inference data acquisition unit 110 of the mental and physical state estimation device 100 acquires the utterance content information of the user 10 transmitted from each speech recognition and utterance device 400 as inference data.

[0063] In this modification, a third trained model is further stored in the trained model storage unit 300. The third trained model is used to infer, from the inference data, the speech pattern of a user 10 who is a person inside a house. The third trained model stored in the trained model storage unit 300 is generated, for example, by the learning device 500 described above.

[0064] The speech pattern of the user 10 is a type of speech that appears repeatedly with a certain frequency or more in the daily life and activities of the user 10. The speech pattern of the user 10 may include elements such as speech content, speech rate, fluency, articulation, clarity of pronunciation, etc.

[0065] In this variant, the mental and physical state estimation device 100 further includes an utterance pattern estimation unit 122. The utterance pattern estimation unit 122 is an inference unit that infers the utterance pattern of user 10, who is a person in a house, from the inference data acquired by the inference data acquisition unit 110, i.e., utterance content information of user 10, using a third learned model stored in the learned model storage unit 300. The utterance pattern estimation unit 122 inputs the input data acquired by the inference data acquisition unit 110 into the third learned model, thereby outputting the utterance pattern of user 10 inferred from the input data. In this way, the utterance pattern estimation unit 122 outputs the utterance pattern of user 10 from the input data acquired by the inference data acquisition unit 110, using the third learned model for inferring the utterance pattern of user 10 from the input data.

[0066] The function of the voice analysis unit 410 that analyzes the voice of the user 10, the function of the user utterance content estimation unit 420 that estimates the content of the utterance of the user 10, and the function of the utterance pattern estimation unit 122 that estimates the speech pattern of the user 10 may be integrated into one unit. In other words, the utterance pattern estimation unit 122 may further include a function for analyzing the voice of the user 10 and a function for estimating the content of the utterance of the user 10. In this case, the voice recognition and utterance device 400 may transmit the voice data of the user 10 to the mental and physical state estimation device 100. Then, the inference data acquisition unit 110 acquires the voice data of the user 10. Furthermore, the utterance pattern estimation unit 122 outputs the speech pattern of the user 10 from the voice data acquired by the inference data acquisition unit 110, using a third trained model for inferring the speech pattern of the user 10 from the voice data of the user 10.

[0067] In this modified example, the mental and physical state change determination units, i.e., the energy decline determination unit 131, the cognitive function decline determination unit 132, and the mental fatigue determination unit 133, determine whether or not there is a change in the mental and physical state of the user 10, based also on the change over time in the speech pattern of the user 10 output by the speech pattern estimation unit 122, which is an inference unit. Specifically, for example, the mental and physical state change determination unit determines that there is a deterioration or decline in the mental and physical state of the user 10, when the frequency of the same speech content appears more frequently in the speech pattern of the user 10, the frequency of inconsistent speech content increases, the speaking speed becomes extremely slow, or the speech becomes inarticulate.

[0068] In this modification, the mental and physical state estimation device 100 may further include an individual identification unit (not shown). The individual identification unit identifies the individual user 10 using the voice data acquired by the inference data acquisition unit 110. Identifying an individual using voice data can be achieved using a known method based on, for example, voiceprints. In this case, the speech pattern estimation unit 122 estimates and outputs the speech pattern of each individual identified by the individual identification unit. Then, the mental and physical state change determination unit determines whether or not there has been a change in the mental and physical state of each individual identified by the individual identification unit using the speech pattern. In this manner, when there are multiple users 10 who are monitoring targets in a house, it is possible to determine whether or not there has been a change in the mental and physical state of each individual. Note that the individual identification unit may also identify the individual user 10 using biometric information of the user 10 other than voice data, such as fingerprints, irises, physique, facial features, etc.

[0069] In this modification, the operation of the electric device 200 may be controlled based on the utterance content of the user 10 estimated by the user utterance content estimation unit 420 or the speech pattern estimation unit 122. In this way, the user 10 can operate the electric device 200 by voice. In this case, if the voice uttered by the user 10 includes an explicit instruction to the electric device 200, the operation of the electric device 200 may be controlled based on the instruction. Including an explicit instruction to the electric device 200 means, for example, that a specific keyword set in advance is included. An example of this keyword is a nickname given to the voice recognition and speech utterance device 400.

[0070] When the voice uttered by the user 10 does not include an explicit instruction to the electric device 200, the mental and physical state estimation device 100 or the voice recognition and speech production device 400 may suggest an operation content for the electric device 200 based on the content of the voice uttered by the user 10. For example, when the user 10 utters a monologue such as "it's hot," "it's cold," or "the air is bad," the mental and physical state estimation device 100 or the voice recognition and speech production device 400 may suggest an operation method for an air conditioner, an air purifier, or the like according to the content of the utterance. The suggestion of the operation content for the electric device 200 can be performed, for example, as follows, similar to the above-described guidance of the operation method to the user 10. That is, information suggesting the operation method is transmitted from the output unit 140 to the mobile terminal 20 and the electric device 200 of the user 10, and the operation method is notified to the user 10 from the mobile terminal 20 and the electric device 200.

[0071] According to such a modified example of the monitoring system, it is possible to determine whether or not there has been a change in the mental and physical state of the user 10 based not only on the behavioral patterns of the user 10 but also on the changes in the speech patterns over time, thereby enabling further improvement in the detection accuracy and early detection of changes in the mental and physical state of the user 10.

[0072] FIG. 8 is a diagram showing an example of a configuration for realizing the respective functions of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500 in this embodiment. The respective functions of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500 are realized, for example, by a processing circuit. The processing circuit may include a processor 101 and a memory 102. The processing circuit may also be dedicated hardware 103. A part of the processing circuit may be formed as dedicated hardware 103, and the processing circuit may further include a processor 101 and a memory 102. In the example shown in the figure, a part of the processing circuit is formed as dedicated hardware 103. Furthermore, in the example shown in the figure, the processing circuit further includes a processor 101 and a memory 102.

[0073] The processing circuit, part of which is at least one dedicated hardware 103, may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. When the processing circuit includes at least one processor 101 and at least one memory 102, the functions of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500 are each realized by software, firmware, or a combination of software and firmware.

[0074] The software and firmware are written as programs and stored in memory 102. Processor 101 realizes the functions of each unit by reading and executing the programs stored in memory 102. Processor 101 is also called a CPU (Central Processing Unit), central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, or DSP. Memory 102 may include, for example, non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, and EEPROM, or a magnetic disk, flexible disk, optical disk, compact disk, minidisk, DVD, etc.

[0075] In this way, the processing circuits of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500 can use hardware, software, firmware, or a combination of these to realize the respective functions of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500. When the processing circuits of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500 each include at least a processor 101 and a memory 102, the processor 101 executes a program stored in the memory 102 in the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500, and the hardware and software of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500 work together to realize the functions of the respective parts of the mental and physical state estimation device 100, the speech recognition and utterance device 400, and the learning device 500.

[0076] In the present disclosure, the configuration examples and modifications according to the embodiments may be combined in any manner without departing from the spirit of the present disclosure. Examples of various aspects of the present disclosure are summarized below as appendices. (Appendix 1) a data acquisition unit that acquires inference data including one or both of an operation history of an electrical device in a house and an operation history of the electrical device; an inference unit that outputs a behavior pattern of a person in the house from the data for inference acquired by the data acquisition unit, using a first trained model for inferring a behavior pattern of a person in the house from the data for inference; a determination unit that determines whether or not there is a change in the mental and physical state of the person in the house based on the time change in the behavior pattern of the person in the house output by the inference unit; A monitoring system comprising: an output unit that outputs the determination result by the determination unit. (Appendix 2) A monitoring system as described in Appendix 1, wherein the judgment unit uses a second trained model for inferring whether or not there has been a change in the mental and physical state of the person from the change in the behavior pattern of the person in the house over time, and determines whether or not there has been a change in the mental and physical state of the person from the change in the behavior pattern of the person in the house over time output by the inference unit. (Appendix 3) The monitoring system described in Appendix 1, wherein the judgment unit compares the time change in an index value that quantifies the behavior pattern of a person in the house output by the inference unit with a predetermined reference value to determine whether or not there has been a change in the person's physical or mental state. (Appendix 4) The data acquisition unit further acquires voice data of voices uttered by people in the house, the inference unit further outputs, from the voice data acquired by the data acquisition unit, the speech patterns of the people in the house, using a third trained model for inferring the speech patterns of the people in the house from the voice data; A monitoring system described in any one of Appendix 1 to Appendix 3, wherein the judgment unit further determines whether or not there has been a change in the physical and mental state of the person in the house based on the change over time in the speech pattern of the person in the house output by the inference unit. (Appendix 5) further comprising an individual identification unit that identifies an individual in the house using the voice data acquired by the data acquisition unit, the inference unit outputs, for each individual identified by the individual identification unit, an utterance pattern of the individual; A monitoring system as described in Appendix 4, wherein the determination unit determines whether or not there has been a change in the physical and mental state of each individual identified by the individual identification unit. (Appendix 6) If the voice uttered by the person in the house includes an explicit instruction to the electrical appliance, controlling the operation of the electrical appliance based on the instruction; A monitoring system as described in Appendix 4 or Appendix 5, which, if the voice made by a person in the house does not include explicit instructions to the electrical appliance, suggests operation instructions for the electrical appliance based on the content of the voice made by the person. (Appendix 7) providing guidance to the person in the house on how to operate the electrical appliance based on the behavior pattern of the person in the house output by the inference unit; A monitoring system as described in any one of Appendix 1 to Appendix 6, wherein the judgment unit judges whether or not there has been a change in the physical and mental state of people in the house based on operations on the electrical appliance when providing instructions on how to operate the electrical appliance. (Appendix 8) A monitoring system according to any one of claims 1 to 7, wherein the output unit outputs a signal to notify a third party other than the person in the house of the determination result by the determination unit. [Explanation of symbols]

[0077] 10 users 20 Mobile devices 30 Third party 31 children 32 Medical professionals 33 Care Manager 34 Civil Welfare Committee 100 Mental and physical state estimation device 101 processors 102 memory 103 Dedicated Hardware 110 Inference data acquisition unit 121 Behavioral Pattern Estimation Unit 122 Speech pattern estimation unit 131 Decline of Willpower 132 Cognitive decline assessment section 133 Mental Fatigue Assessment Unit 140 Output section 200 Electrical Equipment 210 Device storage section 300 Trained model memory unit 400 Voice recognition speech device 410 Audio Analysis Unit 420 User utterance content estimation unit 430 Answer content generation section 440 Audio output section 500 Learning Device 510 Learning data acquisition unit 520 Model Generation Unit

Claims

1. a data acquisition unit that acquires inference data including one or both of an operation history of an electrical device in a house and an operation history of the electrical device; an inference unit that outputs a behavior pattern of a person in the house from the data for inference acquired by the data acquisition unit, using a first trained model for inferring a behavior pattern of a person in the house from the data for inference; a determination unit that determines whether or not there is a change in the mental and physical state of the person in the house based on the time change in the behavior pattern of the person in the house output by the inference unit; A monitoring system comprising: an output unit that outputs the determination result by the determination unit.

2. The monitoring system of claim 1, wherein the judgment unit uses a second trained model for inferring whether or not there has been a change in the mental and physical state of the person from the time changes in the behavior patterns of the person in the house, and determines whether or not there has been a change in the mental and physical state of the person from the time changes in the behavior patterns of the person in the house output by the inference unit.

3. The monitoring system of claim 1, wherein the determination unit compares the time change in an index value that quantifies the behavioral patterns of people in the house output by the inference unit with a predetermined reference value to determine whether or not there has been a change in the physical or mental state of the person.

4. The data acquisition unit further acquires voice data of voices uttered by people in the house, the inference unit further outputs, from the voice data acquired by the data acquisition unit, the speech patterns of the people in the house, using a third trained model for inferring the speech patterns of the people in the house from the voice data; The monitoring system according to any one of claims 1 to 3, wherein the determination unit further determines whether or not there has been a change in the physical or mental state of the person in the house based on the change over time in the speech pattern of the person in the house output by the inference unit.

5. further comprising an individual identification unit that identifies an individual in the house using the voice data acquired by the data acquisition unit, the inference unit outputs, for each individual identified by the individual identification unit, an utterance pattern of the individual; The monitoring system according to claim 4 , wherein the determining unit determines whether or not there is a change in the mental and physical state of each individual identified by the individual identifying unit.

6. If the voice uttered by the person in the house includes an explicit instruction to the electrical appliance, controlling the operation of the electrical appliance based on the instruction; The monitoring system described in claim 4, wherein if the voice uttered by a person in the house does not include explicit instructions to the electrical appliance, the system suggests operation instructions for the electrical appliance based on the content of the voice uttered by the person.

7. providing guidance to the person in the house on how to operate the electrical appliance based on the behavior pattern of the person in the house output by the inference unit; The monitoring system according to any one of claims 1 to 3, wherein the determination unit determines whether or not there has been a change in the physical and mental state of a person in the house based on the operation of the electrical appliance when providing instructions on how to operate the electrical appliance.

8. The monitoring system according to claim 1 , wherein the output unit outputs a signal for notifying a third party other than a person in the house of the determination result made by the determination unit.

Images