Monitoring system, remote control device, robot, and monitoring method

Abstract

The system encourages user 1 to actively communicate with remote control device 2, thereby preventing feelings of isolation and mental health problems, especially among the elderly. [Solution] The system comprises an information input unit 18 that receives instructions or information input from user 1, an information notification unit 14 that notifies user 1 of predetermined information, a control signal output unit 15 that outputs a control signal to control a predetermined controlled device 3, an environmental sensor 17 that detects the environment around user 1, and a control unit (first control unit 10). When the output of the environmental sensor 17 satisfies a first condition, the control unit notifies user 1 via the information notification unit 14 that the controlled device 3 will be controlled to a specific state (control notification), and further, when an affirmative response to the notification is received from user 1 via the information input unit 18, the control signal output unit 15 outputs a control signal to the controlled device 3.

Description

Technical Field

[0001] The present invention relates to a monitoring system for monitoring users such as the elderly, a remote control device, a robot incorporating the remote control device, and a monitoring method.

Background Art

[0002] Elderly people often need assistance in their daily lives due to dementia or physical weakness, and monitoring is essential for maintaining their safety and QOL (Quality of Life). In particular, providing regular communication is considered to greatly contribute to the happiness of the elderly in order to prevent feelings of isolation and mental health problems among the elderly.

[0003] As a technology for communicating between so-called household appliances and their users, for example, a robot that speaks to a user through a voice speaking function, gives an operation instruction by speaking to a second household appliance having a voice recognition function, further recognizes the speech content of the user through the voice recognition function, recognizes the speech of the household appliance having the voice speaking function, and transmits an infrared ray for giving an operation instruction to the household appliance having an infrared ray receiving function through an infrared ray transmitting function is known. (Patent Document 1)

[0004] According to Patent Document 1, after giving an operation instruction to each household appliance, the robot receives a status notification (voice or notification sound) from each household appliance, analyzes the status notification, and if it is found that the household appliance is not operating as instructed, gives an operation instruction to the household appliance again, and if it is found that the household appliance is operating as instructed, stops the operation instruction. Furthermore, the robot may inform the user of the result of the operation instruction, and thereby it is said that it becomes possible to notify the user of the status information of the household appliance.

[0005] Furthermore, an air conditioning control system is known that includes a user consent confirmation unit that confirms the user's consent for each air conditioner in the target area, and the air conditioning control unit targets only the air conditioners in the target area for which the user consent has been confirmed by the user consent confirmation unit, and which repeatedly alternates between normal operation and energy-saving operation. (Patent Document 2)

[0006] According to Patent Document 2, the data storage unit of the central monitoring server stores the address information (email address, IP address, etc.) of users of air conditioners in the target area. Based on this information, the user consent confirmation unit of the central monitoring server sends a confirmation email to each user's terminal (PC or smartphone) in the target area. If a user decides that their air conditioner should be subject to fluctuation control, they reply to the central monitoring server with an email indicating their consent. This allows the user consent confirmation unit to easily and quickly confirm the user's consent to fluctuation control. Furthermore, fluctuation control enables effective energy saving in an air conditioning control system that controls multiple air conditioners in a target area. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2016-76799 [Patent Document 2] Patent No. 5720632 [Disclosure of the Invention] [Problems that the invention aims to solve]

[0008] However, the technology disclosed in Patent Document 1 is limited to improving user convenience, and communication between the user and the robot is unilateral, from the robot to the user. For this reason, it is considered difficult to encourage users, especially the elderly, to actively communicate and to prevent feelings of isolation and mental health problems among the elderly.

[0009] Furthermore, while the technology disclosed in Patent Document 2 obtains user consent regarding whether or not to perform fluctuation control in the operating mode of the air conditioner, the control is based on the result of that consent, and it is not envisioned that the user will be monitored using the consent process. In addition, Patent Document 2 does not suggest a configuration in which the air conditioner is controlled while communicating with the user each time the environment around the user changes.

[0010] The present invention was devised to solve the problems of the prior art, and its purpose is to provide a monitoring system, remote command device, robot, and monitoring method that can encourage users to actively communicate with the remote command device, thereby preventing feelings of isolation and mental health problems, especially among the elderly, and further reducing health risks or life-threatening risks to users. [Means for solving the problem]

[0011] The present invention, made to solve the aforementioned problems, is a monitoring system comprising: an information input unit that receives instructions or information input from a user; an information notification unit that notifies the user of predetermined information; a control signal output unit that outputs a control signal to a predetermined controlled device to control the controlled device; an environmental sensor that detects the environment around the user; and a control unit, wherein the control unit notifies the user via the information notification unit that the controlled device will be controlled to a specific state when the output of the environmental sensor satisfies a first condition, and further, when the control unit receives a positive response to the notification from the user via the information input unit, it outputs the control signal from the control signal output unit to the controlled device, thereby controlling the controlled device to the specific state.

[0012] This encourages user 1 to proactively engage in communication, preventing feelings of isolation and mental health problems, especially among the elderly. Furthermore, it helps maintain a comfortable environment around user 1, preventing, for example, heatstroke.

[0013] Furthermore, the present invention provides that, if the output of the environmental sensor satisfies the second condition, the control unit outputs the control signal from the control signal output unit to the controlled device, even if a negative response to the notification is received from the user, thereby controlling the controlled device to the specific state. This makes it possible to further reduce the health risks and inconveniences to the user's daily life.

[0014] Furthermore, the present invention includes a status detection unit that detects the user's status, and the control unit, if no response is received to the notification, outputs a control signal from the control signal output unit to the controlled device based on the detection result of the status detection unit. This makes it possible to reduce the health risks or life-threatening risks to user 1.

[0015] Furthermore, the present invention comprises a status detection unit for detecting the user's status and a communication unit, and the control unit, if no response is received to the notification, notifies the external party of the user's status via the communication unit based on the output of the status detection unit. This makes it possible to accurately inform family members or others of the user's status, for example, if the user is unconscious.

[0016] Furthermore, in this invention, the control unit outputs the control signal from the control signal output unit to the controlled device based on a positive response to the notification, and then inquires with the user about the response status of the controlled device via the information notification unit. This makes it possible to further activate the interaction between the user and the remote command device.

[0017] Furthermore, in this invention, voice information is input to the information input unit and voice information is output to the information notification unit. This allows the user and the remote command device to communicate through everyday conversation, fostering a sense of familiarity with the remote command device on the user's part and promoting interaction.

[0018] Furthermore, the present invention includes in the notification information based on the output of the environmental sensor, which explains why the controlled device should be controlled to the specific state. This increases the user's interest in their living environment and, moreover, enhances their understanding of, or acceptance of, controlling the controlled device to the specific state.

[0019] Furthermore, the present invention is a remote command device comprising: an information input unit that receives instructions or information input from a user; an information notification unit that notifies the user of predetermined information; a control signal output unit that outputs a control signal to a predetermined controlled device to control the controlled device; an environmental sensor that detects the environment around the user; and a control unit. The control unit notifies the user via the information notification unit that the controlled device will be controlled to a specific state when the output of the environmental sensor satisfies a first condition, and further, when the control unit receives a positive response from the user to the notification via the information input unit, it outputs the control signal from the control signal output unit to the controlled device, thereby controlling the controlled device to the specific state. This encourages users to actively communicate on their own, and in particular makes it possible to prevent feelings of isolation and mental health problems among the elderly.

[0020] Furthermore, the present invention relates to a robot equipped with the aforementioned remote control device. This makes it possible to operate controlled equipment using the robot.

[0021] In addition, when the environment around the user satisfies predetermined conditions, the present invention notifies the user that the controlled device is to be controlled to a specific state, and when a positive response to the notification is obtained from the user, outputs a control signal to the controlled device and monitors the method of controlling the controlled device to the specific state. By doing so, it is possible to encourage the user to actively communicate on their own, and in particular, it is possible to prevent the occurrence of feelings of isolation and mental health problems among the elderly.

Effects of the Invention

[0022] As described above, according to the present invention, it is possible to encourage the user to actively communicate with the remote control device on their own, and in particular, it is possible to prevent the occurrence of feelings of isolation and mental health problems among the elderly. Furthermore, it is possible to reduce the health risks or risks threatening the life of the user.

Brief Description of the Drawings

[0023] [Figure 1] Block diagram showing the configuration of the monitoring system S1 according to the first embodiment of the present invention [Figure 2] Explanatory drawing showing the outline of the remote control device 2 in the first embodiment [Figure 3] Block diagram showing the configuration of the remote control device 2 in the first embodiment [Figure 4] Flowchart showing the operation of the monitoring system S1 in the first embodiment [Figure 5] Block diagram showing the configuration of the monitoring system S1 according to the second embodiment of the present invention [Figure 6] Block diagram showing the configuration of the remote control device 2 in the second embodiment [Figure 7] Flowchart showing the operation of the monitoring system S1 in the third embodiment

Modes for Carrying Out the Invention

[0024] (First Embodiment) Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. Figure 1 is a block diagram showing the configuration of a monitoring system S1 according to the first embodiment of the present invention. The monitoring system S1 consists of a remote command device 2 and a controlled device 3. Here, the remote command device 2 includes the function of a so-called remote controller (hereinafter sometimes referred to as "remote control").

[0025] In the following explanation, User 1 is assumed to be, for example, an elderly person living alone in a residence, but is not limited to this. That is, User 1 refers to a person who uses the remote control function of the remote command device 2, and may include family members or other cohabitants. Examples of controlled devices 3 include home appliances such as televisions 3a, air conditioners 3b, and lighting fixtures 3c. Of course, controlled devices 3 are not limited to these, and any home appliance that can be controlled using a remote control, such as ON / OFF operation or setting of operating settings, is included as a controlled device 3 of the present invention. The remote command device 2 is equipped with the function of a so-called smart remote control that can operate multiple controlled devices 3.

[0026] When user 1 issues instructions to the remote control device 2 for controlling equipment, the remote control device 2 receives the instructions from user 1 and outputs a control signal to control the equipment 3 based on the instructions input by user 1. After outputting the control signal, the remote control device 2 inquires with user 1 about the response status of the equipment 3. This makes it possible to generate interaction between user 1 and the remote control device 2 each time user 1 operates a home appliance or other device via the remote control device 2.

[0027] Figure 2 is an explanatory diagram showing an overview of the remote command device 2 in the first embodiment. As shown in the figure, the remote command device 2 is composed of, for example, a bipedal robot. By making the remote command device 2 a robot, a sense of familiarity with the remote command device 2 is fostered in the user 1, and interaction can be promoted. Of course, there are no particular restrictions on the form of the remote command device 2; it may be composed of a robot toy, or a quadrupedal animal-type robot, or it may be equipped with tracks instead of two or four legs. Furthermore, it may be in a stationary form without a bipedal or other mobility mechanism 16 (described later), a form without arms, a form composed of only one housing part, a form in the form of a so-called home appliance remote control, or a form in the form of a wearable device.

[0028] The remote command device 2 comprises a main body 2BD, a head 2HD, arms 2A, legs 2L, an imaging unit 13, an information notification unit 14, a control signal output unit 15, an environmental sensor 17, an information input unit 18, and a switch unit 21. Joints (not shown) are provided in the leg portion 2L at the engagement point with the main body 2BD (base of the leg portion 2L 2s), the knee 2t, and the ankle 2u. A drive source (not shown), such as a motor, is located near the joint, and the leg portion 2L is rotatable within a predetermined range around the joint. In other words, the remote command device 2 of the first embodiment comprises a movement mechanism 16 consisting of the leg portion 2L, the joint, and the drive source.

[0029] Figure 3 is a block diagram showing the configuration of the remote command device 2 in the first embodiment. Hereinafter, the configuration and operation of the monitoring system S1 will be described in detail using Figure 3 in conjunction with Figures 1 and 2. In addition to the components described above, the remote command device 2 includes a first control unit 10, a storage unit 11, and an inertial sensor 19.

[0030] The first control unit 10 is composed of a CPU (Central Processing Unit) and operates according to a control program stored in a storage unit 11, which is composed of ROM (Read Only Memory), RAM (Random Access Memory), and the like. The first control unit 10 is connected to other components by a bus 20, and the first control unit 10 controls the other components via the bus 20. The first control unit 10 also includes a clock function.

[0031] Furthermore, the memory unit 11 includes non-volatile memory (such as EEPROM (Electrically Erasable Programmable Read-Only Memory)). This non-volatile memory stores pattern data and other information used to generate control signals for controlling the controlled device 3.

[0032] The imaging unit 13 is equipped with an image sensor composed of a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). The imaging unit 13 outputs image data.

[0033] By activating the aforementioned movement mechanism 16, the remote command device 2 can move forward and backward, and further change its posture in the left, right, and up and down directions. In other words, the remote command device 2 can change the imaging range of the imaging unit 13 by moving forward and backward or by essentially panning and tilting. As a result, based on the captured image data, the first control unit 10 can recognize the positional relationship between the remote command device 2 and the controlled device 3.

[0034] The information notification unit 14 is composed of, for example, a speaker. Audio information is provided to the user 1 via the speaker. Of course, the information notification unit 14 may also include a display device composed of, for example, an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) in addition to the speaker. In this case, the user 1 is provided with video and text information in addition to audio information.

[0035] The information input unit 18 is composed of, for example, a microphone. The output of the microphone is converted into digital data by an A / D converter (not shown). The first control unit 10 recognizes the digitized voice of user 1. In speech recognition, phonemes are extracted from the acquired voice information, converted into text, and then vocabulary information is identified. Based on this vocabulary information, the first control unit 10 recognizes the instructions of user 1 and the intent of the information input by user 1.

[0036] In the remote command device 2 of the first embodiment, voice information is input to the information input unit 18, and voice information is output to the information notification unit 14. This enables voice communication between the user 1 and the device. Of course, a touch panel superimposed on a display device may be used as the information input unit 18. In this case, the user 1 can communicate with the remote command device 2 via the touch panel or the like.

[0037] An application for inputting voice information can be realized, for example, by combining natural language processing (NLP) and machine learning models. Furthermore, by using a rule-based system that converts abstract words into specific setting changes for the controlled device 3, the first control unit 10 can determine whether the temperature setting of the air conditioner 3b should be changed to the lower side (cooling) or the luminous intensity value of the lighting fixture 3c should be increased, based on abstract utterances from user 1 that do not specify the controlled device 3, such as "It's kind of hot." or "The room is dark."

[0038] The control signal output unit 15 is composed of an infrared light-emitting diode or a transmitter that outputs radio waves. The first control unit 10 extracts the pattern data, etc., from the storage unit 11 and drives the infrared light-emitting diode or generates a radio wave signal based on the pattern data, etc. That is, the remote command device 2, including the control signal output unit 15, functions as an infrared (IR) remote control or a radio wave (RF) remote control, and an IR or RF control signal is transmitted to the controlled device 3. The control signal output unit 15 may also be composed of a communication module compliant with a short-range wireless standard such as BLE (Bluetooth® Low Energy). When the control signal output unit 15 is composed of an infrared light-emitting diode, the control signal output unit 15 is provided on at least one side (in this case, the front) of the remote command device 2 (see Figure 2).

[0039] The environmental sensor 17 may include, for example, a temperature and humidity sensor that measures the temperature (air temperature) and humidity of the external environment of the remote control device 2 (e.g., the room of user 1), an illuminance sensor that detects light intensity, a motion sensor that detects infrared rays emitted by people, etc., and an odor sensor that detects odors (which may also be a gas sensor that detects a predetermined gas) (none of which are shown). Of course, the remote control device 2 does not have to include all of these sensors. The output of the environmental sensor 17 is converted into digital data by, for example, an A / D converter included in the first control unit 10.

[0040] The inertial sensor 19 is composed of, for example, a three-axis accelerometer and / or a gyroscope. Here, the three-axis accelerometer outputs the acceleration (in acceleration) of the remote command device 2 in the direction and by how much. The gyroscope outputs the angular velocity (in angular velocity) of the remote command device 2 in the direction and by how much speed. Generally, the gyroscope detects the direction of movement and the accelerometer detects the distance traveled. In this way, the inertial sensor 19 outputs three-axis acceleration information and three-axis angular velocity information based on the movement of the remote command device 2.

[0041] The first control unit 10 detects the posture of the remote command device 2 based on the output of the inertial sensor 19. Then, when the remote command device 2 moves forward, backward (walks), or stands on one leg, the first control unit 10 controls it to ensure its balance and prevent it from falling over.

[0042] The switch unit 21 (see Figure 2) is a push switch (toggle switch) located on top of the remote command device 2. When the switch unit 21 is pressed, the remote command device 2 starts / stops operation. Of course, if the remote command device 2 is connected to a power source and power is supplied continuously, the switch unit 21 does not need to be provided.

[0043] Figure 4 is a flowchart illustrating the operation of the monitoring system S1 in the first embodiment. The operation of the monitoring system S1 will be explained below using Figure 4 in conjunction with Figures 1, 2, and 3. First, the first control unit 10 determines whether or not it has received an instruction from user 1 (ST01). Specifically, the first control unit 10 determines whether or not user 1 has given an instruction regarding the operation of the controlled device 3 (hereinafter sometimes referred to as "operation instruction") based on the result of voice recognition of the output of the information input unit 18.

[0044] These operation instructions include, for example, ON / OFF instructions for the controlled device 3, such as "Turn on the air conditioner," or "Turn off the TV," spoken by user 1, as well as instructions regarding the operation settings of the controlled device 3, such as "Set the cooling temperature to 25°C," or "Set the lights to safety lights." Furthermore, as mentioned above, abstract instructions in which the name of the controlled device 3 is not explicitly stated may also be included. Based on the operation instructions, the first control unit 10 uses the NLP, etc., described above to determine a specific operation (for example, turning it ON) for a specific controlled device 3 (for example, air conditioner 3b).

[0045] If the first control unit 10 has not received instructions from user 1 (No in ST01), it returns the process to ST01. On the other hand, if it has received instructions from user 1 (Yes in ST01), it sends a control signal to the controlled device 3 via the control signal output unit 15 (ST02). That is, it sends a control signal to a specific controlled device 3 according to a specific control content. After sending the control signal, the first control unit 10 inquires with user 1 about the response status of the controlled device 3 (ST03). Specifically, if the controlled device 3 to be operated is an air conditioner 3b and the operation content is "turn it ON", the first control unit 10 inquires with user 1 via the information notification unit 14 about the response status of the controlled device 3, such as "Has the air conditioner been turned ON?". In other words, the first control unit 10 confirms with user 1 whether the controlled device 3 has responded as intended by user 1.

[0046] Next, the first control unit 10 receives information input from user 1 (ST04). Specifically, it acquires voice information spoken by user 1 based on the output of the information input unit 18 and recognizes the content of the voice information. Based on this recognition result, the first control unit 10 determines whether the controlled device 3 has responded appropriately (ST05). If the recognition result includes so-called affirmative responses such as "yes" or "okay," or positive responses such as "it's turned on" or "it worked properly" (hereinafter sometimes referred to as "positive responses"), the first control unit 10 determines that the controlled device 3 has responded appropriately (Yes in ST05) and moves the process to ST01.

[0047] During this process, "interaction" occurs between User 1 and the remote control device 2 (robot). The robot initiates communication, and as this becomes established, it is thought that elderly people, especially those living alone, will develop a strong attachment to the robot. As attachment is nurtured, User 1 is expected to respond more actively to inquiries from the robot, and the quality and quantity of communication between them will increase. This may stimulate User 1's cognitive function. Maintaining vocabulary and conversational skills through verbal communication in this way may slow the progression of dementia.

[0048] Generally, the term "interaction" is interpreted as "when a human takes some action (operation or behavior), the system responds accordingly." However, in this specification, "interaction" is understood not only as "when a human takes some action, the system (device) responds accordingly, but also as "when the system (device) takes some action, the human responds accordingly." In other words, the present invention can establish a deeper and stronger relationship between user 1 and robot (remote command device 2 and monitoring system S1) than a typical "interaction."

[0049] On the other hand, if user 1 utters a so-called negative response such as "no" or "no," or if the recognition result includes a negative response such as "it didn't turn on" or "it's not working" (hereinafter sometimes referred to as a "negative response"), the first control unit 10 determines that the controlled device 3 is not responding appropriately (No in ST05). Then, it asks user 1 whether or not to resend the control signal (ST06). If user 1 gives a negative response to this (No in ST07), the process moves to ST01.

[0050] On the other hand, if user 1 gives an affirmative response (Yes in ST07), the first control unit 10 activates the moving mechanism 16 to move the remote command device 2 (ST08). Here, as described above, the control signal output unit 15 is located on one side (in this case, the front) of the remote command device 2. If the control signal output unit 15 includes an IR remote control function, a straight path is required for infrared communication, so if there is an obstacle between the remote command device 2 and the controlled device 3, communication will be hindered. Therefore, in order to reliably transmit the control signal, the first control unit 10 activates the moving mechanism 16 to move the remote command device 2 (for example, in a circle) and recognizes the position of, for example, the air conditioner 3b based on the image data obtained by the imaging unit 13. Then, after moving the remote command device 2 so that the air conditioner 3b is included in the imaging area of ​​the imaging unit 13, it transmits the control signal (ST02).

[0051] In this first embodiment, the robot is equipped with a remote command device 2. This makes it possible to operate the controlled device 3 using the robot. The robot in the first embodiment is also equipped with a movement mechanism 16, and the control unit (first control unit 10) inquires from the user 1 about the response status of the controlled device 3, and then, based on the information input to the information input unit 18, determines that the controlled device 3 is not responding appropriately, activates the movement mechanism 16 to change the positional relationship between the remote command device 2 and the controlled device 3. This makes it possible to secure an infrared communication path when the control signal output unit 15 includes an IR remote control function.

[0052] As described above, the monitoring system S1 (or remote command device 2) of the first embodiment includes an information input unit 18 that receives instructions or information input from user 1, an information notification unit 14 that notifies user 1 of predetermined information, a control signal output unit 15 that outputs a control signal to control a predetermined controlled device 3, and a control unit (first control unit 10). Based on the instructions of user 1 input to the information input unit 18, the control unit, after the control signal output unit 15 outputs a control signal to the controlled device 3, inquires with user 1 via the information notification unit 14 about the response status of the controlled device 3. In this way, the operation of the controlled device 3 by user 1 is supported by the inquiry about the response status. In other words, the monitoring system S1 includes a "device operation support mode". The device operation support mode makes it possible to encourage user 1 to actively communicate with the remote command device 2.

[0053] The following explanation will continue using Figure 3. As mentioned above, the remote command device 2 is equipped with an environmental sensor 17. The environmental sensor 17 includes a temperature and humidity sensor and a human presence sensor. The first control unit 10 periodically monitors the output of the environmental sensor 17. If the human presence sensor detects a user 1 near the remote command device 2, and the output of the temperature and humidity sensor falls outside a predetermined range (in this case, the range of temperature and humidity that allows for comfortable living), the first control unit 10 notifies the user 1 via the information notification unit 14 to set the controlled device 3 (in this case, the air conditioner 3b) to a specific state (hereinafter, this notification may be referred to as a "monitoring notification").

[0054] In other words, the monitoring system S1 (or remote command device 2) includes a "monitoring notification mode" that notifies the user 1 via the information notification unit 14 to set the controlled device 3 to a specific state. This makes it possible to make elderly people, in particular, who may have impaired thermoregulation or temperature sensation, aware of the need to correct the indoor environment.

[0055] Monitoring notifications are issued, for example, when the indoor temperature is higher than 30°C, when the indoor temperature is lower than 15°C, or when the discomfort index falls outside the range of 60-75. The monitoring notification includes information on temperature and humidity and instructions on how to deal with the situation. For example, voice information such as "The discomfort index is high. Please instruct me to turn on the air conditioner" is provided to user 1. In this way, the monitoring notification includes a message prompting user 1 to give instructions to the remote command device 2. When user 1 responds to this (i.e., voice information is input to the information input unit 18 instructing them to turn on the air conditioner), communication takes place between the monitoring system S1 (remote command device 2) and user 1. After this, ST01 in the flowchart shown in Figure 4 is executed, and further communication with user 1 continues.

[0056] Furthermore, in relation to the specific example of operating the air conditioner 3b described above, even if there is no response from user 1, or if user 1 gives a negative response such as "There is no need to turn on the air conditioning," if the output of the environmental sensor 17 indicates, for example, that the temperature is 30°C or higher and the relative humidity is 70% or higher, the first control unit 10 will output a control signal to the air conditioner 3b to turn on the air conditioning. In other words, if the first control unit 10 determines that user 1 is at high risk of health risks such as heatstroke, or a life-threatening risk, it will issue a monitoring notification in order to communicate with user 1, and even if a positive response to the monitoring notification is not received from user 1 (i.e., if no explicit instruction such as "set the controlled device 3 to a specific state" is received via the information input unit 18), it will operate and control the controlled device 3 to a setting that further reduces the risk to user 1.

[0057] Thus, in the monitoring system S1 (remote command device 2) of the first embodiment, if the output of the environmental sensor 17 satisfies predetermined conditions, the control unit (first control unit 10) controls the controlled device 3 to a specific state, even if the user 1 does not input an instruction to the information input unit 18 to set the controlled device 3 to a specific state. This makes it possible to reduce the health risks or life-threatening risks to the user 1.

[0058] (Second Embodiment) A second embodiment of the present invention will now be described with reference to the drawings. Figure 5 is a block diagram showing the configuration of a monitoring system S1 according to the second embodiment of the present invention. As shown in Figure 5, the monitoring system S1 includes, in addition to the components described in the first embodiment, a smart meter 4, a smart plug 5, an information terminal 6, and a server 30. Together with the remote command device 2, these components are connected to a network 50 and exchange information with each other.

[0059] The smart meter 4 includes a measurement unit, a communication module, a data processing unit, etc. (none of which are shown in the figure), and measures the energy consumption of electricity, gas, etc., of the house where user 1 resides in real time or periodically, and transmits it to the server 30 via the network 50. The smart meter 4 can also be controlled from the outside, and for example, it has a function to shut off the energy supply to the house when electricity demand is at its peak.

[0060] The smart plug 5 is a device that interposes itself between a power supply outlet in a house or the like and a designated electrical device 7, supplying power to the electrical device 7, and also works in conjunction with a smartphone or remote control device 2 to remotely turn home appliances and electronic devices ON / OFF. Furthermore, the smart plug 5 is equipped with a power measurement unit (not shown) that measures the power supplied to the electrical device 7. The smart plug 5 also includes a communication module (not shown) that complies with wireless communication standards such as LTE (Long Term Evolution), LTE-M (Long Term Evolution - Machine, LTE Cat.M1), 4G, and 5G.

[0061] In other words, the smart plug 5 turns the electrical equipment 7 connected to it ON / OFF based on control signals input via the network 50, measures the power (energy consumed) consumed by the electrical equipment 7, and transmits the measured power data to an external source via the network 50. Here, the electrical equipment 7 includes, for example, a refrigerator, a television 3a, an air conditioner 3b, a lighting fixture 3c, and other home appliances. Thus, the electrical equipment 7 has aspects of a controlled device 3, and in the second embodiment, the smart plug 5 can control the electrical equipment 7 based on control signals transmitted from the remote command device 2. In the following description, the controlled device 3 and the electrical equipment 7 may be collectively referred to as "controlled device 3, etc."

[0062] Information terminal 6 is, for example, a portable information terminal such as a smartphone or tablet, or a PC (Personal Computer), and is equipped with a display unit, input unit, etc. (not shown). Information terminal 6 is used by a person who is watching over user 1 (family member, caregiver, doctor, etc.; hereinafter sometimes referred to as "family member, etc."). As described above, the present invention aims to enable advanced communication between user 1 and the monitoring system S1 (remote command device 2), but it cannot be denied that some users 1 may feel stressed by frequent interaction with the remote command device 2.

[0063] Therefore, in the second embodiment, the remote command device 2 allows the user 1's family or others to set whether to permit or prohibit the user 1 from inquiring about the response status of the controlled device 3, thereby achieving both enhanced communication and reduced stress for the user 1. The permission or prohibition of inquiries can be set, for example, on a time-by-time basis by linking with a scheduler or the like. This makes it possible to set it so that inquiries are prohibited during the time before bedtime.

[0064] Server 30 is a well-known computer system and consists of a server control unit 30a and a server storage unit 30b. The server control unit 30a consists of a CPU and memory (not shown) and controls the components of Server 30. The server storage unit 30b is equipped with large-capacity storage configured with ROM, RAM, and RAID (Redundant Arrays of Independent Disks), etc. This large-capacity storage stores personal information for multiple users 1, such as their unique identifiers (user IDs), gender, age, and name (hereinafter, these may be collectively referred to as "user information"). User information is managed as a user database. Server 30 obtains time information from an NTP (Network Time Protocol) server (not shown).

[0065] The user database includes a communication history field for each user, which stores the history of communication between the user and the remote command device 2, and an energy consumption history field, which stores the history of energy consumption (amount of electricity used, amount of gas used, etc.).

[0066] When the first control unit 10 receives an operation instruction from user 1, when it inquires with user 1 about the response status of the controlled device 3, and when it issues a monitoring notification, it stores the content of the communication between user 1 and the remote command device 2 (which includes at least voice information or text information generated by voice recognition; hereinafter sometimes referred to as "communication information") in the communication history field, along with user 1's user ID and time information. In other words, the contents of the communication history field are updated each time any communication occurs between user 1 and the remote command device 2.

[0067] Furthermore, the smart meter 4 transmits the amount of electricity consumed in the house to the server 30 at predetermined intervals, and similarly, the smart plug 5 also transmits the amount of electricity consumed by the connected electrical equipment 7 to the server 30. The server control unit 30a stores the received amount of electricity along with time information in the energy consumption history field. The communication information stored in the communication history field is subject to analysis based on instructions from family members, etc. (especially doctors, etc.). When a doctor, etc., enters a predetermined command into the information terminal 6, the command is transmitted to the server 30, and the server control unit 30a performs various statistical and analytical processes on the communication information. The results of these processes are transmitted to the information terminal 6 and used by doctors, etc., to formulate care plans for user 1, etc.

[0068] Furthermore, a natural language processing model (NLP) may be built into the server 30. Alternatively, the server control unit 30a may access external large language models (LLMs) via the network 50. By using large language models, the monitoring system S1 can engage in natural communication (for example, chat) with user 1. That is, the remote command device 2 may have a "chat mode" for conversing with user 1.

[0069] The chat mode may be activated, for example, in the step (ST05) of the flowchart in Figure 4, which determines whether the controlled device 3 has responded appropriately, regardless of the determination result. Furthermore, the chat mode may be activated after the monitoring notification described above. It may also be activated at any time when a person is detected by the environmental sensor 17. As for the content in the chat mode, for example, if ST05 is determined to be No (the controlled device 3 has not responded appropriately), a question may be asked to make the user 1 think about the cause, or news of current events obtained from an external source via the network 50 may be conveyed to the user 1 and their opinion may be asked. Communication conducted using NLP or LLM may also be stored in the communication history field described above along with the time information.

[0070] Figure 6 is a block diagram showing the configuration of the remote command device 2 in the second embodiment. Hereafter, Figure 6 will be used in conjunction with Figure 5 for explanation. The configuration of the remote command device 2 in the second embodiment is basically the same as in the first embodiment. However, in the second embodiment, a communication unit 12 is added. The communication unit 12 includes a communication module (not shown) that conforms to wireless communication standards such as LTE, LTE-M, 4G, and 5G. The communication unit 12 may conform to the WiFi (Wireless Fidelity) standard, and the remote command device 2 may connect to the network 50 via a wireless router or the like. Alternatively, the communication unit 12 may be configured with a communication module (not shown) that conforms to a short-range wireless standard such as BLE, and information may be sent and received directly with, for example, the smart plug 5 without going through the network 50.

[0071] The first control unit 10 accesses the server storage unit 30b via the communication unit 12 and the network 50 and obtains the amount of electricity from the energy consumption history field corresponding to the user ID of user 1. Here, the amount of electricity obtained by the smart meter 4 may be separated into the amount of electricity used by each electrical device 7 (and controlled device 3) by performing a so-called disaggregation process in the server control unit 30a. On the other hand, for the smart plug 5, the amount of electricity corresponding to each electrical device 7 is obtained.

[0072] The first control unit 10 refers to the time information of the electricity consumption measured by the smart meter 4 or smart plug 5, analyzes the acquired electricity consumption in a time series, and if, for example, the amount of electricity consumed by a specific controlled device 3 (in this case, an air conditioner 3b) has increased by more than a predetermined range compared to the same month one year ago, it notifies the user 1 via the information notification unit 14 to change the set temperature of the air conditioner 3b. Of course, if the smart meter 4 measures gas consumption, it can also advise, for example, to set the hot water temperature of the water heater lower. Note that both the amount of electricity consumed and the amount of gas used are considered energy consumption.

[0073] Furthermore, the first control unit 10 can identify which controlled devices 3 consume the most power and identify peak power consumption times by understanding and analyzing the amount of power used and power consumption patterns. It can then suggest actions to save on electricity and gas bills, as well as measures to improve energy efficiency, to the user 1. These suggestions may be used as content in the chat mode described above.

[0074] Thus, in the monitoring system S1 of the second embodiment, the control unit (first control unit 10) acquires the amount of energy consumed in the user's residence and, based on the amount of energy consumed, notifies the user 1 via the information notification unit 14 to set the controlled equipment 3 to a predetermined state. This makes it possible to provide the user 1 with advice on energy conservation.

[0075] Furthermore, similar to the acquisition of power consumption, the first control unit 10 accesses the server storage unit 30b and acquires communication information from the communication history field corresponding to the user ID of user 1. It then derives the frequency with which the same keywords or phrases appear in the communication information for several days, for example. The frequency of the same keywords will increase if user 1 repeatedly gives the same operation instructions or repeatedly talks about the same episode in the chat mode described above. Based on the frequency of the same keywords, the first control unit 10 estimates whether user 1 has any signs of memory impairment or cognitive impairment.

[0076] If the frequency of occurrence of the same keyword or similar exceeds a predetermined value, the first control unit 10 determines that user 1 has signs of cognitive impairment or similar issues and notifies family members or others of this fact via the information terminal 6. If it is determined that there are no signs of cognitive impairment or similar issues, the above notification is not made. Furthermore, if it is determined that there are signs of cognitive impairment or similar issues, the first control unit 10 may extend the duration of the chat mode described above and increase the frequency with which user 1 speaks.

[0077] Thus, in the monitoring system S1 of the second embodiment, the control unit (first control unit 10) estimates whether or not there are signs of cognitive impairment in user 1 based on the information input unit 18, and if it determines that there are signs of cognitive impairment in user 1, it notifies a person other than user 1 of this fact. This makes it possible for family members, etc., to take early action if there are signs of cognitive impairment in user 1.

[0078] The server storage unit 30b contains a device database related to the controlled devices 3, etc. The device database stores information about the devices operated by the remote control, such as the type of device (television 3a, air conditioner 3b, lighting fixture 3c, etc.), manufacturer, product name, product number, multiple images of the device (hereinafter sometimes referred to as "device images"), and pattern data used to generate control signals. The device database allows other items to be referenced, for example, using the product number as a key.

[0079] When user 1 purchases a new controlled device 3 (e.g., a television 3a) or a remote command device 2, for example, user 1 performs a setting operation to associate the controlled device 3 with the remote command device 2. Here, the setting operation refers to the operation of determining the combination of a specific operation of the controlled device 3 with a control signal (pattern data) that will execute this specific operation. In the setting operation, user 1 can instruct the remote command device 2 to "make the television controllable by remote control." The first control unit 10, having received this instruction via the information input unit 18, notifies user 1 to move the remote command device 2 to the vicinity of the controlled device 3.

[0080] Subsequently, the first control unit 10 refers to the output of the inertial sensor 19 and, if it determines that the remote command device 2 has been moved and then placed back in a stationary position, starts taking images with the imaging unit 13. The captured image data, along with the type (in this case, television 3a) input by the user 1 as voice information, is transmitted to the server 30 via the communication unit 12 and the network 50. The server control unit 30a searches the equipment database using the received image data and extracts similar equipment images from the corresponding types. The server control unit 30a also calculates the similarity between images using known methods such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF), and determines that the images are similar if the similarity is greater than a predetermined value.

[0081] The server control unit 30a extracts the part number associated with the equipment image and, using that part number as a key, transmits the type, manufacturer, product name, part number, and pattern data (hereinafter sometimes referred to as "equipment information") to the remote command device 2 via the network 50. If no similar equipment image is extracted from the relevant type, or if the similarity of the image determined to have the highest similarity is below a predetermined value, NULL (no similar images) is returned.

[0082] The server control unit 30a may also extract strings from the captured image data and convert them into text. Based on this text, it may search the equipment database, extract equipment information including the manufacturer, product name, part number, etc., and transmit this equipment information to the first control unit 10.

[0083] Upon receiving the device information, the first control unit 10 notifies the user 1 of the product number via the information notification unit 14 and prompts the user 1 to confirm whether the product number matches the product number of the controlled device 3, etc., that the user 1 intends to remotely control. If the user 1 responds positively, the first control unit 10 stores the controlled device 3, etc., and the device information in association with each other in the storage unit 11. That is, the controlled device 3, etc., and the pattern data (control signal) are associated. In this way, in the second embodiment, the first control unit 10 and the server control unit 30a work together to configure the remote command device 2. Subsequently, when the user 1 says something like, "Turn on the TV," the processing shown in the flowchart of Figure 4 is executed.

[0084] If the first control unit 10 receives NULL from the server 30, it activates the movement mechanism 16 of the remote command device 2 to take images of the area around the remote command device 2. It then sends each of the captured images to the server 30, and when no more NULLs are returned, it performs the verification process described above to check whether the part numbers match.

[0085] Thus, the monitoring system S1 of the second embodiment further includes an imaging unit 13, and the control unit (first control unit 10 and server control unit 30a) associates the controlled equipment 3, etc. with control signals based on images of the controlled equipment 3, etc. captured by the imaging unit 13. In other words, the monitoring system S1 includes a "setting operation support mode". This makes it possible for user 1 to remotely operate the controlled equipment 3, etc. with an extremely simple operation, such as placing the remote command device 2 near the controlled equipment 3, etc.

[0086] In setting operations, when the first control unit 10 receives instructions from user 1, it may access the equipment database built on the server 30 to extract equipment information corresponding to the type included in user 1's instructions (in this case, television 3a), further extract candidate pattern data from the equipment information, and output a control signal from the control signal output unit 15 (i.e., perform a trial) based on the candidate pattern data. After that, the first control unit 10 queries user 1 to see if the controlled equipment 3 has responded appropriately, and if user 1 returns a positive response, it stores the controlled equipment 3 and other equipment information in association with each other in the storage unit 11. On the other hand, if user 1 returns a negative response, the first control unit 10 may select other candidate pattern data and repeat the same trial until a positive response is obtained from user 1.

[0087] Furthermore, if user 1 gives an instruction that does not specify the controlled device 3 or the content of the operation, such as "I would like to be able to operate it with a remote control," the first control unit 10 may inquire about the type, manufacturer, product name, model number, and content of the operation (e.g., ON / OFF, temperature setting, or brightness setting) of the controlled device 3 that is to be operated. If user 1 answers that the type is a television, the manufacturer is company A, and the content of the operation is described, but answers that "I don't know the product name or model number," the first control unit 10 sends search information to the server 30 specifying the type as television 3a and the manufacturer as company A. The server control unit 30a then searches the device database based on this search information, extracts candidate device information, and sends it to the first control unit 10. In this way, the first control unit 10 may use the search information obtained from user 1 to narrow down the candidate device information during the process of obtaining candidate pattern data. Of course, if user 1's initial instruction during the setting operation includes the type, manufacturer, product name, or model number, these may be used as search information. Then, the aforementioned trials may be performed using the pattern data included in the filtered device information.

[0088] Furthermore, in connection with the setting operation, the remote command device 2 may be equipped with a light-receiving element (not shown), such as a photodiode or phototransistor. For example, if the controlled device 3 cannot be operated as intended by the user 1 even after trying with the above-mentioned candidate pattern data, the first control unit 10 prompts the user 1 to hold the IR remote control attached to the controlled device 3 over the light-receiving element to output a control signal such as ON / OFF (to press a button corresponding to the operation). The first control unit 10 generates pattern data from the output of the light-receiving element and prompts the user to input the type of the controlled device 3, manufacturer, product name, product number, and the content of the operation. This makes it possible to simplify the setting operation even if the device information of the controlled device 3 is not stored in the device database. In this way, even in the setting operation, it is possible to encourage the user 1 to actively communicate with the remote command device 2.

[0089] (Third embodiment) Figure 7 is a flowchart showing the operation of the monitoring system S1 in the third embodiment. In the first embodiment, for example, when the indoor temperature is higher than 30°C, the remote command device 2 notifies the user 1 via the information notification unit 14 to set the controlled device 3 (in this case, the air conditioner 3b) to a specific state (monitoring notification). The third embodiment expands on the content of the monitoring notification.

[0090] In the third embodiment, the configuration of the monitoring system S1 and the remote command device 2 is the same as in the second embodiment (Figures 5 and 6), so a detailed explanation will be omitted. Below, the monitoring system S1 and the remote command device 2 in the third embodiment will be described using Figure 7 in conjunction with Figures 5 and 6. In the following example, we will mainly assume that the controlled device 3 is an air conditioner 3b (see Figure 1), and will describe the process of turning on (controlling to ON) an air conditioner 3b that is not operating (in the OFF state).

[0091] As described above, the remote command device 2 is equipped with an environmental sensor 17 that detects the environment around user 1. The environmental sensor 17 includes a temperature and humidity sensor and a human presence sensor. The imaging unit 13 captures images of user 1 and their surrounding environment. Since the first control unit 10 can detect the environment around user 1 based on the captured images, the imaging unit 13 functions as an environmental sensor 17. Furthermore, since the first control unit 10 can detect the situation of user 1 based on the images of user 1, the imaging unit 13 also functions as a situation detection unit.

[0092] The first control unit 10 periodically monitors the output of the environmental sensor 17. It then determines whether the output of the environmental sensor 17 satisfies a first condition (ST101). Here, the first condition is, for example, whether the output of the temperature and humidity sensor falls outside a predetermined range. Specifically, if the output of the temperature and humidity sensor falls outside a predetermined range, it is determined that the first condition is satisfied; otherwise, it is determined that the first condition is not satisfied. In addition, the first condition may be further modified by adding an AND condition that user 1 is detected near the remote control device 2 by the motion sensor. This ensures that if user 1 is absent, such as by going out, the processes from ST102 onwards are not executed.

[0093] If the output of the environmental sensor 17 does not satisfy the first condition (No in ST101), the process in ST101 is repeatedly executed. If the output of the environmental sensor 17 satisfies the first condition (Yes in ST101), the first control unit 10 notifies (previews) the user 1 via the information notification unit 14 that it will "control the controlled device 3 to a specific state" (ST102). Hereafter, the content of this notification may be referred to as a "control preview." For example, from the perspective of avoiding the risk of heatstroke, the first control unit 10 provides a control preview when the heat index (WBGT°C value) derived based on the output of the temperature and humidity sensor reaches 28°C or higher. The control preview may include environmental information (information based on the output of the environmental sensor 17, which explains why the controlled device 3 should be controlled to a specific state). Specifically, for example, voice information such as "The heat index has reached a severe warning level. Is it alright to turn on the air conditioner?" is provided to the user 1 via the speaker constituting the information notification unit 14.

[0094] Thus, the control notification is in the form of a message prompting user 1 to respond. By user 1 responding to the control notification, communication is established between the monitoring system S1 (remote command device 2) and user 1. Furthermore, by including environmental information in the control notification, it is possible to increase user 1's interest in the living environment, etc., and to further enhance their understanding of, i.e., acceptance of, controlling the target device 3 to a specific state.

[0095] Furthermore, the first condition may include, for example, the power values ​​measured by the smart plug 5 or smart meter 4 described in the second embodiment. That is, the smart plug 5 and smart meter 4 can also be included in the environmental sensor 17.

[0096] The combination of the first condition and the control notification is: • If the illuminance of the room detected by the illuminance sensor falls below a predetermined value (e.g., 40 lux): "The room has become quite dark. May I turn on the lights?" Based on the image captured by the imaging unit 13, if the television 3a (see Figure 1) has been ON for longer than a predetermined time (e.g., 3 hours): "It looks like you've been watching TV for quite a long time. Shall we turn it OFF now?" • If the heat index detected by the temperature and humidity sensor is low (e.g., below 28°C) and the health risk is low, but the power used for air conditioning exceeds a predetermined value: "The air conditioning is left on all the time and is wasting electricity. Can we turn off the air conditioning?" Such an example is possible. In this way, the first condition can be set for each controlled device 3, and furthermore, multiple first conditions can be set for a single controlled device 3.

[0097] Of course, the control notification may include information about control parameters in addition to the start (ON) / stop (OFF) control. Examples of control parameters include temperature setting and illumination setting. Specifically, a message such as "The air conditioner has been left ON for a long time and is wasting electricity. Is it okay to raise the air conditioner temperature setting by 1°C?" may be provided to user 1.

[0098] After notifying the control warning, the first control unit 10 acquires voice information spoken by user 1 based on the output of the information input unit 18 during a predetermined period (e.g., 10 seconds), and recognizes the content of the voice information (voice recognition). It then determines whether or not user 1 has responded (ST103). Here, "response received" means that some kind of utterance has been detected from user 1. In this way, in the monitoring system S1 (remote command device 2) of the third embodiment, voice information is input to the information input unit 18, and the information notification unit 14 outputs voice information. This allows user 1 and the remote command device 2 to communicate through everyday conversation, fostering a sense of familiarity between user 1 and the remote command device 2, and promoting interaction.

[0099] If user 1 responds (Yes in ST103), the first control unit 10 determines whether the response is an affirmative response (ST104). Here, for example, if user 1's response includes an affirmative response such as "Yes" or "Okay," or if the control notification includes the phrase "Is it okay to turn on the air conditioner?", if user 1's response includes a phrase indicating a specific state such as "Turn it on," or if the voice recognition results determine that user 1's response includes an expression of agreement to the control notification, then it is determined that an "affirmative response" has been received. If an affirmative response has been received (Yes in ST104), the first control unit 10 outputs a control signal to the controlled device 3 (in this case, the air conditioner 3b) via the control signal output unit 15 (ST105), and controls the controlled device 3 to a specific state (in this case, the ON state).

[0100] As described above, multiple conditions can be set for the first condition. Therefore, for example, when ST105 controls the air conditioner 3b to be turned ON, the state flag corresponding to the state of the target controlled device 3 (in this case, ON / OFF of the air conditioner 3b) is turned ON and processing is moved to ST101. As a result, in ST101, controls that result in turning ON the state flag (i.e., controls that result in a specific state being matched) are excluded from the determination of the first condition.

[0101] As described above, the monitoring system S1 (remote command device 2) of the third embodiment includes an information input unit 18 that receives instructions or information input from user 1, an information notification unit 14 that notifies user 1 of predetermined information, a control signal output unit 15 that outputs a control signal to control a predetermined control target device 3, an environmental sensor 17 that detects the environment around user 1, and a control unit (first control unit 10). When the output of the environmental sensor 17 satisfies a first condition (for example, the heat index becomes 28°C or higher), the control unit notifies user 1 via the information notification unit 14 that the control target device 3 will be controlled to a specific state (control notification). Furthermore, when the control unit obtains an affirmative response (agreement) to the notification from user 1 via the information input unit 18, it outputs a control signal from the control signal output unit 15 to the control target device 3, controlling the control target device 3 to a specific state (for example, ON). This encourages user 1 to actively communicate on their own, and makes it possible to prevent feelings of isolation and mental health problems, especially among the elderly. Furthermore, it becomes possible to maintain a comfortable environment around user 1, for example, preventing heatstroke.

[0102] Furthermore, if the remote command device 2 is in the form of a wearable device such as a smartwatch or smart ring, the information input unit 18 may include a body temperature detection unit (not shown). In this case, the first control unit 10 can determine, for example, that the risk of hypothermia is high if the user 1's body temperature drops to 35°C or below. That is, the monitoring system S1 (remote command device 2) of the third embodiment may notify the user 1 of a control warning when there is a risk of hypothermia, and after obtaining a positive response (consent) from the user 1, control the heating equipment to turn ON. It is known that shivering (shivering for a relatively long period of time) occurs in cases of mild hypothermia. The shivering may be detected by the inertial sensor 19, and a control warning may be notified based on this detection result. Of course, the presence or absence of shivering may also be determined based on images (video in this case) captured by the imaging unit 13 mounted on the remote command device 2 (a robot in this case).

[0103] On the other hand, if User 1's response is not an affirmative response (No in ST104), the first control unit 10 determines whether the response is a negative response (ST106). Here, for example, if User 1's response includes a negative response such as "no" or "no," or if the speech recognition results indicate that User 1's response includes an expression of disagreement with the control notice, it is determined that there was a "negative response." If there was no negative response (No in ST106), the first control unit 10 determines that User 1 has responded but it is neither an affirmative nor a negative response, that is, it has not responded appropriately to the control notice, and returns the process to ST102 to notify the control notice again (ST102).

[0104] Furthermore, if the number of times the loop ST102 → ST103 (Yes) → ST104 (No) → ST106 (No) → ST102 is executed (loop execution count) exceeds a predetermined number of times, it may be determined that User 1 is not responding or speaking normally, and the process may be moved to ST110 (described later), and the situation may be notified to the person monitoring User 1. In particular, if this loop occurs frequently, it may be inferred, for example, that User 1 is showing signs of dementia. The predetermined number of times is arbitrary, and for example, the person monitoring User 1, such as a family member, may operate the information terminal 6 to determine the number of loop executions. Alternatively, the number of loop executions may be determined by agreement between the operator of the monitoring system S1 and the person monitoring User 1.

[0105] If a negative response is received (Yes in ST106), the first control unit 10 determines whether the output of the environmental sensor 17 satisfies the second condition (ST107). Here, the second condition is defined as a condition that poses a greater health risk than the first condition, or a condition that causes greater disruption to daily life than the first condition. Specifically, examples include when the heat index derived based on the output of the temperature and humidity sensor becomes 31°C or higher, when the user's body temperature falls below 32°C and the symptoms of hypothermia become moderate, or when the illuminance of the living room falls below 10 lux.

[0106] If the second condition is met (Yes in ST107), the first control unit 10 outputs a control signal to the controlled device 3 (in this case, the air conditioner 3b) via the control signal output unit 15 (ST105). This controls the air conditioner 3b to turn ON, for example, if the indoor environment poses a risk of heatstroke. In other words, if the first control unit 10 determines that the user 1 is at high risk of health risks such as heatstroke, or a life-threatening risk, it notifies the user 1 of a control warning in order to communicate with the user 1, and even if the user 1 does not give a positive response to the control warning, it operates and controls the controlled device 3 to a setting that further reduces the risk to the user 1.

[0107] Thus, in the third embodiment of the monitoring system (remote command device 2), the control unit (first control unit 10) outputs a control signal from the control signal output unit 15 to the controlled device 3 and controls the controlled device 3 to a specific state, even if a negative response to the notification (control warning) is received from the user 1 when the output of the environmental sensor 17 satisfies the second condition (a condition in which the health risk is greater than that of the first condition, or a condition in which the degree of disruption to daily life is greater). This makes it possible to further reduce the health risks and disruptions to daily life for the user 1.

[0108] On the other hand, if the second condition is not met (No in ST107), the first condition is updated (ST108). Specifically, for example, the heat index that satisfies the first condition is updated to a larger value (e.g., 28.5°C), and the process is moved to ST101. As a result, in ST101, if the updated first condition is met, a control warning is issued again.

[0109] If user 1 does not respond to the control notification (No in ST103), the first control unit 10 detects user 1's movement based on the image captured by the imaging unit 13 (ST109). Of course, before executing ST109, ST102 and ST103 may be executed a predetermined number of times, for example, at intervals of several tens of seconds, and ST109 may be executed only if user 1 does not respond after being notified of the control notification multiple times.

[0110] In the ST109 process, the remote command device 2, which is in the form of a robot, can move forward and backward, and can also change its posture in the left, right, up, and down directions. That is, the remote command device 2 can change the imaging range of the imaging unit 13 by moving forward and backward or by essentially panning and tilting. The first control unit 10 may change the field of view of the imaging unit 13 and take pictures while making some kind of verbal instruction to the user 1 (for example, "Look at me," to guide the user 1 in order to acquire an image that can grasp the situation or movement of the user 1). The captured images may then be transmitted to, for example, the server 30 (see Figure 5).

[0111] In this case, the server control unit 30a detects the movement of user 1 based on the received image (ST109). For motion detection, a pose recognition model such as MediaPipe Pose can be used. For an image (still image or video) of a person, MediaPipe Pose recognizes key points (pose landmarks) from 0.nose to 32.right_foot_index (33 locations in total) and outputs the position information (coordinate values ​​(x,y coordinates)) of the recognized key points. Based on the change in this position information, it is possible to detect the movement of user 1. The detected movement of user 1, or whether or not there is movement, is transmitted to the remote command device 2 (first control unit 10) via the network 50.

[0112] If the first control unit 10 detects movement of user 1 (Yes in ST109), it returns to ST102 and notifies user 1 of the control warning again. However, if the number of times the loop ST102 → ST103 (No) → ST109 (Yes) → ST102 is executed exceeds a predetermined number of times, the process of ST110 described later may be executed. The predetermined number of times is also arbitrary and may be set in the information terminal 6, or it may be decided based on an agreement between the operator of the monitoring system S1 and the person monitoring user 1.

[0113] On the other hand, if no movement of user 1 is detected (No in ST109), the first control unit 10 infers that user 1 is in a situation where they cannot respond to the control notification, such as being unconscious. Then, it notifies the outside of user 1's situation via the communication unit 12 (ST110). Here, "outside" refers to, for example, the information terminal 6 used by someone monitoring user 1. The notification refers to sending an email or voice message to the information terminal 6, and the status of user 1 may include a message stating that "user 1 is not responding to the control notification."

[0114] Furthermore, prior to processing ST109 (detection of user 1's movement), a predetermined safety confirmation process may be performed. A concrete example of this safety confirmation process is, for example... • The information notification unit 14 emits loud music, the name of user 1, etc. • The robot controls the movement mechanism 16 to approach user 1 and make physical contact. Specifically, this involves touching user 1 with the robot's arm 2A (see Figure 2), leaning forward and touching with the user's upper body, etc. The mobile mechanism 16 is controlled to approach the user 1, and the information notification unit 14 emits music, the user 1's name, etc. The control signal is transmitted via the control signal output unit 15 to turn on the television 3a and control the volume to maximum. These are some examples.

[0115] By executing the ST109 process after the safety confirmation process, for example, if user 1 is simply sleeping, it is easy to detect the movement of user 1 once they are awake. In other words, it becomes possible to distinguish whether user 1 is sleeping or unconscious. When the safety confirmation process is executed, the "status of user 1" reported by the ST110 process may include a message stating that "despite the safety confirmation process being executed, no movement of user 1 was detected."

[0116] As described above, the monitoring system S1 (remote command device 2) of the third embodiment includes a situation detection unit (imaging unit 13) that detects the situation of user 1, and a communication unit 12. The control unit (which may include a first control unit 10 or a server control unit 30a) notifies the external party (information terminal 6) of user 1's situation via the communication unit 12 based on the output of the situation detection unit if no response is received to a notification (control warning). This makes it possible to accurately inform family members or others of user 1's situation, for example, if user 1 is unconscious.

[0117] Furthermore, the first control unit 10 transmits a control signal to the controlled device 3 and controls the controlled device 3 to a specific state (ST105). That is, the monitoring system S1 (remote command device 2) of the third embodiment includes a situation detection unit (imaging unit 13) that detects the status of the user 1, and if no response is received to a notification (control warning), the control unit (first control unit 10) outputs a control signal from the control signal output unit 15 to the controlled device 3 (e.g., air conditioner 3b) based on the detection result of the situation detection unit, and controls the controlled device 3 to a specific state (e.g., ON state). This makes it possible to reduce the health risks or life-threatening risks of the user 1.

[0118] Furthermore, after executing the process of ST105, the first control unit 10 may execute the processes from ST03 (see Figure 4) onward, as described in the first embodiment. That is, in the monitoring system S1 (remote command device 2) of the third embodiment, the control unit (first control unit 10) may, based on a positive response to the notification (control warning), output a control signal from the control signal output unit 15 to the controlled device 3, and then inquire with the user 1 about the response status of the controlled device 3 via the information notification unit 14. This makes it possible to further activate the interaction that occurs between the user 1 and the remote command device 2.

[0119] Although the monitoring system S1 according to the present invention has been described in detail based on specific embodiments, these embodiments are merely illustrative, and the present invention is not limited to these embodiments. For example, the division of processing between the first control unit 10 and the server control unit 30a may be changed as appropriate. Specifically, some or all of the processing of ST109 (detection of user 1's movement) described in the third embodiment may be performed by the first control unit 10. Furthermore, the remote command device 2 and the server 30 only need to be connected via the network 50, and for example, the server 30 may be located overseas. [Industrial applicability]

[0120] The monitoring system, remote command device, robot, and monitoring method according to the present invention can encourage users to actively communicate with the remote command device, thereby preventing feelings of isolation and mental health problems, especially among the elderly. Furthermore, it can reduce health risks or life-threatening risks to users, and can therefore be widely used in nursing care facilities, home care settings, and other similar environments. [Explanation of Symbols]

[0121] 1 User 2. Remote control device 3. Controlled devices 5 Smart Plug 6. Information terminals 10 First Control Unit 11 Storage section 13 Imaging Unit 14 Information Notification Department 15 Control signal output section 17 Environmental Sensors 18. Information Input Section 30 servers 30a Server Control Unit S1 Monitoring System

Claims

1. An information input unit that accepts instructions or information input from the user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. If the output of the environmental sensor satisfies the second condition, A monitoring system characterized in that, even if a negative response to the notification is obtained from the user, the system outputs the control signal from the control signal output unit to the controlled device and controls the controlled device to the specific state.

2. An information input unit that accepts instructions or information input from the user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, A status detection unit for detecting the user's status, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. A monitoring system characterized in that, if no response is received to the notification, the control signal output unit outputs the control signal to the controlled device based on the detection result of the status detection unit.

3. An information input unit that accepts instructions or information input from the user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, A status detection unit for detecting the user's status, Communications Department and, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. A monitoring system characterized in that, if no response is received to the notification, the system notifies the external party of the user's status via the communication unit based on the output of the status detection unit.

4. An information input unit that accepts instructions or information input from the user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. A monitoring system characterized by outputting the control signal from the control signal output unit to the controlled device based on a positive response to the notification, and then inquiring with the user about the response status of the controlled device via the information notification unit.

5. The monitoring system according to any one of claims 1 to 4, characterized in that voice information is input to the information input unit and voice information is output to the information notification unit.

6. The monitoring system according to any one of claims 1 to 4, characterized in that the notification includes information based on the output of the environmental sensor, which is the reason why the controlled device should be controlled to the specific state.

7. An information input unit that accepts instructions or information input from the user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. If the output of the environmental sensor satisfies the second condition, A remote command device characterized in that, even if a negative response to the notification is obtained from the user, the control signal output unit outputs the control signal to the controlled device and controls the controlled device to the specific state.

8. An information input unit that receives instructions or information input from a user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, A status detection unit for detecting the user's status, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. A remote command device characterized in that, if no response is received to the notification, the control signal output unit outputs the control signal to the controlled device based on the detection result of the status detection unit.

9. An information input unit that receives instructions or information input from a user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, A status detection unit for detecting the user's status, Communications Department and, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. A remote command device characterized in that, if no response is received to the notification, it notifies the external party of the user's status via the communication unit based on the output of the status detection unit.

10. An information input unit that receives instructions or information input from a user, An information notification unit that notifies the aforementioned user of predetermined information, A control signal output unit that outputs a control signal to control a predetermined controlled device, An environmental sensor that detects the environment surrounding the user, It comprises a control unit and, The control unit, If the output of the environmental sensor satisfies the first condition, The information notification unit notifies the user that the controlled device will be controlled to a specific state. Furthermore, when an affirmative response to the notification is obtained from the user via the information input unit, the control signal output unit outputs the control signal to the controlled device, thereby controlling the controlled device to the specific state. A remote command device characterized by outputting the control signal from the control signal output unit to the controlled device based on a positive response to the notification, and then inquiring with the user about the response status of the controlled device via the information notification unit.

11. A robot characterized by comprising a remote control device according to any one of claims 7 to 8.

12. If the environment surrounding the user meets the specified first condition, The first step is to notify the user that the controlled device will be controlled to a specific state, A second step is to output a control signal to the controlled device and control the controlled device to the specific state when an affirmative response to the notification is obtained from the user, If the environment surrounding the user satisfies the second condition, Even if a negative response to the notification is received from the user, the third step is to output the control signal to the controlled device and control the controlled device to the specific state, Includes, A monitoring method characterized in that the first step, the second step, and the third step are performed by a remote command device.

Images