Monitoring system, remote control device, robot, and monitoring method

Abstract

To provide a watching system, a remote command device, a robot, and a watching method that can urge users themselves to actively establish communication, thereby preventing the problem especially, loneliness and mental health of old people.SOLUTION: A watching system comprises: an information input unit 18 that receives input of an instruction or information given by a user 1; an information notification unit 14 that notifies the user 1 of predetermined information; a control signal output unit 15 that outputs, to a predetermined control target apparatus 3, a control signal for controlling the control target apparatus 3; and a control unit (first control unit 10). After the control signal output unit 15 outputs the control signal to the control target apparatus 3, the control unit inquires the user 1 about the state of response from the control target apparatus 3 through the information notification unit 14 on the basis of the instruction from the user 1 input to the information input unit 18.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a monitoring system for monitoring users such as the elderly, a remote control device, a robot incorporating the remote control device, and a monitoring method.

Background Art

[0002] The elderly often need assistance in their daily lives due to dementia or physical weakness, and monitoring is essential for maintaining their safety and quality of life (QOL). In particular, providing regular communication is considered to greatly contribute to the well-being of the elderly in order to prevent feelings of isolation and mental health problems among them.

[0003] As a technology for communicating between so-called household appliances and their users, for example, a robot is known that speaks to a user through a voice speaking function, gives an operation instruction by speaking to a second household appliance having a voice recognition function, further recognizes the user's speech content through the voice recognition function, recognizes the speech of the household appliance having the voice speaking function, and transmits an infrared ray for giving an operation instruction to a household appliance having an infrared ray receiving function through an infrared ray transmitting function. (Patent Document 1)

[0004] According to Patent Document 1, after giving an operation instruction to each household appliance, the robot receives a status notification (voice or notification sound) from each household appliance, analyzes the status notification, and if it is found that the household appliance is not operating as instructed, gives an operation instruction to the household appliance again. If it is found that the household appliance is operating as instructed, the operation instruction is stopped. Further, the robot may inform the user of the result of the operation instruction, and thereby it is considered that the user can be informed of the status information of the household appliance.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0006] However, the technology disclosed in Patent Document 1 only aims to improve the convenience of users, and the communication between the user and the robot is one-sided from the robot to the user. For this reason, it is considered difficult to actively encourage communication with users, especially the elderly, and to prevent the sense of isolation and mental health problems of the elderly.

[0007] The present invention has been devised to solve such problems of the prior art, and its object is to encourage users to actively communicate on their own, and in particular, to provide a monitoring system, a remote control device, a robot, and a monitoring method capable of preventing the sense of isolation and mental health problems of the elderly.

Means for Solving the Problems

[0008] The present invention made to solve the above problems includes an information input unit that receives an instruction or information input by a user, an information notification unit that notifies the user of predetermined information, a control signal output unit that outputs a control signal for controlling the control target device to the predetermined control target device, and a control unit. The control unit is a monitoring system that, based on the instruction of the user input to the information input unit, after the control signal output unit outputs the control signal to the control target device, inquires the user about the response status of the control target device via the information notification unit. As a result, it becomes possible to encourage the user to actively communicate with the remote control device on their own.

[0009] In addition, the present invention includes an environmental sensor, and the control unit notifies the user via the information notification unit to set the controlled device to a specific state based on the output of the environmental sensor. As a result, it becomes possible to make the user recognize the necessity of correcting the indoor environment, particularly for the elderly whose body temperature regulation function or temperature sensation may decline.

[0010] Further, in the present invention, when the output of the environmental sensor satisfies a predetermined condition, the control unit controls the controlled device to a specific state even when an instruction for setting the controlled device to a specific state by the user is not input to the information input unit. This makes it possible to reduce the risk to the user's health or the risk of threatening life.

[0011] In addition, the present invention is such that the control unit acquires the energy consumption used in the user's residence and notifies the user via the information notification unit to set the controlled device to a predetermined state based on the energy consumption. This makes it possible to give advice on energy conservation to User 1.

[0012] Further, in the present invention, voice information is input to the information input unit, and the information notification unit outputs voice information. This makes it possible to communicate with User 1 by voice.

[0013] In addition, the present invention is such that the control unit estimates the presence or absence of a sign of cognitive impairment of the user based on the information input to the information input unit, and when it is determined that the user has a sign of cognitive impairment, notifies a person other than the user to that effect. As a result, when the user has a sign of cognitive impairment, the family or the like can take early action.

[0014] Furthermore, the present invention further includes an imaging unit, and the control unit associates the controlled device with the control signal based on an image of the controlled device captured by the imaging unit. As a result, it becomes possible for the user to remotely operate the controlled device with extremely simple operations such as placing the remote command device near the controlled device.

[0015] The present invention also includes an information input unit that receives an instruction or information input by a user, an information notification unit that notifies the user of predetermined information, a control signal output unit that outputs a control signal for controlling the controlled device to a predetermined controlled device, and a control unit. The control unit, based on the instruction of the user input to the information input unit, after the control signal output unit outputs the control signal to the controlled device, inquires of the user about the response status of the controlled device via the information notification unit. This makes it possible to prompt the user to actively communicate with the remote command device.

[0016] The present invention also relates to a robot including the remote command device. This makes it possible to operate the controlled device 3 using the robot.

[0017] Furthermore, the present invention further includes a moving mechanism. The control unit, after inquiring of the user about the response status of the controlled device, based on the information input to the information input unit, when it determines that the controlled device is not responding appropriately, operates the moving mechanism to change the positional relationship between the remote command device and the controlled device. This makes it possible to secure an infrared communication path when the control signal output unit includes an IR remote control function.

[0018] Further, the present invention is a monitoring method that receives an instruction from a user, outputs a control signal to a device to be controlled based on the instruction input by the user, and inquires of the user about the response status of the device to be controlled with respect to the control signal. Thereby, every time the user operates a home appliance or the like via a remote control device, it becomes possible to generate an interaction between the user and the remote control device.

Advantages of the Invention

[0019] Thus, according to the present invention, it is possible to encourage the user to actively communicate with the remote control device by himself / herself, and in particular, it is possible to prevent the sense of isolation and mental health problems of the elderly.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0021] (First Embodiment) Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a monitoring system S1 according to the first embodiment of the present invention. The monitoring system S1 is composed of a remote command device 2 and a device to be controlled 3. Here, the remote command device 2 includes the function of a so-called remote controller (hereinafter, may be referred to as a "remote control").

[0022] In the following description, the user 1 is assumed to be, for example, an elderly person living alone in a residence, but is not limited thereto. That is, the user 1 refers to a person who uses the remote control function of the remote command device 2, and may include cohabitants such as family members. Examples of the device to be controlled 3 include home appliances such as a television 3a, an air conditioner 3b, and a lighting fixture 3c. Of course, the device to be controlled 3 is not limited to this, and any home appliance or the like that can perform device control such as ON / OFF operation and operation setting using a remote control is included in the device to be controlled 3 of the present invention. Note that the remote command device 2 has the function of a so-called smart remote control that can operate a plurality of devices to be controlled 3.

[0023] When the user 1 gives an instruction to the remote command device 2 when performing device control, the remote command device 2 receives the instruction from the user 1 and outputs a control signal for controlling the device to be controlled 3 based on the instruction input by the user 1. Then, the remote command device 2 that has output the control signal inquires of the user 1 about the response status of the device to be controlled 3. As a result, every time the user 1 operates a home appliance or the like via the remote command device 2, it is possible to generate an interaction between the user 1 and the remote command device 2.

[0024] FIG. 2 is an explanatory diagram showing an overview of the remote control device 2 in the first embodiment. As shown in the figure, the remote control device 2 is configured by, for example, a bipedal walking robot. By forming the remote control device 2 in the form of a robot, a sense of familiarity of the user 1 with respect to the remote control device 2 can be cultivated, and it becomes possible to promote interaction. Of course, the form of the remote control device 2 is not particularly limited, and it may be configured by a robot toy, or may be a quadruped animal-type robot, or may be provided with an endless track instead of bipedal or quadrupedal. Further, it may be in a form that is stationary without having a moving mechanism 16 (described later) such as bipedal, or in a form that does not have arms, or in a form composed of only one housing part, or in the form of a so-called remote control for household appliances, or in the form of a wearable device.

[0025] The remote control device 2 includes a main body body 2BD, a head 2HD, arm parts 2A, leg parts 2L, an imaging unit 13, an information notification unit 14, a control signal output unit 15, an environment sensor 17, an information input unit 18, and a switch unit 21. Joints (not shown) are provided at the parts corresponding to the engagement part with the main body body 2BD (the base 2s of the leg part 2L), the knee 2t, and the ankle 2u in the leg part 2L. A drive source (not shown) such as a motor is disposed near the joint, and the leg part 2L is rotatable within a predetermined range around the joint. That is, the remote control device 2 of the first embodiment includes a moving mechanism 16 configured by the leg part 2L, the joint, and the drive source.

[0026] FIG. 3 is a block diagram showing the configuration of the remote control device 2 in the first embodiment. Hereinafter, with reference to FIG. 3 in combination with FIGS. 1 and 2, the configuration and operation of the monitoring system S1 will be described in detail. In addition to the above-described components, the remote control device 2 includes a first control unit 10, a storage unit 11, and an inertial sensor 19.

[0027] The first control unit 10 is composed of a CPU (Central Processing Unit) or the like, and operates according to a control program stored in a storage unit 11 composed of a ROM (Read Only Memory), a RAM (Random access memory), and the like. The first control unit 10 and other components are connected by a bus 20 or the like, and the first control unit 10 controls other components via the bus 20 or the like. Note that the first control unit 10 includes a clock function.

[0028] Further, the storage unit 11 includes a non-volatile memory (such as an EEPROM (Electrically Erasable Programmable Read-Only Memory)). Pattern data and the like used for generating a control signal for controlling the controlled device 3 are stored in this non-volatile memory.

[0029] The imaging unit 13 includes an image sensor composed of a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). The imaging unit 13 outputs image data.

[0030] By operating the above-described moving mechanism 16, the remote control device 2 can move forward and backward, and further change its posture in the left-right and up-down directions. That is, the remote control device 2 can change the imaging range of the imaging unit 13 by forward / backward movement or substantial panning and tilting. Thereby, based on the captured image data, the first control unit 10 can recognize the positional relationship between the remote control device 2 and the controlled device 3.

[0031] The information notification unit 14 is composed of, for example, a speaker. Voice information is provided to the user 1 via the speaker. Of course, the information notification unit 14 may include a display device composed of, for example, an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) in addition to the speaker. In this case, video and character information are provided to the user 1 in addition to the voice information.

[0032] The information input unit 18 is composed of, for example, a microphone. The output of the microphone is converted into digital data by an A / D converter (not shown). The first control unit 10 recognizes the digitized voice of user 1. In voice recognition, phonemes are extracted from the acquired voice information, converted into text, and then vocabulary information is specified. Based on this vocabulary information, the first control unit 10 recognizes the instructions of user 1 and the intention of the information input by user 1.

[0033] Thus, in the remote control device 2 of the first embodiment, voice information is input to the information input unit 18, and the information notification unit 14 outputs the voice information. As a result, voice communication with user 1 can be achieved. Of course, as the information input unit 18, a touch panel superimposed on a display device or the like may be used. In this case, user 1 can communicate with the remote control device 2 via the touch panel or the like.

[0034] The application for inputting voice information is realized, for example, by combining natural language processing (NLP) and a machine learning model. By using a rule-based system that converts more abstract words into specific setting changes of the controlled device 3, based on an abstract utterance of user 1 that does not specify the controlled device 3, such as "It seems hot." or "The room is dark.", the first control unit 10 can determine that the temperature setting of the air conditioner 3b should be changed to the low temperature side (cooling), or the light intensity value of the lighting fixture 3c should be increased.

[0035] The control signal output unit 15 is composed of a transmitter that outputs an infrared light-emitting diode or radio waves. The first control unit 10 extracts the above-described pattern data and the like from the storage unit 11, and drives an infrared light-emitting diode or generates a radio wave signal based on the pattern data and the like. That is, the remote control device 2 including the control signal output unit 15 functions as an infrared (IR) remote control or a radio frequency (RF) remote control, and an IR or RF control signal is transmitted to the controlled device 3. Note that the control signal output unit 15 may be composed of a communication module compliant with a short-range wireless standard such as BLE (Bluetooth (registered trademark) Low Energy). When the control signal output unit 15 is composed of an infrared light-emitting diode, the control signal output unit 15 is provided on at least one surface (here, the front surface) of the remote control device 2 (see FIG. 2).

[0036] The environmental sensor 17 may include, for example, a temperature (air temperature) and humidity sensor that measures the temperature and humidity of the external environment of the remote control device 2 (for example, the living room of the user 1), an illuminance sensor that detects illuminance, a human presence sensor that detects infrared rays emitted by a person or the like, and a smell sensor that detects a smell (which may be a gas sensor that detects a predetermined gas) (none of them are shown in the figure). Of course, not all of these sensors need to be included in the remote control device 2. The output of the environmental sensor 17 is converted into digital data by, for example, an A / D converter included in the first control unit 10.

[0037] The inertial sensor 19 is composed of, for example, a three-axis acceleration sensor and / or a gyro sensor. Here, the three-axis acceleration sensor outputs, for the three axes of X, Y, and Z, how much the remote control device 2 changes its speed in which direction (acceleration). The gyro sensor outputs, for the three axes of X, Y, and Z, how much the remote control device 2 rotates in which direction at what speed (angular velocity). Generally, the moving direction is detected by the gyro sensor, and the moving distance is detected by the acceleration sensor. In this way, the inertial sensor 19 outputs three-axis acceleration information and three-axis angular velocity information based on the movement of the remote control device 2.

[0038] The first control unit 10 detects the attitude of the remote control device 2 based on the output of the inertial sensor 19. Then, when the remote control device 2 moves forward / backward (walks) or stands on one leg, etc., it controls to ensure the balance of the remote control device 2 so that it does not fall.

[0039] The switch unit 21 (see FIG. 2) is a push switch (toggle switch) arranged on the top of the remote control device 2. When the switch unit 21 is pressed, the remote control device 2 starts / stops operating. Of course, when the remote control device 2 is connected to the power supply and power is constantly supplied, the switch unit 21 may not be provided.

[0040] FIG. 4 is a flowchart showing the operation of the monitoring system S1 in the first embodiment. Hereinafter, the operation of the monitoring system S1 will be described by using FIGS. 1, 2, and 3 in combination with FIG. 4. First, the first control unit 10 determines whether it has received an instruction from the user 1 (ST01). Specifically, the first control unit 10 determines whether the user 1 has given an instruction regarding the operation of the controlled device 3 (hereinafter, may be referred to as an "operation instruction") based on the result of voice recognition of the output of the information input unit 18.

[0041] This operation instruction includes, for example, ON / OFF instructions for the controlled device 3 such as "Turn on the air conditioner." and "Turn off the TV." uttered by the user 1, and instructions regarding the operation settings of the controlled device 3 such as "Set the cooling temperature to 25°C." and "Change the lighting to a security light." Further, as described above, it may also include an abstract instruction in which the name of the controlled device 3 is not specified. The first control unit 10 determines specific operation content (for example, turn on) for a specific controlled device 3 (for example, the air conditioner 3b) based on the operation instruction by utilizing the above-described NLP, etc.

[0042] When the first control unit 10 has not received an instruction from user 1 (No in ST01), it returns the process to ST01. On the other hand, when it has received an instruction from user 1 (Yes in ST01), it transmits a control signal to the controlled device 3 via the control signal output unit 15 (ST02). That is, it transmits a control signal corresponding to specific control content to a specific controlled device 3. After transmitting the control signal, the first control unit 10 inquires user 1 about the response status of the controlled device 3 (ST03). Specifically, when the controlled device 3 to be operated is the air conditioner 3b and the operation content is "turn it on", the first control unit 10 inquires user 1 about the response status of the controlled device 3, such as "Has the air conditioner been turned on?", via the information notification unit 14. That is, the first control unit 10 confirms with user 1 whether the controlled device 3 has responded as intended by user 1.

[0043] Next, the first control unit 10 receives an information input from user 1 (ST04). Specifically, it acquires the voice information uttered by user 1 based on the output of the information input unit 18 and recognizes the content of the voice information. Based on this recognition result, the first control unit 10 determines whether the controlled device 3 has responded appropriately (ST05). When user 1 utters a so-called affirmative response word such as "Yes." or "Sure.", or when an affirmative response (hereinafter sometimes referred to as an "affirmative response"), such as "It's turned on." or "It's working properly.", is included in the recognition result, the first control unit 10 determines that the controlled device 3 has responded appropriately (Yes in ST05) and moves the process to ST01.

[0044] During this process, "interaction" occurs between user 1 and the remote control device 2 (robot). It is considered that when the robot initiates communication and this becomes established, especially for the elderly living alone, they will develop a strong attachment to the robot. As the attachment grows, user 1 is expected to respond more actively to inquiries from the robot, increasing the quality and quantity of communication between the user and the robot. This can stimulate user 1's cognitive function. Maintaining vocabulary and conversation skills through such language-based communication may slow the progression of dementia.

[0045] In general, the term "interaction" is interpreted as "when a human performs some action (operation or behavior), the system returns a corresponding reaction". However, in this specification, "interaction" is understood as "not only when a human performs some action and the system (device) returns a corresponding reaction, but also when the system (device) performs some action and the human returns a corresponding reaction". That is, according to the present invention, a deeper and stronger relationship can be constructed between user 1 and the robot (remote control device 2 and monitoring system S1) than a general "interaction".

[0046] On the other hand, when user 1 utters so-called negative response words such as "No." or "No.", or when negative responses such as "It didn't turn on." or "It's not moving." (hereinafter sometimes referred to as "negative responses") are included in the recognition result, the first control unit 10 determines that the controlled device 3 is not responding appropriately (No in ST05). Then, it asks user 1 whether to retransmit the control signal (ST06). If user 1 returns a negative response to this (No in ST07), the process proceeds to ST01.

[0047] When the other party, User 1, returns an affirmative response (Yes in ST07), the first control unit 10 activates the moving mechanism 16 to move the remote command device 2 (ST08). Here, as described above, the control signal output unit 15 is provided on one surface (here, the front surface) of the remote command device 2. When the control signal output unit 15 includes an IR remote control function, a direct path is required for communication by infrared rays. Therefore, if there is an obstacle between the remote command device 2 and the controlled device 3, the communication will be hindered. Thus, in order to reliably transmit the control signal, the first control unit 10 activates the moving mechanism 16 to move (e.g., orbit) the remote command device 2, and based on the image data obtained by the imaging unit 13, recognizes the position of, for example, the air conditioner 3b. Then, after moving the remote command device 2 so that the imaging area of the imaging unit 13 includes the air conditioner 3b, the control signal is transmitted (ST02).

[0048] Here, in the first embodiment, it can be said that the robot is equipped with the remote command device 2. As a result, it becomes possible to operate the controlled device 3 using the robot. Further, the robot of the first embodiment is equipped with the moving mechanism 16, and the control unit (first control unit 10), after inquiring the user 1 about the response status of the controlled device 3, based on the information input to the information input unit 18, when it determines that the controlled device 3 is not responding appropriately, activates the moving mechanism 16 to change the positional relationship between the remote command device 2 and the controlled device 3. Thereby, when the control signal output unit 15 includes an IR remote control function, it becomes possible to secure an infrared communication path.

[0049] As described above, the monitoring system S1 (or the remote control device 2) of the first embodiment includes an information input unit 18 that receives an instruction or information input by the user 1, an information notification unit 14 that notifies the user 1 of predetermined information, a control signal output unit 15 that outputs a control signal for controlling the control target device 3 to the control target device 3, and a control unit (the first control unit 10). The control unit, based on the instruction of the user 1 input to the information input unit 18, after the control signal output unit 15 outputs a control signal to the control target device 3, inquires of the user 1 about the response status of the control target device 3 via the information notification unit 14. In this way, the operation of the control target device 3 of the user 1 is supported by inquiring about the response status. That is, the monitoring system S1 has a "device operation support mode". The device operation support mode makes it possible to encourage the user 1 to actively communicate with the remote control device 2 on their own.

[0050] Hereinafter, the description will continue with reference to FIG. 3. As described above, the remote control device 2 includes an environmental sensor 17. The environmental sensor 17 includes a temperature and humidity sensor and a human presence sensor. The first control unit 10 periodically monitors the output of the environmental sensor 17. When the user 1 is detected in the vicinity of the remote control device 2 by the human presence sensor and the output of the temperature and humidity sensor deviates from a predetermined range (here, the range of temperature and humidity in which one can spend comfortably), the user 1 is notified via the information notification unit 14 to set the control target device 3 (here, the air conditioner 3b) to a specific state (hereinafter, this notification may be referred to as a "monitoring notification").

[0051] That is, the monitoring system S1 (or the remote control device 2) has a "monitoring notification mode" in which the user 1 is notified via the information notification unit 14 to set the control target device 3 to a specific state. This makes it possible to make the elderly, in particular, who may have a reduced body temperature regulation function or temperature sensation, recognize the need to correct the indoor environment.

[0052] The monitoring notification is issued, for example, when the indoor temperature is higher than 30°C, when the indoor temperature is lower than 15°C, or when the discomfort index is outside the range of 60 to 75. The monitoring notification includes information on temperature and humidity and coping methods. For example, voice information such as "The discomfort index is high. Please instruct me to turn on the air conditioner." is provided to user 1. In this way, the monitoring notification includes a message prompting the remote control device 2 to give an instruction, and when user 1 responds to this (that is, here, voice information is input to the information input unit 18 to turn on the air conditioner), communication is achieved between the monitoring system S1 (remote control device 2) and user 1. Then, ST01 of the flowchart shown in FIG. 4 is executed, and further communication with user 1 continues.

[0053] Regarding the specific example related to the operation of the air conditioner 3b described above, even when there is no response from user 1 or when user 1 gives a negative response indicating "It is not necessary to turn on the air conditioner.", if the output of the environmental sensor 17 indicates, for example, that the temperature is 30°C or higher and the relative humidity is 70% or higher, the first control unit 10 outputs a control signal to turn on the air conditioner for the air conditioner 3b. That is, when it is determined that the risk to the health of user 1, such as heat stroke, or the risk of threatening life is high, while issuing a monitoring notification from the perspective of communicating with user 1, even in a situation where no positive response to the monitoring notification is obtained from user 1 (that is, when no explicit instruction such as "Set the controlled device 3 to a specific state via the information input unit 18" is received), the controlled device 3 is operated and controlled to a setting that reduces the risk of user 1 more.

[0054] In this way, in the monitoring system S1 (remote control device 2) of the first embodiment, when the output of the environmental sensor 17 satisfies a predetermined condition, the control unit (first control unit 10) controls the control target device 3 to a specific state even when no instruction for setting the control target device 3 to a specific state is input by the user 1 to the information input unit 18. This makes it possible to reduce the health risk or the risk threatening the life of the user 1.

[0055] (Second Embodiment) Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. FIG. 5 is a block diagram showing the configuration of the monitoring system S1 according to the second embodiment of the present invention. As shown in FIG. 5, the monitoring system S1 includes a smart meter 4, a smart plug 5, an information terminal 6, and a server 30 in addition to the components described in the first embodiment. Together with the remote control device 2, these components are connected to the network 50 and exchange information with each other.

[0056] The smart meter 4 includes a measurement unit, a communication module, a data processing unit, etc. (none of which are shown), measures the energy consumption such as electricity and gas in a house where the user 1 lives in real time or periodically, and transmits it to the server 30 via the network 50. The smart meter 4 can also be controlled from the outside, and has a function of cutting off the energy supply to a house, etc., for example, when the power demand reaches a peak.

[0057] The smart plug 5 is a device that is interposed between a power supply port provided in a house or the like and a predetermined electrical device 7, supplies power to the electrical device 7, and cooperates with a smartphone or a remote command device 2 to turn on / off home appliances and electronic devices remotely. Further, the smart plug 5 is provided with a power measurement unit (not shown) that measures the power supplied to the electrical device 7. Also, the smart plug 5 is equipped with a communication module (not shown) compliant with wireless communication standards such as LTE (Long Term Evolution), LTE-M (Long Term Evolution - Machine, LTE Cat.M1), 4G, and 5G.

[0058] That is, based on the control signal input via the network 50, the smart plug 5 turns on / off the electrical device 7 connected to the smart plug 5, further measures the power (consumed energy) consumed by the electrical device 7, and transmits the power data measured via the network 50 to the outside. Here, the electrical device 7 includes, for example, a refrigerator, a TV 3a, an air conditioner 3b, a lighting fixture 3c, and other home appliances. In this way, the electrical device 7 has an aspect as a device to be controlled 3. In the second embodiment, the smart plug 5 can control the electrical device 7 based on the control signal transmitted from the remote command device 2. In the following description, the device to be controlled 3 and the electrical device 7 may be collectively referred to as "device to be controlled 3 etc.".

[0059] The information terminal 6 is, for example, a portable information terminal such as a smartphone or a tablet terminal, or a PC (Personal Computer), and includes a display unit, an input unit, etc. not shown. The information terminal 6 is used by a person (family member, caregiver, doctor, etc. Hereinafter, may be referred to as "family etc.") who watches over the user 1. As described above, the present invention aims to achieve advanced communication between the user 1 and the monitoring system S1 (remote command device 2), but it cannot be denied that there are some users 1 who feel stressed by frequent interactions with the remote command device 2.

[0060] Therefore, in the second embodiment, the remote control device 2 enables family members or the like of the user 1 to set whether to permit or prohibit the remote control device 2 from inquiring the user 1 about the response status of the device 3 to be controlled, thereby achieving both enhanced communication and reduced stress for the user 1. The permission or prohibition of the inquiry can be set for each time period, for example, in association with a scheduler or the like. Thus, it is also possible to set the time period before going to bed to prohibit inquiries.

[0061] The server 30 is a known computer system, and includes a server control unit 30a and a server storage unit 30b. The server control unit 30a is composed of a CPU and a storage unit (not shown), and controls the components of the server 30. The server storage unit 30b includes a large-capacity storage composed of a ROM, a RAM, and a RAID (Redundant Arrays of Independent Disks) or the like. In this large-capacity storage, personal information such as unique identifiers (user IDs), genders, ages, and names of a plurality of users 1 (hereinafter, these may be collectively referred to as "user information") is stored. The user information is managed as a user database. The server 30 obtains time information from an NTP (Network Time Protocol) server (not shown).

[0062] Note that the user database is provided with a communication history field for storing the communication history that occurs between the remote control device 2 and each user 1, and an energy consumption history field for storing the history of energy consumption (such as the amount of electricity used and the amount of gas).

[0063] When the first control unit 10 receives an operation instruction from the user 1, when it inquires the user 1 about the response status of the controlled device 3, and when it issues a monitoring notification, the content of the communication between the user 1 and the remote command device 2 (including at least either voice information or text information generated by performing voice recognition. Hereinafter, it may be referred to as "communication information".) is stored in the communication history field together with the user ID of the user 1 and time information. That is, every time any communication occurs between the user 1 and the remote command device 2, the content of the communication history field is updated.

[0064] Also, the smart meter 4 transmits the amount of power consumed in the house to the server 30 at a predetermined cycle, and similarly, the smart plug 5 also transmits the amount of power consumed by the electrical device 7 to which it is connected to the server 30. The server control unit 30a stores the received power consumption amount in the energy consumption history field together with the time information. Note that the communication information stored in the communication history field is subject to analysis based on the instructions of family members, etc. (especially doctors, etc.). When a doctor or the like inputs a predetermined command to the information terminal 6, the command is transmitted to the server 30, and the server control unit 30a executes various statistical processes and analysis processes on the communication information. The results of these processes are transmitted to the information terminal 6 and are used for formulating a care plan for the user 1 by a doctor or the like.

[0065] Furthermore, a natural language processing model (NLP) may be constructed in the server 30. Alternatively, the server control unit 30a may access an external large language model (LLM: Large Language Models) via the network 50. By using the large language model, the monitoring system S1 can achieve natural communication (here, for example, chat) with the user 1. That is, the remote command device 2 may be provided with a "chat mode" for having a conversation with the user 1.

[0066] The chat mode may be activated regardless of the determination result in the step (ST05) of determining whether the controlled device 3 has appropriately responded, for example, in the flowchart of FIG. 4. Further, the chat mode may be activated after the above-described monitoring notification. Also, it may be activated at any timing when a person is detected by the environmental sensor 17. As the content in the chat mode, for example, when ST05 is a No determination (the controlled device 3 has not appropriately responded), questions may be issued to make the user 1 think about the cause, and current news obtained from the outside via the network 50 may be conveyed to the user 1 and the user's feelings may be solicited. Communication performed using NLP or LLM may also be stored in the above-described communication history field together with the time information.

[0067] FIG. 6 is a block diagram showing the configuration of the remote command device 2 in the second embodiment. Hereinafter, FIG. 6 will be described in combination with FIG. 5. The configuration of the remote command device 2 in the second embodiment is basically the same as that in the first embodiment. However, in the second embodiment, a communication unit 12 is added. The communication unit 12 includes a communication module (not shown) compliant with a wireless communication standard such as LTE, LTE-M, 4G, or 5G. The communication unit 12 may be compliant with the WiFi (Wireless Fidelity) standard, and the remote command device 2 may be connected to the network 50 via a wireless router or the like. Also, the communication unit 12 may be configured by a communication module (not shown) compliant with a short-range wireless standard such as BLE, and information may be directly transmitted and received to and from, for example, the smart plug 5 without going through the network 50.

[0068] The first control unit 10 accesses the server storage unit 30b via the communication unit 12 and the network 50, and acquires the power consumption from the energy consumption history field corresponding to the user ID of the user 1. Here, regarding the power consumption obtained by the smart meter 4, the server control unit 30a may perform so-called disaggregation processing to separate it into the power consumption used by each electric device 7 (as well as the device to be controlled 3). On the other hand, regarding the smart plug 5, the power consumption corresponding to each electric device 7 is acquired.

[0069] The first control unit 10 refers to the time information when the power consumption is measured by the smart meter 4 or the smart plug 5, analyzes the acquired power consumption in time series, and compares it with the same month one year ago, for example. When the power consumption consumed by a specific device to be controlled 3 etc. (here, for example, the air conditioner 3b) has increased beyond a predetermined range, the first control unit 10 notifies the user 1 via the information notification unit 14 to change the set temperature of the air conditioner 3b. Of course, when the gas consumption is measured by the smart meter 4, advice can also be given, for example, to set the hot water supply temperature of the water heater lower. Note that both the consumed power consumption and the gas usage amount are regarded as energy consumption.

[0070] Also, the first control unit 10 can grasp and analyze the used power consumption and power consumption pattern to extract which device to be controlled 3 etc. consumes the most power, the time zone when the power peaks, etc. Then, actions and energy efficiency improvement measures for saving electricity and gas costs can be presented to the user 1. These may be used as one of the contents in the above-mentioned chat mode.

[0071] Thus, in the monitoring system S1 of the second embodiment, the control unit (the first control unit 10) acquires the energy consumption used in the residence of the user 1, and based on the energy consumption, notifies the user 1 via the information notification unit 14 to set the device to be controlled 3 to a predetermined state. Thereby, it becomes possible to give advice on energy saving to the user 1.

[0072] Also, similar to the case of acquiring power consumption, the first control unit 10 accesses the server storage unit 30b and acquires communication information from the communication history field corresponding to the user ID of the user 1. Then, for example, it derives the frequency of appearance of the same keyword or phrase in the communication information for several days. When the user 1 repeatedly performs the same operation instruction many times, or when the user 1 repeatedly talks about the same episode in the above-described chat mode, etc., the frequency of appearance of the same keyword or the like will be high. According to the frequency of appearance of the same keyword or the like, the first control unit 10 estimates whether or not the user 1 has a sign of memory impairment or cognitive impairment.

[0073] When the frequency of appearance of the same keyword or the like becomes larger than a predetermined value, the first control unit 10 determines that there is a sign of cognitive impairment or the like in the user 1, and notifies the family or the like to that effect via the information terminal 6. When it is determined that there is no sign of cognitive impairment or the like, the above-described notification is not performed. Further, when it is determined that there is a sign of cognitive impairment or the like, the first control unit 10 may lengthen the period of the above-described chat mode and increase the frequency at which the user 1 utters words.

[0074] Thus, in the monitoring system S1 of the second embodiment, the control unit (the first control unit 10) estimates the presence or absence of a sign of cognitive impairment of the user 1 based on the information input to the information input unit 18, and when it is determined that the user 1 has a sign of cognitive impairment, notifies the fact to a person other than the user 1. As a result, when there is a sign of cognitive impairment in the user 1, the family or the like can take corresponding measures early.

[0075] Now, in the server storage unit 30b, a device database regarding the controlled device 3 and the like is constructed. In the device database, the type (TV 3a, air conditioner 3b, lighting fixture 3c, etc.), manufacturer, product name, product number, a plurality of images of the device taken (hereinafter, may be referred to as "device images"), pattern data used for generating a control signal, etc. of the device operated by the remote control are stored. The device database is referable to other items using, for example, the product number as a key.

[0076] When user 1 purchases a new device to be controlled 3 (such as TV 3a) or a remote control device 2, for example, a setting operation is performed to associate the device to be controlled 3, etc. with the remote control device 2. Here, the setting operation refers to an operation of determining a combination of a specific operation of the device to be controlled 3, etc. and a control signal (pattern data) for executing this specific operation. In the setting operation, user 1 can give an instruction to the effect of "want to be able to remotely control the TV" to the remote control device 2. The first control unit 10 that has received this instruction via the information input unit 18 notifies user 1 to move the remote control device 2 to the vicinity of the corresponding device to be controlled 3, etc.

[0077] After that, the first control unit 10 refers to the output of the inertial sensor 19. When it determines that the remote control device 2 has been stationary after moving, the imaging unit 13 starts shooting. Then, the captured image data is transmitted to the server 30 via the communication unit 12 and the network 50 together with the type (here, TV 3a) input by user 1 as voice information. The server control unit 30a searches the device database using the received image data and extracts similar device images from the corresponding types. Note that the server control unit 30a calculates the similarity between images using a known method such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or ORB (Oriented FAST and Rotated BRIEF), and determines that the images are similar when the similarity is greater than a predetermined value.

[0078] The server control unit 30a extracts the product number associated with the device image, and uses the product number as a key to transmit the type, manufacturer, product name, product number, pattern data (hereinafter sometimes referred to as "device information") to the remote control device 2 via the network 50. Note that when no similar device image is extracted from the corresponding types, or when the similarity of the image determined to have the highest similarity is less than or equal to the predetermined value, NULL (no similar image) is returned.

[0079] Incidentally, the server control unit 30a may extract a character string from the captured image data, convert it into text, search the device database based on this text, extract device information including the corresponding manufacturer, product name, product number, etc., and transmit the device information to the first control unit 10.

[0080] The first control unit 10 that has received the device information notifies the user 1 of the product number via the information notification unit 14, and prompts the user 1 to confirm whether the product number matches the product number of the control target device 3 or the like that the user 1 intends to remotely control. If the user 1 gives an affirmative response to this, the first control unit 10 stores the control target device 3 or the like and the device information in the storage unit 11 in association with each other. That is, the control target device 3 or the like and the pattern data (control signal) are associated with each other. In this way, in the second embodiment, the first control unit 10 and the server control unit 30a cooperate to execute the setting of the remote command device 2. Thereafter, when the user 1 speaks such as "Turn on the TV.", the processing of the flowchart in FIG. 4 is executed.

[0081] Here, when the first control unit 10 receives NULL from the server 30, it operates the moving mechanism 16 of the remote command device 2 to capture an image around the remote command device 2. Then, it transmits the captured individual images to the server 30, and when NULL is no longer returned, it executes the confirmation process of whether the above-mentioned product numbers match.

[0082] In this way, the monitoring system S1 of the second embodiment further includes an imaging unit 13, and the control unit (the first control unit 10 and the server control unit 30a) associates the control target device 3 or the like with the control signal based on the image captured by the imaging unit 13 of the control target device 3 or the like. That is, the monitoring system S1 has a "setting operation support mode". As a result, it becomes possible for the user 1 to remotely control the control target device 3 or the like in a very simple operation such as placing the remote command device 2 near the control target device 3 or the like.

[0083] In the setting operation, when the first control unit 10 receives an instruction from the user 1, it accesses the device database constructed in the server 30, extracts device information corresponding to the type (here, the TV 3a) included in the instruction of the user 1, further extracts candidates for pattern data from the device information, and based on the candidates for the pattern data, may output (i.e., attempt) a control signal from the control signal output unit 15. Then, the first control unit 10 inquires of the user 1 whether the controlled device 3 has responded appropriately. When the user 1 gives an affirmative response, the storage unit 11 stores the controlled device 3, etc. and the device information in association with each other. On the other hand, when the user 1 gives a negative response, the first control unit 10 may select other candidates for pattern data and repeat the same attempt until an affirmative response is obtained from the user 1.

[0084] Also, when the first control unit 10 receives an instruction from the user 1 that does not specify the controlled device 3, etc. or the content of the operation, such as "I want to be able to perform remote control operations", the first control unit 10 may inquire about the type, manufacturer, product name, product number, and content of the operation (e.g., ON / OFF, temperature setting, or light intensity setting) of the controlled device 3, etc. that is the target of the operation. Here, when the user 1 answers that the type is a TV, the manufacturer is Company A, and the content of the operation, but answers "I don't know the product name and product number.", the first control unit 10 transmits search information with the type being the TV 3a and the manufacturer being Company A to the server 30. Then, the server control unit 30a searches the device database based on the search information, extracts candidate device information, and transmits it to the first control unit 10. In this way, in the process of obtaining candidates for pattern data, the first control unit 10 may narrow down the candidate device information using the search information obtained from the user 1. Of course, in the setting operation, if any of the type, manufacturer, product name, or product number is included in the first instruction of the user 1, these may be used as search information. Then, the above-described attempt may be performed using the pattern data included in the narrowed-down device information.

[0085] Also, in relation to the setting operation, a light receiving element (not shown) such as a photodiode or a phototransistor may be provided in the remote control device 2. For example, even when the control target device 3 or the like cannot be operated as intended by the user 1 by the trial using the candidate of the pattern data described above, the first control unit 10 causes the user 1 to hold the IR remote control attached to the control target device 3 or the like over the light receiving element and output a control signal such as ON / OFF (press a button or the like corresponding to the content of the operation). The first control unit 10 generates pattern data from the output of the light receiving element, and prompts the user to input the type, manufacturer, product name, product number, and content of the operation of the control target device 3 or the like. Thereby, even when the device information of the control target device 3 or the like is not stored in the device database, simplification of the setting operation can be realized. In this way, also in the setting operation, it is possible to prompt the user 1 to actively communicate with the remote control device 2 by himself / herself.

[0086] As described above, the monitoring system S1 according to the present invention has been described in detail based on specific embodiments. However, these embodiments are merely examples, and the present invention is not limited by these embodiments. For example, the division of the processing executed by the first control unit 10 and the server control unit 30a may be changed as appropriate. As a specific example, in the second embodiment, although it has been described that the extraction of the control target device 3 or the like that consumes the most power and the analysis of the time zone when the power peaks are executed by the first control unit 10, this processing may be executed by the server control unit 30a, and the remote control device 2 may receive the result and present it to the user 1.

[0087] Also, in the second embodiment, when it is determined that the control target device 3 does not appropriately respond to the control signal, the control signal may be transmitted again after moving the remote control device 2 (refer to the processing of ST05 (No), ST06, ST07 (Yes), ST08 in the flowchart of FIG. 4 and the subsequent ST02).

[0088] In addition, analysis of energy consumption and estimation of the presence or absence of signs of cognitive impairment in User 1 may be executed by the server control unit 30a instead of the first control unit 10, and the first control unit 10 may be configured to receive the results. That is, the "control unit" in the present invention may be either the first control unit 10 or the server control unit 30a, or these may cooperate to form the "control unit".

Industrial Applicability

[0089] The monitoring system, remote command device, robot, and monitoring method according to the present invention can encourage the user to actively communicate with the remote command device on their own, and can particularly prevent the sense of isolation and mental health problems of the elderly. Therefore, they can be widely used in places such as elderly care facilities and in-home care sites.

Explanation of Reference Numerals

[0090] 1 User 2 Remote Command Device 3 Controlled Device 4 Smart Meter 5 Smart Plug 6 Information Terminal 7 Electric Appliance 10 First Control Unit 13 Imaging Unit 14 Information Notification Unit 15 Control Signal Output Unit 16 Moving Mechanism 17 Environment Sensor 18 Information Input Unit 30 Server 30a Server Control Unit 30b Server Storage Unit 50 Network S1 Monitoring System

Claims

1. An information input unit that receives instructions or information input by a user, An information notification unit that notifies the user of predetermined information, A control signal output unit that outputs a control signal for controlling the control target device to the predetermined control target device, A control unit, and The control unit Based on the instruction of the user input to the information input unit, after the control signal output unit outputs the control signal to the control target device, the control target device is inquired of the response status via the information notification unit to the user. A monitoring system characterized by that.

2. Equipped with an environmental sensor, The control unit Based on the output of the environmental sensor, the monitoring system according to claim 1, characterized in that the user is notified via the information notification unit so that the control target device is in a specific state.

3. When the output of the environmental sensor satisfies a predetermined condition, The control unit The monitoring system according to claim 2, characterized in that the control target device is controlled to a specific state even when an instruction for setting the control target device to a specific state is not input by the user to the information input unit.

4. Equipped with a smart meter that measures the amount of electricity or gas used in the user's residence, The control unit Based on the amount of electricity or gas measured by the smart meter, the monitoring system according to claim 1, characterized in that the user is notified via the information notification unit so that the control target device is in a predetermined state.

5. Voice information is input to the information input unit, and the information notification unit outputs voice information. The monitoring system according to any one of claims 1 to 4.

6. The control unit Based on the information input to the information input unit, the appearance frequency of the same keyword included in the content of the communication with the user is derived. When the appearance frequency of the same keyword becomes larger than a predetermined value, it is determined that the user has a sign of cognitive impairment. The monitoring system according to any one of claims 1 to 4, characterized in that a person other than the user is notified that the user has a sign of cognitive impairment.

7. Further equipped with an imaging unit, The monitoring system according to any one of claims 1 to 4, wherein the control unit associates the controlled device with the control signal based on an image of the controlled device captured by the imaging unit.

8. An information input unit that receives an instruction or information input by a user; An information notification unit that notifies the user of predetermined information; A control signal output unit that outputs a control signal for controlling the controlled device to the predetermined controlled device; A control unit, and The control unit Based on the instruction of the user input to the information input unit, after the control signal output unit outputs the control signal to the controlled device, the user is inquired about the response status of the controlled device via the information notification unit. A remote command device characterized by the above.

9. A robot comprising the remote command device according to claim 8.

10. Furthermore, it is provided with a moving mechanism, The control unit After inquiring the user about the response status of the controlled device, when it is determined that the controlled device is not responding appropriately based on the information input to the information input unit, The robot according to claim 9, characterized in that the moving mechanism is operated to change the positional relationship between the remote command device and the controlled device.

11. A first step of receiving an instruction from a user; A second step of outputting a control signal for controlling the controlled device to the predetermined controlled device based on the instruction input by the user; A third step of inquiring the user about the response status of the controlled device with respect to the control signal, and A monitoring method, characterized in that the first step, the second step, and the third step are executed by a remote command device.

Images