Watching system, remote command device, robot, and watching method

Abstract

To provide a watching system, a remote command device, a robot, and a watching method that can urge users themselves to actively establish communication, thereby preventing the problem especially, loneliness and mental health of old people.SOLUTION: A watching system comprises: an information input unit 18 that receives input of an instruction or information given by a user 1; an information notification unit 14 that notifies the user 1 of predetermined information; a control signal output unit 15 that outputs, to a predetermined control target apparatus 3, a control signal for controlling the control target apparatus 3; and a control unit (first control unit 10). After the control signal output unit 15 outputs the control signal to the control target apparatus 3, the control unit inquires the user 1 about the state of response from the control target apparatus 3 through the information notification unit 14 on the basis of the instruction from the user 1 input to the information input unit 18.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a monitoring system for monitoring users such as elderly people, a remote command device, a robot incorporating a remote command device, and a monitoring method. [Background technology]

[0002] Elderly people often require assistance in daily life due to dementia or physical frailty, and monitoring is essential to maintain their safety and quality of life (QOL). Providing regular communication, particularly to prevent feelings of isolation and mental health problems, is said to contribute greatly to the happiness of elderly people.

[0003] As a technology for communicating between so-called home appliances and their users, for example, a robot is known that uses a voice speech function to speak to a user, issues spoken operational instructions to a second home appliance having a voice recognition function, recognizes the content of the user's speech using the voice recognition function, recognizes the speech of the home appliance having the voice speech function, and transmits infrared rays using an infrared transmission function to issue operational instructions to a home appliance having an infrared receiving function (Patent Document 1).

[0004] According to Patent Document 1, the robot issues operation instructions to each home appliance, then receives a status notification (voice or alarm sound) from each home appliance, analyzes the status notification, and if it finds that the home appliance is not operating according to the operation instructions, issues an operation instruction to the home appliance again, and if it finds that the home appliance is operating according to the operation instructions, stops the operation instructions. Furthermore, the robot may inform the user of the results of the operation instructions, which makes it possible to inform the user of the status information of the home appliances. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-76799 A DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0006] However, the technology disclosed in Patent Document 1 only aims to improve the convenience of the user, and communication between the user and the robot is one-way from the robot to the user. For this reason, it is considered difficult to encourage users, especially elderly people, to actively communicate with the robot and prevent them from feeling isolated or having mental health problems.

[0007] The present invention has been devised to solve such problems in the prior art, and its purpose is to provide a monitoring system, remote control device, robot, and monitoring method that encourages users to actively communicate with each other and can prevent feelings of isolation and mental health problems, particularly among the elderly. [Means for solving the problem]

[0008] The present invention, which has been made to solve the above problems, is a monitoring system that includes an information input unit that accepts input of instructions or information from a user, an information notification unit that notifies the user of predetermined information, a control signal output unit that outputs a control signal to a predetermined controlled device to control the controlled device, and a control unit, and the control unit inquires the user about the response status of the controlled device via the information notification unit after the control signal output unit outputs the control signal to the controlled device based on the instruction of the user input to the information input unit. This makes it possible to encourage the user to actively communicate with the remote command device.

[0009] The present invention further includes an environmental sensor, and the control unit notifies the user through the information notification unit to set the controlled device to a specific state based on the output of the environmental sensor. This makes it possible to make elderly people, who may have a decreased ability to regulate their body temperature or have a decreased sense of temperature, aware of the need to correct the indoor environment.

[0010] In addition, in the present invention, when the output of the environmental sensor satisfies a predetermined condition, the control unit controls the controlled device to a specific state even if the user does not input an instruction to the information input unit to set the controlled device to a specific state, thereby making it possible to reduce risks to the user's health or life.

[0011] In addition, in the present invention, the control unit acquires the amount of energy consumption used in the user's residence, and notifies the user via the information notification unit to set the controlled device to a predetermined state based on the amount of energy consumption. This makes it possible to provide the user 1 with advice on energy conservation.

[0012] In addition, in the present invention, voice information is input to the information input unit, and the information notification unit outputs the voice information. This makes it possible to communicate with the user 1 by voice.

[0013] In addition, in the present invention, the control unit estimates the presence or absence of a sign of cognitive impairment of the user based on the information input to the information input unit, and when it is determined that the user has a sign of cognitive impairment, it notifies a person other than the user to that effect. This enables a family member or the like to take early action when the user has a sign of cognitive impairment.

[0014] The present invention further includes an imaging unit, and the control unit associates the controlled device with the control signal based on an image of the controlled device captured by the imaging unit. This allows a user to remotely operate the controlled device with a very simple operation of placing a remote command device near the controlled device.

[0015] The present invention also provides a remote command device that includes an information input unit that accepts input of instructions or information by a user, an information notification unit that notifies the user of predetermined information, a control signal output unit that outputs to a predetermined controlled device a control signal for controlling the controlled device, and a control unit, wherein the control unit inquires of the user about a response status of the controlled device via the information notification unit after the control signal output unit outputs the control signal to the controlled device based on the instruction of the user input to the information input unit. This makes it possible to encourage the user to actively communicate with the remote command device.

[0016] The present invention also provides a robot including the remote command device, which makes it possible to operate a control target device 3 using the robot.

[0017] The present invention further includes a movement mechanism, and when the control unit inquires of the user about the response status of the controlled device and determines based on the information input to the information input unit that the controlled device is not responding appropriately, the control unit operates the movement mechanism to change the positional relationship between the remote command device and the controlled device. This makes it possible to secure an infrared communication path when the control signal output unit includes an IR remote control function.

[0018] The present invention also provides a monitoring method for receiving an instruction from a user, outputting a control signal to a controlled device based on the instruction input by the user, and inquiring of the user about a response status of the controlled device to the control signal. This makes it possible to generate an interaction between the user and the remote command device every time the user operates a home appliance or the like via the remote command device. Effect of the Invention

[0019] In this way, the present invention encourages users to actively communicate with the remote command device, making it possible to prevent feelings of isolation and mental health problems, particularly among the elderly. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram showing a configuration of a monitoring system S1 according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is an explanatory diagram showing an overview of a remote command device 2 according to a first embodiment. [Diagram 3] FIG. 1 is a block diagram showing a configuration of a remote command device 2 according to a first embodiment. [Figure 4] 1 is a flowchart showing the operation of the monitoring system S1 in the first embodiment. [Diagram 5] A block diagram showing the configuration of a monitoring system S1 according to a second embodiment of the present invention. [Figure 6] FIG. 11 is a block diagram showing the configuration of a remote command device 2 according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] (First embodiment) A first embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a monitoring system S1 according to the first embodiment of the present invention. The monitoring system S1 is made up of a remote command device 2 and a control target device 3. Here, the remote command device 2 includes a function of a so-called remote controller (hereinafter, sometimes referred to as a "remote control").

[0022] In the following description, the user 1 is assumed to be, for example, an elderly person living alone in a residence, but is not limited thereto. In other words, the user 1 refers to a person who uses the remote control function of the remote command device 2, and may include cohabitants such as family members. Examples of the control target devices 3 include home appliances such as a television 3a, an air conditioner 3b, and a lighting fixture 3c. Of course, the control target devices 3 are not limited to these, and any home appliances that can be controlled using a remote control to perform ON / OFF operations, operation settings, etc. are included in the control target devices 3 of the present invention. Note that the remote command device 2 has a function of a so-called smart remote control that can operate multiple control target devices 3.

[0023] When the user 1 gives instructions to the remote command device 2 to control a device, the remote command device 2 accepts the instructions from the user 1 and outputs a control signal to control the controlled device 3 based on the instructions input by the user 1. The remote command device 2 that outputs the control signal then inquires of the user 1 about the response status of the controlled device 3. This makes it possible to generate an interaction between the user 1 and the remote command device 2 every time the user 1 operates a home appliance or the like via the remote command device 2.

[0024] FIG. 2 is an explanatory diagram showing an overview of the remote command device 2 in the first embodiment. As shown in the figure, the remote command device 2 is configured, for example, as a bipedal robot. By making the remote command device 2 in the form of a robot, a sense of familiarity with the remote command device 2 is fostered in the user 1, and it is possible to promote interaction. Of course, there is no particular limitation on the form of the remote command device 2, and it may be configured as a robot toy, or as a four-legged animal-type robot, or may have an endless track instead of two or four legs. In addition, it may be in a fixed form without a moving mechanism 16 (described later) such as two legs, a form without arms, a form composed of only one housing part, a form of a so-called remote control for home appliances, or a form of a wearable device.

[0025] The remote command device 2 includes a main body 2BD, a head 2HD, arms 2A, legs 2L, an imaging unit 13, an information notification unit 14, a control signal output unit 15, an environmental sensor 17, an information input unit 18, and a switch unit 21. Joints (not shown) are provided in the legs 2L at the portions corresponding to the engaging portion with the main body 2BD (the base 2s of the legs 2L), the knees 2t, and the ankles 2u. A driving source (not shown) such as a motor is disposed near the joint, and the legs 2L are rotatable within a predetermined range around the joint. That is, the remote command device 2 of the first embodiment includes a movement mechanism 16 including the legs 2L, the joints, and the driving sources.

[0026] Fig. 3 is a block diagram showing the configuration of the remote command device 2 in the first embodiment. The configuration and operation of the monitoring system S1 will be described in detail below with reference to Fig. 3 in combination with Fig. 1 and Fig. 2. The remote command device 2 includes a first control unit 10, a storage unit 11, and an inertial sensor 19 in addition to the above-mentioned components.

[0027] The first control unit 10 is composed of a CPU (Central Processing Unit) and the like, and operates according to a control program stored in a storage unit 11 composed of a ROM (Read Only Memory), a RAM (Random access memory), etc. The first control unit 10 and other components are connected via a bus 20 and the like, and the first control unit 10 controls the other components via the bus 20 and the like. The first control unit 10 includes a clock function.

[0028] The storage unit 11 also includes a non-volatile memory (such as an EEPROM (Electrically Erasable Programmable Read-Only Memory)). Pattern data and the like used to generate a control signal for controlling the controlled device 3 are stored in this non-volatile memory.

[0029] The imaging unit 13 includes an image sensor configured with a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). The imaging unit 13 outputs image data.

[0030] By operating the above-mentioned moving mechanism 16, the remote command device 2 can move forward and backward, and can also change its posture in the left and right and up and down directions. That is, the remote command device 2 can change the imaging range of the imaging unit 13 by moving forward and backward, or by substantially panning and tilting. This allows the first control unit 10 to recognize the positional relationship between the remote command device 2 and the controlled device 3 based on the captured image data.

[0031] The information notification unit 14 is composed of, for example, a speaker. Audio information is provided to the user 1 via the speaker. Of course, the information notification unit 14 may also include a display device composed of, for example, an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) in addition to the speaker. In this case, video and text information are provided to the user 1 in addition to the audio information.

[0032] The information input unit 18 is composed of, for example, a microphone. The output of the microphone is converted into digital data by an A / D converter (not shown). The first control unit 10 recognizes the digitized voice of the user 1. In the voice recognition, phonemes are extracted from the acquired voice information, converted into text, and then vocabulary information is identified. Based on this vocabulary information, the first control unit 10 recognizes the instructions of the user 1 and the intention of the information input by the user 1.

[0033] In this way, in the remote command device 2 of the first embodiment, voice information is input to the information input unit 18, and the information notification unit 14 outputs the voice information. This allows voice communication with the user 1. Of course, a touch panel or the like superimposed on the display device may be used as the information input unit 18. In this case, the user 1 can communicate with the remote command device 2 via the touch panel or the like.

[0034] An application for inputting voice information is realized by combining, for example, natural language processing (NLP) and a machine learning model. Furthermore, by using a rule-based system that converts abstract words into specific setting changes for the controlled device 3, the first control unit 10 can determine that the temperature setting of the air conditioner 3b should be changed to a lower temperature (cooling) or that the luminance value of the lighting device 3c should be increased, based on the abstract utterance of the user 1 that does not specify the controlled device 3, such as "It's kind of hot" or "The room is dark."

[0035] The control signal output unit 15 is composed of an infrared light emitting diode or a transmitter that outputs a radio wave. The first control unit 10 extracts the above-mentioned pattern data from the storage unit 11, and drives the infrared light emitting diode or generates a radio wave signal based on the pattern data. That is, the remote command device 2 including the control signal output unit 15 functions as an infrared (IR) remote control or a radio wave (RF) remote control, and an IR or RF control signal is transmitted to the control target device 3. The control signal output unit 15 may be composed of a communication module that complies with a short-range wireless standard such as BLE (Bluetooth (registered trademark) Low Energy). When the control signal output unit 15 is composed of an infrared light emitting diode, the control signal output unit 15 is provided on at least one surface (here, the front surface) of the remote command device 2 (see FIG. 2).

[0036] The environmental sensor 17 may include, for example, a temperature and humidity sensor that measures the temperature (air temperature) and humidity of the external environment (for example, the room of the user 1) of the remote command device 2, an illuminance sensor that detects illuminance, a human presence sensor that detects infrared rays emitted by a person or the like, and an odor sensor that detects odors (or a gas sensor that detects a specific gas) (all of which are not shown). Of course, the remote command device 2 does not have to include all of these sensors. The output of the environmental sensor 17 is converted into digital data by, for example, an A / D converter included in the first control unit 10.

[0037] The inertial sensor 19 is composed of, for example, a three-axis acceleration sensor and / or a gyro sensor. Here, the three-axis acceleration sensor outputs the direction and extent of the speed at which the remote command device 2 is changing (acceleration) for the three axes X, Y, and Z. The gyro sensor outputs the direction and speed at which the remote command device 2 is rotating (angular velocity) for the three axes X, Y, and Z. In general, the gyro sensor detects the direction of movement, and the acceleration sensor detects the distance moved. In this way, the inertial sensor 19 outputs three-axis acceleration information and three-axis angular velocity information based on the movement of the remote command device 2.

[0038] The first control unit 10 detects the posture of the remote command device 2 based on the output of the inertial sensor 19. Then, when the remote command device 2 moves forward or backward (walks) or stands on one leg, the first control unit 10 controls the remote command device 2 to ensure balance and prevent it from falling over.

[0039] The switch unit 21 (see FIG. 2) is a push switch (toggle switch) located on the top of the remote command device 2. When the switch unit 21 is pressed, the remote command device 2 starts / stops operation. Of course, if the remote command device 2 is connected to a power source and power is constantly supplied, the switch unit 21 does not have to be provided.

[0040] Fig. 4 is a flowchart showing the operation of the monitoring system S1 in the first embodiment. The operation of the monitoring system S1 will be explained below with reference to Fig. 4 in combination with Figs. 1, 2, and 3. First, the first control unit 10 judges whether or not an instruction from the user 1 has been accepted (ST01). Specifically, the first control unit 10 judges whether or not the user 1 has issued an instruction regarding the operation of the control target device 3 (hereinafter, sometimes referred to as an "operation instruction") based on the result of voice recognition of the output from the information input unit 18.

[0041] The operation instructions include, for example, ON / OFF instructions for the control-target device 3 uttered by the user 1, such as "Turn on the air conditioner" or "Turn off the TV," and instructions related to the operation settings of the control-target device 3, such as "Set the cooling temperature to 25°C" or "Turn on the safety light." Furthermore, as described above, abstract instructions that do not specify the name of the control-target device 3 may also be included. Based on the operation instructions, the first control unit 10 utilizes the above-mentioned NLP, etc., to determine a specific operation content (e.g., turn ON) for a specific control-target device 3 (e.g., air conditioner 3b).

[0042] If the first control unit 10 has not received an instruction from the user 1 (No in ST01), the process returns to ST01. On the other hand, if the first control unit 10 has received an instruction from the user 1 (Yes in ST01), the first control unit 10 transmits a control signal to the control-target device 3 via the control signal output unit 15 (ST02). That is, a control signal according to a specific control content is transmitted to a specific control-target device 3. After transmitting the control signal, the first control unit 10 inquires of the user 1 about the response status of the control-target device 3 (ST03). Specifically, if the control-target device 3 to be operated is the air conditioner 3b and the operation content is "turn ON", the first control unit 10 inquires of the user 1 about the response status of the control-target device 3 via the information notification unit 14, such as "Did the air conditioner turn ON?" That is, the first control unit 10 confirms with the user 1 whether the control-target device 3 responded as the user 1 intended.

[0043] Next, the first control unit 10 accepts input of information from the user 1 (ST04). Specifically, the first control unit 10 acquires voice information uttered by the user 1 based on the output of the information input unit 18, and recognizes the content of the voice information. Based on this recognition result, the first control unit 10 judges whether the controlled device 3 has responded appropriately (ST05). If the user 1 utters a so-called positive response such as "Yup" or "Yes," or the recognition result includes a positive response such as "It's turned on" or "It worked properly" (hereinafter, sometimes referred to as a "positive response"), the first control unit 10 judges that the controlled device 3 has responded appropriately (Yes in ST05), and moves the process to ST01.

[0044] During this process, "interaction" occurs between the user 1 and the remote command device 2 (robot). It is believed that as the robot initiates communication and this becomes established, elderly people living alone, in particular, will develop a strong attachment to the robot. As this attachment is cultivated, it is expected that the user 1 will become more proactive in responding to inquiries from the robot, and the quality and quantity of communication between the user and the robot will increase. This can stimulate the cognitive functions of the user 1. Maintaining vocabulary and conversation skills through verbal communication in this way has the potential to slow the progression of dementia.

[0045] Generally, the term "interaction" is interpreted as "when a human performs some action (operation or behavior), the system responds accordingly." However, in this specification, "interaction" is understood as "not only when a human performs some action and the system (equipment) responds accordingly, but also when the system (equipment) performs some action and the human responds accordingly." In other words, the present invention can build a deeper and stronger relationship between the user 1 and the robot (the remote command device 2 and the monitoring system S1) than a typical "interaction."

[0046] On the other hand, if the user 1 utters a so-called negative response such as "No" or "Nope," or the recognition result includes a negative response such as "It didn't turn on" or "It's not working" (hereinafter sometimes referred to as a "negative response"), the first control unit 10 determines that the controlled device 3 has not responded appropriately (No in ST05). Then, it inquires of the user 1 whether to resend the control signal (ST06). In response to this, if the user 1 returns a negative response (No in ST07), the process proceeds to ST01.

[0047] On the other hand, if the user 1 returns an affirmative response (Yes in ST07), the first control unit 10 operates the movement mechanism 16 to move the remote command device 2 (ST08). Here, as described above, the control signal output unit 15 is provided on one side (here, the front) of the remote command device 2. If the control signal output unit 15 includes an IR remote control function, a linear path is required for infrared communication, and therefore communication is hindered if there is an obstacle between the remote command device 2 and the controlled device 3. Therefore, in order to reliably transmit the control signal, the first control unit 10 operates the movement mechanism 16 to move (for example, orbit) the remote command device 2, and recognizes the position of, for example, the air conditioner 3b based on the image data obtained by the imaging unit 13. Then, after moving the remote command device 2 so that the air conditioner 3b is included in the imaging area of ​​the imaging unit 13, the control signal is transmitted (ST02).

[0048] Here, in the first embodiment, it can be said that the robot is equipped with the remote command device 2. This makes it possible to operate the controlled device 3 using the robot. The robot of the first embodiment also includes a movement mechanism 16, and the control unit (first control unit 10) inquires of the user 1 about the response status of the controlled device 3, and then, when it determines that the controlled device 3 is not responding appropriately based on information input to the information input unit 18, operates the movement mechanism 16 to change the positional relationship between the remote command device 2 and the controlled device 3. This makes it possible to ensure an infrared communication path when the control signal output unit 15 includes an IR remote control function.

[0049] As described above, the monitoring system S1 (or the remote command device 2) of the first embodiment includes an information input unit 18 that accepts an instruction or information input by the user 1, an information notification unit 14 that notifies the user 1 of predetermined information, a control signal output unit 15 that outputs a control signal for controlling the control target device 3 to a predetermined control target device 3, and a control unit (first control unit 10). After the control signal output unit 15 outputs a control signal to the control target device 3 based on the instruction of the user 1 input to the information input unit 18, the control unit inquires the user 1 about the response status of the control target device 3 via the information notification unit 14. In this way, the operation of the control target device 3 by the user 1 is assisted by the inquiry about the response status. That is, the monitoring system S1 includes a "device operation assistance mode". The device operation assistance mode makes it possible to encourage the user 1 to actively communicate with the remote command device 2.

[0050] The explanation will be continued below with reference to FIG. 3. As described above, the remote command device 2 is equipped with an environmental sensor 17. The environmental sensor 17 includes a temperature and humidity sensor and a human presence sensor. The first control unit 10 periodically monitors the output of the environmental sensor 17. When the human presence sensor detects the user 1 in the vicinity of the remote command device 2 and the output of the temperature and humidity sensor is outside a predetermined range (here, a comfortable temperature and humidity range), the first control unit 10 notifies the user 1 via the information notification unit 14 to set the controlled device 3 (here, the air conditioner 3b) to a specific state (hereinafter, this notification may be referred to as a "monitoring notification").

[0051] That is, the monitoring system S1 (or the remote command device 2) has a "monitoring notification mode" that notifies the user 1 via the information notification unit 14 to put the control target device 3 into a specific state. This makes it possible to make elderly people, who may have a decreased ability to regulate their body temperature or a decreased sense of temperature, aware of the need to correct the indoor environment.

[0052] The monitoring notification is issued, for example, when the indoor temperature is higher than 30°C, when the indoor temperature is lower than 15°C, or when the discomfort index is outside the range of 60 to 75. The monitoring notification includes information on temperature and humidity and a method of dealing with the problem, and for example, voice information such as "The discomfort index is high. Please tell me to turn on the air conditioner" is provided to the user 1. In this way, the monitoring notification includes a message urging the user 1 to give an instruction to the remote command device 2, and the user 1 responds to this (i.e., in this case, voice information is input to the information input unit 18 to turn on the air conditioner), thereby enabling communication between the monitoring system S1 (remote command device 2) and the user 1. Then, ST01 of the flowchart shown in FIG. 4 is executed, and further communication with the user 1 is continued.

[0053] In relation to the specific example of the operation of the air conditioner 3b described above, even if there is no response from the user 1 or the user 1 returns a negative response to the effect that "there is no need to turn on the air conditioner," if the output of the environment sensor 17 indicates, for example, that the temperature is 30°C or higher and the relative humidity is 70% or higher, the first control unit 10 outputs a control signal to the air conditioner 3b to turn on the air conditioner. That is, when the first control unit 10 judges that the user 1 is at high risk of health or life-threatening risk such as heat stroke, it issues a monitoring notice from the viewpoint of communicating with the user 1, and further operates and controls the controlled device 3 to a setting that further reduces the risk to the user 1, even in a situation where no positive response to the monitoring notice is received from the user 1 (that is, when no explicit instruction to set the controlled device 3 to a specific state is received via the information input unit 18).

[0054] In this way, in the monitoring system S1 (remote command device 2) of the first embodiment, when the output of the environmental sensor 17 satisfies a predetermined condition, the control unit (first control unit 10) controls the control-target device 3 to a specific state even if the user 1 does not input an instruction to put the control-target device 3 into a specific state to the information input unit 18. This makes it possible to reduce risks to the user 1's health or life.

[0055] Second embodiment A second embodiment of the present invention will be described below with reference to the drawings. Fig. 5 is a block diagram showing the configuration of a monitoring system S1 according to the second embodiment of the present invention. As shown in Fig. 5, the monitoring system S1 includes a smart meter 4, a smart plug 5, an information terminal 6, and a server 30 in addition to the components described in the first embodiment. These components, together with a remote command device 2, are connected to a network 50 and exchange information with each other.

[0056] The smart meter 4 includes a measurement unit, a communication module, a data processing unit, etc. (none of which are shown), and measures the energy consumption of electricity, gas, etc. of the house or the like where the user 1 resides in real time or periodically, and transmits the measured energy consumption to the server 30 via the network 50. The smart meter 4 can also be controlled from the outside, and has a function of cutting off the energy supply to the house or the like when, for example, the demand for electricity reaches a peak.

[0057] The smart plug 5 is a device that is interposed between a power supply port provided in a house or the like and a specific electrical appliance 7 to supply power to the electrical appliance 7 and to remotely turn on / off home appliances and electronic devices in cooperation with a smartphone or the like or a remote command device 2. Furthermore, the smart plug 5 is provided with a power measurement unit (not shown) that measures the power supplied to the electrical appliance 7. The smart plug 5 also includes a communication module (not shown) that complies with wireless communication standards such as LTE (Long Term Evolution), LTE-M (Long Term Evolution-Machine, LTE Cat.M1), 4G, and 5G.

[0058] That is, the smart plug 5 turns on / off the electric appliance 7 connected to the smart plug 5 based on a control signal input via the network 50, and further measures the power (energy consumption) consumed by the electric appliance 7 and transmits the measured power data to the outside via the network 50. Here, the electric appliance 7 includes, for example, a refrigerator, a television 3a, an air conditioner 3b, a lighting fixture 3c, and other home appliances. In this way, the electric appliance 7 has an aspect of being a control target device 3, and in the second embodiment, the smart plug 5 can control the electric appliance 7 based on a control signal transmitted from the remote command device 2. In the following description, the control target device 3 and the electric appliance 7 may be collectively referred to as "the control target device 3, etc."

[0059] The information terminal 6 is, for example, a mobile information terminal such as a smartphone or a tablet terminal, or a PC (Personal Computer), and is equipped with a display unit, an input unit, etc., not shown. The information terminal 6 is used by a person watching over the user 1 (a family member, a caregiver, a doctor, etc., hereinafter sometimes referred to as "family member, etc."). As described above, the present invention aims to achieve advanced communication between the user 1 and the watching system S1 (remote command device 2), but it cannot be denied that some users 1 feel stressed by frequent interactions with the remote command device 2.

[0060] Therefore, in the second embodiment, the family or the like of the user 1 can set permission or prohibition for the remote command device 2 to inquire the user 1 about the response status of the control target device 3, thereby achieving both advanced communication and stress reduction for the user 1. Permission or prohibition of inquiries can be set for each time period, for example, by linking with a scheduler or the like. This makes it possible to set such that inquiries are prohibited during the time period before going to bed.

[0061] The server 30 is a known computer system, and is composed of a server control unit 30a and a server storage unit 30b. The server control unit 30a is composed of a CPU and a storage unit (not shown), and controls the components of the server 30. The server storage unit 30b has a large-capacity storage composed of RAID (Redundant Arrays of Independent Disks) together with ROM and RAM. This large-capacity storage stores personal information such as a unique identifier (user ID), gender, age, name, etc. (hereinafter, these may be collectively referred to as "user information") for each of a plurality of users 1. The user information is managed as a user database. The server 30 obtains time information from an NTP (Network Time Protocol) server (not shown).

[0062] In addition, the user database has a communication history field that stores the history of communication that occurred between each user 1 and the remote command device 2, and an energy consumption history field that stores the history of energy consumption (amount of electricity, amount of gas used, etc.).

[0063] The first control unit 10 stores the contents of communication between the user 1 and the remote command device 2 that occurs when an operation instruction is received from the user 1, when the user 1 is inquired about the response status of the control target device 3, and when a monitoring notification is sent (including at least voice information or text information generated by voice recognition. Hereinafter, these may be referred to as "communication information") together with the user ID of the user 1 and time information in a communication history field. That is, every time some communication occurs between the user 1 and the remote command device 2, the contents of the communication history field are updated.

[0064] In addition, the smart meter 4 transmits the amount of power consumed in the house to the server 30 at a predetermined period, and similarly, the smart plug 5 transmits the amount of power consumed by the connected electrical appliance 7 to the server 30. The server control unit 30a stores the received amount of power together with time information in the energy consumption history field. The communication information stored in the communication history field is subject to analysis based on instructions from family members, etc. (particularly, doctors, etc.). When a doctor, etc. inputs a predetermined command to the information terminal 6, the command is transmitted to the server 30, and the server control unit 30a executes various statistical and analytical processes on the communication information. The results of these processes are transmitted to the information terminal 6 and are used by the doctor, etc. to plan a care plan for the user 1.

[0065] Furthermore, a natural language processing model (NLP) may be built in the server 30. Alternatively, the server control unit 30a may access an external large-scale language model (LLM: Large Language Models) via the network 50. By using the large-scale language model, the monitoring system S1 can achieve natural communication (here, for example, chat) with the user 1. That is, the remote command device 2 may be provided with a "chat mode" for having a conversation with the user 1.

[0066] The chat mode may be started regardless of the result of the judgment in the step (ST05) of judging whether or not the controlled device 3 has responded appropriately in the flowchart of FIG. 4. Furthermore, the chat mode may be started after the above-mentioned monitoring notification. Also, the chat mode may be started at any timing when a person is detected by the environmental sensor 17. As the content in the chat mode, for example, when ST05 is judged as No (the controlled device 3 has not responded appropriately), a question may be asked to make the user 1 think about the cause, or current news acquired from the outside via the network 50 may be conveyed to the user 1 and the user's impression may be asked. The communication performed using NLP or LLM may also be stored in the above-mentioned communication history field together with time information.

[0067] FIG. 6 is a block diagram showing the configuration of the remote command device 2 in the second embodiment. Hereinafter, the description will be given with reference to FIG. 5 in addition to FIG. 6. The configuration of the remote command device 2 in the second embodiment is basically the same as that in the first embodiment. However, in the second embodiment, a communication unit 12 is added. The communication unit 12 includes a communication module (not shown) that complies with a wireless communication standard such as LTE, LTE-M, 4G, or 5G. The communication unit 12 may be configured to comply with the WiFi (Wireless Fidelity) standard, and the remote command device 2 may be connected to the network 50 via a wireless router or the like. In addition, the communication unit 12 may be configured with a communication module (not shown) that complies with a short-range wireless standard such as BLE, and information may be transmitted and received directly between the communication unit 12 and, for example, the smart plug 5 without going through the network 50.

[0068] The first control unit 10 accesses the server storage unit 30b via the communication unit 12 and the network 50, and acquires the amount of power from the energy consumption history field corresponding to the user ID of the user 1. Here, the amount of power acquired by the smart meter 4 may be separated into the amounts of power used by each electric appliance 7 (and the controlled device 3) by performing a so-called disaggregation process in the server control unit 30a. On the other hand, for the smart plug 5, the amount of power corresponding to each electric appliance 7 is acquired.

[0069] The first control unit 10 refers to the time information when the amount of power is measured by the smart meter 4 or smart plug 5, analyzes the acquired amount of power in chronological order, and if the amount of power consumed by a specific controlled device 3 or the like (here, for example, the air conditioner 3b) has increased beyond a predetermined range, for example, compared to the same month one year ago, notifies the user 1 via the information notification unit 14 to change the set temperature of the air conditioner 3b. Of course, if the amount of gas consumption is measured by the smart meter 4, it is also possible to advise, for example, to set the hot water supply temperature of the water heater lower. Note that the amount of power consumed and the amount of gas used are both considered to be energy consumption.

[0070] Furthermore, the first control unit 10 can extract which controlled device 3 is consuming the most power and the time period when power is at its peak by grasping and analyzing the amount of power used and the power consumption pattern. Then, actions to save on electricity and gas bills and measures to improve energy efficiency can be presented to the user 1. These may be used as one of the contents in the chat mode described above.

[0071] As described above, in the monitoring system S1 of the second embodiment, the control unit (first control unit 10) acquires the amount of energy consumption used in the residence of the user 1, and notifies the user 1 via the information notification unit 14 to put the control-target device 3 into a predetermined state based on the amount of energy consumption. This makes it possible to provide the user 1 with advice on energy conservation.

[0072] Similarly to obtaining the amount of power, the first control unit 10 accesses the server storage unit 30b and obtains communication information from the communication history field corresponding to the user ID of the user 1. Then, for example, the frequency with which the same keywords or phrases appear in communication information for several days is derived. When the user 1 repeatedly issues the same operation instruction or talks about the same episode multiple times in the above-mentioned chat mode, the frequency with which the same keywords or phrases appear increases. Depending on the frequency with which the same keywords or phrases appear, the first control unit 10 estimates whether the user 1 has a sign of memory disorder or cognitive disorder.

[0073] When the frequency of appearance of the same keyword or the like becomes greater than a predetermined value, the first control unit 10 determines that the user 1 has a sign of cognitive impairment or the like, and notifies family members or the like of that fact via the information terminal 6. When it is determined that there is no sign of cognitive impairment or the like, the above-mentioned notification is not made. Furthermore, when it is determined that there is a sign of cognitive impairment or the like, the first control unit 10 may lengthen the period of the above-mentioned chat mode, so as to increase the frequency with which the user 1 speaks.

[0074] As described above, in the monitoring system S1 of the second embodiment, the control unit (first control unit 10) estimates the presence or absence of a sign of cognitive impairment in the user 1 based on the information input to the information input unit 18, and if it is determined that the user 1 has a sign of cognitive impairment, it notifies persons other than the user 1 of this fact. This enables family members and the like to take action early when the user 1 has a sign of cognitive impairment.

[0075] Now, an equipment database relating to the control target equipment 3 and the like is constructed in the server storage unit 30b. The equipment database stores the type (television 3a, air conditioner 3b, lighting equipment 3c, etc.) of the equipment operated by the remote control, the manufacturer, the product name, the model number, a plurality of images of the equipment (hereinafter, sometimes referred to as "equipment images"), pattern data used to generate a control signal, etc. In the equipment database, for example, the model number is used as a key to enable reference to other items.

[0076] For example, when the user 1 newly purchases a control target device 3 or the like (e.g., a television 3a) or a remote command device 2, the user 1 performs a setting operation to associate the control target device 3 or the like with the remote command device 2. The setting operation here refers to an operation to determine a combination of a specific operation of the control target device 3 or the like with a control signal (pattern data) for executing this specific operation. In the setting operation, the user 1 can instruct the remote command device 2 to "make it possible to control the television with a remote control." The first control unit 10, which receives this instruction via the information input unit 18, notifies the user 1 to move the remote command device 2 near the corresponding control target device 3 or the like.

[0077] Thereafter, the first control unit 10 refers to the output of the inertial sensor 19, and when it determines that the remote command device 2 has been moved and then placed still, it starts capturing images with the imaging unit 13. Then, the captured image data is transmitted to the server 30 via the communication unit 12 and the network 50 together with the type (here, TV 3a) input by the user 1 as voice information. The server control unit 30a uses the received image data to search the device database and extracts similar device images from the corresponding type. The server control unit 30a calculates the similarity between images using known methods such as SIFT (Scale-Invariant Feature Transform), SURF (Speed ​​Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF), and determines that the images are similar when the similarity is greater than a predetermined value.

[0078] The server control unit 30a extracts the product number associated with the device image, and uses the product number as a key to transmit the type, manufacturer, product name, product number, and pattern data (hereinafter, sometimes referred to as "device information") to the remote command device 2 via the network 50. If no similar device image is extracted from the corresponding type, or if the similarity of the image determined to have the highest similarity is equal to or less than a predetermined value, NULL (no similar image) is returned.

[0079] The server control unit 30a may extract a character string from the captured image data and convert it into text. Then, the server control unit 30a may search the device database based on the text, extract device information including the corresponding manufacturer, product name, product number, etc., and transmit the device information to the first control unit 10.

[0080] The first control unit 10, which has received the device information, notifies the user 1 of the model number via the information notifying unit 14, and prompts the user 1 to confirm whether the model number matches the model number of the controlled device 3, etc., which the user wishes to remotely control. If the user 1 returns an affirmative response to this, the first control unit 10 associates the controlled device 3, etc., with the device information in the storage unit 11 and stores them. That is, the controlled device 3, etc., is associated with the pattern data (control signal). In this way, in the second embodiment, the first control unit 10 and the server control unit 30a work together to execute the setting of the remote command device 2. Thereafter, when the user 1 utters an utterance such as "Turn on the TV," the process of the flowchart in FIG. 4 is executed.

[0081] Here, when the first control unit 10 receives NULL from the server 30, it activates the movement mechanism 16 of the remote command device 2 to capture images of the surroundings of the remote command device 2. Then, it transmits each of the captured images to the server 30, and when NULL is no longer returned, it executes the above-mentioned confirmation process to determine whether the product numbers match.

[0082] In this way, the monitoring system S1 of the second embodiment further includes an imaging unit 13, and the control unit (first control unit 10 and server control unit 30a) associates the control target devices 3, etc. with control signals based on an image of the control target devices 3, etc. captured by the imaging unit 13. That is, the monitoring system S1 includes a "setting operation support mode." This enables the user 1 to remotely operate the control target devices 3, etc. with a very simple task of placing the remote command device 2 near the control target devices 3, etc.

[0083] In the setting operation, when the first control unit 10 receives an instruction from the user 1, it may access the device database constructed in the server 30, extract device information corresponding to the type (here, the TV 3a) included in the instruction from the user 1, further extract pattern data candidates from the device information, and output (i.e., try) a control signal from the control signal output unit 15 based on the pattern data candidates. Thereafter, the first control unit 10 inquires of the user 1 whether the controlled device 3 responded appropriately, and if the user 1 returns a positive response, stores the controlled device 3 and the like and the device information in association with each other in the storage unit 11. On the other hand, if the user 1 returns a negative response, the first control unit 10 may select another pattern data candidate and repeat the same trial until a positive response is obtained from the user 1.

[0084] In addition, when receiving an instruction from the user 1 to the effect that "I want to be able to operate the remote control," without specifying the controlled device 3 or the content of the operation, the first control unit 10 may inquire about the type, manufacturer, product name, model number, and content of the operation (for example, ON / OFF, temperature setting, and brightness setting) of the controlled device 3 to be operated. Here, if the user 1 answers that the type is a television, the manufacturer is Company A, and the content of the operation, but answers that "I do not know the product name and model number," the first control unit 10 transmits search information with the type being television 3a and the manufacturer being Company A to the server 30. Then, the server control unit 30a searches the device database based on the search information, extracts candidate device information, and transmits it to the first control unit 10. In this way, the first control unit 10 may narrow down the candidate device information using the search information obtained from the user 1 in the process of obtaining candidates for pattern data. Of course, if the first instruction from the user 1 in the setting operation includes any of the type, manufacturer, product name, and model number, these may be used as search information. Then, the above-mentioned trial may be performed using the pattern data contained in the narrowed-down device information.

[0085] In addition, in relation to the setting operation, the remote command device 2 may be provided with a light receiving element (not shown) such as a photodiode or a phototransistor. For example, if the user 1 is unable to operate the controlled device 3 as intended even after trials using the above-mentioned pattern data candidates, the first control unit 10 prompts the user 1 to hold an IR remote control attached to the controlled device 3 over the light receiving element and output a control signal such as ON / OFF (to press a button corresponding to the operation). The first control unit 10 generates pattern data from the output of the light receiving element and prompts the user 1 to input the type, manufacturer, product name, product number, and operation content of the controlled device 3. This can simplify the setting operation even if the device information of the controlled device 3 is not stored in the device database. In this way, it is possible to prompt the user 1 to actively communicate with the remote command device 2 even during the setting operation.

[0086] The monitoring system S1 according to the present invention has been described in detail above based on specific embodiments, but these embodiments are merely examples, and the present invention is not limited to these embodiments. For example, the division of the processing performed by the first control unit 10 and the server control unit 30a may be changed as appropriate. As a specific example, in the second embodiment, the extraction of the control target device 3 consuming the most power and the analysis of the time period when power is at its peak are described as being performed by the first control unit 10, but this processing may be performed by the server control unit 30a, and the remote command device 2 may receive the results and present them to the user 1.

[0087] Also, in the second embodiment, when it is determined that the controlled device 3 does not respond appropriately to the control signal, the control signal may be sent again after moving the remote control device 2 (see the processing of ST05 (No), ST06, ST07 (Yes), ST08 and subsequent ST02 in the flowchart of FIG. 4).

[0088] Also, analysis of energy consumption and estimation of the presence or absence of signs of cognitive impairment in the user 1 may be performed by the server control unit 30a instead of the first control unit 10, and the first control unit 10 may receive the results. That is, the "control unit" in the present invention may be either the first control unit 10 or the server control unit 30a, or these may cooperate to form the "control unit." [Industrial Applicability]

[0089] The monitoring system, remote control device, robot, and monitoring method of the present invention encourage users to actively communicate with the remote control device, and are particularly capable of preventing feelings of isolation and mental health problems among the elderly, and therefore can be widely used in elderly care facilities, home care settings, and the like. [Explanation of symbols]

[0090] 1 User 2 Remote control device 3. Controlled Devices 4. Smart meters 5. Smart Plug 6 Information terminal 7. Electrical Equipment 10 First control section 13 Imaging unit 14 Information Notification Department 15 Control signal output section 16 Moving mechanism 17 Environmental Sensors 18 Information input section 30 Servers 30a Server control unit 30b Server storage section 50 Network S1 Monitoring System

Claims

1. an information input unit that accepts input of instructions or information by a user; an information notifying unit that notifies the user of predetermined information; a control signal output unit that outputs a control signal to a predetermined control target device to control the control target device; A control unit, The control unit is A monitoring system characterized in that, after the control signal output unit outputs the control signal to the controlled device based on the user's instructions inputted to the information input unit, the control signal output unit inquires of the user about the response status of the controlled device via the information notification unit.

2. Equipped with environmental sensors, The control unit is The monitoring system according to claim 1, further comprising: notifying the user via the information notification unit to put the controlled device into a specific state based on an output from the environmental sensor.

3. When the output of the environmental sensor satisfies a predetermined condition, The control unit is The monitoring system according to claim 2, characterized in that the control-target device is controlled to a specific state even if the user does not input an instruction to put the control-target device into the information input unit.

4. The control unit is Obtaining energy consumption used in the residence of the user; The monitoring system according to claim 1, further comprising: notifying the user via the information notification unit to put the control-target device into a predetermined state based on the energy consumption.

5. The monitoring system according to any one of claims 1 to 4, characterized in that voice information is input to the information input unit, and the information notification unit outputs the voice information.

6. The control unit is A monitoring system as described in any one of claims 1 to 4, characterized in that the presence or absence of signs of cognitive impairment in the user is estimated based on the information input into the information input unit, and if it is determined that the user has signs of cognitive impairment, a person other than the user is notified of this fact.

7. Further, an imaging unit is provided, The monitoring system according to any one of claims 1 to 4, characterized in that the control unit associates the controlled device with the control signal based on an image of the controlled device captured by the imaging unit.

8. an information input unit that accepts input of instructions or information by a user; an information notifying unit that notifies the user of predetermined information; a control signal output unit that outputs a control signal to a predetermined control target device to control the control target device; A control unit, The control unit is A remote control device characterized in that, after the control signal output unit outputs the control signal to the controlled device based on the user's instructions inputted to the information input unit, the control signal output unit inquires of the user about the response status of the controlled device via the information notification unit.

9. A robot comprising the remote command device according to claim 8.

10. Further, a moving mechanism is provided, The control unit is When it is determined that the control target device is not responding appropriately based on the information input to the information input unit after inquiring of the response status of the control target device to the user, 10. The robot according to claim 9, wherein the positional relationship between the remote command device and the controlled device is changed by operating the moving mechanism.

11. Accept instructions from users, outputting a control signal to a controlled device based on an instruction input by the user; A monitoring method comprising the step of inquiring of the user about the response status of the controlled device to the control signal.

Images