Monitoring device and program

Abstract

The problem is to realize the control of electrical appliances according to various types of gestures of subjects such as residents or pets. [Solution] A monitoring device (40) includes an acquisition means (42) that acquires an image of an object (99) captured by an imaging device (41) in a building from the imaging device (41), an estimation means (42) that estimates the type of gesture of the object (99) captured in the image based on the image acquired by the acquisition means (42), and a control means (42) that controls an electrical appliance (10, 11, 12, 20 or 30) based on the type of gesture estimated by the estimation means (42).

Description

[Technical Field]

[0001] The technology disclosed in this specification relates to a monitoring device and a program. [Background technology]

[0002] Patent Document 1 discloses a monitoring device installed in a room in a nursing home or in the home of an elderly person living alone. The monitoring device includes a camera, a processing unit, and a microSD card. The camera captures images of subjects, such as elderly people. The processing unit performs privacy processing based on the image signal input from the camera and outputs the processed image data to a facility terminal. The privacy processing converts the image of the subject contained in the image data acquired by the camera into a skeletal image and superimposes the skeletal image on a background image corresponding to the room. The facility terminal displays the processed image. The processing unit performs a motion detection process. Specifically, the processing unit determines whether the subject is moving based on the image signal input from the camera and outputs a notification command to the facility terminal as necessary. The facility terminal learns various postures or movements of each subject based on the image data captured by the monitoring device's camera and generates a learning model. The learning model is recorded on the microSD card. The processing unit of the monitoring device recognizes the posture or movement of the subject by determining which posture or movement the subject's current posture or movement corresponds to based on the learning model stored on the microSD card and the image signal input from the camera. The recognition result by the processing unit is output to the facility terminal, for example, as text data indicating the recognized posture or movement. The processing unit outputs a notification command to the facility terminal as necessary. For example, if a supine position for a predetermined period of time or more during the day is detected, or if a movement such as a fall or aspiration is detected, the processing unit outputs a notification command.

[0003] However, the subject may take a posture or movement other than lying down, falling, or aspiration, and the only processing that the processing unit performs upon recognizing the posture or movement of the subject is to output a notification command to the facility terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-100754 Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the technique disclosed in this specification is to realize control of electrical appliances in accordance with various types of gestures of a subject such as a resident or a pet. [Means for solving the problem]

[0006] The reference numerals in parentheses below refer to FIGS. 1 and 2.

[0007] In order to solve the above problems, the present invention provides a system including: an acquisition means (42) that acquires an image of an object (99) captured by an imaging device (41) in a building from the imaging device (41); an estimation means (42) that estimates a type of gesture of the object captured in the image based on the image acquired by the acquisition means (42); and a control means (42) that controls an electric device (10, 11, 12, 20 or 30) based on the type of gesture estimated by the estimation means (42). a recording means for recording, if the estimation means (42) is unable to estimate the type of gesture of the target, an image captured by the imaging device (41), the image comprising a frame group from which the estimation means was unable to estimate the type of gesture of the target, a frame group for a predetermined period before that, and a frame group for a predetermined period after that, in a storage device (45); A monitoring device (40) is provided comprising:

[0008] In order to solve the above problems, a controller (42) connected to an imaging device (41) in a building is configured to transmit an image of an object (99) captured by the imaging device (41) to the imaging device. an acquisition process for acquiring the image from the device (41); an estimation process for estimating a type of gesture of the resident (99) captured in the image based on the image acquired by the acquisition process; and a control process for controlling an electric device (10, 11, 12, 20 or 30) based on the type of gesture estimated by the estimation process. a recording process for recording, in a storage device (45), video captured by the imaging device (41), which comprises a frame group from which the type of gesture of the target has been estimated by the estimation process, a frame group from a predetermined period before that, and a frame group from a predetermined period after that, if the type of gesture of the target cannot be estimated by the estimation process; A program (43) for executing the above is provided. [Effects of the Invention]

[0013] The unexpected kinds of gestures are captured on video as charming memories. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing the network configuration of a network and the Internet connected thereto. [Figure 2] FIG. 2 is a schematic diagram of a model represented by skeletal posture data. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments will be described with reference to the drawings. Features and technical effects of the embodiments will be understood from the following detailed description and drawings. However, the scope of the present invention is not limited to the embodiments disclosed below. Because the drawings are provided for illustrative purposes only, the scope of the present invention is not limited to the examples in the drawings.

[0016] <1. Building network configuration and artificial intelligence devices> FIG. 1 is a diagram showing the network configuration of a building connected to the Internet 1. The building may be a residence such as an apartment building or a detached house. The building may be used as a kindergarten, nursery school, school, daycare center, commercial facility, business establishment, office, or factory. A local network 90 is established in the building. The local network 90 may be established in a private or common area of ​​the building. The local network 90 is a wired LAN (Local Area Network), a wireless LAN, or a combination of these. The local network 90 is connected to the Internet 1 by a communication device such as a home gateway, a router, or a modem.

[0017] Artificial intelligence devices 2 to 4 are connected to the Internet 1. The artificial intelligence devices 2 to 4 are installed in a data center. The artificial intelligence devices 2 to 4 are, for example, GPU (Graphics Processing Unit) servers. The artificial intelligence devices 2 to 4 each have a main board, one or more CPUs (Central Processing Units), one or more GPUs, RAM (Random Access Memory), a storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive), and a communication device. The main board has a bus, a bus controller, an interface circuit, etc. The main board transmits information between the CPU, GPU, RAM, storage device, and communication device.

[0018] A trained model 5 as artificial intelligence (AI) is constructed in the artificial intelligence device 2 by software. The trained model 5 is a model composed of an artificial neural network having a large number of parameters, such as weights or biases. The artificial neural network may be, for example, a feedforward neural network (FNN), a multilayer persetron FNN, a convolutional neural network (CNN), a 3D-CNN, a recurrent neural network (RNN), a long short-term memory (LSTM) RNN, a gated recurrent unit (GRU) RNN, an autoencoder, a transformer, a generative artificial network (GAN), a graph neural network (GNN), or a spiking neural network (SNN). The artificial neural network may be a combination of several of these. The artificial neural network may also be a neural network derived from these. The trained model 5 may be a model obtained by deep machine learning. The machine learning may be supervised learning, unsupervised learning, reinforcement learning, or a combination of two or more of these. The training data or unlabeled data used for training the trained model 5 includes a large number of various still images depicting the entire body or parts of a person, and a large number of skeletal posture data identified by the skeletal postures of the entire body or parts depicted in those still images. As shown in FIG. 2 , for example, the skeletal posture data may include position data of each joint 101 from the central joint 101a to the distal joint 101b, and rotation angle data of each joint 101. The training data or unlabeled data used for training the trained model 5 may also include a large number of feature data representing spatial features of the entire body or parts extracted from those still images. Based on the input still images, the trained model 5 estimates the posture of the human body's skeleton depicted in the input still images to generate skeletal posture information. The skeletal posture information represents the posture of the human body's skeleton. The skeletal posture information may include position data of each joint from the central joint to the distal joint, and rotation angle data of each joint.

[0019] A trained model 6 as artificial intelligence (AI) is constructed in the artificial intelligence device 3 by software. The trained model 6 is a model composed of an artificial neural network having a large number of parameters, such as weights or biases. The artificial neural network may be, for example, a feedforward neural network (FNN), a multilayer persetron FNN, a convolutional neural network (CNN), a 3D-CNN, a recurrent neural network (RNN), a long short-term memory (LSTM) RNN, a gated recurrent unit (GRU) RNN, an autoencoder, a transformer, a generative artificial network (GAN), a graph neural network (GNN), or a spiking neural network (SNN). The artificial neural network may be a combination of several of these. The artificial neural network may also be a neural network derived from these. The trained model 6 is a model that has been created by deep machine learning. The machine learning may be supervised learning, unsupervised learning, reinforcement learning, or a combination of two or more of these. The training data or unlabeled data used to train the trained model 6 includes a large number of different posture transition data representing transitions in the posture of the human body's skeleton, and gesture type data representing types of gestures identified by the transitions in the posture of the human body's skeleton. The posture transition data is skeletal posture data arranged in time series. The training data or unlabeled data used to train the trained model 6 may include a large number of feature data representing spatial features of the posture transition data. The types of gestures may include infants' falling asleep, sleeping, temporary awakening, awakening (excluding temporary awakening), rolling over, thumb sucking, pacifier sucking, sucking reflex, rooting reflex, feeding reflex, lying on their stomachs, sitting, crawling, crawling, starting to stand up, standing up while holding onto something, starting to walk, toddling, falling down, raising one hand, raising both hands, laughing motion, crying motion, starting to cry motion, crying motion, crying and screaming motion, motion after urinary incontinence, motion after fecal incontinence, motion to climb over a fence, and various other motions. Gesture types may include various movements for different ages of infants and children.The gesture types may include various movements of people older than children. The gesture types may include various movements of bedridden people or people requiring care. The gesture types may include various movements of healthy people. The trained model 6 estimates the gesture type of the subject 99 based on the input posture transition data to generate gesture type data. The gesture type data represents the gesture type of the subject 99. The gesture type data generated by the trained model 6 may indicate that the type of human body movement cannot be estimated from the skeletal posture transition represented by the input posture transition data. Such gesture type data is referred to as inestimable data. The gesture type data generated by the trained model 6 may indicate that the input posture transition data does not represent the skeletal posture transition. Such gesture type data is referred to as target absence data. The gesture type data generated by the trained model 6 may also represent stillness. Such gesture type data is referred to as stillness data.

[0020] The trained models 5 and 6 are constructed in separate artificial intelligence devices 2 and 3. Alternatively, the trained models 5 and 6 may be constructed in a common artificial intelligence device.

[0021] A trained model 7 as artificial intelligence (AI) is constructed in the artificial intelligence device 4 by software. The trained model 7 is a model composed of an artificial neural network having a large number of parameters, such as weights or biases. The artificial neural network may be, for example, a feedforward neural network (FNN), a multilayer persetron FNN, a convolutional neural network (CNN), a 3D-CNN, a recurrent neural network (RNN), a long short-term memory (LSTM) RNN, a gated recurrent unit (GRU) RNN, an autoencoder, a transformer, a generative artificial network (GAN), a graph neural network (GNN), or a spiking neural network (SNN). The artificial neural network may be a combination of several of these. The artificial neural network may also be a neural network derived from these. The trained model 7 is a model that has been created by deep machine learning. The machine learning may be supervised learning, unsupervised learning, reinforcement learning, or a combination of two or more of these. The training data or unlabeled data used to train the trained model 7 includes a large number of various videos showing the movements of a person's entire body or parts, and gesture type data representing the types of gestures of the entire body or parts shown in the videos. The training data or unlabeled data used to train the trained model 7 may include temporal features and spatial features of the movements of the entire body or parts extracted from the videos. The training data or unlabeled data used to train the trained model 7 may include still images that are several frames of the videos. The training data or unlabeled data used to train the trained model 7 may include a large number of feature data representing spatial features of the entire body or parts extracted from several frames of the videos. The training data or unlabeled data used to train the trained model 7 may include a large number of skeletal posture data (see FIG. 2) identified by the skeletal postures of the entire body or parts shown in several frames of the videos.The training data or unlabeled data used to train the trained model 7 may include a large number of different posture transition data representing posture transitions of the entire body or parts of the skeleton captured in the video. The posture transition data is a time series of skeletal posture data. The training data or unlabeled data used to train the trained model 7 may include a large number of feature data representing spatial features of the posture transition data. Gesture types may include various movements of infants and young children, such as falling asleep, sleeping, temporary awakening, awakening (excluding temporary awakening), rolling over, thumb sucking, pacifier sucking, sucking reflex, rooting reflex, feeding reflex, lying on stomach, sitting, crawling, crawling, starting to stand up, standing up, starting to walk, toddling, sitting on bottom, falling over, lifting one hand, lifting both hands, laughing movements, crying movements, starting to cry movements, crying movements, screaming movements, movements after urinary incontinence, movements after fecal incontinence, climbing over a fence, and other movements. Gesture types may also include various movements of infants and young children by age. The gesture types may include various movements of people older than children. The gesture types may include various movements of bedridden people or people requiring care. The gesture types may include various movements of healthy people. The trained model 7 estimates the gesture type based on the input video and generates gesture type data. The gesture type data represents the gesture type. The gesture type data generated by the trained model 7 may indicate that the type of movement of a human body captured in the input video cannot be estimated. Such gesture type data is referred to as inestimable data. The gesture type data generated by the trained model 7 may indicate that a human body is not captured in the input video. Such gesture type data is referred to as target absent data. The gesture type data generated by the trained model 7 may indicate stillness. Such gesture type data is referred to as stillness data.

[0022] The building equipment will be described in detail below. An interphone 20 is installed in a building. The interphone 20 is a type of electrical device. The interphone 20 has a base unit 21 and a handset 22.

[0023] The base unit 21 is installed inside the building. The base unit 21 may be installed, for example, in the living room, dining room, kitchen, hallway, or bedroom of the building. The base unit 21 includes a display, a camera, a microphone, a speaker, an operating device, a control unit, and the like. The display may be a dot-matrix display, particularly a monochrome, grayscale, or color liquid crystal display device or an organic electroluminescence (EL) display device. In addition to the dot-matrix display, the display may also include an LED (Light Emitting Diode) display that lights up, turns off, or blinks. The LED display may have a changeable color. The operating device may include various operation buttons. In addition to the operation buttons, the operating device may also include a touch panel provided on the surface of the dot-matrix display. The control unit includes a processor such as a CPU (Central Processing Unit), a memory unit such as a semiconductor memory, a display driver, a camera driver, an audio output circuit, an audio input circuit, a communication device, an interface circuit, a circuit board, and the like. The control unit supervises and controls the operation of the base unit 21. The base unit 21 is connected to the slave unit 22 via a dedicated communication line or a local network 90.

[0024] The handset 22 may be installed outside the building at or near the entrance door of the building. The handset 22 may also be installed outside the building at the exterior, particularly at the entrance to the approach (for example, a gate wall, a gate sleeve, a side wall, a gatepost, or a fence). The handset 22 has a display, a camera, a microphone, a speaker, a call button, a control unit, etc. The display may be an LED display that turns on, off, or flashes. In addition to an LED display, the display may also be a dot matrix display, particularly a monochrome, grayscale, or color liquid crystal display device or an organic EL display device. The LED display may have a changeable color. The control unit has a processor such as a CPU (Central Processing Unit), a memory unit such as a semiconductor memory, a display driver, a camera driver, an audio output circuit, an audio input circuit, a communication device, a circuit board, etc. The control unit supervises and controls the operation of the handset 22.

[0025] The interphone 20 is linked to a mobile terminal 30, which will be described later.

[0026] The set volume of the ringing sound of the base unit 21 and the handset 22 can be manually adjusted by an operating device of the base unit 21. The set volume of the ringing sound of the base unit 21 and the handset 22 may also be set by a device connected to the local network 90.

[0027] When the master unit 21 and the slave unit 22 are set to the normal mode, the master unit 21 and the slave unit 22 operate as follows.

[0028] When a visitor presses the call button on the handset 22, communication is established between the handset 22 and the base unit 21, and then the base unit 21 outputs a ring tone at the set volume from its speaker, and the handset 22 also outputs a ring tone at the set volume from its speaker. Furthermore, the handset 22 receives audio input through its microphone, converts it into an audio signal, and transfers the audio signal to the base unit 21. The base unit 21 outputs audio based on the audio signal transferred from the handset 22 from its speaker. The handset 22 uses its camera to capture an image of the visitor or other person in front of the camera, and transfers the video signal to the base unit 21. The base unit 21 outputs an image based on the video signal transferred from the handset 22 on its display. When a person in the building responds to the ring tone by operating the operating device of the base unit 21, the base unit 21 and the handset 22 establish audio communication using the microphones and speakers of both the base unit 21 and the handset 22. Therefore, the base unit 21 outputs the voice input to the microphone of the handset 22 from the speaker of the base unit, and the handset 22 outputs the voice input to the microphone of the base unit 21 from the speaker of the handset.

[0029] When the base unit 21 is set to silent mode, the base unit 21 does not output a ringtone even if a visitor presses the call button on the handset 22. In silent mode, pressing the call button on the handset 22 by a visitor causes the base unit 21 to perform a notification function to the portable terminal 30 linked to the intercom 20. Therefore, when a visitor presses the call button on the handset 22, the base unit 21 or the handset 22 transmits a notification signal to the portable terminal 30 via the local network 90, and the portable terminal 30 receives and responds to the notification signal. The response of the portable terminal 30 may be, for example, vibration, emission of a notification sound, illumination of a light-emitting element, or a pop-up display. Other operations of the base unit 21 and the handset 22 are the same as those of the base unit 21 and the handset 22 in the normal mode described above.

[0030] When the base unit 21 is set to low volume mode, if a visitor presses the call button on the handset 22, the base unit 21 outputs a ring tone from the speaker at a volume lower than the set volume. However, if the base unit 21 is set to the lowest volume and the base unit 21 is set to low volume mode, the base unit 21 does not output a ring tone even if a visitor presses the call button on the handset 22. In low volume mode, pressing the call button on the handset 22 causes the base unit 21 or the handset 22 to perform a notification function to the portable terminal 30 linked to the intercom 20. Therefore, when a visitor presses the call button on the handset 22, the base unit 21 or the handset 22 transmits a notification signal to the portable terminal 30 via the local network 90, and the portable terminal 30 receives and responds to the notification signal. The response of the portable terminal 30 may be, for example, vibration, emission of a notification sound, illumination of a light-emitting element, or a pop-up display. In the low volume mode, pressing the call button on the handset 22 causes the base unit 21 or handset 22 to perform the blinking control function of the illuminator linked to the intercom 20. Therefore, when a visitor presses the call button on the handset 22, the base unit 21 or handset 22 turns the illuminator on and off several times via the local network 90, causing the illuminator to blink. The illuminator is one of the electrical devices 10 to 11 described below, and may be installed in the living room of the building, for example. The rest of the operation of the base unit 21 and handset 22 is the same as the operation of the base unit 21 and handset 22 in the normal mode described above.

[0031] When the handset 22 is set to the call rejection mode, even if a visitor presses the call button on the handset 22, the handset 22 does not output a ring tone, and the handset 22 displays on its display that it is asleep or sleeping. If the display on the handset 22 is an LED display, the LED display may indicate that it is asleep or sleeping by lighting up a certain color or flashing. If the display on the handset 22 is a dot matrix display, the dot matrix display may display a text string indicating that it is asleep or sleeping. Other operations of the base unit 21 and the handset 22 are the same as those of the base unit 21 and the handset 22 in the normal mode.

[0032] Electrical devices 10 to 12 are placed indoors or outdoors. The number of electrical devices 10 is one or more, the number of electrical devices 11 is one or more, and the number of electrical devices 12 is one or more. The electrical devices 10 to 12 may be installed, fixed, or portable. If the electrical devices 10 to 12 operate with low power consumption, they will contribute to the promotion of carbon neutrality, the realization of a decarbonized society, and the achievement of Sustainable Development Goals (SDGs).

[0033] The electrical devices 10 to 12 are, for example, lighting devices, air conditioners, air purifiers, electric heaters, floor heating devices, gas hot water heaters, heat pump hot water heaters, electric window shutters, electric curtain opening and closing devices, electric locks, ventilation fans, televisions, audio equipment, rice cookers, microwave ovens, electric kettles, electric coffee makers, electric hot plates, electric toasters, refrigerators, electric washing machines, electric vacuum cleaners, alarms, fire alarms, gas leak alarms, or earth leakage alarms.

[0034] The electrical appliance 10 corresponds to a computer network. The electrical appliance 10 is connected to a local network 90. ​​By inputting a control signal from the local network 90, the electrical appliance 10 is remotely controlled in response to the control signal. The electrical appliance 10 may be remotely controlled by a multifunctional portable terminal 30 or a monitoring device 40 (described later) by inputting a control signal via the local network 90 from the portable terminal 30 or a monitoring device 40. Here, remote control includes setting various settings in the electrical appliance 10 and changing the settings of the electrical appliance 10. Note that some electrical appliances 10 may be connected to the Internet 1 instead of the local network 90. ​​Even if the electrical appliance 10 is connected to the Internet 1, the electrical appliance 10 can be remotely controlled by the portable terminal 30 or the monitoring device 40 by inputting a control signal from the portable terminal 30 or the monitoring device 40 via the local network 90 and the Internet 1.

[0035] The electrical appliance 11 does not correspond to a computer network. It does not matter whether the electrical appliance 11 is connected to the local network 90. ​​The electrical appliance 11 is remotely controlled by an attached remote control. The electrical appliance 11 is remotely controlled by a wireless communication device 13 that emits a signal similar to the control signal emitted by the remote control to the electrical appliance 11. The wireless communication device 13 is a communication device that emits infrared or radio waves. The control signal emitted to the electrical appliance 11 is transmitted from the wireless communication device 13 to the electrical appliance 11 via infrared or radio waves. When the electrical appliance 11 receives the control signal, the electrical appliance 11 is remotely controlled in accordance with the control signal. The wireless communication device 13 may transmit a control signal to the electrical appliance 11 via infrared or radio waves in response to a command from the mobile terminal 30 or a monitoring device 40 (described later), thereby remotely controlling the electrical appliance 11 from the mobile terminal 30 or a monitoring device 40 (described later). The infrared rays emitted by the wireless communication device 13 may comply with the IrDA (registered trademark) or IrSS (registered trademark) communication standard. The radio waves emitted by the wireless communication device 13 may conform to the communication standards of Bluetooth (registered trademark), Wi-Fi (registered trademark), Z-Wave (registered trademark), Zigbee (registered trademark), or Matter (registered trademark). Here, the remote control includes making various settings in the electrical device 11 and changing the settings of the electrical device 11. The wireless communication device 13 may be provided individually for each electrical device 11, or may be provided commonly for several electrical devices 11. Note that one or several electrical devices 11 corresponding to a computer network may be remotely controlled by the wireless communication device 13.

[0036] The electrical appliance 12 may or may not be compatible with a computer network. The electrical appliance 12 may or may not be connected to a local network 90. ​​The electrical appliance 12 corresponds to the management device 14. The management device 14 may be a controller also known as a HEMS (Home Energy Management System). The electrical appliance 12 is remotely controlled by the management device 14. That is, when the electrical appliance 12 inputs a control signal output by the management device 14, the electrical appliance 12 is remotely controlled in accordance with the control signal. The management device 14 may transmit a control signal to the electrical appliance 12 in accordance with a command from the mobile terminal 30 or a monitoring device 40 described below, whereby the electrical appliance 12 may be remotely controlled by the mobile terminal 30 or a monitoring device 40 described below. Here, remote control includes making various settings in the electrical appliance 12 and changing the settings of the electrical appliance 12.

[0037] A mobile terminal 30 is connected to a local network 90. ​​The mobile terminal 30 is a type of electrical device. The mobile terminal 30 is a computer system, such as a mobile phone, smartphone, tablet computer, laptop computer, or desktop computer, on which a general-purpose operating system (OS) is installed. The general-purpose OS may be, for example, Windows (registered trademark), Android (registered trademark), iOS (registered trademark), macOS (registered trademark), Linux (registered trademark), or Unix (registered trademark). The mobile terminal 30 is connected to the Internet 1 through the local network 90. ​​An application program associated with the monitoring device 40 or a program installed thereon is installed in the general-purpose OS of the mobile terminal 30. The application program is executable by the mobile terminal 30 on the general-purpose OS. Execution of the application program by the mobile terminal 30 enables the mobile terminal 30 to perform functions such as communicating with the monitoring device 40 through the local network 90, configuring various settings for the monitoring device 40 through the local network 90, and controlling the monitoring device 40 through the local network 90.

[0038] The monitoring device 40 is connected to a local network 90. ​​If the monitoring device 40 operates with low power consumption, the monitoring device 40 will contribute to promoting carbon neutrality, realizing a decarbonized society, and achieving the Sustainable Development Goals (SDGs). The monitoring device 40 is placed indoors or outdoors. The monitoring device 40 monitors a subject 99. The subject 99 may be a resident of a building. The subject 99 may be an infant, toddler, or child. The subject 99 may be older than the pediatric age range. The subject 99 may be bedridden. The subject 99 may be a person requiring care. The subject 99 may be a healthy person.

[0039] The monitoring device 40 may be stationary, fixed, or portable. The monitoring device 40 may be attached to a ceiling, an interior wall, an exterior wall, a beam, a pillar, a floor, a eaves, or a roof porch. The monitoring device 40 may be attached to a bed, a baby cot, a stroller, or a nursing bed. The monitoring device 40 may be mounted on an automatic tracking device that tracks the subject 99. The automatic tracking device is a multi-wing aircraft or a mobile vehicle. The multi-wing aircraft flies to follow the subject 99. The mobile vehicle drives to follow the subject 99. The automatic tracking device may track the subject 99 using video data captured by the monitoring device 40. The monitoring device 40 may be attached to an automatic tracking device that changes the direction and attitude of the monitoring device 40 in accordance with the movement of the subject 99. The automatic tracking device may track the subject 99 using video data captured by the monitoring device 40.

[0040] The monitoring device 40 may be a dedicated or general-purpose computer. A dedicated computer is one specialized for monitoring inside a building. A general-purpose computer is a computer system such as a mobile phone, smartphone, tablet computer, laptop computer, or desktop computer that has a general-purpose operating system (OS) such as Windows (registered trademark), Android (registered trademark), iOS (registered trademark), macOS (registered trademark), Linux (registered trademark), or Unix (registered trademark) installed, and that performs the same functions as a dedicated computer using a program installed on the general-purpose OS.

[0041] The monitoring device 40 includes an imaging device 41 and a controller 42. The monitoring device 40 may include a display device such as a liquid crystal display device or an organic EL display device. The monitoring device 40 may include an input device such as a button or a touch panel. The imaging device 41 includes an imaging lens and a planar image sensor. The imaging device 41 is connected to the controller 42. The imaging device 41 captures an image in front of the imaging lens with the planar image sensor and outputs a signal of the image obtained by the imaging to the controller 42. When the imaging lens of the imaging device 41 is directed toward the subject 99, the subject 99 appears in the image captured by the imaging device 41. The controller 42 includes a main board, a hardware processor such as a CPU, RAM, a storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), an image processing circuit, and a communication device. The controller 42 may include a GPU and a display device. The image processing circuit processes the signal of the image captured by the imaging device 41. The main board includes a bus, a bus controller, an interface circuit, etc. The main board transmits signals between the hardware processor, RAM, storage device, video processing circuit, communication device, and GPU.

[0042] A program 43 is stored in the storage device of the controller 42. The program 43 is executable by the controller 42, particularly the hardware processor. The controller 42, particularly the hardware processor, executes various processes in accordance with the program, causing each component of the monitoring device 40 to operate under the control of the hardware processor, and causing the controller 42 to perform various functions. The controller 42 may have an image recognition function that recognizes an image of the subject 99 in the video captured by the imaging device 41. The controller 42 may have a person identification function that distinguishes the image of the subject 99 from images of other people. The controller 42 may have a function that stores the video captured by the imaging device 41 as a temporary video file. When the controller 42 identifies the subject 99 using the person identification function, the controller 42 may have a function that stores the video captured by the imaging device 41 as a temporary video file. The controller 42 may have a function that trims the video file from the cutting start time to the cutting end time from the temporary video file. The controller 42 may have a function that captures frames from the temporary video file as still image files from the temporary video file.

[0043] The controller 42 is connected to a storage device 45. In the example shown in FIG. 1 , the storage device 45 is connected to the controller 42 via an interface. Alternatively, the storage device 45 may be connected to a local network 90, and the controller 42 may access the storage device 45 through the local network 90 to be connected to the storage device 45. The storage device 45 may be connected to the Internet 1, and the controller 42 may access the storage device 45 through the local network 90 and the Internet 1 to be connected to the storage device 45. The storage device 45 is a semiconductor storage device, a magnetic storage device, a NAS (Network Attached Storage), a data server, a file server, or a cloud computing system.

[0044] The controller 42 records various data in the storage device 45. For example, the controller 42 may add to history data 46 representing a history of gesture types estimated in the past. In the history data 46, the gesture types and the times of actions are associated and arranged in chronological order. When the monitoring device 40 is used to monitor multiple subjects 99 and the controller 42 has a person identification function, the controller 42 records the history data 46 for each subject 99 in the storage device 45. The history data 46 does not include the same gesture types (unpredictable, target absent, and still) as the unpredictable data, target absent data, or still data. Note that when the monitoring device 40 is used for the first time, the history data 46 is not stored in the storage device 45.

[0045] For example, before deleting a temporary video file, the controller 42 transfers the video file to the storage device 45 and records the video file in the storage device 45. For example, the controller 42 may trim the temporary video file, then transfer the cut-out video file to the storage device 45 and record the video file in the storage device 45. The video file 47 stored in the storage device 45 is recorded by the controller 42. The video in the video file 47 captures a type of gesture that the subject 99 has performed for the first time. If the controller 42 has a person identification function, the controller 42 sorts the video files 47 for each subject 99 and records the video files 47 in the storage device 45.

[0046] The controller 42 may edit a plurality of moving image files 47 stored in the storage device 45 to generate a moving image file that is a digest version of these moving image files 47 , and record the generated moving image file in the storage device 45 .

[0047] The controller 42 may transfer the digest version of the video file to the mobile terminal 30 at a set time, for example, on the birthday of the subject 99. When the mobile terminal 30 receives the video file from the controller 42, the video file may be automatically played.

[0048] The controller 42 may periodically play back a plurality of video files 47 stored in the storage device 45 in sequence, and display the video on the display device.

[0049] 2. Operation of the monitoring device (1) Acquisition of video The imaging device 41 continuously captures images of the subject 99 and outputs the signals of the images obtained by capturing the images to the controller 42. The controller 42 continuously receives the video signals transferred from the imaging device 41, thereby continuously acquiring images of the subject 99. The controller 42 stores the images captured by the imaging device 41 as temporary video files.

[0050] (2) Gesture type estimation While the controller 42 is acquiring an image from the imaging device 41, the controller 42 estimates the type of gesture of the subject 99 captured in the image. Specifically, this is as shown in (2-1) or (2-2) below. In (2-1) below, the controller 42 estimates the type of gesture of the subject 99 captured in the image using trained models 5 and 6. In (2-2) below, the controller 42 estimates the type of gesture of the subject 99 captured in the image using trained model 7.

[0051] (2-1) (2-1-1) The controller 42 extracts frames from the video transferred from the imaging device 41 at regular intervals. The regular interval is equal to an integer multiple of the reciprocal of the video frame rate [fps]. If the regular interval is equal to 1 time the frame rate, the controller 42 extracts a frame from the video every other frame. If the regular interval is equal to 2 times the frame rate, the controller 42 extracts a frame from the video every other frame. If the regular interval is equal to 3 times the frame rate, the controller 42 extracts a frame from the video every three frames.

[0052] Each time a frame is extracted, the controller 42 transfers the extracted frame to the artificial intelligence device 2 and inputs the frame into the trained model 5. The artificial intelligence device 2 then uses the trained model 5 to estimate the skeletal posture of the subject 99 appearing in the frame based on the frame, and generates skeletal posture information. The artificial intelligence device 2 transfers the generated skeletal posture information to the controller 42. The controller 42 generates time series data by arranging the skeletal posture information received from the artificial intelligence device 2 in time series. The time series data is posture transition data that represents the transition in the skeletal posture of the subject 99. The controller 42 estimates the transition in the skeletal posture of the subject 99 by acquiring such posture transition data.

[0053] In the above description, the controller 42 executes the frame extraction process. Alternatively, the artificial intelligence device 2 may execute the frame extraction process. In this case, the controller 42 transfers the video transferred from the imaging device 41 as streaming to the artificial intelligence device 2, and the artificial intelligence device 2 extracts frames from the video at regular intervals.

[0054] In the above description, the controller 42 generates posture transition data obtained by arranging the skeletal posture information in chronological order. Alternatively, the artificial intelligence device 2 may generate posture transition data obtained by arranging the skeletal posture information in chronological order. In this case, the artificial intelligence device 2 transfers the posture transition data to the controller 42. The controller 42 obtains such posture transition data and estimates the transition of the skeletal posture of the subject 99.

[0055] (2-1-2) The controller 42 transfers the above-mentioned posture transition data to the artificial intelligence device 3, which inputs the posture transition data into the trained model 6. The artificial intelligence device 3 then uses the trained model 6 to estimate the type of gesture of the subject 99 based on the posture transition data, and generates gesture type data. The artificial intelligence device 3 transfers the generated gesture type data to the controller 42. The controller 42 acquires such gesture type data, thereby estimating the type of gesture of the subject 99.

[0056] (2-2) The controller 42 transfers the video of the subject 99 to the artificial intelligence device 4, which inputs the video into the trained model 7. The video input to the trained model 7 may be streaming or a video file. The artificial intelligence device 4 uses the trained model 7 to estimate the gestures of the subject 99 in the video based on the video, and generates gesture type data. The artificial intelligence device 4 transfers the generated gesture type data to the controller 42. The controller 42 acquires this gesture type data and thereby estimates the type of gesture of the subject 99.

[0057] (2-3) While the controller 42 is acquiring an image from the imaging device 41, the controller 42 repeatedly estimates the gesture type as described above at a fixed cycle. Therefore, the controller 42 acquires gesture type data one after another. Every time the controller 42 acquires gesture type data, the controller 42 executes the following processes (3) to (8).

[0058] (3) Recording a video of a gesture for the first time The controller 42 determines whether the gesture type data is unpredictable data, target non-existent data, or still data. If the gesture type data is not unpredictable data, target non-existent data, or still data, the controller 42 refers to the history data 46 and determines whether the history data 46 contains the same type of gesture as the gesture type data.

[0059] If the history data 46 does not include a type that is the same as the gesture type data, the controller 42 records the video captured by the imaging device 41 as a video file 47 in the storage device 45. Specifically, the controller 42 trims, from the temporarily stored video file, a determined frame group used to determine the type of gesture, a previous frame group of a predetermined period immediately before the determined frame group, and a subsequent frame group of a predetermined period immediately after the determined frame group, and records the video file 47 consisting of the previous frame group, the determined frame group, and the subsequent frame group in the storage device 45. In addition to recording the video file 47 in the storage device 45, the controller 42 may post the video file 47 to an SNS (social networking service) via the Internet 1. In addition to recording the video file 47 in the storage device 45, the controller 42 may post the video file 47 to an SNS (social networking service) via the Internet 1.

[0060] The controller 42 executes the process (3) before the process (8) described below.

[0061] (4) Video recording of gestures of unpredictable types The controller 42 determines whether the gesture type data is non-estimable data.

[0062] If the gesture type data is non-estimable data, the controller 42 records the video captured by the imaging device 41 as a video file 47 in the storage device 45. Specifically, the controller 42 trims, from the temporarily stored video file, a determined frame group used to determine the gesture type, a previous frame group of a predetermined period immediately before the determined frame group, and a subsequent frame group of a predetermined period immediately after the determined frame group, and records a video file 47 consisting of the previous frame group, the determined frame group, and the subsequent frame group in the storage device 45. In addition to recording the video file 47 in the storage device 45, the controller 42 may post the video file 47 to a social networking service (SNS) via the Internet 1. In addition to recording the video file 47 in the storage device 45, the controller 42 may transfer the video file 47 to the portable terminal 30. When the portable terminal 30 receives the video file 47 from the controller 42, the video file 47 may be automatically played.

[0063] (5) Control of electrical equipment The controller 42 determines whether the gesture type data represents the type “falling asleep” or the type “sleeping.” If the gesture type data represents the type “falling asleep” or the type “sleeping,” the controller 42 performs the following processes (5-1) to (5-7).

[0064] After the controller 42 determines that the gesture type data is the type "falling asleep" or the type "sleeping," the controller 42 determines whether the gesture type data represents the type "awakening (excluding temporary awakening)." If the gesture type data represents the type "awakening (excluding temporary awakening)," the controller 42 releases the control performed by the controller 42 in the following processes (5-1) to (5-7).

[0065] (5-1) Intercom control The controller 42 remotely controls the intercom 20 via the local network 90. ​​Specifically, the controller 42 sets the base unit 21 to silent mode or silent mode and sets the handset 22 to call rejection mode. Therefore, even if a visitor presses the call button on the handset 22, the base unit 21 does not ring or rings at a volume lower than the set volume. This allows the target person 99 to continue sleeping stably. Furthermore, the base unit 21 transmits a notification signal to the portable terminal 30 in response to the call button on the handset 22 being pressed, and the portable terminal 30 responds on behalf of the base unit 21. This allows other residents to respond to the visitor by sensing the response of the portable terminal 30. The base unit 21 flashes a light (one of the electrical devices 10-12) in response to the call button on the handset 22 being pressed, allowing other residents to respond to the visitor by sensing the flashing of the light.

[0066] Even if a visitor presses the call button on handset 22, handset 22 does not ring, and handset 22 displays on a display that the visitor has fallen asleep or is asleep, so the visitor will be aware of the situation. Therefore, even if the subject 99 or another resident does not respond to the visitor, the visitor does not have to wait in front of handset 22 for a long time. The display that the visitor has fallen asleep or is asleep may be displayed as text, or may be lit or flashing.

[0067] (5-2) Air conditioning control The air conditioner is one of the electrical devices 10 to 12. The air conditioner may be placed or installed in a room where the subject 99 is present. The controller 42 remotely controls the air conditioner via the local network 90. ​​For example, the controller 42 may reduce or increase the set airflow rate of the air conditioner to a predetermined airflow rate. The predetermined airflow rate may be changeable in advance by the remote controller. Reducing the set airflow rate of the air conditioner reduces the operating noise of the air conditioner, allowing the subject 99 to continue sleeping stably. The controller 42 may reduce the set temperature of the air conditioner, particularly the set temperature for cooling. The controller 42 may increase the set temperature of the air conditioner. The controller 42 may adjust the set temperature of the air conditioner to a predetermined temperature. Such adjustment of the set temperature contributes to a comfortable sleep for the subject 99. The controller 42 may start the air conditioner and operate it at an appropriate set temperature. Such operation contributes to a comfortable sleep for the subject 99. The controller 42 may stop the air conditioner. Shutting down the air conditioning unit causes the sound of the unit to disappear, allowing Subject 99 to continue sleeping stably.

[0068] (5-3) Ventilation fan control The ventilation fan is any one of the electrical devices 10 to 12. The ventilation fan may be installed in the room where the subject 99 is present, or may be installed in a room other than the present room. The controller 42 remotely controls the ventilation fan through the local network 90. ​​For example, the controller 42 may stop the ventilation fan. The controller 42 may reduce the operating speed of the ventilation fan. Stopping or reducing the speed of the ventilation fan eliminates or reduces the operating noise of the air conditioner, allowing the subject 99 to continue sleeping stably.

[0069] (5-4) Control of television or audio equipment The television or sound equipment is any of the electrical devices 10 to 12. The television or sound equipment may be placed or installed in a room where the subject 99 is present. The controller 42 remotely controls the television or sound equipment through the local network 90. ​​For example, the controller 42 may reduce the set volume of the television or sound equipment. The controller 42 may set the set volume of the television or sound equipment to mute. The controller 42 may turn off the power to the television or sound equipment. Remote control of the television or sound equipment by the controller 42 contributes to the subject 99's stable sleep.

[0070] (5-7) Illuminator control The illuminator is any of the electrical devices 10 to 12. The illuminator may be placed or installed in a room where the subject 99 is present. The controller 42 remotely controls the illuminator through the local network 90. ​​For example, the controller 42 may reduce the set brightness of the illuminator. The controller 42 may also turn off the illuminator. Remote control of the illuminator by the controller 42 contributes to the subject 99's stable sleep.

[0071] (6) Alarm The controller 42 determines whether the gesture type data represents a specific type. The specific type may be, for example, the type "sleeping on one's stomach" or the type "action of trying to climb over the rails of a crib." The specific type may also be based on the age of the subject 99. If the gesture type data represents a specific type, the controller 42 transmits an alarm signal to the intercom 20, the portable terminal 30, or the alarm. Upon receiving the alarm, the master unit 21 of the intercom 20 outputs an alarm sound from its speaker. Upon receiving the alarm signal, the portable terminal 30 outputs an alarm sound from its speaker regardless of whether the mute setting is enabled. Upon receiving the alarm signal, the alarm outputs an alarm sound. The alarm is any of the electrical devices 10 to 12.

[0072] (7) Notice The controller 42 transmits a notification signal to the portable terminal 30, the content of which corresponds to the type represented by the gesture type data. When the portable terminal 30 receives the notification signal, the portable terminal 30 executes processing according to the content of the notification signal. For example, if the gesture type data represents the type "awakening (excluding temporary awakening)", the portable terminal 30 displays that the subject 99 has woken up. For example, if the gesture type data represents the type "falling asleep", the portable terminal 30 displays that the subject 99 has fallen asleep. For example, if the gesture type data represents the type "sleeping", the portable terminal 30 displays that the subject 99 is asleep.

[0073] (8) Add to history The controller 42 may associate the action time (specifically, the time when the gesture type data was generated) with the same type as the gesture type data, and add the type and action time to the history data 46.

[0074] <3. Modifications> In the above embodiment, the subject photographed by the imaging device 41 is the subject 99. However, the subject photographed by the imaging device 41 may also be a pet.

[0075] When the subject of a photograph is a pet, the training data or unlabeled data used to train the trained model 5 includes a large number of various still images capturing the entire body or parts of the animal, and a large number of skeletal posture data identified by the skeletal posture of the entire body or parts captured in those still images. The training data or unlabeled data used to train the trained model 5 may also include a large number of feature data representing spatial features of the entire body or parts extracted from those still images. Based on the input still images, the trained model 5 estimates the skeletal posture of the animal, particularly a pet, captured in the input still images to generate skeletal posture information. The skeletal posture information represents the skeletal posture of the animal.

[0076] When the subject of a photograph is a pet, the training data or unlabeled data used to train the trained model 6 includes a large number of different posture transition data representing the transition of posture of the animal's skeleton, and gesture type data representing the type of gesture identified by the transition of posture of the animal's skeleton. The training data may include a large number of feature data representing spatial features of the posture transition data. The types of gestures may include various movements by animal type, size, sex, and age. The trained model 6 estimates the type of gesture of the animal, particularly a pet, based on the input posture transition data and generates gesture type data.

[0077] When the filmed subject is a pet, the training data or unlabeled data used to train the trained model 7 includes a large number of different videos showing the movements of the animal's entire body or parts, and gesture type data representing the types of gestures of the animal's entire body or parts shown in the videos. The training data may include temporal and spatial features of the movements of the animal's entire body or parts extracted from the videos. The training data may include still images that are several frames of the videos. The training data or unlabeled data used to train the trained model 7 may include a large number of feature data representing spatial features of the animal's entire body or parts extracted from several frames of the videos. The training data or unlabeled data used to train the trained model 7 may include a large number of skeletal posture data identified by the skeletal posture of the animal's entire body or parts shown in several frames of the videos. The training data or unlabeled data used to train the trained model 7 may include a large number of different posture transition data representing the posture transitions of the animal's entire body or parts shown in the videos. The types of gestures may include various movements by animal type, size, sex, and age. The trained model 7 estimates the type of gesture based on the input video and generates gesture type data.

[0078] <4. Variations> In the above embodiment, trained models 5 and 6 are constructed in the artificial intelligence devices 2 and 3, respectively. Alternatively, models equivalent to trained models 5 and 6 may be constructed in the controller 42, and the controller 42 may use these models to calculate gesture type data from video. Alternatively, a model equivalent to trained model 7 may be constructed in the controller 42, and the controller 42 may use this model to calculate gesture type data from video.

[0079] <5. Variations> The subject to be photographed may be something other than a resident or a pet. For example, the subject to be photographed may be a suspicious person, and the imaging device 41 may be the exterior wall of a building. When the trained model 6 or the trained model 7 estimates that the type of gesture is suspicious behavior, the controller 42, which receives the estimation result, may activate an alarm. The alarm may be any of the electrical devices 10 to 11. When the trained model 6 or the trained model 7 estimates that the type of gesture is suspicious behavior, the controller 42 may send an alert signal to the portable terminal 30, and the portable terminal 30 may generate an alarm sound in response to the alert signal.

[0080] <6. Summary> (1) The electrical device 10, the electrical device 11, the electrical device 12, the interphone 20, or the portable terminal 30 is controlled based on the type of gesture of the target person 99 estimated by the estimation process of the controller 42. The specific content of the control depends on the type.

[0081] (2) Even if a visitor presses the call button on the handset 22 outside the home of the intercom 20 while the subject 99 is sleeping, the main unit 21 does not ring or rings at a low volume. Therefore, the sleep of the subject 99 or pet is not disturbed.

[0082] (3) While the subject 99 is asleep, even if the base unit 21 does not ring when the call button on the handset 22 is pressed, or even if the volume of the base unit 21 is low, the base unit 21 sends a notification signal to the portable terminal 30, and the portable terminal 30 responds in place of the base unit 21. Therefore, other residents can respond to the visitor by sensing the response of the portable terminal 30.

[0083] (4) While the subject 99 is asleep, even if the call button on the handset 22 is pressed, the base unit 21 does not ring, or even if the volume of the base unit 21 is low, the base unit 21 flashes the light. Therefore, other residents can respond to the visitor by detecting the flashing of the light.

[0084] (5) While the subject 99 is asleep, the handset 22 is set to a call rejection mode. When the handset 22 is set to the call rejection mode, even if a visitor presses the call button on the handset 22, the handset 22 does not ring, and the handset 22 displays on the display that the visitor has fallen asleep or is asleep. Such behavior of the handset 22 gives the visitor some kind of awareness. Therefore, even if the subject 99 or other residents do not answer the visitor, the visitor does not have to wait in front of the handset 22 for a long time.

[0085] (6) While the subject 99 is sleeping, the set airflow rate of the air conditioner is reduced, the set temperature of the air conditioner is adjusted to a predetermined temperature, and the air conditioner is started or stopped. Reducing the set airflow rate of the air conditioner reduces the operating noise of the air conditioner, so the air conditioner does not disturb the sleep of the subject 99. Adjusting the set temperature of the air conditioner contributes to the subject 99's comfortable sleep. Starting the air conditioner contributes to the subject 99's comfortable sleep. Stopping the air conditioner causes the operating noise of the air conditioner to disappear, so the air conditioner does not disturb the sleep of the subject 99.

[0086] (7) While the subject 99 is sleeping, the ventilation fan is turned off or the operating speed of the ventilation fan is reduced, so that the ventilation fan does not disturb the subject 99's sleep.

[0087] (8) While subject 99 is sleeping, the volume of the television is reduced or the television is turned off, so that the television does not disturb subject 99's sleep.

[0088] (9) While subject 99 is sleeping, the volume of the audio equipment is reduced and the audio equipment is turned off, so that the audio equipment does not disturb subject 99's sleep.

[0089] (10) While the subject 99 is sleeping, the set brightness of the lighting device is reduced or the lighting device is turned off, so that the lighting device does not disturb the subject 99's sleep.

[0090] (11) If the controller 42 cannot estimate the type of gesture of the target person 99, the controller 42 records a video file 47 in the storage device 45. The video file 47 is used to estimate the type of gesture of the target person 99. use The video is made up of frames that are being played, frames that precede them, and frames that follow them. Therefore, unexpected kinds of gestures remain as charming memories.

[0091] (12) If the type of gesture that is estimated is the first time that the subject 99 has performed it, the controller 42 records a video file 47 in the storage device 45. The video file 47 is an image made up of a group of frames from which the type of gesture of the subject 99 could not be estimated, the group of frames before that, and the group of frames after that. The type of gesture that the subject 99 has never performed before remains as a charming video file as a memory.

[0092] (13) The trained model 7 contributes to improving the accuracy of video-based gesture type estimation.

[0093] (14) Trained Model 5 contributes to improving the accuracy of estimating the skeletal posture based on video. Trained Model 5 contributes to improving the accuracy of estimating the transition of that posture.

[0094] (15) The trained model 6 contributes to improving the accuracy of gesture type estimation based on skeletal pose transitions.

[0095] (17) As described above in (2-1), the change in the skeletal posture of the subject 99 or pet is estimated based on the video, and the type of gesture is estimated based on the change in the skeletal posture, making it easy to estimate the type of gesture.

[0096] (18) The above-disclosed embodiments have been made for the purpose of illustration and example only, and are not intended to limit the scope of the present invention, which should be interpreted by the terms of the claims.

[0097] <7. Overview> The above description of the embodiments discloses the invention of the monitoring device and the program described below. The reference numerals in parentheses below refer to FIGS.

[0098] (Section 1) an acquisition means (42) for acquiring an image of an object (99) captured by an imaging device (41) in a building from the imaging device (41); an estimation means (42) for estimating a type of gesture of the object (99) captured in the video based on the video acquired by the acquisition means (42); a control means (42) for controlling an electric device (10, 11, 12, 20 or 30) based on the type of gesture estimated by the estimation means (42); A monitoring device (40) comprising:

[0099] According to the first aspect as described above, the electric appliance (10, 11, 12, 20 or 30) is controlled based on the type of gesture of the target (99) such as a resident estimated by the estimation means (42) or estimation process.

[0100] (Section 2) the electrical device (10, 11, 12, 20 or 30) is an intercom (20) installed in the building, If the estimation means (42) estimates that the type of gesture of the target (99) is falling asleep or sleeping, the control means (42) sets the setting of the master unit (21) of the intercom (20) in the house to silent mode or low volume mode. 2. The monitoring device (40) of claim 1.

[0101] According to the second paragraph as described above, even if a visitor presses the call button on the handset (22) outside the home of the intercom (20) while the subject (99) is sleeping, the main unit (21) will not ring or will ring at a low volume, so the subject's sleep will not be disturbed.

[0102] (Section 3) The silent mode or the low volume mode includes a mode in which, when a call button on a handset (22) of the intercom (20) outside the home is pressed, the master unit (21) or the handset (22) transmits a notification signal to a mobile terminal (30) linked to the intercom (20). 3. The monitoring device (40) of claim 2.

[0103] (Section 4) The mobile terminal (30) is a smartphone in the building. 4. The monitoring device (40) of claim 3.

[0104] According to the above third and fourth paragraphs, even if the base unit 21 does not ring when the call button on the handset 22 is pressed while the subject 99 is asleep, or even if the volume of the base unit 21 is low, the base unit 21 or the handset 22 sends a notification signal to the portable terminal 30, and the portable terminal 30 responds in place of the base unit 21. Therefore, someone other than the subject can respond to a visitor by sensing the response of the portable terminal 30.

[0105] (Section 5) The silent mode or the low volume mode includes a mode in which the master unit (21) or the slave unit (22) blinks the illuminator linked to the intercom (20) when a call button on the slave unit (22) outside the home of the intercom (20) is pressed. 10. The monitoring device (40) according to any one of items 2 to 4.

[0106] According to the above-mentioned paragraph 5, while the subject (99) is sleeping, the base unit (21) will not ring even if the call button on the handset (22) is pressed, or even if the volume of the base unit (21) is low, the base unit (21) or handset (22) will flash the light. Therefore, someone other than the subject can sense the flashing of the light and respond to a visitor.

[0107] (Section 6) If the estimation means (42) estimates that the type of gesture of the target (99) is falling asleep or sleeping, the control means (42) sets the setting of the handset (22) outside the home of the intercom (20) to a call rejection mode. 6. A monitoring device (40) according to any one of items 2 to 5.

[0108] According to the sixth paragraph, while the subject (99) is sleeping, the handset (22) outside the home is set to call rejection mode, so that even if a visitor presses the call button on the handset (22), the visitor will have some kind of notice. Therefore, even if the subject (99) or someone else does not answer the visitor, the visitor does not have to wait in front of the handset (22) for a long time.

[0109] (Section 7) The call rejection mode includes a mode in which, when the call button of the handset (22) is pressed, the display of the handset (22) displays the fact that the handset (22) is asleep or asleep. 7. The monitoring device (40) of claim 6.

[0110] According to the seventh paragraph, even if a visitor presses the call button on the handset (22) while the subject (99) is sleeping, the handset (22) will display a message on the display indicating that the subject (99) has fallen asleep or is asleep, so the visitor will have some kind of awareness. Therefore, even if the subject or someone else does not respond to the visitor, the visitor does not have to wait in front of the handset (22) for a long time.

[0111] (Section 8) The call rejection mode includes a mode in which the handset (22) does not emit a call sound even when the call button of the handset (22) is pressed. A monitoring device (40) according to paragraph 6 or paragraph 7 is provided.

[0112] According to the above-mentioned paragraph 8, even if a visitor presses the call button on the handset (22) while the subject (99) is sleeping, the handset (22) will not ring, so the visitor will be aware of the situation. Therefore, even if the subject (99) or someone else does not answer the visitor, the visitor will not have to wait in front of the handset (22) for a long time.

[0113] (Section 9) the electrical equipment (10, 11, 12, 20, or 30) is an air conditioning device (10, 11, or 12) installed in the building, When the estimation means (42) estimates that the type of gesture of the target (99) is falling asleep or sleeping, the control means (42) adjusts the set air volume of the air conditioner (10, 11 or 12) to a predetermined air volume, adjusts the set temperature of the air conditioner (10, 11 or 12) to a predetermined temperature, starts the air conditioner (10, 11 or 12), or stops the air conditioner (10, 11 or 12). 2. The monitoring device (40) of claim 1.

[0114] According to the above paragraph 9, reducing the set airflow rate of the air conditioner (10, 11, or 12) reduces the operating noise of the air conditioner (10, 11, or 12, so the air conditioner (10, 11, or 12) does not disturb the sleep of the subject (99). Adjusting the set airflow rate of the air conditioner (10, 11, or 12) contributes to the subject (99)'s comfortable sleep. Adjusting the set temperature of the air conditioner (10, 11, or 12) contributes to the subject (99)'s comfortable sleep. Starting the air conditioner (10, 11, or 12) contributes to the subject (99)'s comfortable sleep. Stopping the air conditioner (10, 11, or 12) causes the operating noise of the air conditioner (10, 11, or 12 to disappear, so the air conditioner (10, 11, or 12) does not disturb the sleep of the subject (99).

[0115] (Section 10) The electrical equipment (10, 11, 12, 20 or 30) is a ventilation fan (10, 11 or 12) installed in the building, When the estimation means (42) estimates that the gesture type of the target (99) is falling asleep or sleeping, the control means (42) stops the ventilation fan (10, 11 or 12) or adjusts the operating speed of the ventilation fan (10, 11 or 12) to a predetermined speed. A monitoring device (40) according to claim 1 is provided.

[0116] According to the above paragraph 10, stopping the ventilation fan (10, 11, or 12) causes the operating noise of the ventilation fan (10, 11, or 12) to disappear, so the ventilation fan (10, 11, or 12) does not disturb the sleep of the subject (99). Reducing the operating speed of the ventilation fan (10, 11, or 12) causes the operating noise of the ventilation fan (10, 11, or 12) to decrease, so the ventilation fan (10, 11, or 12) does not disturb the sleep of the subject (99). Adjusting the ventilation fan (10, 11, or 12) contributes to the comfortable sleep of the subject (99).

[0117] (Section 11) the electrical appliance (10, 11, 12, 20 or 30) is a television (10, 11 or 12) in the building; When the estimation means (42) estimates that the type of gesture of the target (99) is falling asleep or sleeping, the control means (42) reduces the volume setting of the television (10, 11 or 12) or turns off the power of the television (10, 11 or 12). 2. The monitoring device (40) of claim 1.

[0118] According to the above-described paragraph 11, the volume setting of the television (10, 11 or 12) is reduced or the television (10, 11 or 12) is turned off, so that the television (10, 11 or 12) does not disturb the sleep of the subject (99).

[0119] (Section 12) the electrical equipment (10, 11, 12, 20 or 30) is an acoustic equipment (10, 11 or 12) in the building, When the estimation means (42) estimates that the type of gesture of the target (99) is falling asleep or sleeping, the control means (42) reduces the set volume of the audio device (10, 11 or 12) or turns off the power of the audio device (10, 11 or 12). 2. The monitoring device (40) of claim 1.

[0120] According to the above-mentioned paragraph 12, the set volume of the audio device (10, 11 or 12) is reduced or the audio device (10, 11 or 12) is turned off, so that the audio device (10, 11 or 12) does not disturb the sleep of the subject (99).

[0121] (Section 13) the electrical equipment (10, 11, 12, 20 or 30) is a lighting fixture in the building; The control means (42) reduces the set brightness of the illuminator or turns off the illuminator. A monitoring device (40) as set forth in paragraph 1 is provided.

[0122] According to the above-described item 13, the set brightness of the illuminator is reduced or the illuminator is turned off, so that the illuminator does not disturb the sleep of the subject (99).

[0123] (Section 14) a recording means (42) for recording the image captured by the imaging device (41) in a storage device if the estimation means (42) is unable to estimate the type of gesture of the target (99); The monitoring device (40) according to any one of items 1 to 13 is provided, further comprising:

[0124] (Section 15) The video recorded by the recording means (42) is video consisting of a frame group in which the estimation means (42) was unable to estimate the type of gesture of the target (99), a frame group before that, and a frame group after that. 15. A monitoring device (40) according to paragraph 14.

[0125] (Section 16) an acquisition means (42) for acquiring an image of an object (99) captured by an imaging device (41) in a building from the imaging device (41); an estimation means (42) for estimating a type of gesture of the object (99) captured in the image based on the image acquired by the acquisition means (42); a recording means (42) for recording the image captured by the imaging device (41) in a storage device if the type of gesture estimated by the estimation means (42) is not the type estimated in the past; A monitoring device (40) comprising:

[0126] (Section 17) The video recorded by the recording means (42) is video consisting of a frame group in which the estimation means (42) estimates the type of gesture of the target (99), a frame group before that, and a frame group after that. 17. A monitoring device (40) according to paragraph 16.

[0127] According to the above items 14, 15, 16, and 17, a video file of a type of gesture that the subject (99) has never made before is left as a memory. The video file is interesting.

[0128] (Section 18) The estimation means (42) uses artificial intelligence to estimate the type of gesture of the object (99) captured in the video acquired by the acquisition means (42). A monitoring device (40) according to any one of paragraphs 1 to 17 is provided.

[0129] According to the above paragraph 18, artificial intelligence contributes to improving the estimation accuracy of the estimation means (42).

[0130] (Section 19) The estimation means (42) a posture transition estimation means (42) for estimating a transition of a posture of the skeleton of the object (99) shown in the image based on the image acquired by the acquisition means (42); a gesture estimation means (42) for estimating a type of gesture of the target (99) from the transition estimated by the posture transition estimation means (42); and The type of gesture estimated by the gesture estimation means (42) is the type of gesture estimated by the estimation means (42). A monitoring device (40) according to any one of paragraphs 1 to 17 is provided.

[0131] According to the above-described paragraph 19, the transition of the skeletal posture of the object (99) is estimated based on the image, and the type of gesture is estimated based on the transition of the skeletal posture, which simplifies the estimation of the type of gesture.

[0132] (Section 20) The posture transition estimation means (42) uses artificial intelligence to estimate a transition in the posture of the skeleton of the resident (99) captured in the image acquired by the acquisition means (42). There is provided a monitoring device (40) as set forth in paragraph 19, characterized in that:

[0133] According to the above-mentioned paragraph 20, the artificial intelligence contributes to improving the estimation accuracy of the posture transition estimation means (42).

[0134] (Section 21) The gesture estimation means (42) uses artificial intelligence to estimate the type of gesture of the target (99) from the transition estimated by the posture transition estimation means (42). 21. A monitoring device (40) according to claim 19 or 20, characterized in that:

[0135] According to the above-mentioned paragraph 21, the artificial intelligence contributes to improving the estimation accuracy of the gesture estimation means (42).

[0136] (Section 22) A controller (42) connected to an imaging device (41) in the building, an acquisition process of acquiring, from the imaging device (41), an image of the object captured by the imaging device (41); an estimation process of estimating a type of gesture of the target captured in the video based on the video acquired by the acquisition process; a control process for controlling an electrical device (10, 11, 12, 20, or 30) based on the type of gesture estimated by the estimation process; A program (43) that executes the above.

[0137] According to the above-described paragraph 22, the electrical appliance (10, 11, 12, 20 or 30) is controlled based on the type of gesture of the target (99) such as a resident estimated by the estimation means (42) or the estimation process.

[0138] (Section 23) A controller (42) connected to an imaging device (41) in the building, an acquisition process of acquiring, from the imaging device (41), an image of the object (99) captured by the imaging device (41); an estimation process for estimating a type of gesture of the target (99) captured in the video based on the video acquired by the acquisition process; a recording process of recording the image captured by the imaging device (41) in a storage device if the type of gesture estimated by the estimation process is not the type estimated in the past; A program (43) that executes the above.

[0139] According to the above paragraph 23, a video file of a gesture that the subject (99) has never made before is left as a memory. The video file is interesting. [Explanation of symbols]

[0140] 2, 3, 4 Artificial Intelligence Device 5, 6, 7 Pre-trained model (artificial intelligence) 40 Monitoring equipment 41 Imaging device 42 Controller 43 Programs 45 Storage Devices 46 Historical Data 47 video files 10, 11, 12 Electrical Equipment 20 Intercom 21 Base unit 22 Handset 30 Portable terminals (electrical devices)

Claims

1. an acquisition means for acquiring an image of an object captured by an imaging device in a building from the imaging device; an estimation means for estimating a type of gesture of the target captured in the video based on the video acquired by the acquisition means; a control unit that controls an electrical device based on the type of gesture estimated by the estimation unit; a recording means for recording, if the estimation means is unable to estimate the type of gesture of the target, an image captured by the imaging device, the image comprising a frame group from which the estimation means was unable to estimate the type of gesture of the target, a frame group for a predetermined period before the frame group, and a frame group for a predetermined period after the frame group; A monitoring device comprising:

2. the electrical device is an intercom installed in the building, When the estimation means estimates that the type of gesture of the target is falling asleep or sleeping, the control means sets the setting of the master unit of the intercom in the house to silent mode or low volume mode. The monitoring device of claim 1 .

3. The silent mode or the low volume mode includes a mode in which, when a call button on a handset of the intercom outside the home is pressed, the master unit or the handset transmits a notification signal to a mobile terminal linked to the intercom. The monitoring device according to claim 2 .

4. The mute mode or the low volume mode includes a mode in which, when a call button on a slave unit of the intercom outside the home is pressed, the master unit blinks a lighting device linked to the intercom. The monitoring device according to claim 2 .

5. When the estimation means estimates that the type of gesture of the target is falling asleep or sleeping, the control means sets the setting of the handset of the intercom outside the house to a call rejection mode. The monitoring device according to claim 2 .

6. the electrical equipment is an air conditioning device installed in the building, When the estimation means estimates that the type of the gesture of the target is falling asleep or sleeping, the control means adjusts the set air volume of the air conditioner to a predetermined air volume, adjusts the set temperature of the air conditioner to a predetermined temperature, starts the air conditioner, or stops the air conditioner. The monitoring device of claim 1 .

7. The estimation means estimates the type of gesture of the target captured in the video acquired by the acquisition means using artificial intelligence.

7. A monitoring device according to claim 1, 2, 3, 4, 5 or 6.

8. The estimation means a posture transition estimation means for estimating a posture transition of the object's skeleton shown in the image based on the image acquired by the acquisition means as skeleton posture transition data in which the positions and rotation angles of each joint of the skeleton are arranged in time series; a gesture estimation means for estimating a type of gesture of the target from the transition estimated by the posture transition estimation means; and The type of gesture estimated by the gesture estimation means is the type of gesture estimated by the estimation means. A monitoring device according to any one of claims 1 to 6.

9. The posture transition estimation means estimates a transition of posture of the skeleton of the object captured in the image acquired by the acquisition means using artificial intelligence.

9. The monitoring device according to claim 8.

10. The gesture estimation means estimates the type of gesture of the target from the transition estimated by the posture transition estimation means using artificial intelligence.

9. The monitoring device according to claim 8.

11. A controller connected to the imaging device in the building an acquisition process of acquiring, from the imaging device, an image of the object captured by the imaging device; an estimation process of estimating a type of gesture of the target captured in the video based on the video acquired by the acquisition process; a control process for controlling an electrical device based on the type of gesture estimated by the estimation process; a recording process for recording, in a storage device, a video image captured by the imaging device, the video image including a frame group from which the type of gesture of the target could not be estimated by the estimation process, a frame group from a predetermined period before the frame group, and a frame group from a predetermined period after the frame group; if the type of gesture of the target could not be estimated by the estimation process, the video image captured by the imaging device includes a frame group from which the type of gesture of the target could not be estimated by the estimation process, a frame group from a predetermined period before the frame group, and a frame group from a predetermined period after the frame group; A program that executes the following.

Images