Monitoring Terminal and Monitoring Method

Abstract

To provide a monitoring terminal and monitoring method that perform monitoring while keeping the privacy of an object of monitoring.SOLUTION: A monitoring system has: a monitoring terminal which is installed in a monitoring object facility 9; a controller; and a user terminal, interconnected over a network communicably with one another. The monitoring terminal 10 includes: a control unit including an object area estimation block 161 that estimates object area information representing an area in which an object of monitoring is located, from an image showing the object of monitoring; an abnormality determination block 163 that determines whether or not an abnormality has occurred in the object of monitoring; and a notification block 165 that, when the abnormality determination block 163 determines that an abnormality has occurred, generates a notification image which contains an object area image that is an image showing area information but does not contain an image showing the object of monitoring itself, and transmits the notification image to other devices.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technology for monitoring.

Background Art

[0002] In Japan, which is called an aging society, about 30% of the current population is 65 years old or older. With the increase in the elderly population, the number of elderly people living alone (so-called solitary elderly) is also on the rise, exceeding 6 million as of 2015. Since such solitary elderly people live alone, it is often difficult to take appropriate measures when troubles such as illness or accident occur, and there is also a risk of aggravating injuries or illnesses. Such problems are not limited to the problems of the elderly living alone, but can also occur in facilities premised on living in a single space such as elderly care facilities, and can also occur for the elderly who may be alone during some time periods of the day even when there are cohabitants such as relatives.

[0003] The aggravation of injuries and illnesses leads to an increase in the number of elderly people requiring care. As a result, it causes a large loss to society, such as an increase in the country's social security costs and an increase in the number of caregivers leaving their jobs among the prime working generation. Thus, as one of the social issues, the monitoring of social vulnerable groups centered on the elderly is becoming increasingly important in the future. For such problems, a technology has been proposed to monitor by installing cameras in the living space of the elderly (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, if the situation of the elderly, such as their actions and expressions, is constantly monitored through captured images or the like, although it may be possible to solve problems from the perspective of injuries and illnesses, privacy issues will arise. Such problems have not been solved and sufficient preparations have not been made. As a result, there are cases where the discovery of illnesses and injuries is delayed, leading to aggravation and even death from loneliness. Such problems are not limited to the elderly but are common problems for those being monitored (hereinafter referred to as "monitoring targets").

[0006] In view of the above circumstances, an object of the present invention is to provide a technology that enables monitoring while protecting the privacy of the monitoring target.

Means for Solving the Problems

[0007] One aspect of the present invention is a monitoring terminal including a target area estimation unit that estimates target area information indicating an area where the monitoring target is located from an image of the monitoring target, an abnormality determination unit that determines whether an abnormality has occurred with respect to the monitoring target, and a notification unit that, when the abnormality determination unit determines that an abnormality has occurred, generates a notification image including a target area image that is an image indicating the target area information and not including an image of the monitoring target itself, and transmits the notification image to another device.

[0008] One aspect of the present invention is the above monitoring terminal, which is installed in a monitoring target facility that is a facility where the monitoring target is active.

[0009] One aspect of the present invention is the above monitoring terminal, further including a posture estimation unit that estimates the posture of the monitoring target. The target area estimation unit estimates the area using a learned model obtained by machine learning using a first learning dataset including a teacher image in which the monitoring target is photographed and a teacher image in which the monitoring target is not photographed. The posture estimation unit estimates the posture using a learned model obtained by machine learning using a second learning dataset in which the ratio of the teacher image in which the monitoring target is not photographed to the whole is relatively less than that of the first learning dataset.

[0010] One aspect of the present invention is the above-described monitoring terminal, which further includes a safety confirmation unit that outputs voice content prompting a response as to whether the monitored object is in a safe state from a voice output unit, and then generates information on a confirmation result based on the voice input from the voice input unit. The notification unit transmits the confirmation result to another device.

[0011] One aspect of the present invention is a monitoring method including: a target area estimation step of estimating target area information indicating an area where the monitored object is located from an image obtained by imaging the monitored object; an abnormality determination step of determining whether an abnormality has occurred in the monitored object; and a notification step of, when it is determined in the abnormality determination step that an abnormality has occurred, generating a notification image including a target area image that is an image indicating the target area information and not including an image of the monitored object itself, and transmitting the notification image to another device.

Effect of the Invention

[0012] An object of the present invention is to provide a technology that enables monitoring while protecting the privacy of the monitored object.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0014] Hereinafter, specific configuration examples of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing the system configuration of the monitoring system 100 of the present invention. The monitoring system 100 is a system for monitoring a monitoring target who is active in the monitoring target facility 90. The monitoring target facility 90 may be, for example, a residence where the monitoring target lives, an apartment building or facility (such as a nursing home) where a plurality of monitoring targets live, a facility (such as a nursery, kindergarten, school) where one or more monitoring targets act, or a facility (such as a prison) where one or more monitoring targets live. The monitoring system 100 includes a monitoring terminal 10, a control device 20, and a user terminal 30. The monitoring terminal 10, the control device 20, and the user terminal 30 are communicably connected via a network 40. The network 40 may be a network using wireless communication or a network using wired communication. The network 40 may be configured by combining a plurality of networks.

[0015] The monitoring terminal 10 is installed in the facility 90 to be monitored. The monitoring terminal 10 captures an image of the object to be monitored within the facility 90 to be monitored and converts the image of the object to be monitored into a target area image. The target area image is an image showing an area (hereinafter referred to as "target area") where the whole or a part of the object to be monitored is located. The monitoring terminal 10 generates a notification image including the target area image without including the image of the object to be monitored. The notification image generated by the monitoring terminal 10 is transmitted to the control device 20 and the user terminal 30. Since the notification image does not include the image of the object to be monitored, the privacy of the object to be monitored can be protected. Further, the notification image includes the target area image. Therefore, the state of the object to be monitored can be monitored based on the notification image. The user terminal 30 is operated by a person who plans to respond to an abnormality when an abnormality occurs in the object to be monitored. For example, the user terminal 30 may be used by a relative or acquaintance of the object to be monitored, may be used by a person who performs the task of monitoring (watching over) the object to be monitored, or may be used by another person. With such a mechanism, the monitoring system 100 can perform monitoring while protecting the privacy of the object to be monitored. Hereinafter, the details of the monitoring system 100 will be described.

[0016] FIG. 2 is a schematic block diagram showing a specific example of the functional configuration of the monitoring terminal 10. The monitoring terminal 10 is configured using an information device such as a smartphone, a tablet, a personal computer, a portable game machine, a stationary game machine, or a dedicated device. The monitoring terminal 10 includes a communication unit 11, an imaging unit 12, an audio input unit 13, an audio output unit 14, a display unit 15, a control unit 16, and a storage unit 17.

[0017] The communication unit 11 is a communication device. The communication unit 11 may be configured as, for example, a network interface. The communication unit 11 performs data communication with other devices via the network 40 in accordance with the control of the control unit 16. The communication unit 11 may be a device that performs wireless communication or a device that performs wired communication.

[0018] The imaging unit 12 is configured using a camera. The imaging unit 12 may be configured as the camera itself, or may be configured as an interface for connecting a camera as an external device to the monitoring terminal 10. The camera images a predetermined space within the facility 90 to be monitored. The imaging by the camera may be realized as a process of forming an image by receiving visible light, may be realized as a process of forming an image using a radar, or may be realized by other implementation methods. The predetermined space imaged by the camera is desirably a space where the object to be monitored may be located. For example, when the object to be monitored is an elderly person, it may be the space where the elderly person usually lives (such as a living room, a sitting room, a corridor, a bedroom, etc.). For example, when the object to be monitored is an infant or a young child, it may be the space within the range where the infant or young child is active (such as a living room, a sitting room, on a baby bed, etc.). One camera may be provided, or a plurality of cameras may be provided. For example, when the object to be monitored is an animal, it may be the space within the range where the animal is active (such as a cage, a living room, a shed, a toilet, etc.). The imaging unit 12 outputs the data of the image captured by the camera to the control unit 16.

[0019] The voice input unit 13 is configured using a microphone. The voice input unit 13 may be configured as the microphone itself, or may be configured as an interface for connecting a microphone as an external device to the monitoring terminal 10. The microphone acquires the voice of a predetermined space within the facility 90 to be monitored. The predetermined space where the voice is acquired by the microphone is desirably a space where the object to be monitored may be located. For example, it is desirable that it is the space imaged by the above-described camera. The voice input unit 13 outputs the data of the voice acquired by the microphone to the control unit 16.

[0020] The voice output unit 14 is configured using a speaker. The voice output unit 14 may be configured as the speaker itself, or may be configured as an interface for connecting a speaker as an external device to the monitoring terminal 10. The speaker outputs voice to a predetermined space within the facility 90 to be monitored. The predetermined space where the voice is output by the speaker is desirably a space where the object to be monitored may be located. For example, it is desirable that it is the space that the above-described camera is imaging. The voice output unit 14 outputs voice according to the voice signal output by the control unit 16.

[0021] The display unit 15 displays an image for the object to be monitored according to the control of the control unit 16. The display unit 15 may display, for example, an image for prompting a reaction to the object to be monitored. For example, it may display a character string such as "Are you okay?", or a character string such as "If you are okay, please press the following button" and an operation target such as a button including a character or an image indicating that no problem such as "okay" has occurred.

[0022] The control unit 16 is composed of a processor such as a CPU (Central Processing Unit) and a memory. By executing a program with the processor, the control unit 16 functions as a target area estimation unit 161, a posture estimation unit 162, an abnormality determination unit 163, a notification image generation unit 164, a notification unit 165, a safety confirmation unit 166, a data acquisition unit 167, and an information management unit 168. Note that all or part of each function of the control unit 16 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. A computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a semiconductor storage device (e.g., SSD: Solid State Drive), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may also be transmitted via a telecommunication line.

[0023] The target area estimation unit 161 estimates information (hereinafter referred to as "target area information") indicating an area (target area) in which all or part of the monitoring target is located in the image obtained from the imaging unit 12. The target area may be, for example, a rectangle including an area in which all or part of the monitoring target is located. In this case, the target area information may be information indicating the coordinates of the four corners of the rectangle. The target area is not limited to a rectangle and may be other geometric shapes (e.g., circular, elliptical, triangular, quadrilateral, pentagonal, etc.). The target area may be a shape indicated by a plurality of straight lines or curves along the shape of the monitoring target.

[0024] The target area estimation unit 161 may perform a process of estimating a target area (hereinafter referred to as "target area estimation") by using, for example, a learned model obtained by performing machine learning processing in advance using teacher images (first learning dataset). In this case, as the correct image of the teacher image, an image in which the whole of a living organism of the same type as the monitoring target is photographed or an image in which a part of the living organism of the same type (for example, only the head, only the upper body including the head, the upper body excluding the head, only one of the left and right arms, only the lower body, only the legs and feet) is photographed may be used. In this case, it is desirable that a label indicating that it is correct and information indicating the area where the whole or a part of the living organism of the same type as the monitoring target is located (target area information) are given to the correct image of the teacher image. However, the outer frame of the teacher image itself may be used as the target area information. In this case, it is not necessarily required that target area information is associated with the teacher image.

[0025] Also, as each correct image, an image of the front surface, an image of the side surface, and an image of the back surface may be used for both the whole and a part of the living organism of the same type as the monitoring target. When the attributes of the monitoring target (for example, age, gender, race, etc.) are determined, an image of a living organism whose attributes match those of the monitoring target may be used as the correct image. By using such correct images, the accuracy of target area estimation can be improved.

[0026] Furthermore, as an incorrect image of the teacher image, an image that does not include either the whole or a part of the living organism of the same type as the monitoring target may be used. For example, as an incorrect image of the teacher image, an image that does not include an image of a living organism may be used, or an image in which the whole of a different type of living organism is photographed or an image in which a part of a different type of living organism is photographed may be used. In this case, a label indicating that it is incorrect may be given to the incorrect image of the teacher image. The first learning dataset including the teacher image of the correct image and the teacher image of the incorrect image may be used for generating the learned model used by the target area estimation unit 161. Note that the technology used for the implementation of the target area estimation unit 161 is not necessarily limited to the above-described supervised learning, and any technology may be used.

[0027] The posture estimation unit 162 estimates the posture information of the monitoring target in the image obtained from the imaging unit 12. The posture information is information indicating the posture of the monitoring target. The posture information may be information selected from a plurality of predetermined posture candidates, for example. Such posture candidates may include information indicating a normal posture and information indicating an abnormal posture. Specific examples of the posture candidates include a standing state, a sitting state, a floor-sitting state, a lying state, a fallen state, a crouched state, a collapsed state, etc. As specific examples of the posture candidates, two states, a fallen state (FALL) and a non-fallen state (NOT FALL), may be defined.

[0028] The posture information may be estimated, for example, based on the monitoring target area obtained for the monitoring target. The posture estimation unit 162 may perform a process of estimating the posture information (hereinafter referred to as "posture estimation") by using a learned model obtained by performing machine learning processing using a teacher image (second learning dataset) in advance, for example. In this case, the ratio of the images that do not include either the whole or a part of the same kind of organism as the monitoring target among the teacher images included in the dataset of the teacher images (second learning dataset) used for machine learning may be lower than that of the first learning dataset. This ratio may be zero. That is, the configuration may be such that images in which the monitoring target is not photographed are not used as the teacher image data. Since it is clear that the monitoring target is included in the target area estimated in the process of the target area estimation unit 161, when the posture estimation is executed as a subsequent process, there is no need to use images of organisms of a type different from the monitoring target. Rather, by using only images of organisms of the same kind as the monitoring target, it is possible to improve the accuracy of the posture estimation.

[0029] As teacher image data, information indicating the state of the living thing photographed in each image may be given as a label. In this case, it is desirable that the content of each label be selected from among pre-defined state candidates. Note that the technology used for the implementation of the posture estimation unit 162 does not have to be limited to the processing using supervised learning as described above, and may be any technology.

[0030] The abnormality determination unit 163 determines whether an abnormality has occurred in the monitoring target. The abnormality determination unit 163 may determine an abnormality based on, for example, the target area information that is the estimation result of the target area estimation unit 161. The abnormality determination unit 163 may determine an abnormality based on, for example, the posture information that is the estimation result of the posture estimation unit 162. More specifically, posture information with a high possibility of an abnormality occurring (for example, "FALL") and posture information with a low possibility of an abnormality occurring (for example, "NOT FALL") are defined in advance, and it may be determined whether an abnormality has occurred based on the estimated posture information. The abnormality determination unit 163 may determine an abnormality based on, for example, both the target area information that is the estimation result of the target area estimation unit 161 and the posture information that is the estimation result of the posture estimation unit 162. The abnormality determination unit 163 may determine an abnormality based on, for example, the abnormality determination conditions pre-stored in the storage unit 17. That is, the abnormality determination unit 163 may determine that an abnormality has occurred when the information used for abnormality determination (for example, target area information, posture information, or other information) satisfies the abnormality determination conditions.

[0031] As an abnormality determination condition, for example, target area information or posture information indicating a high possibility that an injury or seizure has occurred to the person being monitored may be registered in advance. As an abnormality determination condition, a condition indicating that the lighting is on for a predetermined period of time even in a time zone indicating a predetermined late night (for example, from 0:00 to 4:00) may be defined. Whether the lighting is on or not may be obtained based on, for example, the brightness of the image captured by the imaging unit 12, or may be obtained based on the output of a brightness sensor (not shown) connected to the monitoring terminal 10, or may be determined by acquiring information indicating the state or power consumption of electrical equipment in the monitored facility 90 from a system such as HEMS. As an abnormality determination condition, it may be defined that the monitored target cannot be detected for a predetermined period of time in the video captured by the imaging unit 12. For example, it may be determined that the monitored target is detected according to the estimation of the target area information by the target area estimation unit 161, or it may be determined that the monitored target is detected when the posture can be determined by the posture estimation unit 162, or the detection of the monitored target may be performed using other human body detection algorithms (for example, detection of a person's face, detection of the silhouette of a person's whole body, use of a human body detection sensor, etc.). As an abnormality determination condition, it may be defined that a phenomenon having a significant difference from normal is detected using a learned model obtained by unsupervised learning based on normal captured images.

[0032] The notification image generation unit 164 generates a notification image by superimposing an image (target area image) indicating the target area information estimated by the target area estimation unit 161 on an image (hereinafter referred to as "background image") previously captured by the imaging unit 12. FIG. 3 is a diagram showing a specific example of the background image. The background image is an image obtained by previously imaging the imaging range of the camera used in the imaging unit 12. The background image does not include an image of the monitoring target itself or an image indicating the monitoring target. The background image may be updated, for example, at a predetermined timing. For example, an image captured at a predetermined time every day (a time when it is generally highly likely that the monitoring target is not active) may be used as the background image. For example, every time a predetermined period (e.g., 24 hours) elapses, an image in a state where the monitoring target area has not been estimated may be recorded as the background image. The target area image may be an image showing a frame indicating the target area, an image showing the outline of the monitoring target, or an image such as an illustration imitating the shape of the monitoring target.

[0033] FIG. 4 is a diagram showing a specific example of an image captured by the imaging unit 12. FIG. 5 is a diagram showing a specific example of the target area estimated by the target area estimation unit 161. In the image of FIG. 4, a person being monitored has fallen sideways in front of the door. A rectangle 51 surrounding the person being monitored who has fallen in front of the door is estimated as the target area by the target area estimation unit 161.

[0034] FIG. 6 is a diagram showing a first specific example of the notification image. In the notification image of FIG. 6, an image of a frame (rectangle 51) indicating the target area estimated in the image of FIG. 5 is superimposed on the background image of FIG. 3.

[0035] FIG. 7 is a diagram showing a second specific example of the notification image. In the notification image of FIG. 7, instead of the target area itself estimated in the image of FIG. 5, an illustration image 52 corresponding to the estimated target area is superimposed on the background image of FIG. 3. A plurality of types of such illustration images may be prepared in advance and recorded in the storage unit 17. For example, based on the shape of the estimated target area, the estimated posture information, etc., the illustration images stored in advance in association with them may be selected. For example, when the target area estimation unit 161 or the posture estimation unit 162 can estimate the orientation of the monitoring target, the illustration image may be further selected based on the estimation result.

[0036] As is clear from the examples of the notification images in FIGS. 6 and 7, in the notification image, the position of the monitoring target is known and the approximate state is also known. In this case, it can be seen that the target is in a state of lying on its side. On the other hand, since the texture and fine movements of the monitoring target itself are unknown, privacy is protected. For example, even if the monitoring target is naked or the monitoring target is dripping snot or saliva, such information is not included in the notification image. Also, since the background image captured in advance is used for the notification image, even if the monitoring target accidentally wets themselves or has diarrhea, the information indicating the filth is not included in the notification image. In this way, information that the monitoring target does not want to be left in the record is not included in the notification image.

[0037] When the notification condition is satisfied, the notification unit 165 transmits notification information including a notification image to another device via the communication unit 11. The notification condition may be, for example, being determined as abnormal by the abnormality determination unit 163. The notification condition may be, for example, being determined as abnormal by the abnormality determination unit 163 and not being confirmed as safe by the safety confirmation unit 166 described later. The notification condition may be defined as other conditions. When the notification unit 165 acquires the confirmation result by the safety confirmation unit 166, it generates notification information including information regarding the confirmation result. The information regarding the confirmation result may be, for example, a character string, an image, or identification information indicating the content of the confirmation result. Examples of the character string indicating the content of the confirmation result include a character string such as "In the result of the safety confirmation, there was a response saying it's okay.", a character string such as "In the result of the safety confirmation, there was a response saying help is needed.", and a character string such as "In the result of the safety confirmation, there was no response."

[0038] When the safety confirmation condition is satisfied, the safety confirmation unit 166 confirms the safety of the monitoring target by outputting a predetermined voice from the voice output unit 14. The safety confirmation condition may be, for example, being determined as abnormal by the abnormality determination unit 163. The predetermined voice output by the safety confirmation unit 166 may be a voice prompting a response as to whether the monitoring target is in a safe state. For example, it may be a voice such as "Are you okay?" or a voice such as "Do you need help?" After outputting the voice, the safety confirmation unit 166 outputs the confirmation result to the notification unit 165.

[0039] The safety confirmation unit 166 may generate a confirmation result through, for example, the following processing. After outputting a voice, the safety confirmation unit 166 acquires the voice data emitted from the monitoring target from the voice input unit 13. If the safety confirmation unit 166 cannot acquire the voice data generated from the monitoring target within a predetermined time, it may output to the notification unit 165 that no response is obtained as the confirmation result. The safety confirmation unit 166 performs voice recognition on the acquired voice, and if it is the predetermined content indicating danger, it may output to the notification unit 165 a confirmation result indicating danger. The safety confirmation unit 166 performs voice recognition on the acquired voice, and if it is the predetermined content indicating safety, it may output to the notification unit 165 a confirmation result indicating safety. The safety confirmation unit 166 performs voice recognition on the acquired voice, and may output the character string that is the result of the voice recognition to the notification unit 165 as the confirmation result. In these cases, the notification unit 165 may generate notification information including the confirmation result output by the safety confirmation unit 166.

[0040] The data acquisition unit 167 acquires various data about the monitoring target and the situation within the monitored facility 90. The data acquisition unit 167 may acquire information indicating the shape and installation position of furniture etc. installed in the space within the shooting range of the camera by detecting an object from the image captured by the imaging unit 12, for example. The data acquisition unit 167 may acquire information indicating the movement path (flow line) of the monitoring target based on, for example, the time-series video captured by the imaging unit 12. The data acquisition unit 167 periodically transmits the acquired data to the control device 20 via the communication unit 11.

[0041] The storage unit 17 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 17 stores the data used by the control unit 16. The storage unit 17 may store, for example, the terminal ID assigned to the monitoring terminal 10 and the attribute information (information such as name, address, gender, age, identification information, etc.) of the monitoring target targeted by the monitoring terminal 10.

[0042] FIG. 8 is a schematic block diagram showing a specific example of the functional configuration of the control device 20. The control device 20 is configured using an information device such as a personal computer or a server device. The control device 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

[0043] The communication unit 21 is a communication device. The communication unit 21 may be configured as a network interface, for example. The communication unit 21 performs data communication with other devices via the network 40 according to the control of the control unit 23. The communication unit 21 may be a device that performs wireless communication or a device that performs wired communication.

[0044] The storage unit 22 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 22 stores data used by the control unit 23. For example, the storage unit 22 may store a transmission destination information table. The transmission destination information table is a table that associates the identification information assigned to the monitoring terminal 10 (hereinafter referred to as "monitoring identification information") with the information of the transmission destination of the user notification information generated based on the notification information transmitted from the monitoring terminal 10 (hereinafter referred to as "notification destination information").

[0045] The control unit 23 is configured using a processor such as a CPU and a memory. The control unit 23 functions as a notification control unit 231 and a data control unit 232 when the processor executes a program. Note that all or part of each function of the control unit 23 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. A computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a semiconductor storage device (e.g., SSD), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0046] When the notification control unit 231 receives notification information from the monitoring terminal 10, it generates user notification information based on the received notification information. The user notification information includes, for example, a character string indicating that an abnormality has occurred in the monitoring target (e.g., "An abnormality has occurred. Please check."), and notification address information indicating the storage area where the notification image is stored. The character string included in the user notification information may be, for example, a fixed character string determined in advance regardless of the monitoring terminal 10 or the user terminal 30. The character string included in the user notification information may be a character string defined in association with the monitoring terminal 10 or the user terminal 30. These character strings may be stored in the storage unit 22.

[0047] The notification address information may be generated by the notification control unit 231 or may be generated by another server that publishes the notification image. For example, the notification control unit 231 may upload the notification image to a storage area accessible from other information devices. In this case, the address information (e.g., URL) indicating the upload destination storage area may be used as the notification address information.

[0048] The notification control unit 231 determines the notification destination based on the identification information included in the received notification information. The notification control unit 231 may determine, for example, based on the transmission destination information table stored in the storage unit 22, the notification destination information associated with the monitoring identification information that is the transmission source of the notification information, as the notification destination. The notification control unit 231 transmits the user notification information to the determined notification destination.

[0049] The data control unit 232 registers the data transmitted from the monitoring terminal 10 in the storage unit 22 in association with the monitoring identification information of the monitoring terminal 10 that is the transmission source. As described above, the data transmitted from the monitoring terminal 10 does not include an image of the monitoring target itself. Therefore, the data registered by the data control unit 232 is unlikely to include information that infringes privacy. The data control unit 232 may register the data transmitted from the monitoring terminal 10 in the storage unit 22 in association with the identification information (e.g., terminal ID) of the monitoring terminal 10 that is the transmission source.

[0050] FIG. 9 is a sequence chart showing a specific example of the operation flow of the monitoring system 100. When the imaging unit 12 of the monitoring terminal 10 captures an image, the control unit 16 acquires the image data captured by the imaging unit 12 (step S101). The target area estimation unit 161 estimates the target area of the monitoring target in the captured image (step S102). The posture estimation unit 162 estimates the posture of the monitoring target in the captured image (step S103). The abnormality determination unit 163 performs an abnormality determination (step S104). If it is not determined as abnormal, that is, if it is determined as normal (step S105-NO), the process of the control unit 16 returns to the process of step S101. On the other hand, if it is determined as abnormal, that is, if it is not determined as normal (step S105-YES), the notification unit 165 generates a notification image (step S106). The notification unit 165 generates notification information including the notification image and transmits the notification information to the control device 20 (step S107).

[0051] When the notification control unit 231 of the control device 20 receives the notification information, it generates user notification information according to the notification information. Also, the notification control unit 231 determines the transmission destination of the user notification information according to the received notification information. Then, the notification control unit 231 transmits the user notification information to the user terminal 30 (step S108). Also, the notification control unit 231 records the notification information in the storage unit 22 (step S109).

[0052] When the user terminal 30 receives user notification information, it outputs an indication that the user notification information has been received. For example, an image or text indicating that the user notification information has been received may be displayed in a predetermined area of the screen of the user terminal 30, or audio output or vibration motor drive may be performed according to a notification method preset in the user terminal 30. When the user instructs the output of the user notification information by operating the user terminal 30, the user terminal 30 outputs the content of the user notification information. For example, the user terminal 30 may display a character string indicating that an abnormality has occurred in the monitoring target and a character string indicating the notification address information on the screen (step S110). When the user operates the user terminal 30 to instruct the display of the notification image, the user terminal 30 accesses the notification address information and requests the transmission of the notification image (step S111).

[0053] When the control device 20 receives a request for transmitting a notification image to the notification address information, it transmits the notification image corresponding to the request to the user terminal 30 (step S112). When the user terminal 30 receives the notification image, it displays the received notification image on the screen (step S113).

[0054] FIG. 10 is a sequence chart showing another specific example of the operation flow of the monitoring system 100. In the sequence chart shown in FIG. 10, the safety confirmation is performed by the monitoring terminal 10. Hereinafter, only the differences from the sequence chart shown in FIG. 9 will be described. Note that the same reference numerals are assigned to the same processes in FIGS. 9 and 10.

[0055] When the safety confirmation unit 166 of the monitoring terminal 10 receives a notification from the notification unit 165, it outputs a safety confirmation voice via the voice output unit 14 (step S201). The safety confirmation unit 166 then accepts the input of the voice uttered thereafter (step S202). The safety confirmation unit 166 generates confirmation notification information based on the input voice. For example, when a voice with a sound pressure equal to or higher than a predetermined sound pressure is input, since it means that the monitoring target has reacted, confirmation information indicating that the safety has been confirmed may be generated. For example, when a voice with a sound pressure equal to or higher than a predetermined sound pressure is not input, since it means that the monitoring target has not reacted, confirmation information indicating that the safety has not been confirmed may be generated. The safety confirmation unit 166 transmits the generated confirmation information to the control device 20 (step S203). The notification control unit 231 of the control device 20 transmits the received confirmation information to the user terminal 30 (step S204). When the user terminal 30 receives the confirmation information, it outputs the received confirmation information (step S205). The output of the confirmation information may be performed by voice output, or by displaying characters or images on the screen, or in other ways. Note that the processing from step S101 to step S113 is the same as that in FIG. 9.

[0056] FIG. 11 is a sequence chart showing another specific example of the operation flow of the monitoring system 100. In the sequence chart shown in FIG. 11, the data generated by the monitoring terminal 10 is stored in the control device 20. Hereinafter, only the differences from the sequence chart shown in FIG. 9 will be described. Note that the same processes in FIGS. 9 and 11 are denoted by the same reference numerals.

[0057] The data acquisition unit 167 of the monitoring terminal 10 detects a flow line indicating the movement path of the monitoring target based on the time-series data of the images captured by the imaging unit 12 and records it in the storage unit 17 (step S301). If it is not determined as abnormal in step S104, the data acquisition unit 167 discards the information of the flow line detected in the past that is greater than a predetermined threshold. On the other hand, if it is determined as abnormal in step S104, the data acquisition unit 167 generates accumulated data including the information of the flow line for a predetermined period recorded in the storage unit 17 (step S302). The predetermined period is, for example, a period of a predetermined number of seconds (such as 5 seconds or 10 seconds) retroactively from the timing when the abnormality is detected. The data acquisition unit 167 transmits the generated accumulated data to the control device 20 (step S303). When receiving the accumulated data, the data control unit 232 of the control device 20 records the received accumulated data in the storage unit 22 (step S304).

[0058] In the monitoring system 100 configured as described above, the notification image used when notifying the outside about the monitoring target does not include an image of the monitoring target itself, but includes an image indicating the target area. In this way, since the notification image does not include an image of the monitoring target itself, the privacy of the monitoring target can be protected. Also, since the notification image includes an image indicating the position of the monitoring target, the state of the monitoring target can be monitored based on the notification image. With such a mechanism, the monitoring system 100 can perform monitoring while protecting the privacy of the monitoring target.

[0059] Also, in the process of estimating the area (target area) where the monitoring target is located, an image in which only a part of the organism of the same type as the monitoring target is photographed is also used as the correct image of the teacher image. Therefore, even if a part of the monitoring target is hidden in the shadow of furniture or the like in the photographed image, it is possible to more accurately estimate the position of the monitoring target based on only the remaining part that is photographed. For the same reason, it is possible to more accurately realize the estimation of the state of the monitoring target and the determination of whether it is abnormal or not.

[0060] In addition, in the monitoring system 100, information indicating the arrangement of furniture and the movement routes of the monitoring targets, etc. are stored in the control device 20 as accumulated data. These data do not include the images of the monitoring targets themselves. Therefore, it is possible to protect the privacy of the monitoring targets even when using these data for analysis and the like. Also, as shown in the sequence chart of FIG. 11, the movement route information accumulates data for a predetermined period retroactively from the timing when an abnormality occurred. Therefore, it is possible to efficiently obtain useful data for analyzing viewpoints such as what movement routes are likely to cause abnormalities and what furniture arrangements have led to movement routes that are likely to cause abnormalities.

[0061] (Modification example) The target area estimation unit 161 and the posture estimation unit 162 may be implemented as one function. That is, the target area estimation and the posture estimation may be performed simultaneously. Such processing may be realized, for example, by using the data of teacher images with labels indicating posture information pre - attached when generating the learned model used for target area estimation.

[0062] The posture estimation unit 162 and the abnormality determination unit 163 may be implemented as one function. That is, the posture estimation and the abnormality determination may be performed simultaneously. Such processing may be realized, for example, by using the data of teacher images with labels indicating whether it is abnormal or not pre - attached when generating the learned model used for posture estimation.

[0063] The control unit 16 may be configured not to perform processing as the posture estimation unit 162. In this case, the abnormality determination unit 163 may be implemented by giving a label indicating whether it is abnormal or not, instead of posture information, to the data of teacher images when learning the above - mentioned learned model used by the posture estimation unit 162. In this case, it is possible to simply determine whether it is abnormal or not without estimating the posture information. The details are as follows.

[0064] The abnormality determination unit 163 may perform abnormality determination by using, for example, a learned model obtained by performing machine learning processing in advance using teacher images. In this case, as data for the teacher images, only images of organisms of the same type as the monitoring target may be used. That is, since it is clear that the monitoring target is included in the target area estimated in the process of the target area estimation unit 161, when abnormality determination is executed as a subsequent process, there is no need to use images of organisms of a type different from the monitoring target. Rather, by using only images of organisms of the same type as the monitoring target, it is possible to improve the accuracy of abnormality determination. As data for the teacher images, information indicating whether the state of the organism photographed in each image is abnormal is given as a label to each image.

[0065] The user notification information may include data of a notification image instead of the notification address information. In this case, the user terminal 30 can display the notification image more smoothly. Further, when the user terminal 30 receives the data of the notification image, it may display the data of the notification image on the screen without receiving a user operation. By being configured in this way, it is possible to notify the user of the abnormality of the monitoring target earlier.

[0066] When the safety confirmation result satisfies a predetermined emergency condition, the notification unit 165 may select a predetermined emergency notification destination instead of the user terminal 30 as the notification destination and notify the emergency notification destination. The predetermined emergency condition may be, for example, that a predetermined keyword indicating a high degree of urgency such as "help" or "die" is included in the character string included in the safety confirmation result.

[0067] The monitoring terminal 10 may include an input device for receiving an operation of the monitoring target. For example, when the monitoring terminal 10 is configured using a terminal device such as a smartphone, such an input device is provided in advance. The monitoring terminal 10 may be configured to temporarily interrupt imaging in the imaging unit 12 in response to receiving a predetermined operation from the monitoring target. By being configured in this way, it is possible to more firmly protect the privacy of the monitoring target.

[0068] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and designs and the like within the scope not departing from the gist of the present invention are also included.

Description of Reference Numerals

[0069] 100…Monitoring system, 10…Monitoring terminal, 20…Control device, 30…User terminal, 40…Network, 90…Facility to be monitored, 11…Communication unit, 12…Imaging unit, 13…Voice input unit, 14…Voice output unit, 15…Display unit, 16…Control unit, 161…Target area estimation unit, 162…Posture estimation unit, 163…Abnormality determination unit, 164…Notification image generation unit, 165…Notification unit, 166…Safety confirmation unit, 167…Data acquisition unit, 17…Storage unit, 21…Communication unit, 22…Storage unit, 23…Control unit, 231…Notification control unit, 232…Data control unit

Claims

1. A target area estimation unit that estimates target area information indicating an area where the monitoring target is located from an image obtained by imaging the monitoring target; A posture estimation unit that estimates posture information indicating the posture of the monitoring target; An abnormality determination unit that determines whether or not an abnormality has occurred in the monitoring target; When it is determined by the abnormality determination unit that an abnormality has occurred, a notification image that is an image of the space where the monitoring target is located, includes a target area image that is an image indicating the target area information, and does not include an image of the monitoring target itself is generated, and the notification image is transmitted to another device; A notification unit; The notification unit is a monitoring terminal that selects, based on an estimation result of the orientation of the monitoring target by the target area estimation unit or the posture estimation unit, an illustration showing the shape of the monitoring target in the area where the monitoring target is located from among a plurality of types prepared in advance based on the estimation result of the orientation and uses it for the notification image.

2. The monitoring terminal according to claim 1, which is installed in a monitoring target facility that is a facility where the monitoring target operates.

3. Further comprising a posture estimation unit that estimates the posture of the monitoring target, The target area estimation unit estimates the area using a learned model obtained by machine learning using a first learning dataset including a teacher image in which the monitoring target is photographed and a teacher image in which the monitoring target is not photographed, The posture estimation unit estimates the posture using a learned model obtained by machine learning using a second learning dataset in which the ratio of the whole of the teacher image in which the monitoring target is not photographed is relatively smaller than that of the first learning dataset. The monitoring terminal according to claim 1 or 2.

4. Further comprising a safety confirmation unit that outputs, from an audio output unit, audio content that prompts a response as to whether the monitoring target is in a safe state, and then generates confirmation result information based on the audio input from the audio input unit, The notification unit also transmits the confirmation result to another device. The monitoring terminal according to any one of claims 1 to 3.

5. A target area estimation step of estimating target area information indicating an area where the monitoring target is located from an image obtained by imaging the monitoring target; A posture estimation step of estimating posture information indicating the posture of the monitoring target; An abnormality determination step of determining whether or not an abnormality has occurred in the monitoring target; When it is determined in the abnormality determination step that an abnormality has occurred, a notification image is generated that includes a target region image, which is an image of the space where the monitoring target is located and shows the target region information, and does not include an image of the monitoring target itself, and the notification image is transmitted to another device; a notification step; having; In the notification step, an illustration showing the shape of the monitoring target in the region where the monitoring target is located is selected from a plurality of types prepared in advance based on the estimation result of the orientation of the monitoring target in the target region estimation step or the posture estimation step, and is used for the notification image based on the estimation result of the orientation. A monitoring method.

Images