A monitoring system and method for reducing false alarms through gesture response confirmation after anomaly detection.

Abstract

To realize a response confirmation method that can be performed even from a lying position on the floor after a fall. [Solution] A monitoring system and method that, after detecting an abnormal state of the person being monitored, automatically determines whether it is a false alarm or not by recognizing the person's own intentional response gesture, and decides whether or not to issue an emergency call. In a preferred embodiment, RGB images acquired from an imaging unit and point cloud data are combined to recognize the upper limb displacement pattern from the floor position after a fall. It distinguishes between gestures indicating safety confirmation (false alarm cancellation) and gestures indicating an emergency state (immediate call), and functions effectively even for elderly people with hearing loss or motor impairments. It operates network-independently through edge processing within the terminal. [Benefits] No additional sensors are required, it responds to falls, protects privacy, and features edge processing, resulting in a simple and secure monitoring system.

Description

Technical Field

[0001] The present invention relates to a system for monitoring an indoor observed person (such as an elderly person) using a depth sensor such as LiDAR, and particularly relates to a technology for reducing false alarms after abnormality detection through response confirmation by the observed person themself. Background Art

[0002] In conventional monitoring systems, methods are known in which after an abnormality is detected by a sensor, an alert is immediately sent to a family member or caregiver, or confirmation is requested through voice or touch operation.

[0003] However, conventional methods have the following problems. With advancing age, elderly people are prone to hearing loss and movement disorders, and there are cases where they are not accustomed to voice response or touch operation. Methods relying on voice response do not function in situations where the observed person cannot speak after a fall, or for observed persons with hearing impairment. Touch operation becomes impossible when the person cannot reach the terminal due to a fall. In addition, a response confirmation sensor is required separately from the abnormality detection sensor, which complicates the system. Furthermore, gesture recognition using a camera raises concerns about acquisition of personal information such as face and body shape. It should be noted that while the movement of the trunk and lower limbs of the observed person after a fall is restricted, the movement ability of the upper limbs (arms and hands) is often relatively maintained. In addition, since the upper limbs displace greatly at positions away from the body, there is a characteristic that it is easy to distinguish intentional gestures from minute movements caused by accidental body motions such as breathing. Prior Art Documents Patent Documents

[0004] Patent Document 1 Japanese Unexamined Patent Application Publication No. 2003-058973 Patent Document 2 Japanese Unexamined Patent Application Publication No. 2001-236583

[0005] Patent Document 1 discloses a system for confirming an emergency situation through voice responses to voice inquiries, but the response means is limited to voice, and a confirmation function using intentional gestures is not disclosed.

[0006] Patent Document 2 discloses a system that determines the state of a subject by confirming their response to minor stimuli, but it does not disclose a configuration that uses the subject's active, intentional response actions as the basis for determining false positives. [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] The present invention aims to provide a monitoring system that functions effectively even in situations where voice responses or touch operations are difficult, such as for elderly people who are prone to hearing loss or motor impairments, by using upper limb gestures as the primary response method. Furthermore, by using the same sensors used for anomaly detection and enabling gesture response confirmation that can be performed even when the observed person is lying on the floor after a fall, the invention aims to achieve both a reduction in false alarms and reliable emergency notification without the need for additional equipment. [Means for solving the problem]

[0008] A monitoring system according to one aspect of the present invention comprises: anomaly detection means for continuously monitoring the state of a person being observed and detecting abnormal conditions; notification means for outputting a confirmation notice to the person being observed in response to the anomaly detection; gesture recognition means for recognizing an intentional response gesture of the person being observed after the anomaly detection; and determination means for determining whether or not to issue a notification based on the content or presence or absence of the response gesture. The determination means cancels the notification if the recognized response gesture indicates safety, and executes the notification if a gesture actively indicating an emergency is recognized or if no response gesture is recognized within a predetermined time. Upper limb gesture response confirmation is a non-contact method that can be performed even if the person being observed is lying on the floor or not holding a terminal, and is effective even for persons being observed with hearing impairments. Voice response and touch operation are positioned as auxiliary means in situations where confirmation by upper limb gesture is difficult.

[0009] The response gestures in this invention primarily target intentional movements of the upper limbs (arms and hands). In a person observed after a fall, the ability to move the upper limbs is more easily preserved compared to the trunk and lower limbs, and the upper limb region is displaced significantly at a distance from the body, making it easy to distinguish from accidental body movements such as breathing. Due to these characteristics, upper limb gestures are the easiest response means for a person observed after a fall to perform and can clearly distinguish between intentional and accidental movements. In a preferred embodiment, the gesture recognition means analyzes the observed person's body movement pattern from image or video data acquired by an imaging means (visible light camera, infrared camera, etc.) and recognizes intentional response gestures. In another embodiment, the gesture recognition means analyzes the observed person's body movement pattern by combining RGB images and three-dimensional point cloud data from a depth sensor, identifies the upper limb region using the RGB image, and then measures the three-dimensional displacement of that region using the point cloud data to improve recognition accuracy. Yet another embodiment also includes a configuration that uses only a depth sensor.

[0010] When using a depth sensor, a predetermined displacement pattern from the reference point cloud is identified as an intentional response gesture by performing a difference analysis between reference point cloud data acquired immediately after anomaly detection (at the start of response confirmation) and response time point cloud data acquired when the observed person performs a predetermined action. This difference ΔP functions as a displacement feature to distinguish between accidental body movements of the observed person (local fluctuations due to breathing, subtle changes in body position, etc.) and intentional response gestures (active displacement of the upper limbs). Point cloud displacements due to accidental body movements are small in magnitude and converge quickly, while displacements in the upper limb region due to intentional gestures are above a predetermined threshold and continue for a predetermined time, allowing for high-precision identification of the two (see Figures 3 and 5). [Effects of the Invention]

[0011] According to the present invention, response confirmation is achieved that combines three characteristics: (a) it can be performed while lying on the floor, (b) it is a contactless method that does not require holding the terminal, and (c) it does not rely on voice, so it functions effectively even when the observed person is in the most severe condition after a fall. In addition, by confirming the observed person's intentional gestures regardless of the type of sensor, false alarms can be reduced and the necessity of emergency calls can be automatically determined. Furthermore, by using upper limb gestures as the primary response means, it functions effectively even when elderly people who are prone to hearing loss or motor impairments cannot use voice responses or touch operations. Moreover, since gestures can recognize not only safety confirmations but also proactive indications of emergency situations, it can transmit highly accurate emergency information in addition to reducing false alarms. In addition, since it is completed by edge processing within the terminal, it can operate even during network failures and eliminate the risk of data leakage. In embodiments that combine RGB images and depth sensors, the recognition accuracy of the upper limb area is improved, and in embodiments that use only depth sensors, RGB images are not acquired, so privacy can be highly protected. [Modes for carrying out the invention]

[0012] The embodiments described below are merely examples of embodiments of the present invention, and the present invention is not limited to these embodiments, but can be implemented in various ways without departing from its spirit. Embodiments of the present invention will now be described with reference to the drawings. Figure 1 is a diagram showing the overall configuration of the monitoring system according to the present invention, and Figure 2 is a processing flowchart. In each embodiment, the monitoring system includes an imaging unit 100, an anomaly detection unit 110, a notification output unit 120, a response recognition unit 130, a determination unit 140, and a notification unit 150. As shown in Figure 1, these components are connected in a vertical order from the imaging unit 100 to the notification unit 150, and each process is completed as an edge processing within the terminal. The detection of an anomaly by the anomaly detection unit 110 is based on embodiments using various known methods such as point cloud analysis, posture estimation, and dwell time determination, and the present invention is mainly characterized by the response confirmation processing by the subsequent response recognition unit 130. [Examples]

[0013] This embodiment is the broadest possible configuration, regardless of the type of sensor. An imaging unit 100 (imaging means: visible light camera, infrared camera, etc.) is permanently installed in the room and continuously acquires the status of the person being observed. When the abnormality detection unit 110 detects a fall or abnormal posture of the person being observed from the image, the notification output unit 120 outputs a confirmation message such as "Are you okay? Please let us know by voice or gesture" as a screen display and voice guidance. Subsequently, the response recognition unit 130 performs skeletal estimation (pose estimation) of the person being observed through image analysis and recognizes a response gesture by tracking the time-series positional changes of the upper limb joints. Specifically, gesture patterns such as raising the hand (vertical displacement of the upper limb) and drawing shapes such as circles and crosses with both arms are defined in advance, and if the movement trajectory of the upper limb from the acquired image frame sequence matches a defined pattern, it is determined to be a response gesture. In the processing flow shown in Figure 2, continuous acquisition of imaging data is performed as continuous monitoring. If an abnormal condition is detected, a confirmation notification is output, and then the response recognition unit 130 performs a response confirmation (maximum 30 seconds). Depending on the result, the judgment unit 140 selects whether to cancel a false alarm or make an emergency call. The judgment unit 140's primary criterion is safety confirmation by response gestures. If a response gesture indicating safety is detected, the false alarm is canceled (auxiliary confirmations such as voice responses and touch operations are handled similarly). If a response that actively indicates an emergency condition is detected, the notification unit 150 immediately makes an emergency call. If no response is received within the predetermined time (30 seconds), the system also proceeds to make an emergency call via the notification unit 150. The response confirmation time is set at 30 seconds because it is the time required for a conscious observed person after a fall to prepare and perform an upper limb gesture. If no response is received within this time, it is highly likely that the person is in a serious condition such as impaired consciousness. Furthermore, the 2-second time window for gesture detection was set as the minimum duration required to distinguish accidental body movements, such as breathing, from intentional movements, taking into account the upper limit of the time that accidental body movements can continue beyond the displacement threshold. [Examples]

[0014] In this embodiment, the imaging unit 100 acquires RGB images and three-dimensional point cloud data from a depth sensor, and combines them to realize gesture recognition. The contour and skeletal information of the observed person is analyzed from the RGB image to identify the upper limb region (arm and hand), and the displacement (ΔP value) of the point cloud data corresponding to that upper limb region is measured. If a displacement pattern of the upper limb region is detected in which the ΔP value exceeds a predetermined threshold, the gesture recognition unit 130 determines that the action is an intentional response gesture. By combining the identification of the upper limb region using RGB images and precise three-dimensional displacement measurement using point cloud data, intentional upper limb gestures can be recognized with high accuracy even when the observed person is near the floor surface (low Y coordinate in the height direction) after a fall. The difference (ΔP value) between the reference point cloud data acquired immediately after anomaly detection (start of response confirmation) and the response time point cloud data acquired when the observed person subsequently performs an action is analyzed (see Figure 5). In Figure 5, panel (A) shows the reference point cloud P_ref after a fall and before the response, panel (B) shows the point cloud P_gesture during response confirmation (upward displacement of the upper limbs), and panel (C) shows the difference ΔP. If the magnitude of the three-dimensional displacement vector |ΔP| in the upper limb region of the difference ΔP is greater than or equal to a predetermined threshold (e.g., 0.2m) and within a predetermined time (e.g., 2 seconds), it is determined to be an intentional gesture. Note that |ΔP| can evaluate the amount of displacement in the three-dimensional direction regardless of the sensor's installation angle (ceiling installation, diagonal installation, etc.), thus accommodating various installation conditions. Examples of response gestures include raising the upper limbs vertically and drawing shapes such as circles and crosses with both arms. Figure 3 shows the coordinate system for gesture recognition in a floor-standing posture, illustrating how the depth sensor detects the three-dimensional displacement vector |ΔP| when the observed person displaces their upper limbs after a fall. This figure illustrates an example of lifting in the vertical direction (Y-axis direction) for illustrative purposes, but regardless of the sensor's installation angle, an intentional gesture is determined when the three-dimensional displacement vector |ΔP| is greater than or equal to a predetermined threshold (see Figure 3). [Examples]

[0015] In embodiments that use only a depth sensor without RGB images, the upper limb region is estimated using only point cloud data. Specifically, the distribution of points in the point cloud of the human body in the post-fall posture is analyzed in the height direction (Y-axis) from the floor surface, and point cloud clusters distributed above the height range of the body are estimated as the upper limb region. By analyzing the point cloud displacement ΔP of the estimated upper limb region, the response recognition unit 130 recognizes an intentional response gesture. In this embodiment, since RGB images are not acquired, the privacy of the observed person can be highly protected. [Examples]

[0016] In this embodiment, when a LiDAR (Light Detection and Ranging) sensor is used for anomaly detection, such as for grasping three-dimensional spatial information, the point cloud data acquisition function of the sensor can also be used for response recognition (gesture recognition). The LiDAR measures the reflection of laser pulses using the time-of-flight (ToF) method and acquires highly accurate three-dimensional point cloud data. By utilizing the point cloud data from the same sensor in both the anomaly detection means and the response recognition means, additional sensors become unnecessary. The processor of the LiDAR-equipped terminal (e.g., LiDAR-equipped smartphone Pro) analyzes the point cloud data through edge processing. [Examples]

[0017] In this embodiment, upper limb gesture recognition is used as the main response confirmation means, and voice response and touch operation are used in combination as auxiliary means in situations where confirmation by gesture is difficult (such as when upper limb movement is restricted due to injury, paralysis, etc. of the upper limb, or when a gesture motion is not detected by a sensor) (see FIG. 4). The multimodal response flow shown in FIG. 4 shows a configuration example in which after an abnormality is detected, gesture recognition by a depth sensor (response means (1), 0 to 30 seconds) is attempted first, followed by voice response by means of a microphone and voice recognition (response means (2), 30 to 60 seconds), and an operation using a touch panel or the like can also be added. The determination unit 140 executes each response confirmation means such as gesture recognition, voice response, and touch operation in an order and combination according to the embodiment. If safety confirmation is obtained by any one of the means, the false alarm is canceled; if a response actively indicating an emergency state is detected, the notification unit 150 immediately executes an emergency call. If no response is obtained in all response confirmations, the notification unit 150 also executes an emergency notification (notification to family members and emergency services). The execution order of the response confirmation means can be changed according to the embodiment, and is not limited to the order described in this embodiment. The present invention is executed as edge processing on the terminal of the observed person, and is completed without transmitting point cloud data outside the network, so that privacy protection can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] [Figure 1] It is a diagram showing the overall configuration of a monitoring system according to the present invention. [Figure 2] It is a processing flowchart showing each stage from abnormality detection to gesture recognition and determination. [Figure 3] It is a diagram showing the principle of gesture recognition from an on-floor posture, showing a detection method based on a three-dimensional displacement vector |ΔP|. [Figure 4] It is a response flow diagram showing an example of the execution order of multimodal response means. [Figure 5] It is a diagram showing the principle of gesture detection by point cloud difference analysis. DESCRIPTION OF REFERENCE SIGNS

[0019] 100 Imaging unit (imaging and measurement means such as camera and depth sensor) 110 Anomaly detection unit 120 Notification output section 130 Response Recognition Unit (Means for confirming responses such as gestures, voice, and touch) 140 Judgment section 150 Reporting Department

Claims

1. An anomaly detection means that continuously monitors the state of the person being observed and detects abnormal conditions, A notification means that outputs a confirmation notice to the person being observed in response to the detection of the anomaly, A gesture recognition means that recognizes the observed person's intentional response gesture after detecting the anomaly, The system includes a determination means for determining whether notification is necessary based on the content or presence of the response gesture, The gesture recognition means uses three-dimensional point cloud data acquired by a depth sensor that does not acquire RGB images, and based on the difference between reference point cloud data acquired immediately after the anomaly detection and response time point cloud data acquired when the observed person subsequently performs an action, it distinguishes between displacement due to accidental body movement of the observed person and displacement due to an intentional response gesture, based on whether the amount of displacement is above a predetermined threshold and whether the displacement continues for a predetermined time, and recognizes the intentional response gesture. The determination means cancels the notification if the intentional response gesture is recognized within a predetermined time after the output of the confirmation notice, and executes the notification if the intentional response gesture is not recognized within the predetermined time. A monitoring system characterized by the following features.

2. The gesture recognition means is In the post-fall posture in which the subject is positioned near the floor, A predetermined displacement pattern of the upper limb region detected within the field of view of the depth sensor Recognize as a response gesture. The monitoring system according to claim 1.

3. The abnormality detection means and the gesture recognition means, Share point cloud data acquired from the same depth sensor. The monitoring system according to claim 1 or 2.

4. The depth sensor is a LiDAR (Light Detection and Ranging) sensor. The monitoring system according to claim 1 or 2.

5. In addition to confirmation using response gestures, a voice notification means that outputs a voice call, A speech recognition means for recognizing a voice response to the aforementioned voice call, Furthermore, The determination means will stop sending the notification if safety confirmation is obtained through a response gesture or voice response. If a response indicating an emergency is detected, or if no response is received by any means, an alert will be issued. The order in which the confirmation by response gesture and the confirmation by voice are performed can be changed depending on the embodiment. The monitoring system according to claim 1.

6. Further comprising UI display means for receiving touch operation input, The determination means will discontinue sending the notification if safety confirmation is obtained through gesture response, voice response, or touch operation. An alert will be issued if no response is received through any means or if an active indication of urgency is detected. The monitoring system according to claim 5.

7. The gesture recognition means is executed as edge processing on the observed person's terminal, Do not send acquired data outside the network. The monitoring system according to claim 1.

8. A step of continuously monitoring the state of the person being observed and detecting an abnormal state, The steps include: outputting a confirmation notice to the person being observed in response to the detection of the anomaly; After detecting the anomaly, the three-dimensional point cloud data acquired by a depth sensor that does not acquire RGB images is used, and based on the difference between the reference point cloud data acquired immediately after the anomaly detection and the response time point cloud data acquired thereafter, the displacement due to the observed person's accidental body movement and the displacement due to an intentional response gesture are identified based on whether the displacement amount is above a predetermined threshold and whether the displacement continues for a predetermined time, and the intentional response gesture is recognized. The steps include: canceling the notification if the intentional response gesture is recognized within a predetermined time after the output of the confirmation notice, and executing the notification if the intentional response gesture is not recognized within the predetermined time; Monitoring methods including those mentioned.

9. The gesture recognition means recognizes a safety confirmation gesture indicating the safety of the person being observed, Identify emergency signal gestures that actively indicate an emergency situation, If the determination means recognizes the safety confirmation gesture, it will stop sending the notification. When the aforementioned emergency signal gesture is recognized, the notification is immediately sent without waiting for the predetermined time to elapse. The monitoring system according to claim 1.

Images