Indoor fall detection method, system, device and medium based on computer vision

Abstract

The present invention discloses a computer vision-based indoor fall detection method, system, device, and medium, which solves the problem of how to accurately detect falls and promptly alarm; it belongs to the field of computer vision; it includes a device end deployed indoors: the device end uses the MoveNet algorithm to extract the position and confidence of human key points from low-resolution video frame images to form an effective key point sequence; the semantic segmentation network in the edge end identifies indoor scene objects in the high-resolution video frame images uploaded by the device end, marks the fall risk area, and classifies the posture of the key point sequence through the constructed deep learning neural network model. When the posture is classified as lying and the position of the human key point is in the fall risk area, the posture is detected as a fall, the posture is verified as a fall, and an alarm signal is generated. On the basis of ensuring user privacy, the present invention improves the accuracy of fall detection, reduces the incidence of false alarms and missed alarms, and buys more time for treatment after a fall.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to a computer vision-based indoor fall detection method, system, device and medium. Background Art

[0002] Timely detection of patient falls is the key to timely subsequent treatment. One of the main methods is fall detection based on computer vision.

[0003] In the prior art, different visual sensors are used to obtain images, and the human features in the images are classified or assembled into sequence data and then posture classification is performed to obtain the patient's fall detection results. For example: Application No. 2021107755248 discloses a fall detection method, system, and terminal based on DVS data; the solution uses an event camera to obtain dynamic visual images; and classification (LSTM network) is performed on the temporal actions of human feature points (obtained through PoseNet). Application No. 2022115626703 discloses a method for identifying people who have fallen down in remote monitoring; the solution uses a pedestrian detection feature frame to make a rough judgment on whether a person has fallen, and controls the camera focal length to zoom in to obtain a large image, and then uses a human key point detection model to obtain human key points, and judges whether a person has fallen based on experience.

[0004] The solution with application number 2021107755248 uses an event camera to acquire images. The problems are as follows: 1. Low accuracy, and the overlap between falling posture and lying posture is not fully considered, which easily leads to false alarms; 2. High cost, DVS cameras are many times more expensive than ordinary RGB cameras, and require expensive computing equipment; 3. Lack of false alarm processing process.

[0005] Application number 2022115626703 has the following issues: 1. It uses a two-stage algorithm to determine falls, and the impact of errors in the first stage is not accounted for in the second stage. That is, if the first stage judgment is wrong, the subsequent algorithm will not work, which can easily lead to missed detections. 2. It does not fully consider the overlap between falling postures and lying postures, which can easily lead to false positives.

[0006] When performing fall detection on a user, the existing technology needs to send all videos of the user at all times to a unified algorithm server for processing, which leads to the problem of user privacy leakage. Due to the problem of recognition accuracy, the algorithm server is prone to false alarms and missed alarms in actual use, and there is no subsequent fall confirmation work. After missed alarms and false alarms occur, the service personnel do not check the real alarm, resulting in the service personnel ignoring the real alarm when encountering it later, which easily misses the best time for treatment and has low accuracy of fall detection. Summary of the Invention

[0007] In response to the problem of how to accurately detect falls and issue alarms in a timely manner, the present invention proposes a computer vision-based indoor fall detection method, system, equipment and medium. This technical solution combines the position of key points of the human body and the position of the fall, adopts collaborative calculation between the device end and the edge end, uses a two-stage algorithm, and adds an automatic alarm review process. The impact caused by the error in the first part of the algorithm can be accommodated in the second stage algorithm, and will not cause missed judgments and misjudgments; when the posture detection is a fall, the review process only needs to upload high-resolution video frame images carrying human features at the current moment, without uploading videos at all times. On the basis of ensuring user privacy, the added automatic fall confirmation and review process improves the accuracy of fall detection, reduces the incidence of false alarms and missed alarms, and ultimately meets the detection accuracy requirements, thereby gaining more time for treatment after a fall.

[0008] The purpose of the present invention is specifically achieved through the following technical solutions:

[0009] The present invention discloses a computer vision-based indoor fall detection method, including a device end deployed indoors and an edge end. The method includes:

[0010] The device periodically captures low-resolution video frames containing human features and high-resolution video frames containing only indoor scene objects. The MoveNet algorithm is used to extract the positions and confidence scores of key points of the human body from the low-resolution video frames to form a valid key point sequence. The key point sequence and high-resolution video frames containing only indoor scene objects are then uploaded to the edge.

[0011] The semantic segmentation network in the edge is used to identify indoor scene objects in the high-resolution video frame images uploaded by the device side, and the fall-hazardous areas in the identified indoor scene objects are marked. The posture of the key point sequence is classified by the constructed deep learning neural network model; when the posture is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is detected as a fall; the control device side is controlled to recapture and upload the high-resolution video frame images carrying human features at the current moment, and the position and confidence of the human body key points are extracted from the recaptured high-resolution video frame images through the pp-tinypose algorithm to form a valid key point sequence, and the posture of the key point sequence is classified by the constructed deep learning neural network model. When the posture in the recaptured high-resolution video frame images is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is reviewed as a fall, and an alarm signal is generated.

[0012] Furthermore, the method of extracting the positions and confidences of key points of the human body from low-resolution video frame images using the MoveNet algorithm to form a key point sequence includes:

[0013] Take the low-resolution video frame image carrying human features captured by the camera in the device end as an input parameter, and use the MoveNet algorithm to output the position coordinates of human key points in the image and the confidence of the detected human key points, forming a key point sequence [(x, y, confidence)].

[0014] Among them, (x, y) are the position coordinates of human key points in the image; confidence is the confidence of the detected human key points.

[0015] Further, the steps for judging the validity of the key point sequence [(x, y, confidence)] include:

[0016] a. Calculate the distance between each point in the key point sequence and find the average distance avg. Judge whether the average distance avg satisfies the preset rule AVGlow < avg < AVGlarge, where AVGlow is the minimum average distance and AVGlarge is the maximum average distance. If the preset rule is satisfied, judge that the key point sequence is valid;

[0017] b. In the video frame image, make an average distance judgment on the point coordinates in the key point sequence corresponding to the previous frame image. If the distance between two frames is greater than the preset maximum average distance, judge that the key point sequence is invalid.

[0018] Further, after the key point sequence is judged to be valid, it also includes the step of classifying the validity of the confidence of human key points. The judgment method is:

[0019] Construct a binary classification algorithm based on SVM;

[0020] Take the confidence configdence of human key points in the key point sequence [(x, y, confidence)] as an input parameter, and output the classification result of the confidence validity of human key points through the constructed binary classification algorithm.

[0021] Further, after the key point sequence and the confidence of human key points are judged to be valid, the method for classifying the pose of the key point sequence through the constructed deep learning neural network model includes:

[0022] The edge end inputs the key point sequence as a parameter into the constructed deep learning neural network model for pose classification, outputs the pose classification result, and inputs the pose classification result into the frame smoothing queue for judging the delay of detection data, and judges whether the pose classification is lying through the frame smoothing queue.

[0023] Further, the method for judging whether the pose classification is lying through the frame smoothing queue includes:

[0024] A. Determine the percentage of people in the current queue who are classified as lying down. If the percentage exceeds one-third, proceed to step B.

[0025] B. Get the position of the lying state data in the current queue and calculate the standard deviation of these positions. If the standard deviation is less than the given threshold, execute step C;

[0026] C. Determine that two-thirds of the current lying data is at the tail end of the queue and classify the posture as lying.

[0027] Furthermore, the method for detecting that the posture is classified as lying and the position of the key points of the human body is in the fall risk area includes:

[0028] If the posture is classified as lying, the position coordinates of the key points of the human body in the key point sequence are taken out in sequence. If the taken position coordinates are in the fall danger zone, the detection result is set to 1, otherwise it is set to 0;

[0029] Add up the detection results corresponding to all position coordinate values. If the sum is greater than the preset value, the posture detection is a fall.

[0030] Furthermore, when the device is deployed indoors, the control steps on the APP side are also included, including:

[0031] Step 1: Use the app to control the device deployed indoors to access the network, and upload the high-resolution video frame images captured by the device, which only contain indoor scene objects, to the edge. Use the semantic segmentation network on the edge to identify indoor scene objects in the high-resolution video frame images uploaded by the device, and mark the fall-hazardous areas in the identified indoor scene objects. The app polls the edge to obtain and display the high-resolution video frame images and the fall-hazardous areas highlighted on the images.

[0032] Step 2: If the fall danger zone displayed by the APP in step 1 is smaller than the actual fall danger zone indoors, the APP calls the edge interface and the edge sends a recapture instruction to the device, repeating step 1 until the actual fall danger zone indoors is completely covered by the fall danger zone displayed by the APP.

[0033] Furthermore, the edge receives the recapture instruction sent by the APP end, sends it to the device end to recapture the high-resolution video frame image that only carries indoor scene objects, and uses the semantic segmentation network in the edge end to identify the indoor scene objects in the high-resolution video frame image uploaded by the device end. If human features are recognized, a failure prompt instruction is sent to the APP end, and the APP end displays a failure prompt.

[0034] Furthermore, when the device captures a video frame image, if the light-sensitive module on the device detects insufficient light, it controls the infrared light switch to provide fill light and submits the infrared light switch event to the edge for storage.

[0035] Furthermore, after using the semantic segmentation network in the edge to identify indoor scene objects in the high-resolution video frame image, it also includes: processing the discontinuity image caused by light spots and reflections through the flooding algorithm, and scaling the identified high-resolution video frame image to a low-resolution space; wherein the semantic segmentation network is DeepLabV3Plus.

[0036] Furthermore, after the alarm signal is generated, the edge terminal further includes a step of displaying an image of the person whose posture is detected as a fall in real time on a corresponding web page.

[0037] The present invention also provides an indoor fall detection system based on computer vision, including a device end deployed indoors and an edge end; wherein,

[0038] The device side is used to periodically capture low-resolution video frames containing human features and high-resolution video frames containing only indoor scene objects. The MoveNet algorithm is used to extract the positions and confidence scores of key points of the human body from the low-resolution video frames to form a valid key point sequence. The key point sequence and high-resolution video frames containing only indoor scene objects are then uploaded to the edge side.

[0039] The edge is used to use a semantic segmentation network to identify indoor scene objects in the high-resolution video frame images uploaded by the device side, and mark the fall-hazardous areas in the identified indoor scene objects, and perform posture classification on the key point sequence through the constructed deep learning neural network model; when the posture is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is detected as a fall; the control device side recaptures and uploads high-resolution video frame images carrying human features, and uses the pp-tinypose algorithm to extract the position and confidence of the human body key points from the recaptured high-resolution video frame images to form a valid key point sequence, and performs posture classification on the key point sequence through the constructed deep learning neural network model. When it is judged that the posture in the recaptured high-resolution video frame image is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is reviewed as a fall, and an alarm signal is generated.

[0040] Furthermore, it also includes an APP end, and the APP end includes a control module and a retake module;

[0041] The control module is used to control the network access of devices deployed indoors through the app, and upload high-resolution video frames captured by the device that only carry indoor scene objects to the edge. The semantic segmentation network in the edge is used to identify indoor scene objects in the high-resolution video frames uploaded by the device, and mark the fall-hazardous areas in the identified indoor scene objects. The app polls the edge to obtain and display the high-resolution video frames and the fall-hazardous areas highlighted in the images.

[0042] The retake module is used to call the edge interface through the APP to send a recapture instruction to the device if the fall hazard area displayed on the APP is smaller than the actual fall hazard area indoors. The control module is called repeatedly until the actual fall hazard area in the room is completely covered by the fall hazard area displayed on the APP.

[0043] The present invention also provides an indoor fall detection device based on computer vision, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium storing a computer program, which implements the steps of the method when executed by a processor.

[0045] The beneficial effects of the present invention are:

[0046] The present invention provides an indoor fall detection method, system, device and medium based on computer vision. To address the problem of how to accurately detect falls and issue alarms in a timely manner, this technical solution combines the positions of key points of the human body and the position of the fall, adopts collaborative calculation between the device end and the edge end, uses a two-stage algorithm, and adds an automatic alarm review process. The impact caused by the error in the first part of the algorithm can be accommodated in the second stage algorithm, and will not cause missed judgments and misjudgments; when the posture detection is a fall, the review process only needs to upload high-resolution video frame images carrying human features at the current moment, without uploading videos at all times. On the basis of ensuring user privacy, the added automatic fall confirmation and review process improves the accuracy of fall detection, reduces the incidence of false alarms and missed alarms, and ultimately meets the detection accuracy requirements, thereby gaining more time for treatment after a fall. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0048] Figure 1 This is a schematic diagram of an indoor fall detection method based on computer vision.

[0049] Figure 2 It is a schematic diagram of a deep learning neural network model. DETAILED DESCRIPTION

[0050] Example 1

[0051] like Figure 1 As shown, the first embodiment of the present invention provides an indoor fall detection method based on computer vision, including a device end deployed indoors and an edge end. The method includes:

[0052] The device periodically captures low-resolution video frames containing human features and high-resolution video frames containing only indoor scene objects. The MoveNet algorithm is used to extract the positions and confidence scores of key points of the human body from the low-resolution video frames to form a valid key point sequence. The key point sequence and high-resolution video frames containing only indoor scene objects are then uploaded to the edge.

[0053] The device side includes camera, movenet, network configuration module, infrared light and photosensitive module;

[0054] Cameras are used to capture high and low resolution video;

[0055] MoveNet is an open-source model released by Google that can detect human poses. It builds on MobileNet V2, a bottom-up human pose detection model with different head types, and provides a corresponding application programming interface (API). The model can run at 50+ fps on a laptop, tablet, or phone. MoveNet can quickly and accurately detect up to 17 key skeletal joints in the human body, including the wrist, shoulder, hip, knee, ankle, and other joints. The Movenet algorithm network model input is a 192*192*3 image and outputs a 17*3 two-dimensional array (17 key points of the human body, each point consists of three pieces of information: x-axis (horizontal axis) bias, y-axis (vertical axis) bias, and confidence. Movenet provides two variants on TF Hub, Lightning and Thunder. Lightning is used for latency-critical applications, while Thunder is used for applications requiring high accuracy. On most modern desktops, laptops, and mobile phones, both models run faster than real-time (30+ FPS), which is crucial for real-time fitness, health, and wellness applications. In this embodiment of the present invention, the physical module that runs the Movenet algorithm is ESP32S3.

[0056] Network configuration module, used to set up Wi-Fi, Bluetooth connection, etc.;

[0057] The infrared light and photosensor module is used to control the infrared light switch to provide fill light when detecting insufficient light when capturing video frame images on the device side, and submit the infrared light switch event to the edge side for storage.

[0058] The semantic segmentation network in the edge is used to identify indoor scene objects in the high-resolution video frame images uploaded by the device side, and the fall-hazardous areas in the identified indoor scene objects are marked. The posture of the key point sequence is classified by the constructed deep learning neural network model; when the posture is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is detected as a fall; the control device side is controlled to recapture and upload the high-resolution video frame images carrying human features at the current moment, and the position and confidence of the human body key points are extracted from the recaptured high-resolution video frame images through the pp-tinypose algorithm to form a valid key point sequence, and the posture of the key point sequence is classified by the constructed deep learning neural network model. When the posture in the recaptured high-resolution video frame images is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is reviewed as a fall, and an alarm signal is generated.

[0059] Among them, the edge is generally deployed in the server within the organization.

[0060] Use the semantic segmentation network on the edge to identify indoor scene objects in the high-resolution video frame images uploaded by the device: that is, use the semantic segmentation network on the high-resolution video frame images to identify objects in the home scene, and identify which ones are beds, sofas, chairs, floors, carpets, etc. Zoom to the low-resolution space and record the spatial information; the semantic segmentation network used in this solution is DeepLabV3Plus, for example: collect more than 3,000 indoor environment photos, and semantically segment and annotate the photos, train the DeepLabV3Plus network, and obtain the network weights. After adjustment, the input of the model is 640*640*3 (size 640*640, 3-channel picture), and the output is 640*640*1 picture (single-channel picture, the value of each pixel in the picture (value range: 0-255), and the predicted indoor value of the pixel Classify interior scene objects. For example, 1 represents the floor. If a value of 1 appears in the image, it indicates that the point is part of the floor. After using the semantic segmentation network in the edge to identify indoor scene objects in high-resolution video frames, the process also includes using a flooding algorithm to address image discontinuities caused by light spots and reflections. Finally, the output 640*640*1 image is scaled to 192*192*1, converting the identified high-resolution video frame image into a low-resolution space.

[0061] The method for the device side to extract the positions and confidences of human key points from a low-resolution video frame image through the MoveNet algorithm to form a key point sequence includes:

[0062] Taking the low-resolution video frame image carrying human features captured by the camera in the device side as an input parameter, and outputting the position coordinates of the human key points in the image and the confidences of the detected human key points through the MoveNet algorithm to form a key point sequence [(x, y, confidence)];

[0063] Among them, (x, y) are the position coordinates of the human key points in the image; confidence is the confidence of the detected human key points.

[0064] Furthermore, the steps for judging the validity of the key point sequence [(x, y, confidence)] include:

[0065] a. Calculate the distances between each point in the key point sequence and find the average distance avg, and judge whether the average distance avg satisfies the preset rule AVGlow < avg < AVGlarge, where AVGlow is the minimum average distance and AVGlarge is the maximum average distance. If the preset rule is satisfied, judge that the key point sequence is valid;

[0066] b. In the video frame image, make an average distance judgment on the point coordinates in the key point sequence corresponding to the previous frame image. If the distance between two frames is greater than the preset maximum average distance, judge that the key point sequence is invalid.

[0067] Furthermore, after the key point sequence is judged to be valid, it also includes the step of classifying the validity of the confidences of the human key points. The judging method is:

[0068] Construct a binary classification algorithm based on SVM;

[0069] Taking the confidence configdence of the human key points in the key point sequence [(x, y, confidence)] as an input parameter, and outputting the classification result of the confidence validity of the human key points through the constructed binary classification algorithm. This classification algorithm actually outputs two values 0 and 1, where 1 represents valid and 0 represents invalid.

[0070] Furthermore, after the key point sequence and the confidences of the human key points are judged to be valid, the method for classifying the pose of the key point sequence through the constructed deep learning neural network model includes:

[0071] The edge inputs the key point sequence as a parameter into the constructed deep learning neural network model for posture classification, outputs the posture classification result, and inputs the posture classification result into the frame smoothing queue used for delay judgment of the detection data. The frame smoothing queue is used to determine whether the posture classification is lying.

[0072] For example: Figure 2 As shown, by building a deep learning neural network model (with 1*51 input and 1*3 output), the key positions and scores of people in a key point sequence are classified as standing, sitting, or lying. For example, at a certain moment, a 192*192 RGB image is sampled from the camera and fed into the MoveNet algorithm as a parameter. The output is a 17*3 two-dimensional array (17 key points of the human body, each consisting of three pieces of information: the x-axis (horizontal axis) offset, the y-axis (vertical axis) offset, and the confidence score). This data is then sent over the network to the edge. The edge feeds the 17*3 data as a parameter into the deep learning neural network model, and the output is a 4*1 array. The position with the highest score in this array corresponds to the standing, sitting, or lying classification (0: standing, 1: sitting, 2: lying, 3: no one).

[0073] Furthermore, the method of determining whether the posture classification is lying down through the frame smoothing queue includes:

[0074] A. Determine the percentage of people in the current queue who are classified as lying down. If the percentage exceeds one-third, proceed to step B.

[0075] B. Get the position of the lying state data in the current queue and calculate the standard deviation of these positions. If the standard deviation is less than the given threshold, execute step C;

[0076] C. Determine that two-thirds of the current lying data is at the tail end of the queue and classify the posture as lying.

[0077] Furthermore, the method for detecting that the posture is classified as lying and the position of the key points of the human body is in the fall risk area includes:

[0078] If the posture is classified as lying, the position coordinates of the key points of the human body in the key point sequence are taken out in sequence. If the taken position coordinates are in the fall danger zone, the detection result is set to 1, otherwise it is set to 0;

[0079] Add up the detection results corresponding to all position coordinate values. If the sum is greater than the preset value, the posture detection is a fall.

[0080] For example, if the posture is classified as lying, the x and y values corresponding to the key points on the body are taken from the semantic segmentation result image (scaled image: 192*192*1) output by the semantic segmentation network. If the value is within the fall risk zone, the result for this point is set to 1, otherwise it is set to 0. The results of all points are added together. If the sum is greater than the preset value of 9, it is considered that the patient is lying in the fall risk zone, that is, a fall has occurred. If a fall has occurred, the posture review continues.

[0081] Posture Verification: For example, a signal is first sent to obtain a high-resolution (256*256) image containing human features captured by the device at the current moment. The human keypoint acquisition algorithm is then run (the difference from the MoveNet algorithm lies in the different input resolution; the human keypoint acquisition algorithm has a higher resolution. This algorithm, pp-tinypose, is an open-source human keypoint acquisition algorithm developed by Baidu and is a top-down algorithm. The input is a 256*256*3 image and the output is 17*3 point data). This yields a high-precision, valid keypoint sequence [(x, y, confidence)]. For validation, refer to the previous section and will not be repeated here. The x and y data of the keypoint sequence are scaled to a 192*192 image size. The keypoint sequence is then subjected to posture classification (192*192) using a constructed deep learning neural network model. If the posture in the recaptured high-resolution video frame is classified as lying down and the human keypoints are located within the fall risk zone, the posture detection is verified as a fall, generating an alarm signal.

[0082] After the alarm signal is generated, the edge also includes the step of displaying the image of the person who has been verified as falling in real time on the corresponding web page. And the device management step mainly manages the camera on the device side and provides the corresponding interface.

[0083] Furthermore, when the device is deployed indoors, the control steps on the APP side are also included, including:

[0084] Step 1: Use the app to control the device deployed indoors to access the network, and upload the high-resolution video frame images captured by the device, which only contain indoor scene objects, to the edge. Use the semantic segmentation network on the edge to identify indoor scene objects in the high-resolution video frame images uploaded by the device, and mark the fall-hazardous areas in the identified indoor scene objects. The app polls the edge to obtain and display the high-resolution video frame images and the fall-hazardous areas highlighted on the images.

[0085] Step 2: If the fall danger zone displayed by the APP in step 1 is smaller than the actual fall danger zone indoors, the APP calls the edge interface and the edge sends a recapture instruction to the device, repeating step 1 until the actual fall danger zone indoors is completely covered by the fall danger zone displayed by the APP.

[0086] Furthermore, the edge receives the recapture instruction sent by the APP end, sends it to the device end to recapture the high-resolution video frame image that only carries indoor scene objects, and uses the semantic segmentation network in the edge end to identify the indoor scene objects in the high-resolution video frame image uploaded by the device end. If human features are recognized, a failure prompt instruction is sent to the APP end, and the APP end displays a failure prompt.

[0087] The present invention provides a specific application example to facilitate those skilled in the art to understand the main workflow of the solution.

[0088] (1) When the device is installed, the APP controls the device to access the network, takes high-resolution video frame images, and uploads them to the edge. The semantic segmentation network in the edge is used to identify indoor scene objects in the high-resolution video frame images uploaded by the device, and marks the fall-hazardous areas in the identified indoor scene objects; the APP polls the edge to obtain and display the high-resolution video frame images and the fall-hazardous areas highlighted on the images; (When displaying, first the 640*640*1 image output by the semantic segmentation network is formed into an RGBA image according to different categories and colors, and the image is superimposed on the original image); the installer checks the superimposed image. If he is not satisfied with the segmentation result, the APP chooses to reshoot. The APP adjusts the edge interface and sends the reshoot instruction (mqtt protocol) from the cloud to the camera on the device. The camera reshoots and processes according to the above process.

[0089] (2) When the device is working normally, it takes 2-5 seconds to obtain a low-resolution video frame image (192.*192*3) from the camera, calls the MoveNet algorithm module on the image, and sends the running results to the edge (http protocol). The edge runs the constructed deep learning neural network model on the received data to perform posture classification. If the person is lying down, it continues to analyze the lying position to determine whether he is lying in a fall-hazardous area. If so, the posture review process is started.

[0090] (3) The posture review process first sends the retake to the camera on the device side to obtain a high-resolution video frame image and upload it to the edge side. After receiving it, the edge side runs the pp-tinypose algorithm to calculate the valid key point sequence, and then runs the constructed deep learning neural network model to combine it with the fall risk area for judgment. If the posture review is a fall, an alarm signal is generated.

[0091] Among them, the main process on the device side

[0092] When running embedded systems on the device, the main steps are:

[0093] (1) The device connects to the network by configuring Wi-Fi access through the APP;

[0094] (2) Receive messages (preferably through the MQTT protocol, determined by the hardware design plan), collect high-resolution video frames, and submit them to the edge through HTTP;

[0095] (3) Regularly collect high-resolution video frame images and submit them to the edge via HTTP;

[0096] (4) Collect low-resolution video frame images and call the MoveNet algorithm module. The MoveNet algorithm will output 17*3 point data of posture data, which is then uploaded to the edge through the HTTP protocol;

[0097] (5) The photosensor controls the infrared light switch and submits the infrared light switch event to the edge.

[0098] Main processes at the edge

[0099] (1) Camera binding and management on the device side, offline management;

[0100] (2) Periodically send commands to the camera on the device, receive the high-resolution video frame images uploaded by the camera, call the semantic segmentation network, output a grayscale image, and save the grayscale image. If a person is detected, the acquisition result is discarded and the next acquisition is waited for.

[0101] (3) Receive the re-collection instruction sent by the APP, send it to the camera on the device to collect data, and run the field semantic segmentation network. If a person is detected, the APP will be prompted and the APP will display an unsuccessful result.

[0102] (4) Receive the key point sequence from the camera on the device side, run the constructed deep learning neural network model, and output four numbers (stored in the database). If the number indicates lying down, expand the posture data to the high-resolution video frame image size to determine whether most of the key point data are in the fall risk area, such as on the floor; if so, start the posture review process; if the posture review indicates a fall, generate an alarm signal;

[0103] (5) Receive the infrared light switch event submitted by the device and save it;

[0104] (6) Instructions are sent regularly to the camera on the device to upload high-resolution video frame images, and the received high-resolution video frame images are run through the pp-tinypose algorithm. The output results are combined with the semantic segmentation network to determine whether a person has fallen.

[0105] (7) Display the page in real time, pull high-resolution video frame images, pull high-resolution video frame images that only carry indoor scene objects, poll and pull key point sequences, and display them on the corresponding web page.

[0106] Main process of APP

[0107] (1) Camera network management and binding on the device side;

[0108] (2) Click to start scene perception, and obtain and display the high-resolution video frame image sent by the edge and the fall risk area highlighted on the image from the edge polling interface: provide display indexes, such as showing the bed and showing the floor, and the corresponding area is highlighted on the image;

[0109] (3) If the perceived area is not appropriate (if the overlay image does not mark all dangerous areas, it can be re-photographed), re-photographing can be performed: re-photographing requires adjusting the edge interface, and then polling to obtain high-resolution video frame images, which is consistent with (2);

[0110] (4) (Optional function) Display near real-time dynamics and call the edge page for direct display.

[0111] The specific application example of the present invention addresses the problem of being unable to determine what kind of fall will trigger an alarm. It proposes that lying on safe items such as beds, sofas, and benches is not considered a fall, but falling on dangerous areas such as the floor and carpet is considered a fall.

[0112] Embodiment 1 of the present invention provides a deep learning neural network model that can cover scenarios where there are problems with key point data of human bodies in complex scenes. The present invention uses a deep learning neural network model to collect many photos of different postures and mark the states of standing, sitting and lying. Movenet is used to extract key points of people from these photos. Some of the extracted key point data have problems, such as missing key points due to occlusion and insufficient lighting, inaccurate key points due to backlighting, and errors caused by the similarity between the color of people's clothes and the background color. Therefore, a deep learning algorithm is directly used, and a network model directly learns the data containing abnormal points, and performs end-to-end processing to avoid the problem of incomplete scene matching caused by human selection of features.

[0113] In order to address the situation where false falls are detected regardless of the algorithm used, this solution adds a posture review process to improve the accuracy of fall detection.

[0114] Example 2

[0115] The second embodiment of the present invention provides an indoor fall detection system based on computer vision, including a device end deployed indoors and an edge end; wherein,

[0116] The device side is used to periodically capture low-resolution video frames containing human features and high-resolution video frames containing only indoor scene objects. The MoveNet algorithm is used to extract the positions and confidence scores of key points of the human body from the low-resolution video frames to form a valid key point sequence. The key point sequence and high-resolution video frames containing only indoor scene objects are then uploaded to the edge side.

[0117] The edge is used to use a semantic segmentation network to identify indoor scene objects in the high-resolution video frame images uploaded by the device side, and mark the fall-hazardous areas in the identified indoor scene objects, and perform posture classification on the key point sequence through the constructed deep learning neural network model; when the posture is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is detected as a fall; the control device side recaptures and uploads high-resolution video frame images carrying human features, and uses the pp-tinypose algorithm to extract the position and confidence of the human body key points from the recaptured high-resolution video frame images to form a valid key point sequence, and performs posture classification on the key point sequence through the constructed deep learning neural network model. When it is judged that the posture in the recaptured high-resolution video frame image is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is reviewed as a fall, and an alarm signal is generated.

[0118] It also includes an APP end, which includes a control module and a retake module;

[0119] The control module is used to control the network access of devices deployed indoors through the app, and upload high-resolution video frames captured by the device that only carry indoor scene objects to the edge. The semantic segmentation network in the edge is used to identify indoor scene objects in the high-resolution video frames uploaded by the device, and mark the fall-hazardous areas in the identified indoor scene objects. The app polls the edge to obtain and display the high-resolution video frames and the fall-hazardous areas highlighted in the images.

[0120] The retake module is used to call the edge interface through the APP to send a recapture instruction to the device if the fall hazard area displayed on the APP is smaller than the actual fall hazard area indoors. The control module is called repeatedly until the actual fall hazard area in the room is completely covered by the fall hazard area displayed on the APP.

[0121] Example 3

[0122] A third embodiment of the present invention provides an indoor fall detection device based on computer vision, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the method described in the first embodiment when executing the computer program.

[0123] The third embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0124] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment 1, and will not be repeated here.

[0125] The beneficial effects of the present invention are:

[0126] The present invention provides an indoor fall detection method, system, device and medium based on computer vision. To address the problem of how to accurately detect falls and issue alarms in a timely manner, this technical solution combines the positions of key points of the human body and the position of the fall, adopts collaborative calculation between the device end and the edge end, uses a two-stage algorithm, and adds an automatic alarm review process. The impact caused by the error in the first part of the algorithm can be accommodated in the second stage algorithm, and will not cause missed judgments and misjudgments; when the posture detection is a fall, the review process only needs to upload high-resolution video frame images carrying human features at the current moment, without uploading videos at all times. On the basis of ensuring user privacy, the added automatic fall confirmation and review process improves the accuracy of fall detection, reduces the incidence of false alarms and missed alarms, and ultimately meets the detection accuracy requirements, thereby gaining more time for treatment after a fall.

[0127] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A computer vision-based indoor fall detection method, comprising a device deployed indoors, characterized in that: It also includes an edge terminal, and the method includes: Timely capturing, by the device terminal, a low-resolution video frame image carrying human features and a high-resolution video frame image carrying only indoor scene objects; extracting the positions and confidences of human key points from the low-resolution video frame image through the movenet algorithm to form an effective key point sequence, and uploading the key point sequence and the high-resolution video frame image carrying only indoor scene objects to the edge terminal; Using the semantic segmentation network in the edge terminal to identify the indoor scene objects in the high-resolution video frame image uploaded by the device terminal, marking the fall hazard areas in the identified indoor scene objects, and classifying the pose of the key point sequence through the constructed deep learning neural network model; when the pose classification is lying and the position of the human key points is in the fall hazard area, the pose is detected as a fall; controlling the device terminal to recapture and upload the high-resolution video frame image carrying human features at the current moment, extracting the positions and confidences of human key points from the recaptured high-resolution video frame image through the pp-tinypose algorithm to form an effective key point sequence, and classifying the pose of the key point sequence through the constructed deep learning neural network model. When it is judged that the pose classification in the recaptured high-resolution video frame image is lying and the position of the human key points is in the fall hazard area, the pose is rechecked as a fall, and an alarm signal is generated.

2. The method according to claim 1, wherein The method of extracting the positions and confidences of human key points from the low-resolution video frame image through the movenet algorithm includes: Taking the low-resolution video frame image carrying human features captured by the camera in the device terminal as an input parameter, and outputting the position coordinates of the human key points in the image and the confidences of the detected human key points through the movenet algorithm to form a key point sequence [(x, y, confidence)]; where, (x, y) are the position coordinates of the human key points in the image; confidence is the confidence of the detected human key points.

3. The method according to claim 1 or 2, wherein: The steps of judging the effectiveness of the key point sequence [(x, y, confidence)] include: a. Calculating the distances between each point in the key point sequence and obtaining the average distance avg, and judging whether the average distance avg satisfies the preset rule AVGlow < avg < AVGlarge, where AVGlow is the minimum average distance and AVGlarge is the maximum average distance. If the preset rule is satisfied, it is judged that the key point sequence is effective; b. In the video frame image, making an average distance judgment on the point coordinates in the key point sequence corresponding to the previous frame image. If the distance between two frames is greater than the preset maximum average distance, it is judged that the key point sequence is invalid.

4. The method according to claim 3, wherein After the key point sequence is judged to be effective, it also includes the step of classifying the effectiveness of the confidences of the human key points. The judging method is: Constructing a binary classification algorithm based on SVM; The confidence configdence of the human key points in the key point sequence [(x, y, confidence)] is used as the input parameter, and the confidence validity classification results of the human key points are output through the constructed binary classification algorithm.

5. The method according to claim 1 or 4, wherein: After the confidence of the key point sequence and the human body key point is judged to be valid, the method of performing posture classification on the key point sequence by using the constructed deep learning neural network model includes: The edge inputs the key point sequence as a parameter into the constructed deep learning neural network model for posture classification, outputs the posture classification result, and inputs the posture classification result into the frame smoothing queue used for delay judgment of the detection data. The frame smoothing queue is used to determine whether the posture classification is lying.

6. The method according to claim 5, wherein The method for determining whether the posture classification is lying down through the frame smoothing queue includes: A. Determine the percentage of people in the current queue who are classified as lying down. If the percentage exceeds one-third, proceed to step B. B. Get the position of the lying state data in the current queue and calculate the standard deviation of these positions. If the standard deviation is less than the given threshold, execute step C; C. Determine that two-thirds of the current lying data is at the tail end of the queue and classify the posture as lying.

7. The method according to claim 6, wherein The detection method for classifying the posture as lying and the position of the key points of the human body in the fall risk area includes: If the posture is classified as lying, the position coordinates of the key points of the human body in the key point sequence are taken out in sequence. If the taken position coordinates are in the fall danger zone, the detection result is set to 1, otherwise it is set to 0; Add up the detection results corresponding to all position coordinate values. If the sum is greater than the preset value, the posture detection is a fall.

8. The method according to claim 1, wherein When the device is deployed indoors, the app also includes control steps, including: Step 1: Use the app to control the device deployed indoors to access the network, and upload the high-resolution video frame images captured by the device, which only contain indoor scene objects, to the edge. Use the semantic segmentation network on the edge to identify indoor scene objects in the high-resolution video frame images uploaded by the device, and mark the fall-hazardous areas in the identified indoor scene objects. The app polls the edge to obtain and display the high-resolution video frame images and the fall-hazardous areas highlighted on the images. Step 2: If the fall danger zone displayed by the APP in step 1 is smaller than the actual fall danger zone indoors, the APP calls the edge interface and the edge sends a recapture instruction to the device, repeating step 1 until the actual fall danger zone indoors is completely covered by the fall danger zone displayed by the APP.

9. The method according to claim 8, wherein The edge receives the recapture instruction sent by the APP end, sends it to the device end to recapture the high-resolution video frame image that only carries indoor scene objects, and uses the semantic segmentation network in the edge end to identify the indoor scene objects in the high-resolution video frame image uploaded by the device end. If human features are recognized, a failure prompt instruction is sent to the APP end, and the APP end displays a failure prompt.

10. The method according to claim 1, wherein When the device captures a video frame image, the light-sensitive module on the device detects insufficient light, controls the infrared light switch to provide fill light, and submits the infrared light switch event to the edge for storage.

11. The method according to claim 1, wherein After using the semantic segmentation network in the edge to identify indoor scene objects in the high-resolution video frame image, it also includes: processing the discontinuous images caused by light spots and reflections through the flooding algorithm, and scaling the identified high-resolution video frame image to a low-resolution space; wherein the semantic segmentation network is DeepLabV3Plus.

12. The method according to claim 1, wherein After the alarm signal is generated, the edge also includes a step of displaying the image of the posture detected as a fall in real time on the corresponding web page.

13. A computer vision-based indoor fall detection system, comprising a device deployed indoors, characterized in that: Also includes the side ends; in, The device side is used to periodically capture low-resolution video frames containing human features and high-resolution video frames containing only indoor scene objects. The MoveNet algorithm is used to extract the positions and confidence scores of key points of the human body from the low-resolution video frames to form a valid key point sequence. The key point sequence and high-resolution video frames containing only indoor scene objects are then uploaded to the edge side. The edge is used to use a semantic segmentation network to identify indoor scene objects in the high-resolution video frame images uploaded by the device side, and mark the fall-hazardous areas in the identified indoor scene objects, and perform posture classification on the key point sequence through the constructed deep learning neural network model; when the posture is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is detected as a fall; the control device side recaptures and uploads high-resolution video frame images carrying human features, and uses the pp-tinypose algorithm to extract the position and confidence of the human body key points from the recaptured high-resolution video frame images to form a valid key point sequence, and performs posture classification on the key point sequence through the constructed deep learning neural network model. When it is judged that the posture in the recaptured high-resolution video frame image is classified as lying and the position of the human body key points is in the fall-hazardous area, the posture is reviewed as a fall, and an alarm signal is generated.

14. The system according to claim 13, wherein: It also includes an APP end, which includes a control module and a retake module; The control module is used to control the network access of devices deployed indoors through the app, and upload high-resolution video frames captured by the devices that only carry indoor scene objects to the edge. The semantic segmentation network on the edge is used to identify indoor scene objects in the high-resolution video frames uploaded by the devices, and mark the fall-hazard areas in the identified indoor scene objects. The app polls the edge to obtain and display high-resolution video frames and highlighted fall-hazard areas. The retake module is used to call the edge interface through the APP to send a recapture instruction to the device if the fall hazard area displayed on the APP is smaller than the actual fall hazard area indoors. The control module is called repeatedly until the actual fall hazard area in the room is completely covered by the fall hazard area displayed on the APP.

15. An indoor fall detection device based on computer vision, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 12 when executed by a processor.

Images