Fall detection method, apparatus, device, and storage medium

Abstract

The embodiment of the application discloses a fall detection method, device, equipment and storage medium, wherein the method comprises the following steps: acquiring a depth video stream and a color video stream of a photographed object; performing fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively; and performing fall detection analysis on the photographed object according to a skeleton detection result obtained from the depth video stream and a first analysis result obtained from the color video stream.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of Internet of Things, and relate to but are not limited to a fall detection method, device, equipment and storage medium. BACKGROUND

[0002] With the deepening of the aging of society, there are more and more empty-nest elderly. Due to the aging of the body function and the influence of diseases, the elderly are prone to fall, and the fall may threaten the life of the elderly. Therefore, it is very meaningful to reduce the number of misjudgments and omissions of the fall detection method to improve the reliability of the fall detection method, so that the elderly can obtain timely assistance.

[0003] At present, in the solution of fall detection in the field of smart care for the elderly, commonly used devices include cameras and wearable devices, and the devices used in the camera solution based on vision include color cameras and depth cameras. The color camera solution generally uses dynamic human body tracking plus threshold rule algorithm for judgment, but since the color camera has limited data acquisition, the detection success rate is low. The existing solution can supplement the depth information through multiple cameras for judgment, but there is still an error in converting two-dimensional data to three-dimensional data, and it is difficult to implement due to the angle of multiple camera deployment. There are open source projects such as Openpose for obtaining bones in 2D, but due to high performance requirements, it cannot be used for real-time detection on the edge side. If the video data is uploaded to the cloud for detection, there is a problem of user privacy. The method based on deep learning for classification to judge the fall requires high device requirements and is usually not deployed on the edge side. SUMMARY

[0004] Therefore, embodiments of the present application provide a fall detection method, device, equipment and storage medium.

[0005] The technical scheme of the embodiments of the present application is as follows:

[0006] In a first aspect, the embodiments of the present application provide a fall detection method, comprising:

[0007] obtaining a depth video stream and a color video stream of a photographed object;

[0008] respectively using the depth video stream and the color video stream to perform fall detection analysis on the photographed object;

[0009] performing fall detection analysis on the photographed object according to the bone detection result obtained from the depth video stream and the first analysis result obtained from the color video stream.

[0010] In a second aspect, the embodiments of the present application provide a fall detection device, comprising:

[0011] obtain a depth video stream and a color video stream of a photographed object;

[0012] perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively.

[0013] perform fall detection analysis on the photographed object by using a skeleton detection result obtained from the depth video stream and a first analysis result obtained from the color video stream.

[0014] In a third aspect, an embodiment of the present application provides a fall detection device, comprising a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the steps in the fall detection method when executing the program.

[0015] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for causing a processor to execute the fall detection method.

[0016] The embodiments of the present application provide a fall detection method, device, equipment and storage medium. First, a depth video stream and a color video stream of a photographed object are obtained. Then, fall detection analysis is performed on the photographed object by using the depth video stream and the color video stream respectively. Finally, fall detection analysis is performed on the photographed object by using a skeleton detection result obtained from the depth video stream and a first analysis result obtained from the color video stream. In this way, the method of using two video stream detection results for comprehensive detection analysis can effectively improve the accuracy of confirming the fall of the photographed object. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An implementation flowchart of a fall detection method provided by an embodiment of the present application;

[0018] Figure 2 An implementation flowchart of another fall detection method provided by an embodiment of the present application;

[0019] Figure 3A An implementation flowchart of another fall detection method provided by an embodiment of the present application;

[0020] Figure 3B An implementation flowchart of another fall detection method provided by an embodiment of the present application;

[0021] Figure 4A An implementation flowchart of another fall detection method provided by an embodiment of the present application;

[0022] Figure 4B An implementation flowchart of another fall detection method provided by an embodiment of the present application;

[0023] Figure 4C An implementation flowchart of another fall detection method provided for an embodiment of the present application is shown in FIG. 6.

[0024] Figure 5 An assembly structure diagram of a fall detection device provided for an embodiment of the present application is shown in FIG. 7.

[0025] Figure 6 A hardware entity diagram of a fall detection device provided for an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0027] It should be understood that some embodiments described herein are merely used to explain the technical solutions of the present application, and are not used to limit the technical scope of the present application.

[0028] Embodiment one

[0029] The current fall detection means is to use a red green blue (RGB) camera to collect fall information. Although the RGB camera has a low cost, the accuracy of fall detection is low due to the lack of depth information. At present, multiple cameras can be used to supplement depth information for judgment, but there is still a two-dimensional to three-dimensional data error, and the angle problem of multiple camera deployment leads to implementation difficulties. Although the two-dimensional bone acquisition method has good accuracy, it cannot be used for end-side real-time detection due to poor performance. Currently, there are open source projects such as human posture recognition (openpose).

[0030] A binocular depth camera can obtain depth information and extract bone nodes with high accuracy, so most of the current posture behavior prediction projects use a depth camera to extract bone points as a basis for judgment. Through the bone node threshold rule judgment method, the fall can be detected well. However, due to the limitations of the imaging distance and angle of the depth camera, it is impossible to extract bone nodes in the actual detection process, which affects the detection accuracy. In addition, the threshold detection based on rules often has the problems of false detection and missed detection. For example, if the falling speed threshold is used to judge the fall, it may not be able to identify if it falls slowly. The above problems result in that the accuracy of most threshold detection schemes is about 90%. The use of deep learning schemes for detection requires high device performance, and is generally placed in the cloud. However, there are privacy problems when uploading videos to the platform, and real-time monitoring requires a large amount of network bandwidth and traffic. Therefore, most deep schemes are still in the research prototype stage, and there is no commercial product.

[0031] In the embodiment of the present application, a binocular depth camera (not dependent on the brand and model of the camera) is used to process the obtained RGB video and depth video stream simultaneously. When the depth video stream cannot extract the skeleton node and cannot be judged, the RGB video stream is processed to complete the judgment and solve the problem of extracting the skeleton of the depth camera. The scheme uses an asynchronous processing method to solve the problem of the performance of the depth learning model on the end side, which can ensure that the data is not uploaded to the cloud and thus protects the privacy of the user, and improves the detection accuracy (verified using the URfalling fall dataset, the fusion detection fall rate is up to 100%, and the detection result of a single depth camera is about 90%). This provides a basis for commercialization.

[0032] The embodiment provides a fall detection method. Figure 1 The implementation flowchart of the fall detection method provided by the embodiment of the present application is shown in Figure 1 The method comprises the following steps.

[0033] In step S101, a depth video stream and a color video stream of a photographed object are obtained.

[0034] A collection device capable of simultaneously collecting a depth video stream and a color video stream is used to simultaneously collect the two video streams of the photographed object. For example, a red green blue depth (RGBD) camera is used. The RGBD camera is a camera that adds a depth measurement function to an RGB ordinary camera. From the technical level of implementing this function, there are the following three mainstream schemes: binocular, structured light, and time of flight (TOF). The ranging method of the RGBD binocular camera is passive ranging, RGBD binocular matching, and triangulation. The resolution of the RGBD binocular camera is medium to high, and the precision is 0.01 mm to 1 mm. The display frame rate is relatively low, about several tens of hertz (Hz). The light resistance is strong, and the power consumption is relatively low. The software is relatively complex, the hardware cost is low, and the outdoor applicability is good, which basically does not affect the use effect. The advantages of using a binocular RGBD camera are low cost, low power consumption, high resolution, and high frame rate.

[0035] The depth video stream and the color video stream of the photographed object are obtained, which can be used for asynchronous detection and analysis of whether the photographed object falls.

[0036] In step S102, the depth video stream and the color video stream are used to perform fall detection analysis on the photographed object, respectively.

[0037] If it is not determined whether the photographed object falls according to the depth video stream or it is determined that the photographed object does not fall, asynchronous detection analysis needs to be performed on the simultaneously collected color video stream to reconfirm whether the photographed object falls to ensure that no falling action is missed.

[0038] Likewise, if it is not determined whether the photographed object falls according to the color video stream or it is determined that the photographed object does not fall, asynchronous detection analysis needs to be performed on the simultaneously collected depth video stream to reconfirm whether the photographed object falls to ensure that no falling action is missed.

[0039] The reconfirmation of whether the photographed object falls needs to use a deep learning method, and since the deep learning method has low detection efficiency and general timeliness, real-time monitoring needs to occupy a large amount of traffic, so the application uses asynchronous detection analysis, uses one of the depth video stream and the color video stream to determine whether the photographed object falls or to determine that the photographed object does not fall as a trigger condition for asynchronous execution of the deep learning model, and when an exception is detected, the deep learning model is used for judgment, solving the performance problem of real-time execution of the deep learning method, and finally the two kinds of inputs are comprehensively analyzed to obtain a better judgment result.

[0040] Step S103, performing fall detection analysis on the photographed object according to the skeleton detection result obtained from the depth video stream and the first analysis result obtained from the color video stream.

[0041] The embodiment of the application provides a fall detection method, which first acquires a depth video stream and a color video stream of a photographed object; then respectively uses the depth video stream and the color video stream to perform fall detection analysis on the photographed object; and finally performs fall detection analysis on the photographed object according to a skeleton detection result obtained from the depth video stream and a first analysis result obtained from the color video stream. In this way, the method of comprehensively detecting and analyzing two kinds of video stream detection results can effectively improve the accuracy of confirming whether the photographed object falls.

[0042] Embodiment two

[0043] The embodiment provides a fall detection method. Figure 2 Another implementation process schematic diagram of a fall detection method provided by the embodiment of the application is shown in FIG. 2, which includes the following steps. Figure 2

[0044] Step S201, acquiring a depth video stream and a color video stream of a photographed object;

[0045] Step S202, respectively using the depth video stream and the color video stream to perform fall detection analysis on the photographed object;

[0046] ​Step S203, if only the skeleton detection result or the first analysis result is detected within the set sliding window, the skeleton detection result or the first analysis result is taken as the fall detection result of the photographed object.

[0047] Since there is a time delay error in the two processing results, to avoid the misjudgment that a fall is considered as a second fall due to the time difference between the decision results of the skeleton detection and the deep learning method detection, a decision maker is designed to save the state, a sliding window mechanism is used, the window length is 1S, the step is 100ms (about 3 frames of data), and the input at the end of a window time is taken as the final input result for judgment and output.

[0048] According to the data of the fall behavior data set, a fall usually occurs in about 1 second, and this window length is sufficient to confirm whether a fall occurs. The trained model is detected and classified through graph convolution and deep learning. The model collects scenes that cannot be obtained by skeleton points or cannot be judged by threshold method, such as too close, too far, occlusion, and slow sitting and falling. If only one result of a video stream is detected within the sliding window, this result is taken as the standard, which can improve the detection efficiency.

[0049] Step S204, if the skeleton detection result of the photographed object falling obtained according to the depth video stream is detected within the set sliding window, a fall detection result of the photographed object falling is output.

[0050] Since the skeleton detection result is more accurate than the color video stream detection result, if the skeleton detection result of the photographed object falling obtained according to the depth video stream is detected within the set sliding window, a fall detection result of the photographed object falling is output. In the embodiment of the application, if only one result of a video stream is detected within the sliding window, this result is taken as the standard. If the depth video stream obtained by the depth camera is analyzed and confirmed to be the photographed object falling within the sliding window, there is no need to use the color image for confirmation. Since the depth camera can obtain more dimensional information for processing, the accuracy of the depth video stream confirmation is high. In this way, the confirmation efficiency is improved without affecting the judgment accuracy.

[0051] Embodiment Three

[0052] The embodiment provides a fall detection method. Figure 3A As shown in the implementation flowchart of another fall detection method provided in the embodiment of the application, Figure 3A The method comprises the following steps.

[0053] Step S300, obtaining a depth video stream and a color video stream of a photographed object.

[0054] Step S301, respectively using the depth video stream and the color video stream to analyze the fall detection of the photographed object;

[0055] Step S302, inputting the depth video stream into a trained skeleton extraction model to obtain the skeleton node coordinates of the photographed object.

[0056] After obtaining a large number of depth map samples through a depth camera, manually labeling the skeleton node positions, and modeling and training the data, a trained model for extracting skeleton samples from a depth map is obtained.

[0057] According to the input depth video stream, a camera bottom-layer hardware method is collected, for example, an interface function GetDepthFrame() is encapsulated to obtain the depth frame of the photographed object.

[0058] The depth frame is input into a trained skeleton extraction model to obtain the skeleton node coordinates of the photographed object.

[0059] Step S303, if the skeleton node coordinates cannot determine whether the photographed object has fallen or determine that the photographed object has not fallen, a method of tracking the human body by foreground detection is used to process the color video stream to obtain the two-dimensional contour of the photographed object.

[0060] The obtained color video and depth video stream are processed at the same time, when the depth video stream cannot extract the skeleton node and cannot be judged, the processing of the color video stream is used to complete the solution to determine, and the problems in the extraction of the skeleton by the depth camera are solved.

[0061] The foreground is the part with stronger visibility and saliency in the image or video scene. Foreground detection is the process of identifying the region of interest or identifying the moving target (scene) and the static part (scene) in the video. It is a key step in motion analysis and video monitoring, and also the most basic and bottom step. The effect of the foreground detection result directly relates to the performance of the subsequent analysis and application processing. Common foreground detection algorithms include frame difference method, background modeling method, optical flow method, average background method, background non-parametric estimation, foreground modeling method, etc.

[0062] According to the input color video stream, a camera bottom-layer hardware method is collected, for example, an interface function GetColorFrame() is encapsulated to obtain the color frame of the photographed object.

[0063] Because the detection accuracy of the color video stream is low, usually there are more false detections and less missed detections, so the abnormality judgment is used as the trigger condition for asynchronous execution of the deep learning model. If it is determined that the photographed object does not fall according to the bone node coordinates, the color video stream of the photographed object is input, and a two-dimensional contour is obtained by using a method of tracking a human body by foreground detection.

[0064] In step S304, the length and the width of the photographed object are determined according to the two-dimensional contour.

[0065] In step S305, a first analysis result is determined according to a size relationship between a ratio between the length and the width of the photographed object and a specific ratio. If the ratio between the length and the width is greater than the specific ratio, it is determined that the first analysis result is that the photographed object does not fall. If the ratio between the length and the width is less than the specific ratio, it is determined that the first analysis result is that the photographed object falls.

[0066] The ratio between the length and the width of the photographed object is analyzed abnormally. When the length-width ratio is about 4.5, it is analyzed that a normal person walks upright, and the process goes to step 407. When the length-width ratio is less than 0.5, it is analyzed that a suspected fall occurs, and the process goes to step 405.

[0067] When a normal person walks upright, the length-width ratio is about 4.5, and when the length-width ratio is less than 0.5 after falling, it is suspected to fall. The abnormality is suspected to be caused by the limitation of two-dimensional data. The length-width ratio may change when the distance of a person changes from far to near, which may cause a misjudgment. Therefore, it is only used as a reference.

[0068] The ratio between the width and the length can also be used for abnormal analysis.

[0069] In step S306, if it is determined that the photographed object falls according to the first analysis result, N frames of pictures before the first frame of the color video stream are extracted, and N is greater than or equal to 2.

[0070] If it is determined that the photographed object falls according to the first analysis result, N frames of pictures before the first frame of the color video stream are extracted, which can more accurately analyze whether the photographed object falls. For example, by using a color video stream to judge suspected first 100 frames, since an ordinary camera can shoot 30 frames per second, about 3 seconds or more are input to a deep learning model to determine whether the photographed object falls.

[0071] In step S307, the N frames of pictures are input into a trained deep learning model for analysis to determine a second analysis result.

[0072] The model established by the deep learning method is a model trained by a computer vision deep learning framework based on PyTorch, which is a deep learning tensor library optimized by using a Graphics Processing Unit (GPU) and a CPU.

[0073] When the color video is detected as an anomaly, the model established by the deep learning method is used for judgment, the performance problem of real-time processing of the deep learning end side is solved, and finally the judgment results with better accuracy are obtained by comprehensively combining the two inputs.

[0074] The N frames of pictures are input into the trained deep learning model for analysis, and whether the photographed object falls is more accurately determined.

[0075] In step S308, if the second analysis result is falling, it is determined that the falling detection result is that the photographed object falls; if the second analysis result is not falling, it is determined that the falling detection result is that the photographed object does not fall.

[0076] In step S309, within the set sliding window, if only the skeleton detection result or the first analysis result is detected, the skeleton detection result or the first analysis result is taken as the falling detection result of the photographed object.

[0077] In step S310, within the set sliding window, if the skeleton detection result of the photographed object falling obtained according to the depth video stream is detected, a falling detection result of the photographed object falling is output.

[0078] In the embodiments of the present application, the obtained color video and depth video stream are processed at the same time. The method of detecting the depth video stream is based on the threshold rule judgment of the human body skeleton node and the machine learning classifier detection method based on the skeleton node. When the depth video cannot extract the skeleton node and cannot judge or determine that the photographed object does not fall, a certain processing and completion solution is performed through the color video to determine whether the photographed object falls. In this way, the problem of missed judgment and misjudgment of falling detection caused by relying on a single detection method is solved, the performance problem of the deep learning method on the end side is solved by using an asynchronous detection analysis method, and the detection accuracy is improved.

[0079] Embodiment Four

[0080] The embodiment provides a falling detection method. Figure 3B As shown in FIG. 5, the method includes the following steps. Figure 3B

[0081] In step S311, a depth video stream and a color video stream of a photographed object are obtained by using a binocular camera.​

[0082] Step S312, respectively using the depth video stream and the color video stream to analyze the fall detection of the photographed object;

[0083] Step S313, in the sliding window, if the first analysis result of the photographed object not falling is detected according to the depth video stream, wait for the first analysis result obtained from the color video stream; if the first analysis result is that the photographed object does not fall, output the fall detection result that the photographed object does not fall;

[0084] When the first analysis result of the photographed object not falling is detected according to the depth video stream, wait for the first analysis result obtained from the color video stream, if the result is also that the photographed object does not fall, the fall detection result that the photographed object does not fall can be output. In this way, the combination of the two video streams for judgment can effectively reduce the missed judgment and the wrong judgment.

[0085] Step S314, in the sliding window, if the first analysis result of the photographed object falling is detected, wait for the skeleton detection result obtained from the depth video stream; if the skeleton detection result is that the photographed object falls, output the fall detection result that the photographed object falls;

[0086] When the first analysis result of the photographed object not falling is detected according to the color video stream, wait for the skeleton detection result obtained from the depth video stream, if the result is also that the photographed object does not fall, the fall detection result that the photographed object does not fall can be output. In this way, the combination of the two video streams for judgment can effectively reduce the missed judgment and the wrong judgment.

[0087] Step S315, in the sliding window, if the first analysis result of the photographed object falling is detected, wait for the skeleton detection result obtained from the depth video stream; if the skeleton detection result is that the photographed object does not fall, input the color video stream into the trained deep learning model for analysis to determine the second analysis result; if the second analysis result is that the photographed object falls, output the fall detection result that the photographed object falls; if the second analysis result is that the photographed object does not fall, determine the fall detection result that the photographed object does not fall;

[0088] The first analysis result detects that the photographed object falls down, and the bone detection result needs to be waited for. If the bone detection result is that the photographed object falls down, a fall detection result that the photographed object falls down is output. If the bone detection result is that the photographed object does not fall down, the color video stream is input into the trained deep learning model for analysis to determine a second analysis result. The first analysis result is a method of tracking a human body by using foreground detection to analyze the color video stream. Because the detection accuracy of the color video stream is low, false detection is usually more, and missed detection is less, and therefore, this abnormality judgment is used as a trigger condition for asynchronous execution of the deep learning model. When the first analysis result is that the photographed object falls down, the deep learning model is used for judgment to determine the second analysis result. If the second analysis result is that the photographed object falls down, a fall detection result that the photographed object falls down is output. If the second analysis result is that the photographed object does not fall down, a fall detection result that the photographed object does not fall down is output.

[0089] In step S316, if only the bone detection result or the first analysis result is detected in the set sliding window, the bone detection result or the first analysis result is taken as a fall detection result of the photographed object.

[0090] In step S317, if the bone detection result that the photographed object falls down is detected according to the depth video stream in the set sliding window, a fall detection result that the photographed object falls down is output.

[0091] In the embodiment of the application, because the binocular camera is used to obtain the depth video stream and the color video stream of the photographed object, there is a time difference between the two videos, and therefore, the sliding window needs to be set, and whether the photographed object falls down is judged according to the depth video stream and the color video stream data in the sliding window. Because the detection method of the depth video stream is a threshold rule judgment based on a human body bone node and a machine learning classifier detection method based on a bone node, false detection of the color video stream is less, and therefore, if the depth video stream determines that the photographed object falls down, a fall result is output. When the depth video stream cannot determine or determines that the photographed object does not fall down in the sliding window, the first analysis and the second analysis of the color video are used to judge whether the photographed object falls down.

[0092] The embodiments of the present application involve four results, which are a skeleton detection result, a first analysis result, a second analysis result and a fall detection result. The skeleton detection result is obtained according to a depth video stream, the first analysis result and the second analysis result are obtained according to color video stream analysis, and the fall detection result is finally determined by comprehensively analyzing the skeleton detection result, the first analysis result and the second analysis result. In this way, the misjudgment that a fall is considered twice due to the time difference in video collection by the binocular camera is solved, and at the same time, the problems of missed judgment and misjudgment of fall detection caused by relying on a single detection method are solved. The asynchronous detection analysis method is used to solve the problem of end-side performance of the deep learning method, and the detection accuracy is improved.

[0093] Embodiment five

[0094] Figure 4A To realize the hardware schematic diagram used in the embodiments of the present application, as shown in Figure 4A , the hardware includes an RGBD camera 401 and a JetsonNano development board 402.

[0095] Figure 4B To realize the system architecture schematic diagram of the development board in the embodiments of the present application, as shown in Figure 4B , the development board includes a data acquisition layer 411, a business logic processing layer 412, a deep learning processing layer 413, a cache queue 414, a database 415, a resource task scheduling layer 416, an operating system 417, a central processing unit (CPU) 418 and a graphics processing unit (GPU) 419.

[0096] Figure 4C To realize the implementation flow schematic diagram of another fall detection method provided by the embodiments of the present application, as shown in Figure 4C , the method includes:

[0097] Step S400, using a binocular camera to simultaneously collect an RGB color video stream and a depth video stream frame by frame;

[0098] The binocular camera uses two cameras to run simultaneously to perceive the surrounding environment. The advantages of the visual system composed of the binocular camera are obvious. Compared with the monocular system, the binocular system can extract more information related to the real world environment, especially the depth information of the target object. The binocular vision system can even restore the depth of the observation point normally in the case of inaccurate camera calibration, and minimize the human-caused error.

[0099] RGB color mode is a color standard in the industry, which is to get all kinds of colors by changing the three color channels of red (R), green (G) and blue (B) and superimposing them on each other. RGB represents the three channels of red, green and blue. This standard includes almost all colors that can be perceived by human vision, and is one of the most widely used color systems.

[0100] RGBD camera is an RGB camera plus a depth camera. In 3D computer graphics, a depth map is an image or image channel that contains information about the distance of the scene object's surface from the viewpoint. The depth map is similar to a grayscale image, except that each pixel value is the actual distance from the sensor to the object. Usually, the RGB image and the depth map are registered, so there is a one-to-one correspondence between the pixels.

[0101] The hardware for implementing the embodiments of the present application is shown in Figure 4A , wherein the RGBD camera 401 is used to collect RGB color video stream and depth video stream; the JetsonNano development board 402 is used to develop programs for processing color video stream and depth video stream.

[0102] The system architecture for implementing the present application is shown in Figure 4B , wherein the data acquisition layer 411 is used to adapt the video stream acquisition of the camera, and has an interactive interface with the business logic layer and the depth learning layer; the business logic processing layer 412 is used to input, judge and execute the fall detection rules and decision maker rules; the depth learning processing layer 413 is used to interface with the depth learning open source project, process the video data format, and serve as the input and output feedback of the business logic layer and the data acquisition layer; the cache queue 414 is used for video stream storage, and adopts the principle of first-in first-out, with a queue length of 100 frames; the database 415 is used to store business data and log data; the resource task scheduling layer 416 is used to monitor the system resource situation, including CPU and GPU memory usage, to allocate processing tasks between CPU and GPU and ensure hardware resource utilization; the operating system 417 can use Ubuntu operating system and QT / C++ development language; the central processing unit (CPU) 418 is the operation and control core of the computer system, and is the final execution unit of information processing and program running; the graphics processing unit (GPU) 419 is a microprocessor that is specially used for image and graphics related operations on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.).

[0103] The data acquisition layer uses a binocular RGBD camera to simultaneously collect two video streams of the photographed object in frames: an RGB color video stream and a depth video stream.

[0104] Step S401: processing according to the input depth video stream to obtain a depth frame;

[0105] The service logic processing layer collects the camera bottom layer hardware method according to the input depth video stream, for example: encapsulating as an interface function GetDepthFrame() to obtain the depth frame of the photographed object.

[0106] Step S402: inputting the depth frame into a trained model to obtain skeleton node coordinates;

[0107] After the service logic processing layer obtains a large number of depth map samples through the depth camera, the skeleton node positions are manually labeled, the data is modeled and trained, and a trained model for extracting skeleton samples from a depth map is obtained.

[0108] The service logic processing layer inputs the depth frame of the photographed object into the trained model to obtain skeleton node coordinates.

[0109] Step S403: determining whether the skeleton node coordinates are successfully obtained, and if yes, proceeding to step 404, and if not, proceeding to step 402;

[0110] Step S404: inputting the obtained skeleton node coordinates and performing skeleton fall analysis according to a rule model;

[0111] Here, the rule model refers to the change in the height of the center of gravity, the change in the speed of the head, the inclination of the torso to the ground, etc.

[0112] The obtained skeleton node coordinates of the photographed object are inputted, fall analysis is performed according to the rule model, and the result is inputted to 409 for fall decision analysis, regardless of whether the analysis result is a fall or no fall.

[0113] Step S405: processing according to the input RGB color video stream to obtain a color frame;

[0114] The service logic processing layer collects the camera bottom layer hardware method according to the input RGB color video stream, for example: encapsulating as an interface function GetColorFrame() to obtain the color frame of the photographed object.

[0115] Step S406: inputting the color frame and obtaining a two-dimensional contour by using a foreground detection method to track the human body, and calibrating the length and width according to the two-dimensional contour;

[0116] The foreground is the part with stronger visibility and saliency in an image or video scene. Foreground detection is a process of identifying the region of interest or identifying moving objects (foreground) and static parts (background) in a video. It is a key step of motion analysis and video monitoring, and is also the most basic and bottom step. The effect of the foreground detection result directly relates to the performance of subsequent analysis and application processing. Common foreground detection algorithms include frame difference method, background modeling method, optical flow method, average background method, background non-parametric estimation, foreground modeling method, etc.

[0117] The color frame of the photographed object is input, the foreground detection is used to track the human body by the service logic processing layer to obtain a two-dimensional contour, and the two-dimensional contour length-width of the photographed object is calibrated according to the two-dimensional contour.

[0118] In step S407, the two-dimensional contour length-width is input, and the two-dimensional contour of the human body is established.

[0119] The two-dimensional contour length-width of the photographed object is input, and the two-dimensional contour of the human body is established according to the two-dimensional contour length-width by the service logic processing layer.

[0120] In step S408, the two-dimensional contour of the human body is subjected to abnormal analysis.

[0121] The two-dimensional contour of the human body is subjected to abnormal analysis: the length-width ratio value is about 4.5 to analyze the normal person walking upright, and the length-width ratio value is less than 0.5 to analyze the suspected fall, and the process is turned to step 405.

[0122] The 2D human body tracking module in the service logic layer judges (CPU calculation) the two-dimensional contour calibrated by the simultaneously conducted foreground tracking method, and the rules are as follows: when the normal person walks upright, the length-width ratio value is about 4.5, and the length-width ratio value is less than 0.5 after falling down, which is suspected to fall, and the abnormal suspected reason is that the two-dimensional data has limitations, and the distance of the person changes from far to near, which may cause misjudgment due to the change of the length-width ratio, so it is only used as a reference.

[0123] In step S409, the fall analysis result of the skeleton, the abnormal analysis result of the RGB video stream, and the deep learning result are input to make a fall decision.

[0124] Because there are sometimes time delay errors in the two processing results, to avoid the situation that a fall is considered as a second fall due to time difference, which leads to a misjudgment of the decision results of the bone fall analysis and the deep learning method analysis, a decision maker is designed to save the state, using a sliding window mechanism, the window length is 1S, and the step is 100ms (about 3 frames of data), if only one kind of video stream collected by the binocular camera is detected in the window time, whether the fall is judged by the collected video stream; if two kinds of video streams are collected in the window time, the fall rule module in the business logic layer judges the suspected fall, and inputs the result to the decision maker module, if the bone fall result in the current window of the decision maker is YES, the fall result is output by the decision maker without processing; if the bone fall analysis result is NO, the first 100 frames (30 frames / s) are extracted for judgment by the cache.

[0125] The fall decision rule is as follows:

[0126] In the window, the bone fall analysis result is no fall, and the RGB video stream abnormal analysis result is fall, the cache extraction is performed asynchronously for deep fall analysis, and step 410 is turned to;

[0127] In the window, the bone fall analysis result and the deep learning result are both no fall, and the judgment result is no fall;

[0128] In the window, the bone fall analysis result is fall or the deep learning result is fall, and the fall is directly judged.

[0129] According to the data set of the fall behavior data, the fall usually occurs in about 1 second, and this window is sufficient to confirm whether the fall occurs. The deep learning processing layer detects and classifies the trained model by means of graph convolution and deep learning, and the model collects scenes such as close distance, too far distance, occlusion, etc. which cannot be obtained by bone points or cannot be judged by threshold method, and slowly sitting down, etc.

[0130] Step S410, the first 100 frames (30 frames / s) are extracted for judgment by the cache;

[0131] The first 100 frames of the RGB video stream are extracted for judgment by the cache, which is used for input of the deep learning method.

[0132] Step S411, the model established by the deep learning method is used for detection, and the judgment result is input into the decision maker, and the judgment is continued after the judgment is finished.

[0133] The model established by the deep learning method is a model trained by a computer vision deep learning framework based on PyTorch, wherein PyTorch is a deep learning tensor library optimized by a graphics processing unit (GPU) and a CPU.

[0134] According to the input RGB video stream, the first 100 frames are determined to determine whether the photographed object falls.

[0135] The application embodiment provides a fusion detection method, uses an asynchronous and multi-thread mechanism to design a video queue cache mode, balances the performance consumption of a deep learning model detection, uses a dynamic monitoring mechanism to allocate CPU / GPU resources, improves hardware utilization (rules, real-time frame input uses CPU multi-thread processing, and the cache video deep learning model uses GPU processing, if there is an exception in resource monitoring, the two resource reallocations can be performed), uses a sliding window and a state retention mechanism to design a decision maker, shields secondary misjudgment caused by the asynchronous results of the two processing modes, through continuous optimization and testing, two video stream inputs test data sets respectively, finds the missed and misjudged points in a single determination method, adjusts the influence on the decision maker through threshold adjustment or model retraining processing, improves the detection accuracy, and the hardware power consumption is low, the development board only consumes 5w of power, is energy-saving, and is suitable for long-term operation of monitoring products.

[0136] Embodiment six

[0137] Based on the foregoing embodiments, the application embodiment provides a fall detection device, which includes various modules and can be realized by a processor in a fall detection device (such as a computer, a mobile phone, or other electronic devices with information processing capability); of course, it can also be realized by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0138] Figure 5 The composition structure diagram of the fall detection device provided by the application embodiment is shown in Figure 5 The device 500 includes an acquisition module 501, a first detection analysis module 502, and a second detection analysis module 503, wherein:

[0139] The acquisition module 501 is configured to acquire a depth video stream and a color video stream of a photographed object.

[0140] The first detection analysis module 502 is configured to perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively.

[0141] The second detection analysis module 503 is configured to perform fall detection analysis on the photographed object according to a skeletal detection result obtained from the depth video stream and a first analysis result obtained from the color video stream.

[0142] Based on the foregoing embodiments, the embodiments of the present application provide a fall detection device, the device comprising: an acquisition module, a first detection analysis module, and a second detection analysis module, wherein the second detection analysis module comprises a first determination submodule and a first output submodule:

[0143] The acquisition module is configured to acquire a depth video stream and a color video stream of a photographed object.

[0144] The first detection analysis module is configured to perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively.

[0145] The first determination submodule is configured to, within a set sliding window, if only the skeleton detection result or the first analysis result is detected, take the skeleton detection result or the first analysis result as a fall detection result of the photographed object.

[0146] The first output submodule is configured to, within the set sliding window, if the skeleton detection result of the photographed object falling is detected according to the depth video stream, output a fall detection result of the photographed object falling.

[0147] Based on the foregoing embodiments, the embodiments of the present application provide a fall detection device, the device comprising: an acquisition module, a first detection analysis module, and a second detection analysis module, wherein the second detection analysis module comprises a first determination submodule, a first output submodule, a second determination submodule, and a third determination submodule, and wherein:

[0148] The acquisition module is configured to acquire a depth video stream and a color video stream of a photographed object.

[0149] The first detection analysis module is configured to perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively.

[0150] The first determination submodule is configured to, within a set sliding window, if only the skeleton detection result or the first analysis result is detected, take the skeleton detection result or the first analysis result as a fall detection result of the photographed object.

[0151] The first output submodule is configured to, within the set sliding window, if the skeleton detection result of the photographed object falling is detected according to the depth video stream, output a fall detection result of the photographed object falling.

[0152] a second determining submodule, configured to, when it is determined that the photographed object does not fall according to the depth video stream within the set sliding window, and it is determined that the photographed object falls according to the first analysis result, input the color video stream into the trained deep learning model for analysis to obtain a second analysis result;

[0153] a third determining submodule, configured to, when the second analysis result is falling, determine that the fall detection result is that the photographed object falls; and when the second analysis result is not falling, determine that the fall detection result is that the photographed object does not fall.

[0154] Based on the foregoing embodiments, the embodiments of the present application provide a fall detection device, which comprises an acquisition module, a first detection and analysis module, and a second detection and analysis module, wherein the second detection and analysis module comprises a first determining submodule, a first output submodule, a second determining submodule, and a third determining submodule, wherein the second determining submodule comprises a first obtaining unit, a first analysis unit, an extracting unit, and a second analysis unit, and wherein:

[0155] the acquisition module is configured to acquire a depth video stream and a color video stream of a photographed object;

[0156] the first detection and analysis module is configured to perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively;

[0157] the first determining submodule is configured to, within a set sliding window, take the bone detection result or the first analysis result as a fall detection result of the photographed object if only the bone detection result or the first analysis result is detected;

[0158] the first output submodule is configured to, within the set sliding window, output a fall detection result that the photographed object falls if the bone detection result that the photographed object falls is detected according to the depth video stream;

[0159] the first obtaining unit is configured to input the depth video stream into a trained bone extracting model to obtain bone node coordinates of the photographed object;

[0160] the first analysis unit is configured to, when it is determined that the photographed object does not fall according to the bone node coordinates or it is determined that the photographed object does not fall, perform first analysis on the color video stream by using a human body tracking method based on foreground detection to obtain a first analysis result;

[0161] extracting N frames of pictures before a first frame of the color video stream if it is determined that the photographed object falls according to the first analysis result, wherein N is greater than or equal to 2;

[0162] a second analysis unit configured to input the N frames of pictures into a trained deep learning model to perform analysis and determine a second analysis result;

[0163] a third determination submodule configured to determine that the fall detection result is that the photographed object falls if the second analysis result is that the photographed object falls, and determine that the fall detection result is that the photographed object does not fall if the second analysis result is that the photographed object does not fall.

[0164] Based on the foregoing embodiments, the embodiments of the present application provide a fall detection device, which comprises an acquisition module, a first detection and analysis module, and a second detection and analysis module, wherein the second detection and analysis module comprises a first determination submodule, a first output submodule, a second determination submodule, a third determination submodule, a first obtaining submodule, a fourth determination submodule, and a fifth determination submodule, and wherein:

[0165] the acquisition module is configured to acquire a depth video stream and a color video stream of a photographed object;

[0166] the first detection and analysis module is configured to perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively;

[0167] the first determination submodule is configured to, in a set sliding window, take the skeleton detection result or the first analysis result as a fall detection result of the photographed object if only the skeleton detection result or the first analysis result is detected;

[0168] the first output submodule is configured to, in the set sliding window, output a fall detection result that the photographed object falls if the skeleton detection result that the photographed object falls is detected according to the depth video stream;

[0169] the first obtaining submodule is configured to obtain a two-dimensional contour of the photographed object by processing the color video stream by using a method of tracking a human body by using foreground detection;

[0170] the fourth determination submodule is configured to determine a length and a width of the photographed object according to the two-dimensional contour;

[0171] a fifth determining sub-module, configured to determine a first analysis result according to a size relationship between a ratio between a length and a width of the photographed object and a specific ratio; if the ratio between the length and the width is greater than the specific ratio, determining that the first analysis result is that the photographed object does not fall; if the ratio between the length and the width is less than the specific ratio, determining that the first analysis result is that the photographed object does not fall;

[0172] a second determining sub-module, configured to, if it is unable to determine whether the photographed object falls or determines that the photographed object does not fall according to the depth video stream, and determines that the photographed object falls according to the first analysis result, input the color video stream into a trained deep learning model to perform analysis and determine a second analysis result within a set sliding window;

[0173] a third determining sub-module, configured to, if the second analysis result is falling, determine that the fall detection result is that the photographed object falls; if the second analysis result is not falling, determine that the fall detection result is that the photographed object does not fall.

[0174] Based on the foregoing embodiments, the embodiments of the present application provide a fall detection device, the device comprising: an acquisition module, a first detection and analysis module, and a second detection and analysis module, wherein the second detection and analysis module comprises a first determining sub-module, a first output sub-module, a second determining sub-module, a third determining sub-module, a second output sub-module, a third output sub-module, and a fourth output sub-module, and wherein:

[0175] the acquisition module is configured to acquire a depth video stream and a color video stream of a photographed object;

[0176] the first detection and analysis module is configured to perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively;

[0177] the first determining sub-module is configured to, within a set sliding window, if only the skeleton detection result or the first analysis result is detected, take the skeleton detection result or the first analysis result as a fall detection result of the photographed object;

[0178] the first output sub-module is configured to, within a set sliding window, if the skeleton detection result of the photographed object falling is detected according to the depth video stream, output a fall detection result of the photographed object falling;

[0179] a second determining sub-module, configured to, when it is determined that the photographed object does not fall according to the first analysis result and it is unable to determine whether the photographed object falls or not according to the depth video stream, input the color video stream into the trained deep learning model to analyze and determine a second analysis result within the set sliding window;

[0180] a third determining sub-module, configured to, when the second analysis result is falling, determine that the fall detection result is that the photographed object falls; and when the second analysis result is not falling, determine that the fall detection result is that the photographed object does not fall.

[0181] a second outputting sub-module, configured to, within the sliding window, when the skeleton detection result that the photographed object does not fall according to the depth video stream is detected first, wait for the first analysis result according to the color video stream; and when the first analysis result is that the photographed object does not fall, output the fall detection result that the photographed object does not fall.

[0182] a third outputting sub-module, configured to, within the sliding window, when the first analysis result that the photographed object falls is detected first, wait for the skeleton detection result according to the depth video stream; and when the skeleton detection result is that the photographed object falls, output the fall detection result that the photographed object falls.

[0183] a fourth outputting sub-module, configured to, within the sliding window, when the first analysis result that the photographed object falls is detected first, wait for the skeleton detection result according to the depth video stream; when the skeleton detection result is that the photographed object does not fall, input the color video stream into the trained deep learning model to analyze and determine a second analysis result; when the second analysis result is that the photographed object falls, output the fall detection result that the photographed object falls; and when the second analysis result is that the photographed object does not fall, determine the fall detection result that the photographed object does not fall.

[0184] The above description of the device embodiment is similar to the description of the method embodiment, and has similar beneficial effects. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment.

[0185] It should be noted that, in the embodiments of the present application, if the fall detection method described above is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a plurality of instructions for causing an electronic device (which can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a robot, a drone, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read Only Memory, ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific hardware and software combination.

[0186] Correspondingly, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the client-side or server-side fall detection method provided in the above embodiments.

[0187] Correspondingly, the embodiments of the present application provide a fall detection device, Figure 6 A hardware entity diagram of the fall detection device of the embodiments of the present application is shown in FIG. 6, which includes a memory 601 and a processor 602, the memory 601 stores a computer program executable on the processor 602, and the processor 602 implements the steps of the fall detection method provided in the above embodiments when executing the program. Figure 6

[0188] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data (for example, image data, audio data, voice communication data and video communication data) to be processed by the processor 602 and each module in the fall detection device 600, which can be implemented by a flash (FLASH) or a random access memory (Random Access Memory, RAM).

[0189] It should be noted that: the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0190] ​It should be understood that every embodiment mentioned in the specification throughout the description means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the occurrence of in one embodiment or in an embodiment throughout the specification does not necessarily mean the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that the size of the sequence number of each process in various embodiments of the present application does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The sequence number of the above embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0191] It should be noted that in this text, the terms include, contain or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element limited by the sentence including one does not exclude the presence of another identical element in the process, method, article or device including the element.

[0192] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0193] The units described above as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units; they can be located in one place or distributed on multiple network units; part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0194] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a unit alone, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or hardware plus software functional unit.

[0195] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes a mobile storage device, a read only memory (ROM), a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0196] Alternatively, the integrated units described above in the present application can be stored in a computer readable storage medium if they are realized in the form of software function modules and sold or used as independent products. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a plurality of instructions for causing the fall detection device (which can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a robot, a drone, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes a mobile storage device, a ROM, a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0197] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0198] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0199] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method or device embodiments.

[0200] The above is only an implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A fall detection method, characterized by, The method applied to a terminal comprises: obtaining a depth video stream and a color video stream of a photographed object; respectively using the depth video stream and the color video stream to perform fall detection analysis on the photographed object; performing fall detection analysis on the photographed object according to a skeleton detection result obtained from the depth video stream and a first analysis result obtained from the color video stream; wherein the fall detection analysis on the photographed object according to the skeleton detection result obtained from the depth video stream and the first analysis result obtained from the color video stream comprises: if only the skeleton detection result or the first analysis result is detected within a set sliding window, taking the skeleton detection result or the first analysis result as a fall detection result of the photographed object; if the skeleton detection result of the photographed object falling obtained from the depth video stream is detected within the set sliding window, outputting a fall detection result of the photographed object falling; if it is determined that the photographed object does not fall or it is determined that the photographed object does not fall according to the depth video stream within the set sliding window, and it is determined that the photographed object falls according to the first analysis result, inputting the color video stream into a trained deep learning model to perform analysis and determine a second analysis result; the first analysis result is obtained by performing first analysis on the color video stream using a human body tracking method based on foreground detection; if the second analysis result is falling, determining that the fall detection result is the photographed object falling; if the second analysis result is not falling, determining that the fall detection result is the photographed object not falling.

2. The method of claim 1, wherein, the inputting the color video stream into the trained deep learning model to perform analysis and determine the second analysis result comprises: extracting N frames of pictures before the first frame of the color video stream, wherein N is greater than or equal to 2; inputting the N frames of pictures into the trained deep learning model to perform analysis and determine the second analysis result.

3. The method of claim 1, wherein, the determining that the photographed object does not fall or it is determined that the photographed object does not fall according to the depth video stream comprises: inputting the depth video stream into a trained skeleton extraction model to obtain skeleton node coordinates of the photographed object; if it is determined that the photographed object does not fall or it is determined that the photographed object does not fall according to the skeleton node coordinates, performing first analysis on the color video stream using a human body tracking method based on foreground detection to obtain a first analysis result.

4. The method of claim 1, wherein, the method further comprises: performing processing on the color video stream using a human body tracking method based on foreground detection to obtain a two-dimensional contour of the photographed object; determining the length and width of the photographed object according to the two-dimensional contour; determining a first analysis result according to the size relationship between the ratio of the length to the width of the photographed object and a specific ratio; wherein, If the ratio between the length and the width is greater than the specific ratio, the first analysis result is determined as that the photographed object does not fall; if the ratio between the length and the width is less than the specific ratio, the first analysis result is determined as that the photographed object falls.

5. The method of claim 1, wherein, The depth video stream and the color video stream of the photographed object are obtained by using a binocular camera. Correspondingly, the fall detection analysis on the photographed object is performed according to the skeleton detection result obtained from the depth video stream and the first analysis result obtained from the color video stream, including: In the sliding window, if the skeleton detection result that the photographed object does not fall is detected first, the first analysis result is waited for; if the first analysis result is that the photographed object does not fall, a fall detection result that the photographed object does not fall is output.

6. The method of claim 1, wherein, The fall detection analysis on the photographed object is performed according to the skeleton detection result obtained from the depth video stream and the first analysis result obtained from the color video stream, further including: In the sliding window, if the first analysis result that the photographed object falls is detected first, the skeleton detection result is waited for; if the skeleton detection result is that the photographed object falls, a fall detection result that the photographed object falls is output.

7. The method of claim 6, wherein, The fall detection analysis on the photographed object is performed according to the skeleton detection result obtained from the depth video stream and the first analysis result obtained from the color video stream, further including: If the skeleton detection result is that the photographed object does not fall, the color video stream is input into a trained deep learning model for analysis to determine a second analysis result; If the second analysis result is that the photographed object falls, a fall detection result that the photographed object falls is output; if the second analysis result is that the photographed object does not fall, a fall detection result that the photographed object does not fall is determined.

8. A fall detection apparatus characterized by comprising: The method comprises: obtaining a depth video stream and a color video stream of a photographed object; a first detection analysis module is configured to perform fall detection analysis on the photographed object by using the depth video stream and the color video stream respectively, and perform fall detection analysis on the photographed object according to a skeleton detection result obtained from the depth video stream and a first analysis result obtained from the color video stream; a first determination submodule is configured to, in a set sliding window, take the skeleton detection result or the first analysis result as a fall detection result of the photographed object if only the skeleton detection result or the first analysis result is detected; a first output submodule is configured to, in the set sliding window, output a fall detection result that the photographed object falls if the skeleton detection result that the photographed object falls is detected from the depth video stream; and a second determination submodule is configured to, in the set sliding window, if the first analysis result that the photographed object does not fall is detected first, wait for the skeleton detection result obtained from the depth video stream; if the skeleton detection result is that the photographed object does not fall, output a fall detection result that the photographed object does not fall. a second determining sub-module, configured to input the color video stream into a trained deep learning model for analysis to determine a second analysis result, if it is unable to determine whether the photographed object has fallen according to the depth video stream, or it is determined that the photographed object has not fallen according to the depth video stream, and it is determined that the photographed object has fallen according to the first analysis result, within a set sliding window; the first analysis result is obtained by performing first analysis on the color video stream by using a foreground detection method to track a human body; a third determining sub-module, configured to determine that the fall detection result is that the photographed object has fallen, if the second analysis result is that the photographed object has fallen; determine that the fall detection result is that the photographed object has not fallen, if the second analysis result is that the photographed object has not fallen.

9. A fall detection device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, the processor implements the steps in the fall detection method in any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, the computer program implements the steps in the fall detection method in any one of claims 1 to 7 when executed by the processor.

Images