A method and system for intelligently identifying falling persons based on human skeleton points

Abstract

The present invention discloses a method and system for intelligently identifying a person falling down based on human skeleton points, and relates to the field of computer vision technology, including: using a human body detection model to judge the real-time acquired surveillance camera video frame, if the human body detection model judges that there is a human body in the surveillance camera video frame, then intercepting the human body area image; extracting the position coordinates of key human body joint points in the human body area image through a human body joint point detection model; based on the position coordinates of the key human body joint points, through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, calculating the distance, angle and relative position relationship between the key human body joint points, and comprehensively judging whether the human body presents a falling action. The present invention provides an efficient, accurate and practical method for detecting a person falling down by combining human body detection, skeleton point extraction and multi-dimensional judgment algorithms, which provides strong support for improving personnel safety and quality of life.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for intelligently identifying a person falling based on human skeleton points. Background Art

[0002] Falling is one of the main causes of many accidental injuries and accidents, especially for the elderly and people with limited mobility. Detecting people falling is crucial for people's life safety and health management. In the field of people falling detection, traditional methods mainly rely on sensor devices or video surveillance systems. Sensor devices are usually based on the principle of accelerometers, which detect sudden changes in gravity acceleration to identify falling events.

[0003] However, this method has many limitations: first, wearing sensor devices may cause inconvenience to users and affect their daily activities; second, the battery life of sensors is limited and needs to be replaced or recharged regularly, which increases maintenance costs; third, sensors may generate false alarms or missed alarms due to misoperation or equipment failure. Video surveillance systems use computer vision technology to analyze the behavior of people in surveillance videos to detect falls. Although this method does not require users to wear additional equipment, it also faces many challenges: traditional video analysis algorithms often rely on background modeling and motion detection, which are easily interfered in complex environments, resulting in a high misrecognition rate; at the same time, these algorithms usually require a lot of computing resources and are difficult to achieve real-time processing.

[0004] With the development of deep learning technology, computer vision-based methods have made significant progress, such as human posture estimation and behavior recognition, providing new ideas and methods for fall detection. However, existing deep learning methods still have some problems: first, most models require a large amount of labeled data for training, and the collection and labeling of fall data is a huge workload; second, deep learning models are usually computationally complex and have high hardware requirements, making it difficult to run in real time on edge devices; third, these models are often end-to-end black box models, lacking interpretability, and difficult to optimize and improve in a targeted manner. In addition, existing methods still have other challenges in fall detection: low accuracy. Behavior recognition methods based on images or videos are usually sensitive to lighting changes, occlusions, and complex backgrounds, resulting in low recognition accuracy; insufficient real-time performance. Some methods may require a large number of manually designed features or complex model structures, which is not conducive to scenarios with high real-time requirements; privacy protection issues. Directly using raw video data for analysis may involve the risk of privacy leakage; and insufficient robustness. Existing methods are often difficult to adapt to different camera angles, different human postures, and different environmental conditions. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is: how to solve the problems existing in the existing personnel fall detection technology, such as large data processing volume, poor system real-time performance, sensitivity to environmental changes, insufficient privacy protection, low fall detection accuracy, and poor interpretability.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a method for intelligently identifying a person falling based on human skeleton points, comprising: using a human body detection model to judge the surveillance camera video frame acquired in real time, if the human body detection model judges that there is a human body in the surveillance camera video frame, then intercepting the human body area image; extracting the position coordinates of key human body joints in the human body area image through a human body joint point detection model; based on the position coordinates of the key human body joints, calculating the distance, angle and relative position relationship between the key human body joints through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and comprehensively judging whether the human body is in a falling action.

[0008] As a preferred solution of the method for intelligently identifying a person falling based on human skeleton points described in the present invention, before judging the real-time acquired surveillance camera video frame using the human body detection model, the following steps are included:

[0009] Get the video stream of the surveillance camera in real time:

[0010] V = {v(t)|t = 1, 2, ..., T};

[0011] Where v(t) represents the video frame at time t, and T is the total number of frames in the video stream.

[0012] A single frame image is extracted from the video stream as a video frame to be processed, specifically, at time t, a video frame v(t) is extracted:

[0013] v(t) = ExtractFrame(V, t);

[0014] Among them, ExtractFrame is a function that extracts a single frame from a video stream.

[0015] As a preferred solution of the method for intelligently identifying a person falling based on human skeleton points described in the present invention, the method comprises: using a human body detection model to judge the real-time acquired surveillance camera video frame, if the human body detection model judges that there is a human body in the surveillance camera video frame, then intercepting the human body area image includes the following steps: inputting the video frame to be processed into a pre-trained human body detection model for multi-scale feature extraction, CNN detector processing and outputting the detection result; judging whether there is a human body in the video frame by the human body detection model; if a human body is detected, determining the position and size of the human body in the video frame; intercepting the area image containing the complete human body in the video frame according to the determined human body position and size; if no human body is detected, continuing to process the next frame of the image.

[0016] As a preferred solution of the method for intelligently identifying falling persons based on human skeleton points described in the present invention, the method of judging whether there is a human body in a video frame by using a human body detection model includes weighted combination of the detection results of all CNN detectors to obtain a comprehensive score:

[0017]

[0018] Among them, S(v(t),x,y,s) is the comprehensive score function; N is the number of detectors; w i is the weight of the i-th CNN detector; ∈ is a constant; x, y are the position coordinates of the candidate region containing the human body; s is the scale of the candidate region containing the human body; D i (v(t),x,y,s) is the detection result of CNN detector.

[0019] Determine whether there is a human body:

[0020] H(v(t))=I(max (x,y,s)∈R S(v(t),x,y,s)>0);

[0021] Among them, H(v(t)) is the judgment function; I(max (x,y,s)∈R S(v(t),x,y,s)>0) is the indicator function.

[0022] When the maximum comprehensive score max (x,y,s)∈ When RS(v(t),x,y,s)>0, the indicator function returns 1, so H(v(t))=1, and it is considered that a human body is detected; when the maximum comprehensive score max (x,y,s)∈R When S(v(t),x,y,s)≤0, the indicator function returns 0, so H(v(t))=0, indicating that no human body is detected.

[0023] If a human body is detected, determining the position and size of the human body in the video frame includes:

[0024] If H(v(t))=1, then:

[0025] (x * ,y * ,s * ) = argmax (x,y,s)∈R S(v(t),x,y,s);

[0026] Among them, (x * ,y * ,s * ) are the center coordinates and scale of the detected human body; argmax is the parameter that returns the maximum value of the function.

[0027] The method of intercepting a region image containing a complete human body in a video frame according to the determined human body position and size comprises the following steps: obtaining (x * ,y * ,s * ); * ,y * ,s * ) Combined with the cropping region expansion factor α, the Crop function is called to extract the region corresponding to (x * ,y * ,s * ) The corresponding area. The specific position and size of the intercepted area are determined by the Crop function; the intercepted result is assigned to I human (t), and obtain the cropped human body image; where,

[0028] I humax (t)=Crop(v(t),x * ,y * ,s * ,α);

[0029] Where α is the cropping area expansion factor; (x * ,y * ,s * ) are the center coordinates and scale of the detected human body; I human (t) is the cropped human image; Crop is the cropping function.

[0030] As a preferred solution of the method for intelligently identifying a person falling down based on human skeleton points of the present invention, wherein: extracting the position coordinates of key human joint points in the human region image by using a human joint point detection model comprises the following steps: human(t) Preprocessing; using multiple pre-trained human joint detection models to output multiple key human joint detection results; weighted fusion of multiple key human joint detection results; extracting coordinates of key human joints and calculating confidence; normalizing the coordinates of key human joints; introducing spatiotemporal consistency check, and adjusting the positions of key human joints of the current frame by speed constraints and motion smoothness. For the positions of key human joints with low confidence or not detected, estimation is performed by the positional relationship of adjacent joints; wherein the key human joints are at least one of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0031] As a preferred solution of the method for intelligently identifying a person falling based on human skeleton points described in the present invention, wherein: based on the position coordinates of the key human joint points, the distance, angle and relative position relationship between the key human joint points are calculated through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and a comprehensive judgment on whether the human body presents a falling action includes the following steps: based on the position coordinates of the key human joint points, the distance, angle and relative position relationship between the key human joint points are calculated through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and a comprehensive judgment on whether the human body presents a falling action includes: position judgment is: if multiple consecutive video frames meet: the y coordinate of the nose is greater than the average y coordinate of the two shoulders and the two hips, and the angle between the torso and the vertical line is greater than twice the first preset angle A, it is considered that the head is facing down and the human body is falling; the torso refers to the line from the midpoint of the shoulder to the midpoint of the hip; when the y coordinate of the nose is greater than the average y coordinate of the ankle, and the y coordinates of the eyes and ears are both greater than the y coordinate of the shoulder, and the duration exceeds the first preset threshold, it is considered that the head is facing down and the human body is falling.

[0032] The posture judgment is as follows: judge whether the distance between the left hip and the right hip is greater than one third of the maximum value between the left knee and the left hip, and the right knee and the right hip. If greater than, calculate the angle between the line connecting the higher and the lower of the left hip and the right hip and the horizontal line; if the angle is greater than twice the first preset angle A and less than four times the first preset angle A, it indicates that the human body is in a lying posture; calculate the angle between the left ankle and the left hip, calculate the angle between the right ankle and the right hip, and calculate the angle between the trunk and the ground. If the angle between the left ankle and the left hip and the angle between the right ankle and the right hip are both less than the first preset angle A, the angle between the trunk and the ground is less than half of the first preset angle A, the y coordinate of the nose is greater than the y coordinate of the left hip and greater than the y coordinate of the right hip, and the y coordinates of the left shoulder and the right shoulder are greater than the y coordinate of the left hip and greater than the y coordinate of the right hip, then it is considered that the human body is in a prone posture.

[0033] As a preferred embodiment of the method for intelligently identifying a person falling based on human skeleton points described in the present invention, the method comprises: based on the position coordinates of the key human joint points, the distance, angle and relative position relationship between the key human joint points are calculated through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and the comprehensive judgment of whether the human body presents a falling action also includes: body folding judgment is: calculating the sum of the distances from the shoulder to the elbow, the elbow to the hip, and the hip to the ankle on the left side of the human body to obtain a first body distance; calculating the sum of the distances from the shoulder to the elbow, the elbow to the hip, and the hip to the ankle on the right side of the human body to obtain a second body distance; comparing the first body distance and the second body distance and taking the maximum value thereof, calculating the actual height of the human body, calculating the angles between the left ankle, the left knee and the left hip, and calculating the angles between the right ankle, the right knee and the right hip. If both are less than 1.5 times the first preset angle A, and the actual height of the human body is less than 0.75 times the theoretical height, it is considered that the human body is in a body folding state.

[0034] To further solve the above technical problems, the present invention provides the following technical solutions: a system for intelligently identifying people falling based on human skeleton points, comprising: a human body detection and extraction module, used to use a human body detection model to judge the real-time acquired surveillance camera video frame, if the human body detection model determines that there is a human body in the surveillance camera video frame, then intercept the human body area image; a joint point detection module, used to extract the position coordinates of key human body joints in the human body area image through the human body joint point detection model; a fall judgment module, based on the position coordinates of the key human body joints, through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, calculates the distance, angle and relative position relationship between the key human body joints, and comprehensively judges whether the human body is in a falling action.

[0035] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the method for intelligently identifying a person falling based on human skeleton points as described above are implemented.

[0036] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method for intelligently identifying a person falling based on human skeleton points as described above are implemented.

[0037] Beneficial effects of the present invention: The present invention first uses a human body detection model to judge the surveillance camera video frames acquired in real time. If the presence of a human body is detected, the human body area image is captured. This step effectively reduces the amount of data that needs to be processed and improves the real-time performance of the system. Subsequently, the position coordinates of key human body joints in the human body area image are extracted through the human body joint point detection model. This method based on skeleton points is more robust than directly using the original image, and can effectively cope with lighting changes and partial occlusion. Finally, based on the position coordinates of key human body joints, the distance, angle and relative position relationship between key human body joints are calculated through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and a comprehensive judgment is made as to whether the human body is in a falling action. This multi-dimensional judgment method greatly improves the accuracy and reliability of fall detection. Compared with the prior art, the present invention has the following advantages: first, by using human skeleton points instead of original images for analysis, the privacy protection capability of the system is improved and the demand for computing resources is reduced; second, a combined tuning algorithm is used for multi-dimensional judgment to improve the accuracy and robustness of detection; third, the entire processing flow is reasonably designed to achieve real-time detection and meet the needs of actual applications; finally, the present invention has good interpretability and optimizability, which facilitates further improvement and adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0039] Figure 1 An overall flow chart of a method for intelligently identifying a person falling based on human skeleton points provided by an embodiment of the present invention.

[0040] Figure 2 A schematic diagram of human skeleton joint points of the method for intelligently identifying a person falling based on human skeleton points of the present invention.

[0041] Figure 3 A diagram of a computer device for the method of intelligently identifying a person falling based on human skeleton points of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] Example 1, reference Figure 1 and Figure 2 , which is an embodiment of the present invention, provides a method for intelligently identifying a person falling based on human skeleton points, comprising:

[0045] Among the related technologies, the technology for detecting human falls has problems to be solved, such as large data processing volume, poor system real-time performance, sensitivity to environmental changes, insufficient privacy protection, low accuracy of fall detection, and poor interpretability.

[0046] The present application provides a method that can effectively solve the above-mentioned problems. Next, a plurality of embodiments will be combined to explain in detail how to implement the method for intelligently identifying a person falling based on human skeleton points.

[0047] Figure 1 The overall flow chart of a method for intelligently identifying a person falling based on human skeleton points is shown, including:

[0048] S1: Use the human body detection model to judge the real-time acquired surveillance camera video frame. If the human body detection model determines that there is a human body in the surveillance camera video frame, the human body area image is intercepted;

[0049] S2: Extract the position coordinates of key human joints in the human region image through the human joint detection model;

[0050] S3: Based on the position coordinates of key human joints, a combined tuning algorithm of position judgment, posture judgment and body folding judgment is used to calculate the distance, angle and relative position relationship between key human joints, and comprehensively judge whether the human body is in a falling action.

[0051] Next, we will analyze each step in detail:

[0052] S1: Use the human body detection model to judge the surveillance camera video frame obtained in real time. If the human body detection model determines that there is a human body in the surveillance camera video frame, the human body area image is captured.

[0053] Specifically, the video stream of the surveillance camera is obtained in real time, and the video stream V is defined as continuous video frames:

[0054] V = {v(t)|t = 1, 2, ..., T};

[0055] Where v(t) represents the video frame at time t, and T is the total number of frames in the video stream.

[0056] Extract a single frame image from the video stream as the video frame to be processed. Specifically, extract the video frame at time t:

[0057] v(t) = ExtractFrame(V, t);

[0058] Among them, ExtractFrame is a function that extracts a single frame from a video stream.

[0059] Input the video frame to be processed into the pre-trained human detection model and perform the following operations:

[0060] Multi-scale feature extraction:

[0061]

[0062] Among them, x, y are the position coordinates of the candidate region containing the human body, indicating the center position of the candidate region; s is the scale of the candidate region containing the human body, indicating the size of the candidate region; PANet is a path aggregation network; FPN is a feature pyramid.

[0063] Each CNN detector processes the extracted multi-scale features:

[0064] CNN i (X) = SENet(ResNeXt(X));

[0065] Among them, X is the input feature; ResNeXt refers to the ResNeXt network; SENet refers to the Squeeze-and-Excitation network; i is the index of the CNN detector.

[0066] Each CNN detector outputs a detection result D i (v(t),x,y,s) is:

[0067]

[0068] Among them, D i is the output of the i-th CNN detector; CNNi is the i-th CNN model.

[0069] The detection results of all CNN detectors are weighted and combined to obtain a comprehensive score:

[0070]

[0071] Where S is the comprehensive score; S(v(t),x,y,s) is the comprehensive score function; N is the number of detectors; w i is the weight of the i-th detector; ∈ is a constant.

[0072] Preferably, the comprehensive score function is a continuous function, which is conducive to model training and optimization. The comprehensive score function combines the outputs of multiple detectors to improve the robustness of detection; by learning the weights w i , the importance of different detectors can be automatically adjusted; the normalization operation ensures the comparability of the scores and does not cause deviations due to changes in the number of CNN detectors. In actual use, S(v(t),x,y,s) of all possible positions and scales in the video frame is calculated, and then the position with the highest score is selected as the possible position of the human body.

[0073] If the highest score exceeds a certain threshold (0 in this embodiment), it is considered that a human body is detected, that is, the maximum comprehensive score is found in all positions and scales:

[0074]

[0075] Among them, R is the set of positions and scales of all candidate regions containing the human body; I(max (x,y,s)∈R S(v(t),x,y,s)>0) is an indicator function that returns 1 if the condition in the parentheses is true, otherwise it returns 0.

[0076] It should be noted that

[0077] H(v(t))=I(max (x,y,s)∈R S(v(t),x,y,s)>0);

[0078] When the maximum comprehensive score max (x,y,s)∈R When S(v(t),x,y,s)>0, the indicator function returns 1, so H(v(t))=1, and it is considered that a human body is detected.

[0079] When the maximum comprehensive score max (x,y,s)∈R When S(v(t),x,y,s)≤0, the indicator function returns 0, so H(v(t))=0, indicating that no human body is detected.

[0080] Among them, H(v(t)) is the judgment function.

[0081] If H(v(t))=1, then:

[0082] (x * ,y * ,s * ) = argmax (x,y,s)∈R S(v(t),x,y,s);

[0083] Among them, (x * ,y * ,s * ) are the center coordinates and scale of the detected human body; argmax is the parameter that returns the maximum value of the function.

[0084] According to the determined position and size of the human body, the regional image containing the complete human body is intercepted in the video frame, specifically:

[0085] I human (t)=Crop(v(t),x * ,y * ,s * ,α);

[0086] Among them, α is the cropping area expansion factor, which is used to ensure that the intercepted area contains the complete human body; (x * ,y * ,s * ) are the center coordinates and scale of the detected human body; I human (t) is the cropped human body image; Crop is the cropping function, which adopts the conventional technology in this field and will not be described here.

[0087] S2: Extract the position coordinates of key human joint points in the human area image through the human joint point detection model.

[0088] The joints of the human body include the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. For the corresponding serial number relationship of the joints of the human skeleton, see Figure 2 , as shown in Table 1,

[0089] Table 1 Human skeleton joint position table

[0090]

[0091]

[0092] Figure 2Schematic diagram of human skeleton joints. In this embodiment, the key human joints are at least one of nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0093] The human body region image I obtained in step S1 human (t) Perform preprocessing, including normalization and data augmentation:

[0094] I preprocessed (t)=Preprocess(I h uman(t));

[0095] Among them, I preprocessed (t) is the preprocessed human body area image; Preprocess is the preprocessing function, including: 1. Resizing: adjusting the image to a fixed size; 2. Color normalization: performing color space conversion and normalization on the image; 3. Data enhancement: applying random rotation, scaling, flipping, etc.

[0096] Use multiple pre-trained human joint detection models, such as HRNet, OpenPose, and AlphaPose:

[0097] J i (t) = KeypointDetector(I p reprocessed(t)),i=1,2,...,M;

[0098] Among them, M is the number of human joint point detection models used, KeypointDetector i represents the i-th human joint detection model; J i (t) is the key human joint point detection result of the i-th model at time t.

[0099] Weighted fusion of multiple key human joint detection results:

[0100] J f used(t)=FuseOutputs(J1(t),J2(t),...,J M (t));

[0101] Among them, the FuseOutputs function can adopt weighted average or more complex fusion strategies; f used(t) is the key human joint point detection result after weighted fusion.

[0102] Extract the coordinates of key human joint points and calculate the confidence:

[0103] From J f The coordinates and confidence of each key human joint point are extracted from used(t):

[0104] K=k1,k2,...,k n ,C=c1,c2,...,c n ;

[0105] Among them, k is the coordinate set of key human joint points; k1, k2, ..., k n Indicates k i The collection of k i =(x i ,y i ) are the coordinates of the key human joint points, i.e., the key point coordinates; C is the confidence set of the coordinates of the key human joint points; c1, c2, ..., c n Indicates c i The collection of c i is the confidence level corresponding to the coordinates of the key points; n is the number of key points, usually including the head, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles, etc.

[0106] Normalized keypoint coordinates:

[0107]

[0108] Among them, W and H are I preprocessed (t) width and height; is the normalized coordinate of the key human joint points; x i and i Represents the coordinates of the i-th key human joint point.

[0109] A spatiotemporal consistency check is introduced to reduce detection jitter and anomalies. Specifically, the ConsistencyCheck function is used to consider the joint position and confidence of the current frame and previous frames, and the position of key human joints in the current frame is adjusted through speed constraints and motion smoothness.

[0110] For the positions of key human joints with low confidence or not detected, the positions of the missing key human joints are estimated by the positional relationship of the adjacent joints, mainly using the CompleteKeypoints function, using the prior knowledge of the human skeletal structure to estimate the positions of the missing key human joints. In this embodiment, a confidence level lower than 0.5 can be considered as low confidence.

[0111] Generates a joint point data structure containing normalized coordinates, confidence, and additional metadata, caches the processing results, and supports asynchronous processing to improve system responsiveness.

[0112] Preferably, S2 improves the accuracy and robustness of joint point detection by introducing technologies such as multi-model integration, spatiotemporal consistency check, and missing joint point completion. At the same time, the overall processing efficiency is improved through optimizations such as preprocessing and asynchronous processing. This provides more reliable and complete human skeletal structure information for the S3 stage, which helps to improve the accuracy of subsequent fall action judgment.

[0113] S3: Based on the position coordinates of key human joints, a combined tuning algorithm of position judgment, posture judgment and body folding judgment is used to calculate the distance, angle and relative position relationship between key human joints, and comprehensively judge whether the human body is in a falling action.

[0114] Specifically, the first step is to determine the position. If multiple consecutive video frames meet the following conditions (in this embodiment, it can be set to 3 frames, or it can be set dynamically according to actual needs): the y coordinate of the nose is greater than the average y coordinate of the two shoulders and two hips, and the angle between the trunk and the vertical line is greater than twice the first preset angle A, then it is considered that the head is facing down and the human body is falling. When the y coordinate of the nose is greater than the average y coordinate of the ankle, and the y coordinates of the eyes and ears are greater than the y coordinate of the shoulders, and the duration exceeds the first preset threshold, then it is considered that the head is facing down and the human body is falling; the first preset threshold here can be 1 second, or it can be set according to actual conditions.

[0115] It should be noted that the trunk refers to the line from the midpoint of the shoulder to the midpoint of the hip. The first preset angle A is 30°; twice the first preset angle A, that is, 2A, is 60°; one and a half times the first preset angle A, that is, 1.5A, is 45°; four times the first preset angle A, that is, 4A, is 120°.

[0116] The second step is posture judgment, which mainly includes two types: falling posture and prone posture. The falling posture judgment includes the following contents:

[0117] Calculate the distance between the left hip and the right hip, and the distance between the left knee and the left hip, and the distance between the right knee and the right hip. If the distance between the left hip and the right hip is far, and the angle between the hip and the ground meets the characteristics of the posture of falling down, then determine whether the human body is in a posture of falling down. Specifically, determine whether the distance between the left hip and the right hip is greater than one-third of the maximum value between the left knee and the left hip, and the right knee and the right hip. If greater than, calculate the angle between the connecting line of the higher and the lower of the left hip and the right hip and the horizontal line; if the angle is greater than twice the first preset angle A and less than four times the first preset angle A, it means that the human body is in a state of falling down.

[0118] The judgment of the prone posture includes the following contents: judging whether the human body is prone on the ground by calculating the angle between the ankle and the hip, specifically calculating the angle between the left ankle and the left hip (left leg angle), calculating the angle between the right ankle and the right hip (right leg angle), and calculating the angle between the trunk (the line connecting the midpoint of the shoulder to the hip) and the ground. If the following conditions are met: the angles between the left and right hips and the ankles are both less than the first preset angle A, the angle between the trunk and the ground is less than half of the first preset angle A, the y-coordinate of the nose is greater than the y-coordinate of the left hip and greater than the y-coordinate of the right hip, and the y-coordinates of the left shoulder and the right shoulder are greater than the y-coordinate of the left hip and greater than the y-coordinate of the right hip, then the human body is considered to be in a prone posture.

[0119] Finally, the body folding judgment. By calculating the angles and distances of various parts of the body, it is judged whether the human body is in a state of body folding. If the angles between the left and right ankles and the knees and hips are small, and the ratio of the actual height to the theoretical height also meets certain conditions, the human body is considered to be in a state of falling, specifically, the sum of the distances from the shoulder to the elbow, elbow to the hip, and hip to the ankle on the left side of the human body is calculated to obtain the first body distance; the sum of the distances from the shoulder to the elbow, elbow to the hip, and hip to the ankle on the right side of the human body is calculated to obtain the second body distance; compare the first body distance and the second body distance and take the maximum value of the two, calculate the actual height of the human body, calculate the angles between the left ankle, left knee and left hip, and calculate the angles between the right ankle, right knee and right hip. If both are less than 1.5 times the first preset angle A, and the actual height of the human body is less than 0.75 times the theoretical height, the human body is considered to be in a state of body folding.

[0120] In summary, the present invention first uses a human body detection model to judge the surveillance camera video frame acquired in real time. If the presence of a human body is detected, the human body area image is intercepted. This step effectively reduces the amount of data that needs to be processed and improves the real-time performance of the system. Subsequently, the position coordinates of key human body joints in the human body area image are extracted through the human body joint point detection model. This method based on skeleton points is more robust than directly using the original image, and can effectively cope with lighting changes and partial occlusion. Finally, based on the position coordinates of key human body joints, the distance, angle and relative position relationship between key human body joints are calculated through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and a comprehensive judgment is made as to whether the human body is showing a fall action. This multi-dimensional judgment method greatly improves the accuracy and reliability of fall detection. Compared with the prior art, the present invention has the following advantages: first, by using human skeleton points instead of original images for analysis, the privacy protection capability of the system is improved and the demand for computing resources is reduced; second, a combined tuning algorithm is used for multi-dimensional judgment to improve the accuracy and robustness of detection; third, the entire processing flow is reasonably designed to achieve real-time detection and meet the needs of actual applications; finally, the present invention has good interpretability and optimizability, which facilitates further improvement and adjustment.

[0121] Embodiment 2 is an embodiment of the present invention, which provides a system for intelligently identifying human falls based on human skeleton points, including: a human body detection and extraction module, which is used to use a human body detection model to judge the real-time acquired surveillance camera video frames, and if it is judged that there is a human body in the video frame, the human body area image is captured; a joint point detection module, which is used to extract the position coordinates of key human body joints in the human body area image through the human body joint point detection model; a fall judgment module, which is used to calculate the distance, angle and relative position relationship between key human body joints based on the position coordinates of key human body joints through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and comprehensively judge whether the human body is in a fall action.

[0122] Embodiment 3 is an embodiment of the present invention, which is different from the previous embodiment in that:

[0123] like Figure 3 As shown, if the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer,

[0124] The server, or network device, etc.) performs all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0126] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0127] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0128] Example 4 is an embodiment of the present invention, which provides a method for intelligently identifying a person falling based on human skeleton points. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0129] First, we use a self-built dataset of 10,000 video clips covering a variety of scenarios and fall types.

[0130] Secondly, three current mainstream fall detection methods are selected for comparison, namely, the acceleration sensor-based method (Method A), the single CNN-based visual method (Method B), and the optical flow-based method (Method C).

[0131] Finally, the performance of each method is evaluated using accuracy, precision, recall, F1 score, average processing time, and privacy protection.

[0132] Table 2 Comparison with existing technology

[0133] method Accuracy Accuracy Recall F1 score Average processing time (ms) Privacy protection level This method 97.8％ 96.5％ 98.2％ 97.3％ 45 high Method A 89.2％ 87.6％ 90.1％ 88.8％ 10 Low Method B 93.5％ 92.8％ 93.9％ 93.3％ 80 middle MethodC 91.7％ 90.5％ 92.3％ 91.4％ 65 middle

[0134] As shown in Table 2, the present invention is significantly superior to the existing technical solutions in multiple key indicators, and has achieved the highest levels in terms of accuracy, precision, recall and F1 score. In particular, the F1 score is as high as 97.3%, far exceeding other methods. Although the average processing time (45ms) is slightly higher than the sensor-based method, it is still better than other visual methods and is sufficient to support real-time monitoring. More importantly, while ensuring high performance, the present invention provides extremely high privacy protection through skeleton point recognition technology. In addition, although not directly reflected in the table, the present invention also has excellent environmental adaptability and interpretability. Overall, the present invention shows obvious advantages in detection performance, real-time, privacy protection, environmental adaptability and interpretability, providing a more comprehensive and effective solution for the field of fall detection.

[0135] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for intelligently identifying a person falling down based on human skeleton points, characterized in that: include: The human body detection model is used to judge the real-time acquired surveillance camera video frame, and if the human body detection model determines that there is a human body in the surveillance camera video frame, an image of the human body area is captured; Extracting the position coordinates of key human joint points in the human body region image through a human joint point detection model; Based on the position coordinates of the key human joints, the distance, angle and relative position relationship between the key human joints are calculated through a combined optimization algorithm of position judgment, posture judgment and body folding judgment, and a comprehensive judgment is made as to whether the human body is in a falling action; Before using the human body detection model to judge the real-time acquired surveillance camera video frame, the following steps are included: Get the video stream of the surveillance camera in real time: V={v(t)|t=1,2,...,T}; Where v(t) represents the video frame at time t, and T is the total number of frames in the video stream; A single frame image is extracted from the video stream as a video frame to be processed, specifically, at time t, a video frame v(t) is extracted: v(t) = ExtractFrame(V, t); Among them, ExtractFrame is a function to extract a single frame from a video stream; The method of using the human body detection model to judge the real-time acquired surveillance camera video frame, if the human body detection model judges that there is a human body in the surveillance camera video frame, then intercepting the human body area image includes the following steps: Input the video frame to be processed into the pre-trained human detection model for multi-scale feature extraction, CNN detector processing and output of detection results; Use the human body detection model to determine whether there is a human body in the video frame; If a human body is detected, the position and size of the human body in the video frame are determined; According to the determined position and size of the human body, a region image containing the complete human body is intercepted in the video frame; If no human body is detected, continue processing the next frame of image; The determining whether there is a human body in the video frame by using a human body detection model includes: The detection results of all CNN detectors are weighted and combined to obtain a comprehensive score: Where S(v(t), x, y, s) is the comprehensive score function; N is the number of detectors; w i is the weight of the i-th CNN detector; is a constant; x, y are the position coordinates of the candidate region containing the human body; s is the scale of the candidate region containing the human body; D i (v(t), x, y, s) is the detection result of the CNN detector; determine whether there is a human body: H(v(t))=I(max (x，y,s)∈R S(v(t),x,y,s)>0); Among them, H(v(t)) is the judgment function; I(max (x，y,s)∈R S(v(t), x, y, s)>0) is the indicator function; When the maximum comprehensive score max (x,y,s)∈R When S(v(t), x, y, s)>0, the indicator function returns 1, so H(v(t))=1, and it is considered that a human body is detected; When the maximum comprehensive score max (x,y,s)∈R When S(v(t), x, y, s)≤0, the indicator function returns 0, so H(v(t))=0, indicating that no human body is detected; If a human body is detected, determining the position and size of the human body in the video frame includes: If H(v(t))=1, then: (x * ,y * ,s * )=argmax (x,y，s)∈R S(v(t),x,y,s); Among them, (x * ,y * ,s * ) is the center coordinate and scale of the detected human body; argmax is the parameter that returns the maximum value of the function; R is the set of positions and scales of all candidate regions containing the human body; The method of intercepting a region image containing a complete human body in a video frame according to the determined human body position and size comprises the following steps: Through the human body detection model, we can get (x * ,y * ,s * ); (x * ,y * ,s * ) Combined with the cropping region expansion factor α, the Crop function is called to extract the region corresponding to (x * ,y * ,s * ) corresponding area; The specific position and size of the intercepted area are determined by the Crop function; Assign the interception result to I human (t), and obtain the cropped human body image; where, I human (t)=Crop(v(t),x * ,y * ,s * ,α); Where α is the cropping area expansion factor; (x * ,y * ,s * ) are the center coordinates and scale of the detected human body; I human (t) is the cropped human image; Crop is the cropping function.

2. The method for intelligently identifying a person falling down based on human skeleton points as claimed in claim 1, characterized in that: Extracting the position coordinates of key human joint points in the human region image by using a human joint point detection model comprises the following steps: For the cropped human image I human (t) performing pre-processing; Use multiple pre-trained human joint point detection models to output multiple key human joint point detection results; Perform weighted fusion on the detection results of multiple key human joints; Extract the coordinates of key human joints and calculate the confidence level; Normalize the coordinates of key human joint points; Introduce spatiotemporal consistency checks to adjust the positions of key human joints in the current frame through speed constraints and motion smoothness; For the positions of key human joints with low confidence or not detected, they are estimated through the positional relationship of adjacent joints; Among them, the key human body joints are at least one of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

3. The method for intelligently identifying a person falling down based on human skeleton points as claimed in claim 2, characterized in that: Based on the position coordinates of the key human joints, the distance, angle and relative position relationship between the key human joints are calculated through a combined optimization algorithm of position judgment, posture judgment and body folding judgment, and the comprehensive judgment of whether the human body is in a falling action includes: The position is determined as: If multiple consecutive video frames satisfy the following conditions: the y coordinate of the nose is greater than the average y coordinate of the two shoulders and the two hips, and the angle between the trunk and the vertical line is greater than twice the first preset angle A, then it is considered that the head is facing downward and the human body is in a falling motion; the trunk refers to the line from the midpoint of the shoulder to the midpoint of the hip; When the y coordinate of the nose is greater than the average of the y coordinates of the ankles, and the y coordinates of the eyes and ears are greater than the y coordinates of the shoulders, and the duration exceeds a first preset threshold, it is considered that the head is facing downward and the human body is in a falling motion; Posture judgment is: Determine whether the distance between the left hip and the right hip is greater than one third of the maximum value between the left knee and the left hip, or between the right knee and the right hip; if so, calculate the angle between the line connecting the higher and the lower of the left hip and the right hip and the horizontal line; if the angle is greater than two times the first preset angle A and less than four times the first preset angle A, it indicates that the human body is in a lying position; Calculate the angle between the left ankle and the left hip, calculate the angle between the right ankle and the right hip, and calculate the angle between the torso and the ground. If the angle between the left ankle and the left hip and the angle between the right ankle and the right hip are both smaller than the first preset angle A, the angle between the torso and the ground is smaller than half of the first preset angle A, the y-coordinate of the nose is larger than the y-coordinate of the left hip and larger than the y-coordinate of the right hip, and the y-coordinates of the left shoulder and the right shoulder are larger than the y-coordinate of the left hip and larger than the y-coordinate of the right hip, then it is considered that the human body is in a prone position.

4. The method for intelligently identifying a person falling down based on human skeleton points as claimed in claim 3, characterized in that: Based on the position coordinates of the key human joints, the distance, angle and relative position relationship between the key human joints are calculated through a combined optimization algorithm of position judgment, posture judgment and body folding judgment, and the comprehensive judgment of whether the human body is in a falling action also includes: The body folding judgment is as follows: the sum of the distances from the shoulder to the elbow, the elbow to the hip, and the hip to the ankle on the left side of the human body is calculated to obtain a first body distance; the sum of the distances from the shoulder to the elbow, the elbow to the hip, and the hip to the ankle on the right side of the human body is calculated to obtain a second body distance; the first body distance and the second body distance are compared and the maximum value is taken, the actual height of the human body is calculated, the angles between the left ankle, the left knee and the left hip are calculated, and the angles between the right ankle, the right knee and the right hip are calculated. If both are less than 1.5 times the first preset angle A, and the actual height of the human body is less than 0.75 times the theoretical height, it is considered that the human body is in a state of body folding.

5. A system for intelligently identifying a person falling down based on human skeleton points as claimed in any one of claims 1 to 4, characterized in that: include: A human body detection and extraction module is used to use a human body detection model to judge the real-time acquired surveillance camera video frame, and if the human body detection model determines that there is a human body in the surveillance camera video frame, an image of the human body area is captured; A joint point detection module is used to extract the position coordinates of key human joint points in the human body region image through a human joint point detection model; The fall judgment module calculates the distance, angle and relative position relationship between the key human joint points based on the position coordinates of the key human joint points through a combined tuning algorithm of position judgment, posture judgment and body folding judgment, and comprehensively judges whether the human body is in a falling action.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method for intelligently identifying a person falling based on human skeleton points described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for intelligently identifying a person falling based on human skeleton points described in any one of claims 1 to 4 are implemented.

Images