Fall detection model training improvement method and fall detection method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN VIRTUAL CLUSTERS INFORMATION TECH
- Filing Date
- 2023-12-07
- Publication Date
- 2026-07-21
AI Technical Summary
Existing fall detection methods are insufficient in terms of detection accuracy and cost, and are prone to false detection, especially in complex environments, making it difficult to achieve efficient and accurate fall recognition.
A deep learning method based on skeletal keypoint sequences is adopted, using the graph convolutional network GCN and the TGN module of the temporal convolutional network TCN as the backbone network, and adding a spatiotemporal decoupling module and a feature refinement module to optimize the action classification model. The consistency loss is calculated using TP, FN and FP samples to improve the model's ability to distinguish similar poses.
While ensuring real-time detection, the accuracy of fall detection has been significantly improved, and false detections of postures such as sitting and squatting have been effectively reduced, achieving high-precision fall recognition.
Smart Images
Figure CN117671794B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fall detection technology, and in particular to an improved method for training a fall detection model and a fall detection method. Background Technology
[0002] In recent years, safety and health issues have received increasing attention, and emergency response to falls is an important safety measure. With the rapid development of computer technology and artificial intelligence, it has become possible to use deep learning tools to detect and warn of pedestrian falls, thereby protecting pedestrian safety.
[0003] Existing fall detection methods can be divided into two main directions: wearable sensor-based and vision-based methods. Wearable sensor-based methods mostly use devices such as accelerometers, pressure sensors, and microphones to calculate changes in the human body's position or acceleration, thereby determining whether a person has fallen. However, these methods are designed from the perspective of the person being detected, relying not only on the person's cooperation but also being very costly to implement in high-traffic environments such as construction sites or public places where falls are likely. Vision-based methods, on the other hand, use cameras to capture video images and analyze the video to detect fall behavior, avoiding the inconvenience of wearing devices and can be widely applied in various daily life and industrial scenarios. Among these, methods based on skeletal keypoint sequences and deep learning technology show significantly improved detection accuracy compared to traditional methods and are much less computationally expensive than methods based on RGB video, making them practically valuable. However, in actual fall detection, pedestrian movements are diverse, and many actions such as sitting, squatting, and lying down have skeletal keypoint sequences similar to those of a fall, easily leading to false positives. Summary of the Invention
[0004] The purpose of this invention is to address the technical problems existing in the background art by proposing an improved method for training a fall detection model, comprising the following steps:
[0005] Collect video data, which includes fall data as positive samples and non-fall data as negative samples;
[0006] Extract the frame sequence containing the human body;
[0007] Construct a sequence of key points for the human skeleton based on the frame sequence;
[0008] The action classification model is trained using the sequence of key points on the human skeleton. During the training phase, a spatiotemporal decoupling module and a feature refinement module are added to enhance the action classification model's ability to distinguish human postures.
[0009] Furthermore, the process of "training an action classification model using human skeletal keypoint sequences, and incorporating a spatiotemporal decoupling module and a feature refinement module during the training phase to enhance the action classification model's ability to discriminate human postures" also includes the following steps:
[0010] The action classification model uses the graph convolutional network GCN and the TGN module of the temporal convolutional network TCN as the backbone network;
[0011] Add the following modules after the TGN module: "Spatiotemporal Decoupling Module" and "Feature Refinement Module". The "Spatiotemporal Decoupling Module" and "Feature Refinement Module" are used to calculate the contrastive classification loss.
[0012] The contrastive classification losses calculated by the "spatiotemporal decoupling module" and the "feature refinement module" are weighted and summed to obtain the total contrastive classification loss, which is used to optimize the action classification model.
[0013] Furthermore, adding a "spatiotemporal decoupling module" and a "feature refinement module" after the TGN module, which are used to calculate the contrastive classification loss, also includes the following steps:
[0014] The human skeleton keypoint sequence is input into the TGN module to obtain a feature map of shape N*C*T*V, where N represents the number of sequences, C represents the number of channels, T represents the number of frames contained in each sequence, and V represents the number of keypoints contained in each skeleton.
[0015] The feature map is decomposed into an N*C*1*V "temporal component" and an N*C*T*1 "spatial component". Temporal pooling and convolution modules are used to represent the "temporal component" and "spatial component" as one-dimensional features of the same length, denoted as F. te i and F sp i , where i represents the sequence number of the current spatiotemporal decoupling module;
[0016] F is calculated separately using the "feature refinement module". te i and F sp i The feature similarity is used to obtain the predicted category results. The predicted category results are compared with the true results. The samples labeled as falling and predicted as falling are denoted as TP samples, the samples labeled as falling but incorrectly predicted as not falling are denoted as FN samples, and the samples labeled as not falling but incorrectly predicted as falling are denoted as FP samples.
[0017] Using the TP sample set as the anchor sample set and the sample set easily identifiable by the model, and the FN and FP samples as the difficult samples that are hard to identify by the model, the centers of the three sample sets are calculated. The consistency loss L1 is calculated based on the center distance between the TP and FN samples and the center distance between the TP and FP samples. i L2 i The results are summed to obtain the contrast classification loss L. con i .
[0018] Furthermore, the step of "weightedly summing the contrastive classification losses calculated by the 'empty decoupling module' and the 'feature refinement module' to obtain the total contrastive classification loss, which is then used to optimize the action classification model" also includes the following steps:
[0019] The contrastive classification loss at each stage is calculated, and the total contrastive classification loss L is obtained by weighting the results using a multiplicative matrix. con ;
[0020] The total contrast classification loss L con The total loss is obtained by adding the classification loss of the original model before adding the "empty decoupling module" and the "feature refinement module".
[0021] The total loss is trained using an action classification model to repeatedly enhance the use of spatiotemporal information of difficult samples with fuzzy recognition, thereby improving the action classification model's ability to distinguish human postures.
[0022] Furthermore, before constructing the skeletal keypoint sequence, the following steps are also included:
[0023] Based on the YOLOv5s algorithm, the bounding boxes of human bodies in the image are labeled, and a human detection model is trained.
[0024] Human Keypoint Detection Model: Lite-HRNet Model Calculates Human Skeletal Keypoints.
[0025] Furthermore, key points of the human skeleton include the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.
[0026] Calculate the center points of the nose, left eye, right eye, left ear, and right ear, and use them as head key points to replace the original key points;
[0027] Remove the original key point connection lines for the left and right shoulders and the left and right hips, and add key point connection lines for the head and shoulders and the head and hips.
[0028] Furthermore, the present invention also provides a fall detection method, which uses an "improved fall detection model training method" of the present invention. The detection method includes the following steps:
[0029] Collect surveillance video data and construct a video frame sequence;
[0030] Human detection based on human detection models;
[0031] Keypoint retrieval is performed based on a human keypoint detection model, and a sequence of human skeletal keypoints is constructed.
[0032] Action classification is performed based on an action classification model to determine whether a person has fallen.
[0033] Furthermore, "collecting surveillance video data and constructing a video frame sequence" also includes the following steps:
[0034] Set the length L of the video frame sequence;
[0035] Detect pedestrians in video frames, start counting frames from the time a pedestrian is detected, construct the first video frame sequence V0 starting from frame L, and denote the sequence from frame i to frame L+i as Vi. If an interruption occurs in the middle, recalculate.
[0036] Furthermore, "retrieving key points based on a human key point detection model and constructing a skeletal key point sequence" also includes the following steps:
[0037] Based on the Lite-HRNet model, the coordinates of human key points are calculated for each of the L pedestrian bounding boxes in sequence Vi.
[0038] Using the obtained L sets of key point coordinates, construct the skeleton key point sequence Si;
[0039] Input the skeletal keypoint sequence Si into the action classification model.
[0040] Furthermore, it also includes the following steps:
[0041] If a fall occurs, a fall alarm will be triggered.
[0042] If no fall occurs, continue building the video frame sequence for fall detection.
[0043] Compared with existing technologies, this invention has the following beneficial technical effects: This invention utilizes "spatiotemporal decoupling" and a "feature refinement module" to enhance the deep learning model's learning of spatiotemporal information and fuzzy samples, improving the model's ability to distinguish between fall sequences and fall-like non-fall sequences. Finally, by combining a human detection model, a keypoint detection model, and an action classification model, end-to-end intelligent analysis of surveillance videos is performed, improving the accuracy of pedestrian fall detection in real-world scenarios without changing computational costs. Attached Figure Description
[0044] Figure 1 A fall detection process based on key point sequences of the human skeleton;
[0045] Figure 2 Improved comparison diagram of key points of the human skeleton;
[0046] Figure 3 An improved diagram of the action classification model structure;
[0047] Figure 4 This is a complete flowchart of the improved training method for the fall detection model and the fall detection method of the present invention. Detailed Implementation
[0048] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0049] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more features. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.
[0050] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or a specific connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0051] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0052] It is understandable that traditional fall detection schemes based on static images and centroid and acceleration calculations are easily affected by perspective, lighting, etc. Schemes based on general deep learning models have high computational costs and are difficult to achieve real-time detection. Therefore, this invention selects a deep learning method based on skeletal keypoint sequences to achieve dynamic video detection. In order to address the problem of false detection of similar postures in real-world fall detection scenarios, this invention proposes an improved method for training fall detection models.
[0053] Specifically, such as Figures 1-4 As shown, this invention uses a "spatiotemporal decoupling module" and a "feature refinement module" to enhance the detection model's ability to distinguish similar postures, thereby improving the effectiveness of fall detection in practical applications. While ensuring real-time detection, it effectively reduces false detections of postures such as sitting, squatting, and lying down, achieving high detection accuracy. Specifically, the detection model training process is as follows:
[0054] 1. Collect video data, which includes fall data as positive samples and non-fall data as negative samples. Non-fall data includes various actions in daily life, such as standing, sitting, squatting, walking, running, etc.
[0055] 2. Extract the frame sequence: First, extract valid frames based on the actual events, and then use sampling and augmentation strategies to construct the frame sequence.
[0056] 3. Mark the bounding boxes of the human figures in the image;
[0057] 4. Train the human detection model;
[0058] 5. Calculate key points of the human skeleton based on the key point detection model;
[0059] 6. Construct a sequence of skeletal key points;
[0060] 7. Train the action classification model.
[0061] In this embodiment, as Figure 2 As shown, training the human detection model and calculating human skeletal key points can be achieved using the YOLOv5s algorithm and the Lite-HRNet model, respectively. The Lite-HRNet model can calculate human skeletal key points, including 17 key points: nose, eyes (left and right), ears (left and right), shoulders (left and right), elbows (left and right), wrists (left and right), hips (left and right), knees (left and right), and ankles (left and right). Considering the different degrees of influence of falls on the position changes of each key point, as well as the interaction between skeletal key points, this invention reconstructs the skeletal key point map. Specifically, the center points of the nose, eyes (left and right), and ears (left and right) are calculated as head key points and replaced with the original key points. The original lines connecting the two shoulders and two hips key points are removed, and new lines connecting the head and shoulders and the head and hips key points are added.
[0062] Furthermore, the action classification model can use TGN, which combines Graph Convolutional Network (GCN) and Temporal Convolutional Network (TCN), as the backbone network. After the TGN module, a "spatiotemporal decoupling module" and a "feature refinement module" are added. The "spatiotemporal decoupling module" and the "feature refinement module" are used to calculate the contrastive classification loss. The contrastive classification losses calculated by the "spatiotemporal decoupling module" and the "feature refinement module" are then weighted and summed to obtain the total contrastive classification loss, which is used to optimize the action classification model.
[0063] This is understandable, as the newly added module operates during the model's training phase and is masked during the detection phase, thus not affecting real-time performance in real-world applications.
[0064] Specifically, such as Figure 3 As shown, in this embodiment, the skeletal keypoint sequence can be input into the TGN module. After obtaining a feature map of shape N*C*T*V through the TGN module, the feature map is decomposed into a temporal component of N*C*1*V and a spatial component of N*C*T*1, where N represents the number of sequences, C represents the number of channels, and in the detection process of this embodiment, two channels, x and y coordinates, are used to correspond to the temporal component and the spatial component, respectively. T represents the number of frames contained in each sequence, and V represents the number of keypoints contained in each skeleton. It can be understood that in this embodiment, the temporal component can include the position of the same skeletal keypoint in different frames, and the spatial component can include the position of all skeletal keypoints in the same frame. Further, temporal / spatial pooling and convolutional network layers are used to represent the temporal / spatial components as one-dimensional features of the same length, denoted as F. te i and F sp i In this embodiment, multiple "spatiotemporal decoupling modules" and corresponding "feature refinement modules" may be included to achieve parallel data processing and improve processing efficiency. It can be understood that F te i and F sp i In this context, 'i' represents the sequence number of the current spatiotemporal decoupling module.
[0065] Furthermore, by calculating feature similarity to predict the action category separately, the prediction results are compared with the actual results. Samples labeled as "falling" and predicted as "falling" are denoted as TP samples, samples labeled as "falling" but incorrectly predicted as "not falling" are denoted as FN samples, and samples labeled as "not falling" but incorrectly predicted as "falling" are denoted as FP samples.
[0066] The TP sample set is considered the anchor sample set, which is the sample set that the model can easily identify. The FN and FP samples are the difficult samples that the model has difficulty identifying. The centers of the three sample sets are calculated, and the consistency loss L1 is calculated based on the center distance between the TP and FN samples and the center distance between the TP and FP samples. i L2 i The sum of these values yields the contrast classification loss L. con i Finally, the contrastive classification loss for each stage is calculated, and the total contrastive classification loss L is obtained by weighting the results using the weight matrix. con Then, the total loss is added to the original model's classification loss (i.e., the loss before adding the "spatiotemporal decoupling module" and "feature refinement module"). During the optimization process based on this loss, the action classification model can repeatedly enhance its utilization of spatiotemporal information for difficult-to-identify samples with ambiguity, ultimately learning discriminative features.
[0067] Furthermore, this embodiment also provides a fall detection method, which uses an improved fall detection model training method of the present invention. The detection method first detects pedestrians in the surveillance video based on the YOLOv5s algorithm, and then uses the Lite-HRNet model to perform key point detection on the pedestrians in the current frame. The human skeleton is constructed using these key points, and then the skeletal key point map of each pedestrian in consecutive video frames is collected to form the skeletal key point sequence of the pedestrian. This sequence is used as the input of the improved action classification model of this patent, and the model calculates and predicts whether the pedestrian has fallen.
[0068] The fall detection method specifically includes the following steps:
[0069] 1. Collect surveillance video data;
[0070] 2. Construct a video frame sequence;
[0071] 3. Perform human detection based on the human detection model obtained during the training process;
[0072] 4. Key point detection based on key point detection model;
[0073] 5. Construct a sequence of skeletal key points;
[0074] 6. Perform action classification based on the action classification model obtained during the training process;
[0075] 7. Determine whether the pedestrian has fallen based on the model classification results. If a fall has occurred, output a fall alarm; otherwise, continue constructing a video frame sequence for fall detection.
[0076] Specifically, for constructing the video frame sequence and the skeletal keypoint sequence, the length L of the video frame sequence / skeletal keypoint sequence can be set. Pedestrians are detected in the video frames, and the frame number is counted from the detected pedestrian. The first video frame sequence V0 is constructed starting from the Lth frame, and the sequence from the i-th frame to the (L+i)-th frame is denoted as Vi. If an interruption occurs, the calculation is restarted. For the L pedestrian bounding boxes in sequence Vi, the keypoint coordinates are calculated respectively. Considering the detection accuracy, generalization ability, and computational cost of the human detection model, the YOLOv5s algorithm is preferred to predict the human bounding boxes. Then, the obtained L sets of keypoint coordinates are used to construct the skeletal keypoint sequence Si. The skeletal keypoint sequence Si is then used as input, and the action classification model predicts whether the current sequence corresponds to a fallen state or a non-fallen state.
[0077] As a preferred option, such as Figure 2 As shown, considering the accuracy, stability, and computational cost of the keypoint detection model, the Lite-HRNet model is preferred for calculating human keypoints, including 17 keypoints: nose, eyes (left and right), ears (left and right), shoulders (left and right), elbows (left and right), wrists (left and right), hips (left and right), knees (left and right), and ankles (left and right). Considering the different degrees of impact of falls on the position changes of each keypoint, as well as the interaction between skeletal keypoints, this invention reconstructs the skeletal keypoint map. Specifically, the center points of the nose, eyes (left and right), and ears (left and right) are calculated as head keypoints and replaced with the original keypoints. The original lines connecting the two shoulders and two hips keypoints are removed, and new lines connecting the head and shoulders and the head and hips keypoints are added.
[0078] In summary, this invention utilizes a "spatiotemporal decoupling and feature refinement module" to enhance the deep learning model's learning of spatiotemporal information and fuzzy samples, thereby improving the model's ability to distinguish between fall sequences and fall-like non-fall sequences. Finally, by combining a human detection model, a keypoint detection model, and an action classification model, end-to-end intelligent analysis of surveillance video is performed, improving the accuracy of pedestrian fall detection in real-world scenarios without changing computational costs.
[0079] The above descriptions provide one or more embodiments in conjunction with specific details, but do not imply that the specific implementation of the present invention is limited to these descriptions. Any methods or structures that are similar to or identical to those of the present invention, or any technical deductions or substitutions made based on the concept of the present invention, should be considered within the scope of protection of the present invention.
Claims
1. An improved method for training a fall detection model, characterized in that, Includes the following steps: Collect video data, which includes fall data as positive samples and non-fall data as negative samples; Extract the frame sequence containing the human body; Construct a sequence of key points for the human skeleton based on the frame sequence; The action classification model is trained by using the sequence of key points of the human skeleton. During the training phase, a spatiotemporal decoupling module and a feature refinement module are added to the action classification model to enhance the model's ability to distinguish human postures. The phrase "training an action classification model using human skeletal key point sequences, with a spatiotemporal decoupling module and a feature refinement module added during the training phase to enhance the action classification model's ability to discriminate human postures" further includes the following steps: The action classification model uses the graph convolutional network GCN and the TGN module of the temporal convolutional network TCN as the backbone network; Add the following modules after the TGN module: "Spatiotemporal Decoupling Module" and "Feature Refinement Module". The "Spatiotemporal Decoupling Module" and "Feature Refinement Module" are used to calculate the contrastive classification loss. The contrastive classification losses calculated by the "spatiotemporal decoupling module" and the "feature refinement module" are weighted and summed to obtain the total contrastive classification loss, which is used to optimize the action classification model. The phrase "adding a 'spatiotemporal decoupling module' and a 'feature refinement module' after the TGN module, the 'spatiotemporal decoupling module' and the 'feature refinement module' being used to calculate the contrastive classification loss" further includes the following steps: The human skeleton keypoint sequence is input into the TGN module to obtain a feature map of shape N*C*T*V, where N represents the number of sequences, C represents the number of channels, T represents the number of frames contained in each sequence, and V represents the number of keypoints contained in each skeleton. The feature map is decomposed into a "temporal component" of N*C*1*V and a "spatial component" of N*C*T*1, and the "temporal component" and the "spatial component" are respectively represented into one-dimensional features of the same length by using a "temporal pooling and convolution module" and a "spatial pooling and convolution module", and are respectively denoted as F te i and F sp i wherein i represents the serial number of the current space-time decoupling module. F is calculated separately using the "feature refinement module". te i and F sp i The feature similarity is used to obtain the predicted category results. The predicted category results are compared with the true results. The samples labeled as falling and predicted as falling are denoted as TP samples, the samples labeled as falling but incorrectly predicted as not falling are denoted as FN samples, and the samples labeled as not falling but incorrectly predicted as falling are denoted as FP samples. Using the TP sample set as the anchor sample set and the sample set easily identifiable by the model, and the FN and FP samples as the difficult samples that are hard to identify by the model, the centers of the three sample sets are calculated. The consistency loss L1 is calculated based on the center distance between the TP and FN samples and the center distance between the TP and FP samples. i L2 i The results are summed to obtain the contrast classification loss L. con i ; The step of "weightedly summing the contrastive classification losses calculated by the "empty decoupling module" and the "feature refinement module" to obtain the total contrastive classification loss for optimizing the action classification model" further includes the following steps: The contrastive classification loss at each stage is calculated, and the total contrastive classification loss L is obtained by weighting the results using a multiplicative matrix. con ; The total contrast classification loss L con The total loss is obtained by adding the classification loss of the original model before adding the "empty decoupling module" and the "feature refinement module". The total loss is trained using an action classification model to repeatedly enhance the use of spatiotemporal information of difficult samples with fuzzy recognition, thereby improving the action classification model's ability to distinguish human postures.
2. The improved method for training a fall detection model according to claim 1, characterized in that, Before constructing the skeletal keypoint sequence, the following steps are also included: Based on the YOLOv5s algorithm, the bounding boxes of human bodies in the image are labeled, and a human detection model is trained. Human Keypoint Detection Model: Lite-HRNet Model Calculates Human Skeletal Keypoints.
3. The improved method for training a fall detection model according to claim 2, characterized in that, Key points of the human skeleton include the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. Calculate the center points of the nose, left eye, right eye, left ear, and right ear, and use them as head key points to replace the original key points; Remove the original key point connection lines for the left and right shoulders and the left and right hips, and add key point connection lines for the head and shoulders and the head and hips.
4. A fall detection method, characterized in that, The "improved method for training a fall detection model" as described in any one of claims 1-3 above further includes the following steps before constructing the skeletal keypoint sequence: Based on the YOLOv5s algorithm, the bounding boxes of human bodies in the image are labeled, and a human detection model is trained. Based on the human keypoint detection model: Lite-HRNet model calculates human skeletal keypoints; The detection method includes the following steps: Collect surveillance video data and construct a video frame sequence; Human detection based on human detection models; Keypoint retrieval is performed based on a human keypoint detection model, and a sequence of human skeletal keypoints is constructed. Action classification is performed based on an action classification model to determine whether a person has fallen.
5. A fall detection method according to claim 4, characterized in that, The "collecting surveillance video data and constructing a video frame sequence" also includes the following steps: Set the length L of the video frame sequence; Detect pedestrians in video frames, start counting frames from the time a pedestrian is detected, construct the first video frame sequence V0 starting from frame L, and denote the sequence from frame i to frame L+i as Vi. If an interruption occurs in the middle, recalculate.
6. The fall detection method according to claim 5, characterized in that, The step of "retrieving key points based on the human key point detection model and constructing a sequence of human skeletal key points" further includes the following steps: Based on the Lite-HRNet model, the coordinates of human key points are calculated for each of the L pedestrian bounding boxes in sequence Vi. Using the obtained L sets of key point coordinates, construct the skeleton key point sequence Si; Input the skeletal keypoint sequence Si into the action classification model.
7. A fall detection method according to claim 4, characterized in that, It also includes the following steps: If a fall occurs, a fall alarm will be triggered. If no fall occurs, continue building the video frame sequence for fall detection.