Double-stage tumble detection method fusing key point detection and space-time diagram convolution

Abstract

The invention discloses a two-stage tumble detection method fusing key point detection and space-time diagram convolution, and is applied to the technical field of computer vision. Comprising the following steps: designing a YOLO-PoseLiteNet model used for extracting a human skeleton key point sequence and an ST-GCN model used for detecting a tumble behavior; acquiring a data set for human body key point detection and marking human body behaviors; a YOLO-PoseLiteNet model and an ST-GCN model are trained, and the YOLO-PoseLiteNet model and Obtaining video data of a detection target, and inputting the video data into the YOLO-PoseLiteNet model to obtain a human skeleton key point sequence; and inputting the human skeleton key point sequence into the ST-GCN model to obtain a tumble detection result. The advantages of YOLO-PoseLiteNet and ST-GCN are combined, and more reliable technical support is provided for the fields of health monitoring of old people, smart home and the like.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a two-stage fall detection method that integrates key point detection and spatiotemporal graph convolution. Background Technology

[0002] With the accelerating pace of aging and the increasing size of the elderly population, the phenomena of "empty nest syndrome" and "population cliff" are becoming increasingly apparent. Many elderly people face numerous risks and hidden dangers in their lives due to declining physical fitness, insufficient balance, first aid skills, and reaction time, as well as complex environments and chronic illnesses. If any inconvenience experienced by the elderly is not effectively addressed and promptly assisted, even minor accidents can have a "magnifying effect," leading to serious consequences and even endangering their lives. Falls are a significant threat to the health and safety of the elderly in their daily lives, not only easily causing serious physical injuries but also triggering other more serious complications.

[0003] Deep learning-based fall detection methods are currently mainstream. Deep learning algorithms construct deep networks, learn from data, and use backpropagation to adjust network parameters and correct errors, ultimately obtaining the optimal network output. Falling is a complex dynamic behavior with obvious spatiotemporal characteristics. The human body's joints can naturally be viewed as a graph, composed of multiple joints and their connecting edges. Each joint can be considered a node in the graph, while the connections between joints are its edges. As the human body moves, the positions and relationships of the joints in the skeletal structure change over time. Especially in fall detection scenarios, the displacement of joints and the relationships between them can capture subtle changes in human posture. For example, during a fall, key joints (such as the head, hips, and knees) undergo drastic changes. However, existing fall detection algorithms do not capture the temporal changes and dynamic changes in spatial relationships of joints. Therefore, how to provide a two-stage fall detection method that integrates keypoint detection and spatiotemporal graph convolution is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a two-stage fall detection method that integrates keypoint detection and spatiotemporal graph convolution. In the first stage, the YOLO-PoseLiteNet keypoint detection algorithm is used to accurately detect the joints of the human body. In the second stage, the spatiotemporal feature analysis of the keypoint sequence is performed through a spatiotemporal graph convolutional network.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution includes the following steps:

[0007] S1. Design a YOLO-PoseLiteNet model for extracting key point sequences of human skeletons;

[0008] S2. Design an ST-GCN model for detecting fall behavior;

[0009] S3. Obtain the dataset for human keypoint detection and label human behaviors;

[0010] S4. Train the YOLO-PoseLiteNet model and the ST-GCN model based on the labeled dataset;

[0011] S5. Obtain video data of the detected target, input it into the trained YOLO-PoseLiteNet model, and obtain the human skeleton key point sequence;

[0012] S6. Input the sequence of key points of the human skeleton into the trained ST-GCN model to obtain the fall detection results.

[0013] Optionally, S1 is as follows:

[0014] S11. Design a lightweight detection head LiteDetect, improve the performance of detection head localization and classification by GroupNorm, and improve the detection head's detail capture capability and detection accuracy by using dynamic full-dimensional convolution ODConv;

[0015] S12. Design the RFCBAM-Net backbone network and introduce receptive field attention convolution RFCBAMConv to focus on the spatial features of the receptive field.

[0016] S13. Design a neck network with a multi-branch assisted FPN structure, including shallow assisted fusion, which combines the outputs of the backbone network and the neck network, retains the best level of shallow information, and then outputs diverse gradient information through high-level assisted fusion of the neck network.

[0017] Optionally, S11 specifically refers to:

[0018] In the YOLOv8-Pose detection head, a shared convolutional structure is introduced, allowing YOLOv8-Pose to share some convolutional layers for bounding box prediction and class prediction at three different scales. This reduces the computational burden on the network and improves efficiency. GroupNorm normalization is used instead of BN normalization, and full-dimensional dynamic convolution ODConv is introduced. GroupNorm normalization is also used to further improve the detail capture capability of the detection head and improve detection accuracy. The lightweight detection head LiteDetect reduces the number of 3×3 convolutions by using a shared convolutional structure, thereby reducing computational complexity and parameter count while maintaining the stability of detection accuracy.

[0019] Optionally, S12 specifically refers to:

[0020] The ordinary convolutions in the backbone network and the C2f module are improved into receptive field attention convolutions RFCBAMConv, and the improved C2f module is named C2f-RFCBAMConv. By introducing the receptive field attention mechanism, the receptive field attention convolution RFCBAMConv can enhance the ability to express global information while capturing local features, thereby improving the effectiveness of feature extraction and the overall performance of the network. Three residual connections are designed in the network, and the residual learning mechanism is used to alleviate the gradient vanishing problem.

[0021] Optionally, S13 specifically refers to:

[0022] The backbone network consists of four stages: P2, P3, P4, and P5. The outputs of stages P2, P3, and P4 are used to adjust the feature dimensions and sizes through convolution operations to achieve feature fusion. In the bottom-up path, the shallow auxiliary fusion module extracts multi-scale features from the backbone network and performs preliminary fusion in the shallow layers of the neck network, combining the outputs of the backbone network and the neck network to ensure optimal preservation of shallow features. In the top-down path, the high-level auxiliary fusion module collects gradient information from each layer through a dense connection mechanism. Both the shallow auxiliary fusion module and the high-level auxiliary fusion module use C2f for feature extraction.

[0023] Optionally, the ST-GCN model includes a BN normalization layer, stacked ST-GCN modules, a global average pooling layer, and a fully connected layer connected in sequence. The ST-GCN module consists of an attention layer, a GCN spatial graph convolution, and a TCN temporal graph convolution. The attention layer is used to generate an attention matrix that adjusts the weights of the edges in the graph structure. By dynamically adjusting the weights of the edges in the graph, the network can better capture the feature relationships in the spatial and temporal dimensions and enhance its sensitivity to action details. The GCN spatial convolution performs convolution operations on the spatial dimension of the input to extract and aggregate the relational features between joints. In the initial input, the spatial dimension features are 3, representing the three-dimensional coordinate information and confidence of the joints. The TCN temporal convolution performs convolution operations in the temporal dimension to model the dynamic features of the action at different times.

[0024] Optionally, the ST-GCN model employs a spatial configuration strategy to group the label mappings, dividing the nodes within the domain into three subsets based on their spatial positional relationships: root nodes, centripetal nodes, and centrifugal nodes. The root node represents the node currently undergoing convolution, the centripetal node represents the node closer to the center of gravity of the skeleton, and the centrifugal node represents the node farther away from the center of gravity of the skeleton. This expands the domain span D of the spatial graph convolution from 1 to 2, enabling each node to establish connections with more distant related nodes, enhancing the ST-GCN model's global perception of actions, and helping to capture cross-node features of fall behavior.

[0025] Optionally, after model training is completed in S4, model performance is evaluated using the average precision (AP) and average recall (AR) metrics, and the similarity (OKS) between the ground truth keypoints and the predicted keypoints is set:

[0026]

[0027] In the formula, d i v represents the Euclidean distance between the i-th labeled keypoint and the predicted keypoint. i The visible marker of the i-th labeled key point is represented by sk, where sk is the target scale and δ is the standard error.

[0028] Top1 and Top5 metrics are used to measure the classification accuracy of the model. Top1 assesses whether the model's predicted most likely class matches the actual class. If the model's highest probability class matches the true class, the Top1 accuracy is 1; otherwise, it is 0. The Top5 metric measures whether the model's predicted top five likely classes include the true class. If the actual class appears in these five classes, the Top5 accuracy is 1; otherwise, it is 0. The formulas for calculating both metrics are as follows:

[0029]

[0030] In the formula, Indicates the true category of the i-th behavior. This represents the predicted category for the i-th action. For the category whose predicted value ranks first in the i-th behavior prediction, Let N represent the top 5 categories in the prediction of the i-th behavior, and let N represent the total number of behaviors.

[0031] As can be seen from the above technical solution, compared with the prior art, the present invention provides a two-stage fall detection method that integrates key point detection and spatiotemporal graph convolution, which has the following beneficial effects: The present invention improves the YOLO-PoseLiteNet network and ST-GCN network for fall detection by designing and improving them. Combining the advantages of YOLO-PoseLiteNet and ST-GCN, the detection accuracy of each behavior category is improved, and the human posture can be accurately identified, providing more reliable technical support for elderly health monitoring, smart homes and other fields; the classification is performed by the spatial configuration relationship of the human skeleton, which enables the model to more accurately identify fall behavior. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0033] Figure 1 This is a flowchart of the two-stage fall detection method of the present invention;

[0034] Figure 2 This is a schematic diagram of the YOLO-PoseLiteNet model structure of the present invention;

[0035] Figure 3 This is a schematic diagram of the fine dynamic full-dimensional convolution ODConv structure of the present invention;

[0036] Figure 4 This is a schematic diagram of the LiteDetect detection head structure of the present invention;

[0037] Figure 5 This is a schematic diagram of the receptive field attention convolution RFCBAMConv structure in an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of the improved backbone network in an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram of the multi-branch auxiliary FPN structure of the present invention;

[0040] Figure 8 This is a representation of the spatiotemporal graph convolutional skeleton sequence in an embodiment of the present invention;

[0041] Figure 9 This is a schematic diagram of the temporal convolution operation in an embodiment of the present invention;

[0042] Figure 10 This is a flowchart of the overall algorithm of the ST-GCN model of the present invention;

[0043] Figure 11 This is a schematic diagram of the partitioning strategy in an embodiment of the present invention;

[0044] Figure 12 This is a key point diagram of the COCO dataset in an embodiment of the present invention;

[0045] Figure 13 This is a key point map of the NTU RGB+D dataset in an embodiment of the present invention;

[0046] Figure 14 This is a schematic diagram of the experimental results of the configuration strategy improvement in this embodiment of the invention;

[0047] Figure 15 This is a schematic diagram of the experimental results of the two-stage fall detection method in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] This invention discloses a two-stage fall detection method that integrates keypoint detection and spatiotemporal graph convolution, such as... Figure 1 As shown, it includes the following steps:

[0050] S1. Design a YOLO-PoseLiteNet model for extracting key point sequences of human skeletons;

[0051] The YOLO-PoseLiteNet model structure is as follows: Figure 2 As shown;

[0052] S2. Design an ST-GCN model for detecting fall behavior;

[0053] S3. Obtain the dataset for human keypoint detection and label human behaviors;

[0054] S4. Train the YOLO-PoseLiteNet model and the ST-GCN model based on the labeled dataset;

[0055] S5. Obtain video data of the detected target, input it into the trained YOLO-PoseLiteNet model, and obtain the human skeleton key point sequence;

[0056] S6. Input the sequence of key points of the human skeleton into the trained ST-GCN model to obtain the fall detection results.

[0057] Furthermore, S1 specifically refers to:

[0058] S11. Design a lightweight detection head, LiteDetect. Improve the head's localization and classification performance using GroupNorm, and enhance its detail capture capability and detection accuracy using dynamic full-dimensional convolution ODConv. The structure of dynamic full-dimensional convolution ODConv is as follows: Figure 3 As shown;

[0059] S12. Design the RFCBAM-Net backbone network and introduce receptive field attention convolution RFCBAMConv to focus on the spatial features of the receptive field.

[0060] S13. Design a neck network with a multi-branch assisted FPN structure, including shallow assisted fusion, which combines the outputs of the backbone network and the neck network, retains the best level of shallow information, and then outputs diverse gradient information through high-level assisted fusion of the neck network.

[0061] Furthermore, such as Figure 4 As shown, S11 specifically refers to:

[0062] In the YOLOv8-Pose detection head, a shared convolutional structure is introduced, allowing YOLOv8-Pose to share some convolutional layers for bounding box prediction and class prediction at three different scales. This reduces the computational burden on the network and improves efficiency. GroupNorm normalization is used instead of BN normalization, and full-dimensional dynamic convolution ODConv is introduced. GroupNorm normalization is also used to further improve the detail capture capability of the detection head and improve detection accuracy. The lightweight detection head LiteDetect reduces the number of 3×3 convolutions by using a shared convolutional structure, thereby reducing computational complexity and parameter count while maintaining the stability of detection accuracy.

[0063] In this embodiment of the invention, the lightweight detection head LiteDetect is an improvement on the detection head of the YOLOv8-Pose model. YOLOv8-Pose is an extension of the YOLOv8 series of algorithms in the field of human pose estimation, used to estimate the positions of various key points of the human body. The YOLOv8-Pose detection head can simultaneously output the human body bounding box and key point information, outputting three parts of information: Target bounding box: This is a four-dimensional vector representing the bounding box position of the target object, including (x, y) coordinates and width and height (w, h); Class probability: Outputs the probability value of the target's category, indicating which object category the detection result belongs to (e.g., human or other objects); Key point coordinates: The (x, y) coordinates of each key point position. Typically, 17 key points are predicted, covering the main joints of the human body. Key point information can help the model estimate human pose.

[0064] The YOLOv8-Pose detection head contains a series of convolutional layers responsible for predicting the bounding box regression value, target class confidence, and human keypoint information for each scale. However, since the detection head at each scale calculates the bounding box loss, class loss, and keypoint loss using two 3×3 convolutions and one 1×1 convolution, it requires 18 3×3 convolutions and 9 1×1 convolutions for computation, resulting in a significant computational overhead for the YOLOv8-Pose detection head. In YOLOv8-Pose, only the human region needs to be detected, thus simplifying the class prediction to a single class, namely human. This simplifies the class detection task, and the original network design is redundant in this respect compared to conventional multi-class detection networks. To address this issue and optimize the network structure, a shared convolutional structure is introduced, allowing YOLOv8-Pose to share some convolutional layers for bounding box prediction and class prediction at three different scales, reducing the network's computational burden and improving efficiency. Simultaneously, to improve the performance of detection head localization and classification, GN (GroupNorm) normalization is used instead of BN normalization. Batch normalization (BN) is affected by batch processing, and excessively large or small batch sizes can lead to a decrease in model performance. Normalization (GN), on the other hand, solves the problem of BN's normalization being affected by batch processing.

[0065] During the LiteDetect structure optimization process, the design of the keypoint branch remained unchanged. This is because keypoint detection is crucial to the overall detection accuracy, and improper modifications could significantly impact the final detection results. Therefore, the keypoint output branch was not modified, but future optimization schemes can be further explored to improve detection efficiency while ensuring accuracy remains unaffected. In summary, the optimized LiteDetect head scheme minimizes accuracy loss while reducing the number of parameters and computational load. This design not only improves the model's inference speed, making it more suitable for resource-constrained environments, but also provides a feasible direction for subsequent lightweight improvements.

[0066] Furthermore, S12 specifically refers to:

[0067] The ordinary convolutions in the backbone network and the C2f module are improved to receptive field attention convolutions RFCBAMConv, with the structure as follows: Figure 5 As shown, the improved C2f module is named C2f-RFCBAMConv, and the improved backbone network is as follows. Figure 6 As shown, by introducing a receptive field attention mechanism, the receptive field attention convolution RFCBAMConv can enhance the expressive power of global information while capturing local features, thereby improving the effectiveness of feature extraction and the overall performance of the network. However, the introduction of RFCBAMConv inevitably increases the number of network layers, which may lead to potential network saturation and performance degradation. Therefore, three residual connections are designed in the network to alleviate the gradient vanishing problem using a residual learning mechanism. The introduction of residual connections helps to effectively transfer features, ensuring stable information propagation in deep networks, while avoiding gradient loss during forward or backward propagation.

[0068] Because the convolution kernel uses the same parameters to extract information in each receptive field during the convolution operation, without considering the differences between different locations, the network performance is limited. Furthermore, the convolution process does not consider the importance of each feature, which further reduces the efficiency of feature extraction and ultimately limits the model's performance. Therefore, a novel convolution based on receptive field attention (RFAConv) can be adopted to solve the parameter sharing problem of the convolution kernel and consider the importance of each feature in the receptive field. The convolution operation designed by RFAConv is a breakthrough method that can replace the standard convolution operation in current neural networks. Taking a 3×3 convolution kernel as an example, the spatial feature refers to the original feature map "receptive field spatial feature," which is a feature map transformed from the spatial feature and consists of non-overlapping sliding windows. Each 3×3 window in the receptive field spatial feature represents a receptive field slider. This highlights the importance of different features in the receptive field slider and prioritizes the receptive field spatial feature, thus solving the convolution kernel parameter sharing problem.

[0069] When learning the attention map, the interaction of each receptive field feature introduces additional computational overhead. To minimize computational burden and parameter count, average pooling is used to summarize the global information of each receptive field feature. Next, feature interactions are completed through 1×1 group convolutions. Finally, a softmax mechanism is used to highlight the importance of each receptive field feature. When extracting receptive field spatial features, group convolution (GroupConv) is used for fast extraction, mapping the original features to new features—a faster, more efficient, and lightweight approach. This yields the attention map and receptive field spatial features, with the number of channels being the square of the kernel size (the number of channels in the input feature map). For example, with input features of C×H×W and a kernel size of 3×3, the attention map and receptive field spatial features are 9C×H×W. The attention map and receptive field spatial features are then reparameterized (i.e., multiplied) and reshaped to C×3H×3W. Finally, a convolution operation is performed with a regular convolution with a kernel size of 3×3 and a stride of 3, thus resolving the kernel parameter sharing problem.

[0070] Receptive field spatial feature convolution eliminates the problem of convolution parameter sharing. Furthermore, current spatial attention mechanisms already consider long-range information, obtaining global information through global average pooling or global max pooling, explicitly taking long-range information into account. Therefore, in the RFCBAM model, a k×k final convolution operation with stride k is used to extract feature information. In this embodiment, these two novel convolution operations are referred to as RFCBAMConv. In RFCBAM, channel and spatial attention are not performed as separate steps. Instead, they are weighted simultaneously, allowing for different attention maps to be obtained for each channel.

[0071] Furthermore, such as Figure 7 As shown, S13 specifically refers to:

[0072] The backbone network consists of four stages: P2, P3, P4, and P5. The outputs of stages P2, P3, and P4 are used to adjust the feature dimensions and sizes through convolution operations to achieve feature fusion. In the bottom-up path, the shallow auxiliary fusion module extracts multi-scale features from the backbone network and performs preliminary fusion in the shallow layers of the neck network, combining the outputs of the backbone network and the neck network to ensure optimal preservation of shallow features. In the top-down path, the high-level auxiliary fusion module collects gradient information from each layer through a dense connection mechanism. Both the shallow auxiliary fusion module and the high-level auxiliary fusion module use C2f for feature extraction.

[0073] Furthermore, such as Figure 10 As shown, the ST-GCN model consists of a BN normalization layer, stacked ST-GCN modules, a global average pooling layer, and a fully connected layer connected in sequence. The ST-GCN module is composed of an attention layer, a GCN spatial graph convolution, and a TCN temporal graph convolution. The attention layer is used to generate an attention matrix that adjusts the weights of the edges in the graph structure. By dynamically adjusting the weights of the edges in the graph, the network can better capture the feature relationships in the spatial and temporal dimensions and enhance its sensitivity to action details. The GCN spatial convolution performs convolution operations on the spatial dimension of the input to extract and aggregate the relational features between joints. In the initial input, the spatial dimension features are 3, representing the three-dimensional coordinate information and confidence of the joints. The TCN temporal convolution performs convolution operations in the temporal dimension to model the dynamic features of the action at different times.

[0074] By alternately stacking ST-GCN modules, the ST-GCN model progressively extracts and enhances the spatiotemporal features of the input data. In this embodiment, the initial stage has a time dimension of 150 frames, corresponding to the number of keyframes in the input, while the feature dimension of the joints is 3. After passing through multiple layers of ST-GCN units, the feature dimension of the joints expands to 256, while the time dimension is compressed to 38. The network integrates all spatiotemporal features through global average pooling, and then classifies the extracted high-level semantic information through a fully connected layer to achieve action recognition.

[0075] In this embodiment of the invention, the spatiotemporal graph model of ST-GCN is as follows: Figure 8As shown, nodes represent corresponding human joints; edges are divided into two types: spatial edges and temporal edges. Spatial edges define the topological structure of the human skeleton within each time frame; temporal edges connect the same nodes in consecutive time frames, capturing the dynamic changes of human joints, such as the movement trajectory of the elbow joint across multiple time frames. This dynamic feature of human skeletal joints has unique advantages in pose estimation, as it can convey important motion-related information in a structured way, avoiding interference from appearance and background.

[0076] To apply convolution operations to graph data, the sampling function p and the weight function w need to be redefined. First, the sampling function is defined: in an image, p(h,w) is defined on the neighboring pixels relative to the center position x. Similarly, in a graph, the node v can be defined... ti The neighbor set B(v) ti )={v tj |d(v tj ,v ti The sampling function on d(v) ≤ D} (D = 1), where d(v) tj ,v ti ) represents v tj to v ti The minimum length of any path. Therefore, the sampling function can be written as: p(v ti ,v tj ) = v ti Defining the weight function: Compared to the sampling function, the weight function is easier to define. This is achieved by jointly defining the node v. ti The neighbor set B(v) ti To simplify the process, the system is divided into a fixed number of K subsets, each with a label. Specifically, since the body skeleton is spatially localized, this spatial configuration can be utilized during partitioning to divide the neighbor set into three subsets: the root node itself; the centripetal group (neighboring nodes closer to the skeleton's center of gravity than the root node); and the centrifugal group (nodes farther from the center of gravity than the root node). The specific formula is as follows:

[0077]

[0078] r j It is the distance from the root node to the centroid, r i It is the distance from the node to the centroid;

[0079] Finally, the spatial map is extended to the time domain, such as... Figure 9 As shown, the concept of domain is also extended to the time domain B(v_ti)=v_qj|d(v tj ,v ti )≤K, The parameter Γ controls the temporal range of adjacent graphs, and can also be called the temporal kernel size. To perform the convolution operation on the temporal graph, a sampling function and a weighting function are also needed. The basic principle is the same; only the mapping function for label grouping needs to be redefined, and the rest of the calculation is the same. Since the time axis is ordered, the labels can be directly mapped to l. ST Modified to use v ti The spacetime domain as the root: l ti (v tj Grouping labels in space. Grouping by temporal labels. This constructs a well-defined convolution operation on the spatiotemporal graph. In temporal convolution, the kernel size is the number of keyframes, and the operation of one node is completed at a time, moving one frame to perform the convolution of the next node.

[0080] Furthermore, the ST-GCN model employs a spatial configuration strategy to group the label mappings, dividing the nodes within the domain into three subsets based on their spatial positional relationships: root nodes, centripetal nodes, and centrifugal nodes. The root node represents the node currently undergoing convolution, the centripetal node represents the node closer to the center of gravity of the skeleton, and the centrifugal node represents the node farther away from the center of gravity of the skeleton. To better capture the dynamic features in fall behavior, the domain span D of the spatial graph convolution is expanded from 1 to 2, enabling each node to establish connections with more distant related nodes. This enhances the ST-GCN model's global perception of actions and helps capture the cross-node features of fall behavior.

[0081] See Figure 11 The nodes within the domain are relabeled as follows: centripetal nodes (blue, labeled 1): closer to the center of gravity of the skeleton, representing inward or lateral movement characteristics; centrifugal nodes (yellow, labeled 2): farther from the center of gravity of the skeleton, representing outward or lateral movement trends; root nodes (green, labeled 0): key joints in the skeleton, serving as important cores of motion features.

[0082] In this embodiment of the invention, the spatial allocation strategy is obtained by improving the partitioning strategy of the traditional ST-GCN model, such as... Figure 11As shown, the traditional ST-GCN proposes three partitioning strategies: First, the Uni-labeling strategy: all nodes in the domain are treated as the same category and assigned the same label. Since it ignores the spatial positional relationship and relative motion characteristics between nodes, it cannot effectively capture the complex features of the spatial graph and has limitations in performance. Second, the distance strategy: partitioning the nodes into subsets based on the distance between the nodes and the root node. The distance strategy divides the domain nodes into two subsets according to the domain span D (default is 1): neighboring nodes within 1, reflecting closely related local features, and nodes with a distance greater than 1, representing more distant non-local features. This strategy can enhance the perception of features of different ranges to a certain extent, but it is limited when the domain span is limited. Third, the spatial configuration strategy: dividing the nodes in the domain into three subsets according to the spatial positional relationship of the nodes: (1) root node: representing the node to be convolved; (2) centripetal node: the node closer to the center of gravity of the skeleton, usually related to inward movement, reflecting the centripetal movement trend; (3) centrifugal node: the node farther from the center of gravity of the skeleton, usually related to outward movement, reflecting the centrifugal movement trend. This strategy can capture finer-grained interactions between nodes when constructing spatiotemporal graph convolutions, improving the ability to express human skeletal features. However, since the domain span of spatial graph convolutions is set to D=1, this means that graph convolution operations can only focus on nodes within a local range. However, for fall behavior detection, action features usually have the following characteristics: (1) large amplitude of movement, manifested as the coordinated movement of multiple joints; (2) action changes are concentrated in a short period of time, requiring the model to be more sensitive to global features.

[0083] Furthermore, after model training is completed in S4, model performance is evaluated using the average precision (AP) and average recall (AR) metrics, and the similarity (OKS) between the ground truth keypoints and the predicted keypoints is set:

[0084]

[0085] In the formula, d i v represents the Euclidean distance between the i-th labeled keypoint and the predicted keypoint. i The visible marker of the i-th labeled key point is represented by sk, where sk is the target scale and δ is the standard error.

[0086] Top1 and Top5 metrics are used to measure the classification accuracy of the model. Top1 assesses whether the model's predicted most likely class matches the actual class. If the model's highest probability class matches the true class, the Top1 accuracy is 1; otherwise, it is 0. The Top5 metric measures whether the model's predicted top five likely classes include the true class. If the actual class appears in these five classes, the Top5 accuracy is 1; otherwise, it is 0. The formulas for calculating both metrics are as follows:

[0087]

[0088] In the formula, Indicates the true category of the i-th behavior. This represents the predicted category for the i-th action. For the category whose predicted value ranks first in the i-th behavior prediction, Let N represent the top 5 categories in the prediction of the i-th behavior, and let N represent the total number of behaviors.

[0089] In this embodiment of the invention, two public datasets are used to train the model: the COCOKeypoints dataset and the NTURGB+D dataset; a subset of the NTURGB+D dataset, which is constructed by selecting categories on the basis of NTU RGB+D, is named the Fall DeductionNTU dataset.

[0090] The COCOKeypoints dataset is part of the COCO (Common Objects in Context) dataset, focusing on human keypoint detection, such as... Figure 12 As shown. The COCOKeypoints dataset annotates key points of the human body, such as the head, shoulders, elbows, wrists, hips, knees, and ankles. The COCO2017 dataset, released after 2017, is currently widely used. The training set trian2017 of the COCO2017 dataset contains 57K images and 150K human pose instances, while the validation set val2017 contains 5000 images. Each human instance has 17 key points: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. In the annotation information of each key point, the coordinates of the human key point are defined as (x, y, v), where (x, y) represents the spatial coordinate information of the human key point, and v represents the visibility of the key point in the image. The value can be 0, 1, or 2. 0 means that the human key point is not in the image, 1 means that the human key point is occluded but its location can be inferred, and 2 means that the key point is visible in the image and can be accurately marked.

[0091] NTURGB+D dataset 2D key point information, such as Figure 13As shown. The NTURGB+D dataset is divided into two versions: NTURGB+D and NTURGB+D120. The NTURGB+D dataset contains 56,880 samples covering 60 action categories; while NTURGB+D120 is an expanded version of the former, retaining all the original data and adding 60 new categories, increasing the total number of samples to 114,480. Both datasets provide multimodal information for each sample, including RGB video, depth map sequences, 3D skeleton data, and infrared (IR) video. The 3D skeleton data contains the 3D coordinates of 25 body joints per frame. The dataset contains two subsets, X-Sub and X-View. The X-Sub subset contains 40 different participants, each performing each action category 3 times, for a total of 21,120 samples. This subset is widely used for participant human behavior recognition. The X-View subset uses multiple cameras to capture information about human behavior from different angles and poses. This subset contains 25 different perspectives, with each participant performing each action category 3 times in each perspective, totaling 35,760 samples. This subset has been widely used for human behavior recognition across multiple perspectives.

[0092] Because the NTURGB+D dataset covers a wide range of action categories, including many complex behaviors unrelated to falls, this embodiment selects four typical actions closely related to fall detection: falling, sitting, standing up, and walking. Ultimately, the Fall Detection NTU dataset contains 946 fall samples, 941 sitting samples, 936 standing samples, and 947 walking samples, totaling 3770 data points. By training ST-GCN on this dataset, the model can more effectively learn the spatiotemporal features of fall-related scenarios.

[0093] In one embodiment of the present invention, an experiment was conducted on LiteDetect, a lightweight detection head of YOLO-PoseLiteNet, and the experimental results are shown in Table 1:

[0094] Table 1. Experimental results of the lightweight detection head LiteDetect

[0095]

[0096] Experimental results show that the lightweight detection head LiteDetect slightly decreases the mAP@0.5 and mAP@0.5:0.95 metrics, but significantly improves the lightweight performance of the model. Specifically, GFLOPs and Params decrease by 36% and 24%, respectively, while detection speed (FPS) increases by 23% and 39% on GPU and CPU, respectively. Based on LiteDetect-1, GN normalization is added to improve LiteDetect-2, effectively enhancing the localization performance of the detection head and improving both mAP@0.5 and mAP@0.5:0.95 metrics. Further, LiteDetect-3 is designed by combining dynamic full-dimensional convolution ODConv, enhancing the model's representation ability and generalization performance. LiteDetect significantly improves the model's lightweightness while maintaining an acceptable performance decrease; compared to the original model, the mAP@0.5 metric decreases by 0.5%, and the mAP@0.5:0.95 metric decreases by 1.1%.

[0097] The modified LiteDetect approach retains the shared convolutional structure but removes the keypoint output branch, and is then applied to the object detection model YOLOv8. Experimental results are shown in Table 2.

[0098] Table 2. Experimental results of LiteDetect on the YOLOv8 object detection model.

[0099]

[0100] In the table, (LD) indicates the application of LiteDetect. It can be seen that while using LiteDetect in the YOLOv8 model significantly reduces computational overhead and greatly improves detection speed, the decrease in detection accuracy is excessive. The mAP@0.5 and mAP@0.5:0.95 metrics decreased by 7% and 5% respectively, far exceeding the performance degradation caused by using LiteDetect in YOLOv8-Pose. The main reason for the performance degradation is that LiteDetect simplifies the category output branch in the detection head, which has a very negative impact on the classification performance of the object detection network. In contrast, in keypoint detection tasks, since complex category classification is not required (focusing only on single human category information), LiteDetect has a smaller impact on performance and can better achieve a balance between accuracy and lightweight design. Therefore, although LiteDetect fails to achieve an effective balance between detection accuracy and lightweight design in the YOLOv8 object detection model, its performance in human joint keypoint detection tasks remains excellent, demonstrating its potential for keypoint detection.

[0101] In one embodiment of the present invention, experiments were conducted on the backbone network of YOLO-PoseLiteNet, and the experimental results are shown in Table 3:

[0102] Table 3. Backbone Network Experiment Results

[0103]

[0104] RFCBAM-Net-1, lacking residual connections, achieves improvements of 1% and 0.9% in mAP@0.5 and mAP@0.5:0.95 respectively, with FPS reaching 225.1 on GPU and 9.3 on CPU. RFCBAM-Net-2, introducing residual connections, reduces the risk of network learning degradation due to increased network depth, enhancing network stability and generalization ability. Compared to YOLOv8-pose, it achieves improvements of 1.2% and 1.5% in mAP@0.5 and mAP@0.5:0.95 respectively.

[0105] In one embodiment of the present invention, experiments were conducted on the multi-branch auxiliary FPN structure of YOLO-PoseLiteNet, and the experimental results are shown in Table 4:

[0106] Table 4 Experimental results of multi-branch auxiliary FPN structure

[0107]

[0108] Experimental results show that the multi-branch assisted FPN structure disclosed in this embodiment of the invention achieves improved accuracy with an increase of 0.6 in GFLOPs and 0.1 in Params / M, with an increase of 0.9% in mAP@0.5 and 0.4% in mAP@0.5:0.95, and the overall performance is the best.

[0109] In one embodiment of the present invention, an ablation experiment was performed on YOLO-PoseLiteNe, and the experimental results are shown in Table 5:

[0110] Table 5. Results of YOLO-PoseLiteNe ablation experiments

[0111]

[0112] Experimental results show that the LiteDetect head reduces the detection accuracy of YOLOv8-pose but significantly reduces the computational burden and substantially improves the frame rate. The RFCBAM-Net backbone network design improves mAP@0.5 to 0.817 and mAP@0.5:0.95 to 0.505. Using the MAFPN structure, mAP@0.5 improves from 0.794 to 0.803, and mAP@0.5:0.95 to 0.494. In summary, the LiteDetect lightweight head, RFCBAM-Net backbone network, and MAFPN structure achieve the best performance with a 2.4% improvement in mAP@0.5, a 1.7% improvement in mAP@0.5:0.95, and a frame rate increase of approximately 31 frames per second.

[0113] In one embodiment of the present invention, a comparative experiment was conducted on the spatial configuration strategy of ST-GCN. The experimental results are shown in Table 6 and... Figure 14 :

[0114] Table 6 ST-GCN Configuration Experiment Results

[0115]

[0116] It can be seen that, compared with the Uni-labeling strategy, the Distance strategy, and the Spatial strategy, on the NTU RGB+D dataset, the Uni-labeling strategy, the Distance strategy, and the Spatial strategy achieved Top 5 metrics of 68.04%, 96.22%, and 96.25%, respectively. The Uni-labeling strategy is inferior to the Distance and Spatial strategies because the Uni-labeling strategy treats all neighboring nodes as the same category, making it difficult to understand human actions. When the neighborhood span D=2, nodes can be connected with more distant related nodes, enhancing the model's global perception of actions. This is particularly crucial for detecting large-amplitude human actions, such as falls. Selecting representative small-amplitude human actions ( Figure 14 The first 6 categories of the X-axis) and large-amplitude human movements ( Figure 14The performance improvement of the ST-GCN for detecting the last five classes of the X-coordinate (the last five classes) is visually presented, showing that the improvement is more significant for large-amplitude human behaviors such as sitting down and falling. On the NTURGB+D dataset, both the Distance and Spatial strategies achieved performance improvements in the Top-1 metric, with Distance improving by 0.87% and Spatial by 2.34%. On the Fall NTU dataset, the Distance and Spatial strategies improved by 2.03% and 2.57% respectively in the Top-1 metric, showing even more significant improvements. This demonstrates the importance of spatial information in fall detection tasks. In particular, the Spatial strategy, which classifies based on the spatial configuration of the human skeleton, enables the model to more accurately identify fall behaviors.

[0117] In one embodiment of the present invention, a comparative experiment was conducted on the two-stage fall detection method disclosed in the present invention, and the experimental results are shown in Table 7:

[0118] Table 7 Comparative Experiment Results

[0119]

[0120] As can be seen, compared with other models, the biggest advantage of the method proposed in this embodiment is its low GFLOPs and lighter weight. On the NTU RGB+D dataset, the method proposed in this embodiment achieves a high accuracy of 88.1%, which is 3.5% and 1.2% higher than PoseC3D and AGCN, respectively. On the FallNTU dataset, the method proposed in this embodiment improves the detection accuracy of each behavior category compared with the original ST-GCN model (the human joint detector uses the OpenPose algorithm). Moreover, in comparison with other models, the detection accuracy of the fall and walking categories is optimal, which can accurately identify human posture.

[0121] The detection results of the two-stage fall detection method based on the fusion of keypoint detection and spatiotemporal graph convolution are as follows: Figure 15As shown, the video segments of each type of dynamic behavior in the image are detected in five stages. The YOLO-PoseLiteNet keypoint detection algorithm is responsible for detecting human keypoints in each frame of the image, forming a human keypoint sequence; the spatiotemporal graph convolution algorithm performs temporal modeling of human posture and fall detection. In this paper, the detection of human fall actions is mainly studied, while the NTURGB+D dataset contains many complex behaviors unrelated to falls. Therefore, the FallNTU dataset is used for pre-training of the spatiotemporal graph convolution algorithm ST-GCN. Detection was performed for four human states: falling, sitting down, standing up, and walking. The results show that the fall detection method disclosed in this embodiment can accurately distinguish falls from other non-fall behaviors. This method of posture estimation based on human keypoints has stronger resistance to interference from lighting changes and complex environments compared to the original image data. Even under partial occlusion or unstable lighting conditions, as long as the keypoint detector has high accuracy, the overall detection performance can still maintain a good level.

[0122] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution, characterized in that, Includes the following steps: S1. Design a YOLO-PoseLiteNet model for extracting key point sequences of human skeletons; S2. Design an ST-GCN model for detecting fall behavior; S3. Obtain the dataset for human keypoint detection and label human behaviors; S4. Train the YOLO-PoseLiteNet model and the ST-GCN model based on the labeled dataset; S5. Obtain video data of the detected target, input it into the trained YOLO-PoseLiteNet model, and obtain the human skeleton key point sequence; S6. Input the sequence of key points of the human skeleton into the trained ST-GCN model to obtain the fall detection results.

2. The two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution as described in claim 1, characterized in that, S1 specifically refers to: S11. Design a lightweight detection head LiteDetect, improve the performance of detection head localization and classification by GroupNorm, and improve the detection head's detail capture capability and detection accuracy by using dynamic full-dimensional convolution ODConv; S12. Design the RFCBAM-Net backbone network and introduce receptive field attention convolution RFCBAMConv to focus on the spatial features of the receptive field. S13. Design a neck network with a multi-branch assisted FPN structure, including shallow assisted fusion, which combines the outputs of the backbone network and the neck network, retains the best level of shallow information, and then outputs diverse gradient information through high-level assisted fusion of the neck network.

3. The two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution as described in claim 2, characterized in that, S11 specifically refers to: In the YOLOv8-Pose detection head, a shared convolutional structure is introduced, allowing YOLOv8-Pose to share some convolutional layers for bounding box prediction and class prediction at three different scales. This reduces the computational burden on the network and improves efficiency. GroupNorm normalization is used instead of BN normalization, and full-dimensional dynamic convolution ODConv is introduced. GroupNorm normalization is also used to further improve the detail capture capability of the detection head and improve detection accuracy. The lightweight detection head LiteDetect reduces the number of 3×3 convolutions by using a shared convolutional structure, thereby reducing computational complexity and parameter count while maintaining the stability of detection accuracy.

4. The two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution as described in claim 2, characterized in that, S12 specifically refers to: The ordinary convolutions in the backbone network and the C2f module are improved into receptive field attention convolutions RFCBAMConv, and the improved C2f module is named C2f-RFCBAMConv. By introducing the receptive field attention mechanism, the receptive field attention convolution RFCBAMConv can enhance the ability to express global information while capturing local features, thereby improving the effectiveness of feature extraction and the overall performance of the network. Three residual connections are designed in the network, and the residual learning mechanism is used to alleviate the gradient vanishing problem.

5. The two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution according to claim 2, characterized in that, S13 specifically refers to: The backbone network consists of four stages: P2, P3, P4, and P5. The outputs of stages P2, P3, and P4 are used to adjust the feature dimensions and sizes through convolution operations to achieve feature fusion. In the bottom-up path, the shallow auxiliary fusion module extracts multi-scale features from the backbone network and performs preliminary fusion in the shallow layers of the neck network, combining the outputs of the backbone network and the neck network to ensure optimal preservation of shallow features. In the top-down path, the high-level auxiliary fusion module collects gradient information from each layer through a dense connection mechanism. Both the shallow auxiliary fusion module and the high-level auxiliary fusion module use C2f for feature extraction.

6. The two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution as described in claim 1, characterized in that, The ST-GCN model consists of sequentially connected BN normalization layers, stacked ST-GCN modules, global average pooling layers, and fully connected layers. Each ST-GCN module comprises an attention layer, a GCN spatial graph convolution, and a TCN temporal graph convolution. The attention layer generates an attention matrix that adjusts the weights of edges in the graph structure. By dynamically adjusting the weights of edges in the graph, the network can better capture feature relationships in both spatial and temporal dimensions, enhancing its sensitivity to action details. The GCN spatial convolution performs convolution operations on the spatial dimension of the input to extract and aggregate relational features between joints. In the initial input, the spatial dimension features are 3, representing the three-dimensional coordinate information and confidence level of the joints. The TCN temporal convolution performs convolution operations in the temporal dimension, modeling the dynamic features of the action at different times.

7. The two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution as described in claim 6, characterized in that, The ST-GCN model employs a spatial configuration strategy to group label mappings, dividing nodes within the domain into three subsets based on their spatial positional relationships: root nodes, centripetal nodes, and centrifugal nodes. The root node represents the node currently undergoing convolution, the centripetal node represents the node closer to the skeleton's centroid, and the centrifugal node represents the node farther away from the skeleton's centroid. This expands the domain span D of the spatial graph convolution from 1 to 2, enabling each node to establish connections with more distant related nodes. This enhances the ST-GCN model's global perception of actions and helps capture cross-node features of fall behavior.

8. The two-stage fall detection method integrating keypoint detection and spatiotemporal graph convolution as described in claim 1, characterized in that, After model training is completed in S4, model performance is evaluated using the Mean Precision (AP) and Mean Recall (AR) metrics. The similarity between ground truth keypoints and predicted keypoints is set to OKS. In the formula, d i v represents the Euclidean distance between the i-th labeled keypoint and the predicted keypoint. i The visible marker of the i-th labeled key point is represented by sk, where sk is the target scale and δ is the standard error. Top1 and Top5 metrics are used to measure the classification accuracy of the model. Top1 assesses whether the model's predicted most likely class matches the actual class. If the model's highest probability class matches the true class, the Top1 accuracy is 1; otherwise, it is 0. The Top5 metric measures whether the model's predicted top five likely classes include the true class. If the actual class appears in these five classes, the Top5 accuracy is 1; otherwise, it is 0. The formulas for calculating both metrics are as follows: In the formula, Indicates the true category of the i-th behavior. This represents the predicted category for the i-th action. For the category whose predicted value ranks first in the i-th behavior prediction, Let N represent the top 5 categories in the prediction of the i-th behavior, and let N represent the total number of behaviors.

Images