Fall detection method and system based on millimeter wave radar fused with human body posture

Abstract

The invention provides a falling detection method and system based on millimeter wave radar fusion human body postures, and relates to the technical field of falling detection, and the method comprises the steps: collecting human body echo signals, and constructing three-dimensional point cloud data through distance, angle and speed estimation; the point cloud is processed to generate a dynamic sequence, and skeleton features and key point coordinates are extracted in combination with a graph convolutional network. A skeleton connection relation is constructed based on the key points, attitude features are calculated, key change features are extracted through a self-attention mechanism, and the key change features are fused with a point cloud sequence to construct multi-modal features. And the double-branch state recognition network processes the fusion features, and when abnormality is detected, further analysis is carried out through residual attention and space-time diagram convolution, and finally, the falling state is recognized and the risk level is evaluated.

Description

Technical Field

[0001] This invention relates to the field of fall detection technology, and in particular to a fall detection method and system based on millimeter-wave radar fusion of human posture. Background Technology

[0002] Fall detection is an important research area in elderly safety monitoring, aiming to identify fall events in the elderly in a timely manner and issue an alarm so that necessary rescue measures can be taken. Existing fall detection methods are mainly based on technologies such as video images, wearable sensors, and environmental sensors. However, existing technologies still have limitations in accurately identifying complex fall postures and movements, poor adaptability to environmental changes, and a lack of fall risk assessment, making it difficult to provide personalized health monitoring and intervention measures. Therefore, a solution is urgently needed to address the problems existing in the current technology. Summary of the Invention

[0003] This invention provides a fall detection method and system based on millimeter-wave radar fusion of human posture, which can at least solve some of the problems existing in the prior art.

[0004] A first aspect of this invention provides a fall detection method based on millimeter-wave radar fusion of human posture, comprising: The target distance information is obtained by collecting human echo signals with millimeter-wave radar and estimating parameters. The angle information is obtained by adaptive beamforming. The velocity information is obtained by phase difference velocimetry. Based on the target distance information, angle information and velocity information, the point cloud mapping relationship is constructed by dense convolution to obtain three-dimensional point cloud data. After dynamic threshold segmentation, establishment of local coordinate system and iterative nearest point registration, a dynamic point cloud sequence is generated. Combined with graph convolutional network, the skeletal features are extracted and the three-dimensional coordinates of key points are output. The skeleton connection relationship is constructed based on the 3D coordinates of key points, and the spatial vector and angle are calculated to obtain the initial pose features. An adaptive feature sequence is generated through dynamic sampling. Key pose change features are extracted through a self-attention mechanism. A spatial pyramid pooling network is constructed in combination with the dynamic point cloud sequence to extract multi-scale features. A motion feature descriptor is constructed based on the centroid trajectory of the point cloud. Multimodal fusion features are obtained through graph attention network. The multimodal fusion features are input into the dual-branch state recognition network. The initial judgment result and anomaly probability are obtained through lightweight convolution. When the anomaly probability exceeds the threshold, the features are enhanced by residual attention and spatiotemporal combined features are extracted by spatiotemporal graph convolution. The recognition result is obtained by posture classification, action classification and Bayesian fusion. If it is a fall state, the fall action is determined and the risk level is output.

[0005] In one alternative implementation, Human body echo signals are acquired using millimeter-wave radar, and target distance information is obtained through parameter estimation. Angle information is obtained through adaptive beamforming, and velocity information is obtained based on phase difference velocimetry. Based on the target distance, angle, and velocity information, a point cloud mapping relationship is constructed using dense convolution to obtain 3D point cloud data. Dynamic point cloud sequences are generated after dynamic threshold segmentation, establishment of a local coordinate system, and iterative nearest-point registration. Finally, a graph convolutional network is used to extract skeletal features and output the 3D coordinates of key points, including: Collect human echo signals of the target object, construct the signal covariance matrix, determine the signal subspace and noise subspace through eigenvalue decomposition, construct the distance spectrum function to search for peaks and obtain target distance information; The target angle information is obtained by spatial filtering of the human body echo signal through an adaptive beamforming algorithm, and the phase difference sequence is extracted from the echo signal at adjacent time points and Doppler processing is performed to obtain the target velocity information. Dense convolution is performed on the target distance information, target angle information and target velocity information to extract multi-scale features and multi-level combination of features through dense connections. Deconvolution operation is used to construct point cloud mapping relationship to obtain three-dimensional point cloud data. The density distribution characteristics of the three-dimensional point cloud data are calculated. The valley point method is used to determine the initial segmentation threshold and dynamic threshold segmentation is performed according to local density changes. The target point cloud cluster is determined by combining the region growing algorithm and a local coordinate system is established. Based on the local coordinate system, the transformation matrix of the point cloud of two adjacent frames is estimated and iteratively optimized until the registration error converges to generate a dynamic point cloud sequence. Combined with the graph convolutional network, the human body is modeled as a set of key points to construct the skeletal topology. The node features are locally aggregated to extract the skeletal features and the key node features are enhanced to obtain the three-dimensional coordinates of the key points.

[0006] In one alternative implementation, The calculation of the density distribution characteristics of the 3D point cloud data, the determination of the initial segmentation threshold using the valley point method, and the dynamic threshold segmentation based on local density changes, combined with the region growing algorithm to determine the target point cloud cluster and establish a local coordinate system, includes: Based on the 3D point cloud data, a 3D mesh is divided, the number of points contained in each mesh cell is calculated to obtain the local density value of the point cloud, the local density value of the point cloud is divided into density intervals according to the numerical value, and the distribution of the number of points in each density interval is statistically analyzed to establish a density histogram. Local peak points are detected and valley points are determined in the density histogram. The valley point method is used to determine the initial segmentation threshold and a sliding window is set. The density variance of the point cloud within the sliding window is calculated, and the initial segmentation threshold is corrected based on the density variance and the segmentation is repeated. In the segmented high-density region, a local density maximum point is selected as a seed point. A spherical search region is established with the seed point as the center to determine candidate points. The density value of the candidate point is calculated as the difference between the density value of the candidate point and the average density value of all points in the high-density region, as well as the Euclidean distance from the candidate point to the centroid of the high-density region. The candidate points are selected and incorporated into the high-density region using the region growing algorithm to obtain the target point cloud. The shape compactness parameters and surface smoothness parameters of the target point cloud are calculated by shape analysis. Abnormal point clusters are eliminated by combining the preset multidimensional feature constraint threshold to obtain the target point cloud cluster. The centroid position of the target point cloud cluster is calculated. The scatter matrix is ​​calculated as the spatial distribution covariance matrix and eigenvalue decomposition is performed. The eigenvector with the largest eigenvalue is used as the principal axis direction to establish a local coordinate system.

[0007] In one alternative implementation, Based on the 3D coordinates of key points, a skeleton connection relationship is constructed, and spatial vectors and angles are calculated to obtain initial pose features. An adaptive feature sequence is generated through dynamic sampling, and key pose change features are extracted through a self-attention mechanism. A spatial pyramid pooling network is constructed in conjunction with the dynamic point cloud sequence to extract multi-scale features, and a motion feature descriptor is constructed based on the centroid trajectory of the point cloud. Feature fusion is performed through a graph attention network to obtain multimodal fused features, including: Obtain the 3D coordinates of the key points corresponding to the key points of the human body, establish the skeletal connection relationship between adjacent key points of the human body, and obtain the initial pose features by combining the spatial vectors and angles between adjacent key points of the human body. Calculate the displacement of the human body key points between adjacent time frames, determine the sampling frequency based on the displacement, and perform dynamic sampling. Arrange the sampled initial posture features in chronological order to generate an adaptive feature sequence. The adaptive feature sequence is segmented by sliding based on the self-attention mechanism. Within each time window, the similarity score between feature vectors is calculated to construct an attention weight matrix. The adaptive feature sequence is then weighted and aggregated to obtain key pose change features. A spatial pyramid pooling network is constructed by combining dynamic point cloud sequences. Multi-level downsampling is performed through max pooling, and the point density distribution, normal vector distribution, and curvature distribution in each pooling region are calculated to extract multi-scale features. Based on the dynamic point cloud sequence, the centroid position of the point cloud is calculated to obtain the centroid trajectory of the point cloud. A smooth trajectory is obtained by spline interpolation, and the curvature and torsion are calculated to construct a motion feature descriptor. The key posture change features, multi-scale features, and motion feature descriptors are used as graph nodes. The connection relationship between nodes is established based on feature similarity, and the information transmission weight is calculated to perform feature fusion to obtain multimodal fusion features.

[0008] In one alternative implementation, The process involves calculating the centroid position of the point cloud to obtain its trajectory, uniformly sampling control points along this trajectory, and obtaining a smooth trajectory through spline interpolation. The curvature and torsion of the smooth trajectory are then calculated. A multidimensional motion feature descriptor is constructed by combining the geometric features and motion parameters of the trajectory. The process includes calculating the centroid position of the point cloud based on the dynamic point cloud sequence, obtaining the smooth trajectory through spline interpolation, calculating curvature and torsion, and constructing a motion feature descriptor. The centroid position of the point cloud is obtained by calculating the mean coordinate value of all points in the point cloud in each frame of the point cloud sequence, and the centroid positions of the point cloud are connected in chronological order to form the centroid trajectory of the point cloud. Control points are obtained by uniform sampling on the centroid trajectory of the point cloud. Multiple interpolation points are generated between adjacent control points using spline interpolation. At each interpolation point, a tangent vector, a principal normal vector, and a secondary normal vector are calculated. The tangent vector is obtained by the position difference between adjacent interpolation points. The principal normal vector is perpendicular to the tangent vector and points to the curvature center of the smooth trajectory. The secondary normal vector is obtained by the cross product of the tangent vector and the principal normal vector. The tangent vector, the principal normal vector, and the secondary normal vector are normalized to construct a local coordinate system. The curvature and torsion of the smooth trajectory at each interpolation point are calculated based on the local coordinate system. The geometric features and motion parameters of the smooth trajectory are extracted, and the curvature, torsion, geometric features and motion parameters are combined to construct a multidimensional motion feature descriptor.

[0009] In one alternative implementation, Multimodal fusion features are input into a dual-branch state recognition network. Lightweight convolutions are used to obtain initial judgment results and anomaly probabilities. When the anomaly probability exceeds a threshold, residual attention is used to enhance features, and spatiotemporal combined features are extracted using spatiotemporal graph convolutions. The recognition result is obtained through pose classification, action classification, and Bayesian fusion. If a fall is identified, the fall action is determined, and the risk level is output, including: Multimodal fusion features are input into a dual-branch state recognition network, and after performing lightweight convolution, they are normalized and activated. The spatial dimension is reduced by pooling, and the initial judgment result and anomaly probability are output through a fully connected layer. When the anomaly probability exceeds a preset threshold, the multimodal fusion feature is enhanced by residual attention. A spatial attention map is generated by convolution, channel weights are generated by global pooling and a two-layer perceptron, and the temporal dimension attention score is calculated by a sliding window. The spatial attention map, channel weights, and temporal dimension attention score are multiplied by the multimodal fusion feature to obtain the enhanced feature. The human skeletal structure is modeled as a graph by combining spatiotemporal graph convolution, the enhanced features are mapped to graph nodes, graph convolution is performed in the spatial domain through adjacency matrix, and temporal modeling is performed in the temporal domain to extract spatiotemporal combined features. Based on the spatiotemporal combination features, the probability distribution of basic posture categories is output through posture classification, the probability distribution of action categories is output through action classification, the basic posture categories and action categories are combined, the posterior probability is calculated through Bayesian fusion, and the combination with the highest posterior probability is selected to obtain the recognition result. If the identification result is a fall, the curvature, torsion, velocity, and acceleration in the pre-acquired multidimensional motion feature descriptor are analyzed to determine the fall action and output the risk level.

[0010] In one alternative implementation, By combining spatiotemporal graph convolution to model the human skeletal structure as a graph, mapping the enhanced features to graph nodes, performing graph convolution in the spatial domain through an adjacency matrix, and performing temporal modeling and extracting spatiotemporal combined features in the temporal domain, the following steps are taken: Human key points are used as nodes of a graph structure, and the skeletal connection relationship between adjacent human key points is defined as the edge of the graph structure. An adjacency matrix is ​​generated based on the skeletal connection relationship, and the enhanced features are mapped to graph nodes to construct a node feature sequence. By combining spatiotemporal graph convolution in the spatial domain, the spatial association strength between the target node and the directly connected node is calculated through the adjacency matrix to obtain the spatial attention coefficient. The spatial attention coefficient is then weighted and combined with the node features corresponding to the directly connected nodes to obtain the spatial aggregation feature. The node feature sequence is extracted from consecutive frames on the time axis using a sliding window. The node feature sequence is then subjected to a standard convolution operation to extract local temporal patterns. Motion features within different time ranges are extracted layer by layer through multiple layers of standard convolution. Spatial feature extraction and temporal feature extraction are alternately chained together. Spatial feature extraction includes graph convolution, normalization, and activation functions, while temporal feature extraction includes standard convolution, normalization, and deactivation. Shortcut connection paths are added between adjacent feature extractions to directly pass the input features to the output and add them to the transformed features. The output features of each layer are combined, and pooling is performed on the features of all nodes to extract global features. The pooled features are then mapped to a unified dimensional space through feature transformation to obtain spatiotemporal combined features.

[0011] A second aspect of the present invention provides a fall detection system based on millimeter-wave radar fusion of human posture, comprising: The first unit is used to collect human echo signals through millimeter-wave radar and perform parameter estimation to obtain target distance information, obtain angle information through adaptive beamforming, obtain velocity information based on phase difference velocimetry, and obtain three-dimensional point cloud data by constructing point cloud mapping relationship through dense convolution based on target distance information, angle information and velocity information. After dynamic threshold segmentation, establishment of local coordinate system and iterative nearest point registration, dynamic point cloud sequence is generated. Combined with graph convolutional network, skeletal features are extracted and the three-dimensional coordinates of key points are output. The second unit is used to construct the skeleton connection relationship based on the three-dimensional coordinates of key points and calculate the spatial vector and the included angle to obtain the initial pose features. It generates an adaptive feature sequence through dynamic sampling, extracts key pose change features through a self-attention mechanism, constructs a spatial pyramid pooling network in combination with the dynamic point cloud sequence to extract multi-scale features, constructs a motion feature descriptor based on the centroid trajectory of the point cloud, and obtains multi-modal fusion features through graph attention network. The third unit is used to input multimodal fusion features into the dual-branch state recognition network. The initial judgment result and anomaly probability are obtained through lightweight convolution. When the anomaly probability exceeds the threshold, the feature is enhanced by residual attention and spatiotemporal combined features are extracted by spatiotemporal graph convolution. The recognition result is obtained by posture classification, action classification and Bayesian fusion. If it is a fall state, the fall action is determined and the risk level is output.

[0012] A third aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0013] Fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0014] In this invention, three-dimensional point cloud data of the human body is acquired through millimeter-wave radar, and human skeletal features are extracted by combining graph convolutional networks to construct a multi-dimensional motion feature descriptor. This enables more accurate capture of human posture and motion information, thereby improving the accuracy of fall detection and effectively reducing false alarms and false negatives. Employing techniques such as dynamic threshold segmentation, iterative nearest-point registration algorithms, and adaptive beamforming algorithms, the system effectively addresses interference from different environments and individual differences, enhancing the robustness and adaptability of the fall detection system. Utilizing a dynamic sampling module and a spatial pyramid pooling network, the system adaptively extracts key posture change features and multi-scale spatial features, further improving its adaptability. A risk assessment model is constructed based on confidence scores to output fall risk levels, providing more comprehensive information for elderly safety monitoring and timely assistance. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the fall detection method based on millimeter-wave radar fusion of human posture according to an embodiment of the present invention. Figure 2 This is a comparison chart of the point cloud segmentation accuracy of the fall detection method based on millimeter-wave radar fusion of human posture according to an embodiment of the present invention. Figure 3 This is a comparison chart of trajectory fitting accuracy for the fall detection method based on millimeter-wave radar fusion of human posture according to an embodiment of the present invention. Figure 4 This is a comparison chart showing the overall performance of posture recognition algorithms corresponding to the fall detection method based on millimeter-wave radar fusion of human posture in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0018] Figure 1 This is a flowchart illustrating the fall detection method based on millimeter-wave radar fusion of human posture according to an embodiment of the present invention. Figure 1 As shown, the method includes: S1. The target distance information is obtained by collecting human echo signals through millimeter-wave radar and estimating parameters. The angle information is obtained by adaptive beamforming. The velocity information is obtained by phase difference velocimetry. Based on the target distance information, angle information and velocity information, the point cloud mapping relationship is constructed by dense convolution to obtain three-dimensional point cloud data. After dynamic threshold segmentation, establishment of local coordinate system and iterative nearest point registration, a dynamic point cloud sequence is generated. Combined with graph convolutional network, the skeletal features are extracted and the three-dimensional coordinates of key points are output. In one alternative implementation, Human body echo signals are acquired using millimeter-wave radar, and target distance information is obtained through parameter estimation. Angle information is obtained through adaptive beamforming, and velocity information is obtained based on phase difference velocimetry. Based on the target distance, angle, and velocity information, a point cloud mapping relationship is constructed using dense convolution to obtain 3D point cloud data. Dynamic point cloud sequences are generated after dynamic threshold segmentation, establishment of a local coordinate system, and iterative nearest-point registration. Finally, a graph convolutional network is used to extract skeletal features and output the 3D coordinates of key points, including: Collect human echo signals of the target object, construct the signal covariance matrix, determine the signal subspace and noise subspace through eigenvalue decomposition, construct the distance spectrum function to search for peaks and obtain target distance information; The target angle information is obtained by spatial filtering of the human body echo signal through an adaptive beamforming algorithm, and the phase difference sequence is extracted from the echo signal at adjacent time points and Doppler processing is performed to obtain the target velocity information. Dense convolution is performed on the target distance information, target angle information and target velocity information to extract multi-scale features and multi-level combination of features through dense connections. Deconvolution operation is used to construct point cloud mapping relationship to obtain three-dimensional point cloud data. The density distribution characteristics of the three-dimensional point cloud data are calculated. The valley point method is used to determine the initial segmentation threshold and dynamic threshold segmentation is performed according to local density changes. The target point cloud cluster is determined by combining the region growing algorithm and a local coordinate system is established. Based on the local coordinate system, the transformation matrix of the point cloud of two adjacent frames is estimated and iteratively optimized until the registration error converges to generate a dynamic point cloud sequence. Combined with the graph convolutional network, the human body is modeled as a set of key points to construct the skeletal topology. The node features are locally aggregated to extract the skeletal features and the key node features are enhanced to obtain the three-dimensional coordinates of the key points.

[0019] Millimeter-wave radar is used to collect human echo signals from the target object. Taking a 77GHz millimeter-wave radar as an example, the radar is set to transmit a linear frequency modulated continuous wave signal and receive the reflected echo signal. Assume the sampling frequency is 100MHz and the number of sampling points is 1024.

[0020] The acquired echo signals are preprocessed. A moving average filtering method is used to reduce noise in the echo signals, with a window length of 5. For example, the signal values ​​of 5 consecutive sampling points are averaged to obtain a new signal value, and this process is repeated to smooth the entire echo signal. Next, the covariance matrix of the signal is constructed, and eigenvalue decomposition is performed. The eigenvalues ​​are arranged in descending order, and the eigenvector is divided into a signal subspace and a noise subspace based on the magnitude of the eigenvalues. Assume the signal subspace has a dimension of 3 and the noise subspace has a dimension of 1021. Based on the principle of subspace orthogonality, a range spectrum function is constructed. By searching for the peak position of the range spectrum function, the target's distance information can be obtained. Assuming the range spectrum function reaches its peak at the 50th sampling point, the target distance is approximately 7.5 meters (calculated based on radar parameters and signal processing methods).

[0021] Spatial filtering is performed on the human body echo signal to obtain target angle information. A spatial sampling model of the received signal is established, and the array manifold vector is calculated. An 8-element uniform linear array with an element spacing of half a wavelength is assumed. The spatial covariance matrix is ​​constructed, and an iterative optimization method, such as the Capon algorithm, is used to solve for the optimal weight vector. The amplitude and phase of the optimal weight vector are adaptively adjusted according to the spatial distribution characteristics of the signal and interference. The optimal weight vector is applied to the echo signal for spatial filtering to obtain the target angle information. The target angle is assumed to be 30 degrees.

[0022] The filtered signal is subjected to Doppler processing to obtain target velocity information. Phase extraction is performed on the echo signals from adjacent time points to obtain a phase difference sequence. A sliding window method is used to smooth the phase difference sequence, with a window length of 10. Based on the principle of phase difference velocimetry, the target velocity information is calculated. It is assumed that the target velocity is 1 m / s.

[0023] Target distance information (7.5 meters), target angle information (30 degrees), and target velocity information (1 m / s) are input into a dense convolutional network to construct a point cloud mapping. The dense convolutional network includes a feature extraction branch and a feature fusion branch. The feature extraction branch uses convolutional kernels of different scales (e.g., 3x3, 5x5) to extract multi-scale features. The feature fusion branch achieves multi-level combination of features through dense connections and establishes cross-layer connections between network layers to achieve feature reuse. Deconvolution operations are used to upsample the feature map, constructing a point cloud mapping to obtain the 3D point cloud data of the object to be detected.

[0024] Dynamic thresholding segmentation is performed on 3D point cloud data. The density distribution characteristics of the 3D point cloud data are calculated, and a density histogram is constructed. The valley point method is used to determine the initial segmentation threshold, and dynamic thresholding segmentation is performed based on local density changes. High-density regions are used as seed points, and a region growing algorithm is employed for point cloud segmentation. Abnormal point clusters are removed through shape analysis to obtain the target point cloud clusters. A local coordinate system is established, and the normalized point cloud pose is obtained.

[0025] Temporal registration is performed. Based on the normalized point cloud pose, the Iterative Closest Point (ICP) registration algorithm is used to establish point correspondences between adjacent two frames of point clouds and perform feature matching. The transformation matrix between the two sets of point clouds is estimated, and the transformation parameters are iteratively optimized and updated until the registration error converges, thus completing the temporal registration of the point cloud sequence and generating a dynamic point cloud sequence.

[0026] This method combines graph convolutional networks to extract human skeletal features. The human body is modeled as a set of keypoints with spatial connections, constructing a graph model of the skeletal topology. Graph convolution operators are used to propagate features in space by locally aggregating node features. Human skeletal features are extracted through a multi-level graph convolutional structure. An attention mechanism is employed to enhance the feature representation of important nodes and suppress the influence of noisy nodes. The output includes the 3D coordinates and confidence scores of the human keypoints. For example, the 3D coordinates of the left shoulder joint are output as (0.1, 0.2, 0.3), with a confidence score of 0.9.

[0027] In this embodiment, by combining millimeter-wave radar signal processing, point cloud processing, and graph convolutional networks, human posture features can be effectively extracted, improving the accuracy of human posture recognition. The use of dynamic threshold segmentation and temporal registration techniques can effectively cope with complex scenarios such as noise and occlusion, enhancing the robustness of human posture recognition. The use of millimeter-wave radar for non-contact measurement avoids interference with human activities and is suitable for various application scenarios.

[0028] In one alternative implementation, The calculation of the density distribution characteristics of the 3D point cloud data, the determination of the initial segmentation threshold using the valley point method, and the dynamic threshold segmentation based on local density changes, combined with the region growing algorithm to determine the target point cloud cluster and establish a local coordinate system, includes: Based on the 3D point cloud data, a 3D mesh is divided, the number of points contained in each mesh cell is calculated to obtain the local density value of the point cloud, the local density value of the point cloud is divided into density intervals according to the numerical value, and the distribution of the number of points in each density interval is statistically analyzed to establish a density histogram. Local peak points are detected and valley points are determined in the density histogram. The valley point method is used to determine the initial segmentation threshold and a sliding window is set. The density variance of the point cloud within the sliding window is calculated, and the initial segmentation threshold is corrected based on the density variance and the segmentation is repeated. In the segmented high-density region, a local density maximum point is selected as a seed point. A spherical search region is established with the seed point as the center to determine candidate points. The density value of the candidate point is calculated as the difference between the density value of the candidate point and the average density value of all points in the high-density region, as well as the Euclidean distance from the candidate point to the centroid of the high-density region. The candidate points are selected and incorporated into the high-density region using the region growing algorithm to obtain the target point cloud. The shape compactness parameters and surface smoothness parameters of the target point cloud are calculated by shape analysis. Abnormal point clusters are eliminated by combining the preset multidimensional feature constraint threshold to obtain the target point cloud cluster. The centroid position of the target point cloud cluster is calculated. The scatter matrix is ​​calculated as the spatial distribution covariance matrix and eigenvalue decomposition is performed. The eigenvector with the largest eigenvalue is used as the principal axis direction to establish a local coordinate system.

[0029] The input 3D point cloud data undergoes structured processing. To effectively analyze the local density characteristics of the point cloud data, the space containing the 3D point cloud data is divided into a series of equal-sized 3D grids. For example, the point cloud data space can be divided into 100x100x100 small cubes, each of which is a grid cell. Then, the number of point cloud data points contained in each grid cell is counted. This number reflects the point cloud density within that grid cell. Suppose one grid cell contains 5 points, while another grid cell contains 20 points; the latter's point cloud density is significantly higher than the former's. By collecting the point cloud density values ​​of all grid cells, the local density distribution of the point cloud is obtained.

[0030] A point cloud density histogram is constructed based on local density values. The local density values ​​are divided into certain intervals, for example, the density values ​​from 0 to 100 are divided into 10 intervals, each containing 10 density values. Then, the number of grid cells contained in each density interval is counted, and the results are plotted as a histogram. The horizontal axis of the histogram represents the density interval, and the vertical axis represents the number of grid cells within that interval. The histogram provides a visual understanding of the density distribution of the point cloud data. For example, if the histogram shows high values ​​in both low-density and high-density intervals, but low values ​​in the medium-density interval, it indicates that the point cloud data contains both low-density and high-density areas, with fewer medium-density areas.

[0031] The initial segmentation threshold is determined from the density histogram. A density histogram typically contains multiple local peaks, which correspond to high-density regions in the point cloud data. The lowest point between any two adjacent peaks, known as the valley, is identified, and the density value corresponding to this valley is used as the initial segmentation threshold. For example, if there is a peak at density values ​​of 20 and 50, and a valley at density value 30, then 30 is used as the initial segmentation threshold.

[0032] The segmentation threshold is dynamically adjusted based on local density variations. A variable-size sliding window, such as an initial 10x10x10 cube, is set in the 3D point cloud space. The average density and density variance of the point cloud data within the sliding window are calculated. If the density variance is large, it indicates that the density distribution within the window is uneven, and the segmentation threshold needs to be increased, for example, by 10%. Conversely, if the density variance is small, it indicates that the density distribution within the window is uniform, and the segmentation threshold can be decreased, for example, by 5%. By continuously adjusting the segmentation threshold, it is possible to better adapt to local density variations in the point cloud data.

[0033] Perform point cloud segmentation. Select the point with the highest density in the high-density region as the seed point. Establish a spherical search region centered on the seed point, for example, a spherical region with a radius of 5. Use the set of point cloud data points within the search region as the candidate point set. For each candidate point, calculate the difference between its density value and the average density value of all points in the current growth region, as well as the distance from the candidate point to the centroid of the current growth region. If the density difference is less than a first preset threshold (e.g., 0.1) and the distance is less than a second preset threshold (e.g., 2), then add the candidate point to the current growth region. Repeat the above growth process until no new points are added, thus obtaining a target point cloud.

[0034] Shape analysis is performed on the segmented target point cloud to remove outlier clusters. The spatial dimensions of the target point cloud are obtained by calculating the difference between the maximum and minimum coordinate values ​​along the three coordinate axes. The consistency of the normal vectors of the surface points of the target point cloud is calculated to obtain the shape compactness parameter. The curvature distribution of the target point cloud is calculated to obtain the surface smoothness parameter. The spatial dimensions, shape compactness parameter, and surface smoothness parameter are combined into a single feature vector. Multidimensional feature constraint thresholds are set, for example, spatial dimensions less than 10, shape compactness parameter greater than 0.8, and surface smoothness parameter less than 0.2. If the feature vector of the target point cloud does not meet the preset constraints, the point cloud is considered an outlier cluster and is removed.

[0035] The pose of the preserved target point cloud cluster is normalized. The centroid position of the target point cloud cluster is calculated and translated to the origin. The scatter matrix of the target point cloud cluster is calculated and used as the spatial distribution covariance matrix. Eigenvalue decomposition is performed on the covariance matrix to obtain three eigenvectors and their corresponding eigenvalues. The eigenvector with the largest eigenvalue is selected as the principal axis direction. A local coordinate system is established based on the principal axis direction. The point cloud coordinates are mapped from the global coordinate system to the local coordinate system through rotation transformation, thus obtaining the normalized point cloud pose.

[0036] In this embodiment, by dynamically adjusting the segmentation threshold, the local density changes of point cloud data can be better adapted, thereby improving the accuracy of point cloud segmentation. By removing abnormal point clusters through shape analysis, noise points and outliers can be effectively removed, improving the robustness of the algorithm. By establishing a local coordinate system, target point clouds with different poses can be normalized, which facilitates subsequent feature extraction and recognition.

[0037] S2. Construct the skeleton connection relationship based on the 3D coordinates of key points and calculate the spatial vector and angle to obtain the initial pose features. Generate an adaptive feature sequence through dynamic sampling. Extract key pose change features through a self-attention mechanism. Combine the dynamic point cloud sequence to construct a spatial pyramid pooling network to extract multi-scale features. Construct a motion feature descriptor based on the centroid trajectory of the point cloud. Perform feature fusion through a graph attention network to obtain multi-modal fusion features. In one alternative implementation, Based on the 3D coordinates of key points, a skeleton connection relationship is constructed, and spatial vectors and angles are calculated to obtain initial pose features. An adaptive feature sequence is generated through dynamic sampling, and key pose change features are extracted through a self-attention mechanism. A spatial pyramid pooling network is constructed in conjunction with the dynamic point cloud sequence to extract multi-scale features, and a motion feature descriptor is constructed based on the centroid trajectory of the point cloud. Feature fusion is performed through a graph attention network to obtain multimodal fused features, including: Obtain the 3D coordinates of the key points corresponding to the key points of the human body, establish the skeletal connection relationship between adjacent key points of the human body, and obtain the initial pose features by combining the spatial vectors and angles between adjacent key points of the human body. Calculate the displacement of the human body key points between adjacent time frames, determine the sampling frequency based on the displacement, and perform dynamic sampling. Arrange the sampled initial posture features in chronological order to generate an adaptive feature sequence. The adaptive feature sequence is segmented by sliding based on the self-attention mechanism. Within each time window, the similarity score between feature vectors is calculated to construct an attention weight matrix. The adaptive feature sequence is then weighted and aggregated to obtain key pose change features. A spatial pyramid pooling network is constructed by combining dynamic point cloud sequences. Multi-level downsampling is performed through max pooling, and the point density distribution, normal vector distribution, and curvature distribution in each pooling region are calculated to extract multi-scale features. Based on the dynamic point cloud sequence, the centroid position of the point cloud is calculated to obtain the centroid trajectory of the point cloud. A smooth trajectory is obtained by spline interpolation, and the curvature and torsion are calculated to construct a motion feature descriptor. The key posture change features, multi-scale features, and motion feature descriptors are used as graph nodes. The connection relationship between nodes is established based on feature similarity, and the information transmission weight is calculated to perform feature fusion to obtain multimodal fusion features.

[0038] Obtain the 3D coordinates of key points on the human body. These key points include 15 points: head, neck, shoulders, elbows, wrists, center of the torso, hips, knees, and ankles. For example, at a certain moment, the 3D coordinates of the head key point are (0.1, 0.2, 1.7), the 3D coordinates of the neck key point are (0.15, 0.22, 1.65), and so on, to obtain the 3D coordinates of all 15 key points.

[0039] A human skeletal model is constructed based on the connections between adjacent keypoints. For example, the head and neck are connected, the neck and left shoulder are connected, and the neck and right shoulder are connected. The spatial vectors and angles between adjacent keypoints are calculated to obtain initial pose features. For example, the spatial vector between the head and neck is (0.05, 0.02, -0.05), and the angle between them can be calculated using vectors. The spatial vectors and angles between all adjacent keypoints are combined to form the initial pose feature vector.

[0040] The initial pose features are fed into the dynamic sampling module. This module contains three sampling sub-modules: high-frequency, mid-frequency, and low-frequency. An appropriate sampling frequency is selected based on the displacement of human keypoints between adjacent time frames. For example, an upper threshold of 0.2 and a lower threshold of 0.05 are set. If the displacement of a keypoint between two adjacent frames is greater than 0.2, high-frequency sampling is used; if the displacement is between 0.05 and 0.2, mid-frequency sampling is used; and if the displacement is less than 0.05, low-frequency sampling is used. Each keypoint is dynamically sampled independently, and the sampled pose features are arranged in chronological order to generate an adaptive feature sequence.

[0041] Key pose change features are extracted using a self-attention mechanism. Short, medium, and long time windows are set, for example, 0.5 seconds, 1 second, and 2 seconds, respectively. The adaptive feature sequence is segmented by sliding, and the similarity score between feature vectors is calculated within each time window to construct an attention weight matrix. The features are then weighted and aggregated based on the attention weight matrix to obtain key pose change features at different time scales.

[0042] Based on key attitude change features and dynamic point cloud sequences, a spatial pyramid pooling network is constructed. This network comprises multiple layers. The first layer uses the original point cloud resolution, and subsequent layers are downsampled using max pooling. For example, the first layer has a resolution of 256x256, the second layer 128x128, and so on. Within each pooling region, statistical features such as point density distribution, normal vector distribution, and curvature distribution are calculated. The features from each layer are upsampled to the same resolution and combined to obtain multi-scale spatial features. The centroid position of the point cloud is calculated, resulting in the centroid trajectory. Control points are uniformly sampled on the trajectory, and a smooth trajectory is obtained through spline interpolation. The curvature and torsion of the smooth trajectory are calculated, and a multi-dimensional motion feature descriptor is constructed by combining the geometric features and motion parameters of the trajectory.

[0043] A graph attention network is used for multimodal feature fusion. Adaptive feature sequences, multi-scale spatial features, and multidimensional motion feature descriptors are used as graph nodes. Connections between nodes are established based on feature similarity, and the information transfer weights between nodes are calculated through a graph attention mechanism. The three types of features are then fused hierarchically to obtain the final multimodal fused feature.

[0044] In this embodiment, by fusing multimodal features, human actions can be described more comprehensively, thereby improving recognition accuracy. The dynamic sampling module can adaptively adjust the sampling frequency to effectively handle actions at different speeds, improving the robustness of the algorithm. The spatial pyramid pooling network can effectively extract multi-scale spatial features, reducing computational complexity and improving the efficiency of the algorithm.

[0045] In one alternative implementation, The process involves calculating the centroid position of the point cloud to obtain its trajectory, uniformly sampling control points along this trajectory, and obtaining a smooth trajectory through spline interpolation. The curvature and torsion of the smooth trajectory are then calculated. A multidimensional motion feature descriptor is constructed by combining the geometric features and motion parameters of the trajectory. The process includes calculating the centroid position of the point cloud based on the dynamic point cloud sequence, obtaining the smooth trajectory through spline interpolation, calculating curvature and torsion, and constructing a motion feature descriptor. The centroid position of the point cloud is obtained by calculating the mean coordinate value of all points in the point cloud in each frame of the point cloud sequence, and the centroid positions of the point cloud are connected in chronological order to form the centroid trajectory of the point cloud. Control points are obtained by uniform sampling on the centroid trajectory of the point cloud. Multiple interpolation points are generated between adjacent control points using spline interpolation. At each interpolation point, a tangent vector, a principal normal vector, and a secondary normal vector are calculated. The tangent vector is obtained by the position difference between adjacent interpolation points. The principal normal vector is perpendicular to the tangent vector and points to the curvature center of the smooth trajectory. The secondary normal vector is obtained by the cross product of the tangent vector and the principal normal vector. The tangent vector, the principal normal vector, and the secondary normal vector are normalized to construct a local coordinate system. The curvature and torsion of the smooth trajectory at each interpolation point are calculated based on the local coordinate system. The geometric features and motion parameters of the smooth trajectory are extracted, and the curvature, torsion, geometric features and motion parameters are combined to construct a multidimensional motion feature descriptor.

[0046] Obtain the point cloud sequence. Assume you have a point cloud sequence containing 100 frames, each frame containing hundreds to thousands of 3D points. Calculate the centroid position of each frame. For example, the first frame contains 500 points, each with its own 3D coordinates (x, y, z). Sum the x-coordinates of all 500 points and divide by 500 to obtain the mean in the x-direction. The y and z coordinates are calculated similarly. This gives the centroid coordinates (x1, y1, z1) of the first frame. Continue this process, calculating the centroid coordinates of all 100 frames to obtain 100 centroid positions.

[0047] Connect the centroid trajectories of the point cloud. Connect these 100 centroid coordinates in chronological order to form a centroid trajectory of the point cloud. Sample control points uniformly along this trajectory. Assume 10 control points are sampled uniformly along the trajectory. The total length of the trajectory can be divided into 10 equal parts, and a control point is selected at each division point.

[0048] Spline interpolation is performed on the control points. Cubic spline interpolation is used to interpolate between adjacent control points. Assuming 9 points are interpolated between any two adjacent control points, a total of 90 points are interpolated between the original 10 control points. Adding these to the original 10 control points, a total of 100 interpolated points are obtained, forming a smooth trajectory. The generation of interpolated points ensures positional continuity and tangent continuity, meaning that the coordinate values ​​of adjacent interpolated points and the tangent direction transition smoothly.

[0049] Construct a local coordinate system. At each interpolation point, calculate the tangent vector, principal normal vector, and binormal vector. The tangent vector can be obtained by the position difference between two adjacent interpolation points; for example, the tangent vector at the i-th interpolation point can be obtained by subtracting the coordinates of the i-th interpolation point from the coordinates of the (i+1)-th interpolation point. The principal normal vector is perpendicular to the tangent vector and points towards the center of curvature of the trajectory; it can be obtained by calculating the rate of change of the tangent vector. The binormal vector is obtained by the cross product of the tangent vector and the principal normal vector. Normalize the tangent vector, principal normal vector, and binormal vector (i.e., normalize their lengths) to construct the local coordinate system.

[0050] Calculate curvature and torsion. Curvature represents the degree of trajectory bending and can be obtained by calculating the second derivative of the trajectory at the interpolation point. The second derivative reflects the rate of change of the tangent direction of the trajectory. Torsion represents the degree of trajectory distortion and can be obtained by calculating the third derivative of the trajectory at the interpolation point. The third derivative reflects the rate of change of the trajectory bending.

[0051] Extract geometric features. Calculate the total arc length of the smooth trajectory, which is the sum of the distances between all adjacent interpolation points. Identify the location and number of inflection points on the trajectory; inflection points are points where the direction of the trajectory's curvature changes. Calculate the cumulative turning angle of the trajectory, which is the sum of the angles between the tangent vectors of all adjacent interpolation points.

[0052] Obtain motion parameters. Calculate the trajectory velocity using the position differences between adjacent interpolation points. For example, the velocity of the i-th interpolation point can be obtained by subtracting the coordinates of the i-th interpolation point from the coordinates of the (i+1)-th interpolation point and then dividing by the time difference. Calculate the trajectory acceleration using the velocity differences. For example, the acceleration of the i-th interpolation point can be obtained by subtracting the velocity of the i-th interpolation point from the velocity of the (i+1)-th interpolation point and then dividing by the time difference.

[0053] Construct a multidimensional motion feature descriptor. Combine the calculated curvature, torsion, geometric features (total arc length, inflection point location and number, cumulative rotation angle) and motion parameters (velocity, acceleration) to construct a multidimensional motion feature descriptor. This descriptor can comprehensively describe the motion characteristics of the trajectory.

[0054] In this embodiment, by combining the geometric features and motion parameters of the trajectory, a multidimensional motion feature descriptor is constructed, which can more comprehensively describe the motion characteristics of the trajectory and avoid the limitations of single feature description. The multidimensional motion feature descriptor contains richer trajectory information, which can effectively distinguish different types of motion trajectories, improve the accuracy of trajectory recognition, and transform the complex motion pattern of the trajectory into a multidimensional feature vector, which can simplify the complexity of trajectory analysis and facilitate subsequent applications such as pattern recognition and machine learning.

[0055] S3. Input the multimodal fusion features into the dual-branch state recognition network, and obtain the initial judgment result and anomaly probability through lightweight convolution. When the anomaly probability exceeds the threshold, enhance the features through residual attention and extract the spatiotemporal combined features by combining spatiotemporal graph convolution. Obtain the recognition result through posture classification, action classification and Bayesian fusion. If it is a fall state, determine the fall action and output the risk level.

[0056] In one alternative implementation, Multimodal fusion features are input into a dual-branch state recognition network. Lightweight convolutions are used to obtain initial judgment results and anomaly probabilities. When the anomaly probability exceeds a threshold, residual attention is used to enhance features, and spatiotemporal combined features are extracted using spatiotemporal graph convolutions. The recognition result is obtained through pose classification, action classification, and Bayesian fusion. If a fall is identified, the fall action is determined, and the risk level is output, including: Multimodal fusion features are input into a dual-branch state recognition network, and after performing lightweight convolution, they are normalized and activated. The spatial dimension is reduced by pooling, and the initial judgment result and anomaly probability are output through a fully connected layer. When the anomaly probability exceeds a preset threshold, the multimodal fusion feature is enhanced by residual attention. A spatial attention map is generated by convolution, channel weights are generated by global pooling and a two-layer perceptron, and the temporal dimension attention score is calculated by a sliding window. The spatial attention map, channel weights, and temporal dimension attention score are multiplied by the multimodal fusion feature to obtain the enhanced feature. The human skeletal structure is modeled as a graph by combining spatiotemporal graph convolution, the enhanced features are mapped to graph nodes, graph convolution is performed in the spatial domain through adjacency matrix, and temporal modeling is performed in the temporal domain to extract spatiotemporal combined features. Based on the spatiotemporal combination features, the probability distribution of basic posture categories is output through posture classification, the probability distribution of action categories is output through action classification, the basic posture categories and action categories are combined, the posterior probability is calculated through Bayesian fusion, and the combination with the highest posterior probability is selected to obtain the recognition result. If the identification result is a fall, the curvature, torsion, velocity, and acceleration in the pre-acquired multidimensional motion feature descriptor are analyzed to determine the fall action and output the risk level.

[0057] Acquire multimodal data. Use a camera to acquire RGB video data, a depth sensor to acquire depth image data, and an inertial measurement unit (IMU) to acquire acceleration and angular velocity data. For example, use a Kinectv2 camera to simultaneously acquire RGB video and depth images, and use an MPU6050 IMU to acquire acceleration and angular velocity data, with the acquisition frequency set to 30 frames per second.

[0058] Multimodal feature extraction and fusion are performed. For RGB video data, the OpenPose human pose estimation library is used to extract the 2D coordinate information of 25 keypoints. For depth image data, the human body region is segmented, and the depth histogram features of the human body region are calculated. For inertial measurement unit data, statistical features such as the mean, variance, and peak value of acceleration and angular velocity are calculated. The extracted multimodal features are concatenated into a fusion feature vector. For example, the 2D coordinates (50-dimensional), depth histogram features (16-dimensional), and statistical features of acceleration and angular velocity (12-dimensional) of the 25 keypoints are concatenated into a 78-dimensional fusion feature vector.

[0059] The fused features are input into a pre-constructed two-branch state recognition network. This network consists of a lightweight convolutional neural network. Each convolutional operation is followed by a batch normalization layer and a ReLU activation function. Max pooling layers are used to reduce the spatial dimensionality of the feature map and increase the number of channels. Finally, a fully connected layer outputs the initial state judgment result and the state anomaly probability score. For example, using MobileNetV3 as a lightweight convolutional neural network, the output is the initial state judgment result (standing, walking, sitting, etc.) and the state anomaly probability score (between 0 and 1). Assume the first anomaly threshold is set to 0.8.

[0060] If the state anomaly probability score is greater than the first anomaly threshold, feature enhancement and spatiotemporal feature extraction are performed. A residual attention module is used to enhance the fused features. The residual attention module includes spatial attention, channel attention, and temporal attention submodules. The spatial attention submodule extracts spatial information through max pooling and average pooling, respectively, and then generates a spatial attention map via convolutional layers. The channel attention submodule extracts channel statistics through global average pooling and then generates channel weights via a two-layer perceptron. The temporal attention submodule calculates the temporal dimension attention score using a sliding window. For example, the sliding window size is set to 5 frames.

[0061] A spatiotemporal feature correlation model is constructed by combining a spatiotemporal graph convolutional network. The human skeletal structure is modeled as a graph structure, with nodes representing key points and edges representing skeletal connections. In the spatial domain, graph convolution operations are performed to pass information between nodes through an adjacency matrix. In the temporal domain, standard convolution is used to perform temporal modeling of the node feature sequences. Spatiotemporal combined features are obtained by concatenating multiple layers of spatiotemporal graph convolutional modules. For example, a 3-layer spatiotemporal graph convolutional module is used, with each layer outputting 64-dimensional features.

[0062] A pose classifier and an action classifier are constructed based on spatiotemporal combined features. Both classifiers employ a multilayer perceptron structure, with random deactivation layers added between each layer, and ReLU activation function used in the intermediate layers. The pose classifier outputs the probability distribution of basic pose categories, and the action classifier outputs the probability distribution of action categories. For example, the pose classifier outputs the probability distribution of basic poses such as standing, bending over, and lying down, while the action classifier outputs the probability distribution of actions such as walking, sitting, and falling.

[0063] A Bayesian fusion algorithm is used to combine basic pose categories and action categories. The posterior probability of each state combination is calculated based on the state transition probability matrix, and the combination with the highest posterior probability is selected as the state recognition result. For example, the state transition probability matrix represents the probability of transitioning from one state to another.

[0064] If the state recognition result indicates a fall-related state, then the fall action is determined and a risk assessment is performed. The fall action is determined by analyzing the trajectory curvature, torsion change, velocity, and acceleration information in the multidimensional motion feature descriptor. The feature vector corresponding to the fall action is input into a multi-layer neural network to extract deep feature representations. Prior rule knowledge is combined to assess the motion characteristics, posture changes, and impact intensity during the fall. A risk assessment model is constructed based on the confidence score to output the fall risk level (e.g., low risk, medium risk, high risk). For example, if the body's center of gravity drops rapidly while the impact acceleration is high, it is judged as a high-risk fall.

[0065] In this embodiment, by using multimodal fusion and spatiotemporal feature association, human state information can be captured more comprehensively, thereby improving the accuracy of state recognition. Especially in complex scenarios, it can not only identify the fall state, but also assess the fall risk level based on the characteristics of the fall action, providing a basis for timely intervention. The use of lightweight convolutional neural networks and efficient spatiotemporal graph convolution operations ensure the computational efficiency of the method, enabling it to be applied to real-time scenarios.

[0066] In one alternative implementation, By combining spatiotemporal graph convolution to model the human skeletal structure as a graph, mapping the enhanced features to graph nodes, performing graph convolution in the spatial domain through an adjacency matrix, and performing temporal modeling and extracting spatiotemporal combined features in the temporal domain, the following steps are taken: Human key points are used as nodes of a graph structure, and the skeletal connection relationship between adjacent human key points is defined as the edge of the graph structure. An adjacency matrix is ​​generated based on the skeletal connection relationship, and the enhanced features are mapped to graph nodes to construct a node feature sequence. By combining spatiotemporal graph convolution in the spatial domain, the spatial association strength between the target node and the directly connected node is calculated through the adjacency matrix to obtain the spatial attention coefficient. The spatial attention coefficient is then weighted and combined with the node features corresponding to the directly connected nodes to obtain the spatial aggregation feature. The node feature sequence is extracted from consecutive frames on the time axis using a sliding window. The node feature sequence is then subjected to a standard convolution operation to extract local temporal patterns. Motion features within different time ranges are extracted layer by layer through multiple layers of standard convolution. Spatial feature extraction and temporal feature extraction are alternately chained together. Spatial feature extraction includes graph convolution, normalization, and activation functions, while temporal feature extraction includes standard convolution, normalization, and deactivation. Shortcut connection paths are added between adjacent feature extractions to directly pass the input features to the output and add them to the transformed features. The output features of each layer are combined, and pooling is performed on the features of all nodes to extract global features. The pooled features are then mapped to a unified dimensional space through feature transformation to obtain spatiotemporal combined features.

[0067] Obtain pre-determined human keypoint data. Human keypoint data can be extracted from videos or images using pose estimation algorithms such as OpenPose and AlphaPose. Assume 18 human keypoints are extracted, including the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Each keypoint contains its two-dimensional coordinates (x, y) in the image or video frame and its confidence value c, representing the detection reliability of that keypoint. These 18 keypoints are then used as nodes in a graph structure.

[0068] The edges of the graph structure are defined based on the skeletal connections of the human body. Adjacent key points are connected by skeletal structures; for example, the nose is connected to the left eye, the left eye to the left ear, the left shoulder to the left elbow, and so on. These connections generate an adjacency matrix A. The adjacency matrix is ​​an 18x18 matrix. If there is a connection between node i and node j, then A(i, j) = 1; otherwise, A(i, j) = 0. The initial feature vector of each node is formed by combining its spatial coordinates (x, y) and confidence value c, creating a three-dimensional vector (x, y, c). For example, if the nose's coordinates are (100, 50) and its confidence value is 0.9, then its initial feature vector is (100, 50, 0.9).

[0069] Spatial domain feature extraction is performed. For each target node, the spatial association strength between it and its directly connected nodes is calculated to obtain a spatial attention coefficient. The spatial association strength can be calculated based on factors such as distance and angle between nodes. The spatial attention coefficient is then weighted and combined with the node features corresponding to the directly connected nodes to obtain the spatial aggregation feature of the target node. For example, the spatial association strength between the left shoulder node and its connected nodes (left elbow, right shoulder, neck) is calculated to obtain the corresponding attention coefficient. These coefficients are then multiplied by the feature vectors of the left elbow, right shoulder, and neck nodes respectively, and the results are summed to obtain the spatial aggregation feature of the left shoulder node. To extract spatial association patterns between nodes from multiple dimensions, multiple sets of feature transformation and attention calculation units can be used. The output features of multiple sets of units are combined to obtain an enhanced node representation.

[0070] Temporal feature extraction is performed. A sliding window is used to process the enhanced node representation. Assuming the sliding window size is 5 frames, a sequence of node features from 5 consecutive frames is extracted each time. Standard convolution operations are then performed on this sequence to extract local temporal patterns. For example, a one-dimensional convolution with a kernel size of 3 is used to convolve the feature sequence from 5 frames. Motion features within different time ranges are extracted layer by layer through multiple layers of standard convolution to obtain the node motion trajectory features.

[0071] A spatiotemporal graph convolutional module is constructed by alternately cascading spatial feature extraction units and temporal feature extraction units. The spatial feature extraction unit includes graph convolutional layers, normalization layers, and activation function layers. The temporal feature extraction unit includes standard convolutional layers, normalization layers, and deactivation layers. Shortcut connections are added between adjacent feature extraction units to directly pass the input features to the output and add them to the transformed features.

[0072] The output features of each spatiotemporal graph convolutional module are combined to obtain a multi-scale feature representation. Pooling operations, such as average pooling or max pooling, are performed on the features of all nodes to extract global features. The pooled features are then mapped to a unified dimensional space through feature transformation to obtain the final spatiotemporal combined features.

[0073] In this embodiment, by modeling the human skeletal structure and spatiotemporal correlation, subtle differences in human movements can be better captured, thereby improving the accuracy of action recognition. Graph structures can effectively represent the relationships between key points of the human body, while spatiotemporal convolution can capture motion patterns at different time scales with lower computational complexity because it only focuses on information about key points of the human body and does not need to process a large amount of video pixel data.

[0074] A second aspect of the present invention provides a fall detection system based on millimeter-wave radar fusion of human posture, comprising: The first unit is used to collect human echo signals through millimeter-wave radar and perform parameter estimation to obtain target distance information, obtain angle information through adaptive beamforming, obtain velocity information based on phase difference velocimetry, and obtain three-dimensional point cloud data by constructing point cloud mapping relationship through dense convolution based on target distance information, angle information and velocity information. After dynamic threshold segmentation, establishment of local coordinate system and iterative nearest point registration, dynamic point cloud sequence is generated. Combined with graph convolutional network, skeletal features are extracted and the three-dimensional coordinates of key points are output. The second unit is used to construct the skeleton connection relationship based on the three-dimensional coordinates of key points and calculate the spatial vector and the included angle to obtain the initial pose features. It generates an adaptive feature sequence through dynamic sampling, extracts key pose change features through a self-attention mechanism, constructs a spatial pyramid pooling network in combination with the dynamic point cloud sequence to extract multi-scale features, constructs a motion feature descriptor based on the centroid trajectory of the point cloud, and obtains multi-modal fusion features through graph attention network. The third unit is used to input multimodal fusion features into the dual-branch state recognition network. The initial judgment result and anomaly probability are obtained through lightweight convolution. When the anomaly probability exceeds the threshold, the feature is enhanced by residual attention and spatiotemporal combined features are extracted by spatiotemporal graph convolution. The recognition result is obtained by posture classification, action classification and Bayesian fusion. If it is a fall state, the fall action is determined and the risk level is output.

[0075] Figure 2 This image shows a comparison of the point cloud segmentation accuracy of the fall detection method based on millimeter-wave radar fusion of human posture according to an embodiment of the present invention. It illustrates the accuracy comparison of three point cloud segmentation methods at different point cloud scales. The proposed solution (circular markers), through dynamic density thresholding and multi-dimensional feature constraints, steadily improves its accuracy from 92% to 98% as the number of point clouds increases from 0 to 400. The DBSCAN algorithm (square markers) improves its accuracy from 88% to 90%, and the region growing algorithm (triangle markers) improves its accuracy from 90% to 93%. When the number of point clouds is 400, the accuracy of the proposed solution reaches 98%, significantly higher than the 90% of the DBSCAN algorithm and the 93% of the region growing algorithm. Experimental data shows that the proposed solution has better scalability in large-scale point cloud processing, and its accuracy shows a stable upward trend with increasing point cloud scale. Especially after the number of point clouds exceeds 200, the advantages of the proposed solution are more obvious, improving the accuracy by an average of 5-8 percentage points compared to existing methods.

[0076] Figure 3This diagram compares the trajectory fitting accuracy of the fall detection method based on millimeter-wave radar fusion with human posture according to an embodiment of the present invention. It shows the comparison of fitting accuracy of three different trajectory fitting methods under different numbers of control points. The proposed solution (circular marker) uses spline interpolation combined with a dynamic local coordinate system. As the number of control points increases from 20 to 100, the fitting accuracy steadily improves from 0.65 to 0.95. The traditional B-spline fitting method (square marker) improves its accuracy from 0.60 to 0.78, and the least squares fitting method (triangle marker) improves its accuracy from 0.62 to 0.85. When the number of control points is 100, the fitting accuracy of the proposed solution reaches 0.95, significantly better than the 0.78 of B-spline fitting and the 0.85 of least squares fitting. The proposed solution, by introducing a local coordinate system and multi-dimensional feature description, exhibits better adaptability in trajectory fitting tasks, especially when the number of control points is large. When the number of control points is above 60, the proposed solution improves accuracy by an average of 0.15-0.2 compared to existing methods. Figure 3 The horizontal axis represents the number of control points (20-100), and the vertical axis represents the trajectory fitting accuracy (0.5-1.0).

[0077] Figure 4 This chart compares the overall performance of posture recognition algorithms corresponding to the fall detection method based on millimeter-wave radar fusion of human posture according to embodiments of the present invention. The proposed solution (circular marker) achieves high levels in all indicators: positioning accuracy 95%, time efficiency 92%, anti-occlusion capability 90%, adaptability 88%, and real-time performance 94%. The traditional ST-GCN method (square marker) achieves performance indicators of 88%, 85%, 82%, 80%, and 86%, respectively; the 2s-AGCN method (triangle marker) achieves performance indicators of 90%, 87%, 85%, 83%, and 89%. The proposed solution, through the combination of multi-scale feature extraction and spatiotemporal attention mechanisms, achieves significant improvements in all performance indicators, with an average improvement of 5-8 percentage points. The advantages of the proposed solution are particularly evident in the two key indicators of positioning accuracy and time efficiency, demonstrating its strong practical value in real-world applications.

[0078] A third aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0079] Fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0080] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fall detection method based on millimeter-wave radar fusion of human posture, characterized in that, include: The target distance information is obtained by collecting human echo signals with millimeter-wave radar and estimating parameters. The angle information is obtained by adaptive beamforming. The velocity information is obtained by phase difference velocimetry. Based on the target distance information, angle information and velocity information, the point cloud mapping relationship is constructed by dense convolution to obtain three-dimensional point cloud data. After dynamic threshold segmentation, establishment of local coordinate system and iterative nearest point registration, a dynamic point cloud sequence is generated. Combined with graph convolutional network, the skeletal features are extracted and the three-dimensional coordinates of key points are output. The skeleton connection relationship is constructed based on the 3D coordinates of key points, and the spatial vector and angle are calculated to obtain the initial pose features. An adaptive feature sequence is generated through dynamic sampling. Key pose change features are extracted through a self-attention mechanism. A spatial pyramid pooling network is constructed in combination with the dynamic point cloud sequence to extract multi-scale features. A motion feature descriptor is constructed based on the centroid trajectory of the point cloud. Multimodal fusion features are obtained through graph attention network. The multimodal fusion features are input into the dual-branch state recognition network. The initial judgment result and anomaly probability are obtained through lightweight convolution. When the anomaly probability exceeds the threshold, the features are enhanced by residual attention and spatiotemporal combined features are extracted by spatiotemporal graph convolution. The recognition result is obtained by posture classification, action classification and Bayesian fusion. If it is a fall state, the fall action is determined and the risk level is output.

2. The method according to claim 1, characterized in that, Human body echo signals are acquired using millimeter-wave radar, and target distance information is obtained through parameter estimation. Angle information is obtained through adaptive beamforming, and velocity information is obtained based on phase difference velocimetry. Based on the target distance, angle, and velocity information, a point cloud mapping relationship is constructed using dense convolution to obtain 3D point cloud data. Dynamic point cloud sequences are generated after dynamic threshold segmentation, establishment of a local coordinate system, and iterative nearest-point registration. Finally, a graph convolutional network is used to extract skeletal features and output the 3D coordinates of key points, including: Collect human echo signals of the target object, construct the signal covariance matrix, determine the signal subspace and noise subspace through eigenvalue decomposition, construct the distance spectrum function to search for peaks and obtain target distance information; The target angle information is obtained by spatial filtering of the human body echo signal through an adaptive beamforming algorithm, and the phase difference sequence is extracted from the echo signal at adjacent time points and Doppler processing is performed to obtain the target velocity information. Dense convolution is performed on the target distance information, target angle information and target velocity information to extract multi-scale features and multi-level combination of features through dense connections. Deconvolution operation is used to construct point cloud mapping relationship to obtain three-dimensional point cloud data. The density distribution characteristics of the three-dimensional point cloud data are calculated. The valley point method is used to determine the initial segmentation threshold and dynamic threshold segmentation is performed according to local density changes. The target point cloud cluster is determined by combining the region growing algorithm and a local coordinate system is established. Based on the local coordinate system, the transformation matrix of the point cloud of two adjacent frames is estimated and iteratively optimized until the registration error converges to generate a dynamic point cloud sequence. Combined with the graph convolutional network, the human body is modeled as a set of key points to construct the skeletal topology. The node features are locally aggregated to extract the skeletal features and the key node features are enhanced to obtain the three-dimensional coordinates of the key points.

3. The method according to claim 2, characterized in that, The calculation of the density distribution characteristics of the 3D point cloud data, the determination of the initial segmentation threshold using the valley point method, and the dynamic threshold segmentation based on local density changes, combined with the region growing algorithm to determine the target point cloud cluster and establish a local coordinate system, includes: Based on the 3D point cloud data, a 3D mesh is divided, the number of points contained in each mesh cell is calculated to obtain the local density value of the point cloud, the local density value of the point cloud is divided into density intervals according to the numerical value, and the distribution of the number of points in each density interval is statistically analyzed to establish a density histogram. Local peak points are detected and valley points are determined in the density histogram. The valley point method is used to determine the initial segmentation threshold and a sliding window is set. The density variance of the point cloud within the sliding window is calculated, and the initial segmentation threshold is corrected based on the density variance and the segmentation is repeated. In the segmented high-density region, a local density maximum point is selected as a seed point. A spherical search region is established with the seed point as the center to determine candidate points. The density value of the candidate point is calculated as the difference between the density value of the candidate point and the average density value of all points in the high-density region, as well as the Euclidean distance from the candidate point to the centroid of the high-density region. The candidate points are selected and incorporated into the high-density region using the region growing algorithm to obtain the target point cloud. The shape compactness parameters and surface smoothness parameters of the target point cloud are calculated by shape analysis. Abnormal point clusters are eliminated by combining the preset multidimensional feature constraint threshold to obtain the target point cloud cluster. The centroid position of the target point cloud cluster is calculated. The scatter matrix is ​​calculated as the spatial distribution covariance matrix and eigenvalue decomposition is performed. The eigenvector with the largest eigenvalue is used as the principal axis direction to establish a local coordinate system.

4. The method according to claim 1, characterized in that, Based on the 3D coordinates of key points, a skeleton connection relationship is constructed, and spatial vectors and angles are calculated to obtain initial pose features. An adaptive feature sequence is generated through dynamic sampling, and key pose change features are extracted through a self-attention mechanism. A spatial pyramid pooling network is constructed in conjunction with the dynamic point cloud sequence to extract multi-scale features, and a motion feature descriptor is constructed based on the centroid trajectory of the point cloud. Feature fusion is performed through a graph attention network to obtain multimodal fused features, including: Obtain the 3D coordinates of the key points corresponding to the key points of the human body, establish the skeletal connection relationship between adjacent key points of the human body, and obtain the initial pose features by combining the spatial vectors and angles between adjacent key points of the human body. Calculate the displacement of the human body key points between adjacent time frames, determine the sampling frequency based on the displacement, and perform dynamic sampling. Arrange the sampled initial posture features in chronological order to generate an adaptive feature sequence. The adaptive feature sequence is segmented by sliding based on the self-attention mechanism. Within each time window, the similarity score between feature vectors is calculated to construct an attention weight matrix. The adaptive feature sequence is then weighted and aggregated to obtain key pose change features. A spatial pyramid pooling network is constructed by combining dynamic point cloud sequences. Multi-level downsampling is performed through max pooling, and the point density distribution, normal vector distribution, and curvature distribution in each pooling region are calculated to extract multi-scale features. Based on the dynamic point cloud sequence, the centroid position of the point cloud is calculated to obtain the centroid trajectory of the point cloud. A smooth trajectory is obtained by spline interpolation, and the curvature and torsion are calculated to construct a motion feature descriptor. The key posture change features, multi-scale features, and motion feature descriptors are used as graph nodes. The connection relationship between nodes is established based on feature similarity, and the information transmission weight is calculated to perform feature fusion to obtain multimodal fusion features.

5. The method according to claim 4, characterized in that, The process involves calculating the centroid position of the point cloud to obtain its trajectory, uniformly sampling control points along this trajectory, and obtaining a smooth trajectory through spline interpolation. The curvature and torsion of the smooth trajectory are then calculated. A multidimensional motion feature descriptor is constructed by combining the geometric features and motion parameters of the trajectory. The process includes calculating the centroid position of the point cloud based on the dynamic point cloud sequence, obtaining the smooth trajectory through spline interpolation, calculating curvature and torsion, and constructing a motion feature descriptor. The centroid position of the point cloud is obtained by calculating the mean coordinate value of all points in the point cloud in each frame of the point cloud sequence, and the centroid positions of the point cloud are connected in chronological order to form the centroid trajectory of the point cloud. Control points are obtained by uniform sampling on the centroid trajectory of the point cloud. Multiple interpolation points are generated between adjacent control points using spline interpolation. At each interpolation point, a tangent vector, a principal normal vector, and a secondary normal vector are calculated. The tangent vector is obtained by the position difference between adjacent interpolation points. The principal normal vector is perpendicular to the tangent vector and points to the curvature center of the smooth trajectory. The secondary normal vector is obtained by the cross product of the tangent vector and the principal normal vector. The tangent vector, the principal normal vector, and the secondary normal vector are normalized to construct a local coordinate system. The curvature and torsion of the smooth trajectory at each interpolation point are calculated based on the local coordinate system. The geometric features and motion parameters of the smooth trajectory are extracted, and the curvature, torsion, geometric features and motion parameters are combined to construct a multidimensional motion feature descriptor.

6. The method according to claim 1, characterized in that, Multimodal fusion features are input into a dual-branch state recognition network. Lightweight convolutions are used to obtain initial judgment results and anomaly probabilities. When the anomaly probability exceeds a threshold, residual attention is used to enhance features, and spatiotemporal combined features are extracted using spatiotemporal graph convolutions. The recognition result is obtained through pose classification, action classification, and Bayesian fusion. If a fall is identified, the fall action is determined, and the risk level is output, including: Multimodal fusion features are input into a dual-branch state recognition network, and after performing lightweight convolution, they are normalized and activated. The spatial dimension is reduced by pooling, and the initial judgment result and anomaly probability are output through a fully connected layer. When the anomaly probability exceeds a preset threshold, the multimodal fusion feature is enhanced by residual attention. A spatial attention map is generated by convolution, channel weights are generated by global pooling and a two-layer perceptron, and the temporal dimension attention score is calculated by a sliding window. The spatial attention map, channel weights, and temporal dimension attention score are multiplied by the multimodal fusion feature to obtain the enhanced feature. The human skeletal structure is modeled as a graph by combining spatiotemporal graph convolution, the enhanced features are mapped to graph nodes, graph convolution is performed in the spatial domain through adjacency matrix, and temporal modeling is performed in the temporal domain to extract spatiotemporal combined features. Based on the spatiotemporal combination features, the probability distribution of basic posture categories is output through posture classification, the probability distribution of action categories is output through action classification, the basic posture categories and action categories are combined, the posterior probability is calculated through Bayesian fusion, and the combination with the highest posterior probability is selected to obtain the recognition result. If the identification result is a fall, the curvature, torsion, velocity, and acceleration in the pre-acquired multidimensional motion feature descriptor are analyzed to determine the fall action and output the risk level.

7. The method according to claim 6, characterized in that, By combining spatiotemporal graph convolution to model the human skeletal structure as a graph, mapping the enhanced features to graph nodes, performing graph convolution in the spatial domain through an adjacency matrix, and performing temporal modeling and extracting spatiotemporal combined features in the temporal domain, the following steps are taken: Human key points are used as nodes of a graph structure, and the skeletal connection relationship between adjacent human key points is defined as the edge of the graph structure. An adjacency matrix is ​​generated based on the skeletal connection relationship, and the enhanced features are mapped to graph nodes to construct a node feature sequence. By combining spatiotemporal graph convolution in the spatial domain, the spatial association strength between the target node and the directly connected node is calculated through the adjacency matrix to obtain the spatial attention coefficient. The spatial attention coefficient is then weighted and combined with the node features corresponding to the directly connected nodes to obtain the spatial aggregation feature. The node feature sequence is extracted from consecutive frames on the time axis using a sliding window. The node feature sequence is then subjected to a standard convolution operation to extract local temporal patterns. Motion features within different time ranges are extracted layer by layer through multiple layers of standard convolution. Spatial feature extraction and temporal feature extraction are alternately chained together. Spatial feature extraction includes graph convolution, normalization, and activation functions, while temporal feature extraction includes standard convolution, normalization, and deactivation. Shortcut connection paths are added between adjacent feature extractions to directly pass the input features to the output and add them to the transformed features. The output features of each layer are combined, and pooling is performed on the features of all nodes to extract global features. The pooled features are then mapped to a unified dimensional space through feature transformation to obtain spatiotemporal combined features.

8. A fall detection system based on millimeter-wave radar fusion of human posture, used to implement the method of any one of claims 1-7, characterized in that, include: The first unit is used to collect human echo signals through millimeter-wave radar and perform parameter estimation to obtain target distance information, obtain angle information through adaptive beamforming, obtain velocity information based on phase difference velocimetry, and obtain three-dimensional point cloud data by constructing point cloud mapping relationship through dense convolution based on target distance information, angle information and velocity information. After dynamic threshold segmentation, establishment of local coordinate system and iterative nearest point registration, dynamic point cloud sequence is generated. Combined with graph convolutional network, skeletal features are extracted and the three-dimensional coordinates of key points are output. The second unit is used to construct the skeleton connection relationship based on the three-dimensional coordinates of key points and calculate the spatial vector and the included angle to obtain the initial pose features. It generates an adaptive feature sequence through dynamic sampling, extracts key pose change features through a self-attention mechanism, constructs a spatial pyramid pooling network in combination with the dynamic point cloud sequence to extract multi-scale features, constructs a motion feature descriptor based on the centroid trajectory of the point cloud, and obtains multi-modal fusion features through graph attention network. The third unit is used to input multimodal fusion features into the dual-branch state recognition network. The initial judgment result and anomaly probability are obtained through lightweight convolution. When the anomaly probability exceeds the threshold, the feature is enhanced by residual attention and spatiotemporal combined features are extracted by spatiotemporal graph convolution. The recognition result is obtained by posture classification, action classification and Bayesian fusion. If it is a fall state, the fall action is determined and the risk level is output.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images