Millimeter wave radar human body tumble detection method based on multi-feature fusion

Abstract

The invention discloses a millimeter wave radar human body tumble detection method based on multi-feature fusion, and the method comprises the steps: obtaining an original point cloud frame from each radar, and carrying out the timestamp calibration and space coordinate system conversion; performing noise filtering, ground segmentation and human body point cloud extraction on each frame of point cloud; dividing the preprocessed point cloud into a static cluster and a dynamic cluster; filtering false dynamic clusters; extracting a residual dynamic cluster set, and identifying and tracking the human body dynamic clusters in continuous frames by adopting a tracking algorithm; extracting a corrected time sequence feature from the tracked human body cluster; inputting the time sequence characteristics into a pre-trained deep time sequence network, learning a falling time sequence dependency relationship, and outputting an abnormal index; and performing multi-source fusion with the abnormal index to obtain a comprehensive index to judge whether to trigger an alarm. The method can adapt to a complex home environment, improves the detection accuracy and real-time performance, and reduces the false alarm and missing alarm.

Description

Technical Field

[0001] The present invention relates to a millimeter wave radar human fall detection method based on multi-feature fusion, and belongs to the technical field of intelligent recognition. Background Art

[0002] In a home environment, it is difficult to distinguish between falling and lying flat. When a user slowly lies down on a mattress or sofa, the change in their z-axis height may be similar to a real fall. Traditional detection methods often rely on the body's offset state, and if a person lies down slowly, it is easy to cause false alarms. In addition, reflections generated by walls, floors and metal furniture will partially overlap with the human body echo in space, which may cause the furniture to be mistakenly identified as a human figure.

[0003] Although multi-feature fusion methods attempt to combine multiple features such as Doppler spectrum, point cloud data, and temporal information, they still face the following challenges: The sparse point cloud data generated by millimeter-wave radar is difficult to effectively extract spatial and temporal features in complex environments, which affects recognition performance; existing methods do not fuse the extracted features deeply enough and fail to fully explore the correlations between features, which limits the improvement of model performance. Existing models perform well on training datasets, but their performance degrades significantly in complex real-world environments, such as furniture occlusion and lighting changes. They also have limited adaptability to environmental noise and interference and are easily affected by non-target motion, resulting in false positives and false negatives. To improve detection accuracy, many methods use complex deep learning models, but this brings the following problems: deep models have high hardware resource requirements and are difficult to run in real time on resource-constrained devices; in real-world applications, the system needs to respond quickly to fall events, and complex models may cause response delays, affecting practicality.

[0004] In addition, existing technologies perform poorly in complex and changeable environments in actual applications. In environments with dense furniture or obstructions, radar signals are easily interfered with, affecting detection accuracy. For slow falls or atypical falls, the existing models have limited recognition capabilities and are prone to missed detections. Summary of the Invention

[0005] The purpose of the present invention is to provide a millimeter-wave radar human fall detection method based on multi-feature fusion to address the above-mentioned problems, which can adapt to complex home environments, improve detection accuracy and real-time performance, and reduce false alarms and missed alarms.

[0006] The technical solution adopted in the present invention is as follows: A millimeter-wave radar human fall detection method based on multi-feature fusion, characterized by comprising the following steps: S1. Data Acquisition and Preprocessing: Obtain raw point cloud frames containing distance, velocity, and reflection intensity from each radar, perform timestamp calibration, and convert spatial coordinates. Perform noise filtering, ground segmentation, and human point cloud extraction on each point cloud frame to obtain clean data. S2. Dynamic clustering: Based on a density clustering algorithm with an adaptive neighborhood radius, the pre-processed point cloud is divided into static clusters and dynamic clusters. S3. Multipath False Cluster Filtering: Filter out false dynamic clusters caused by multipath reflections using temporal features and multi-radar fusion information. S4. Time-series tracking: Extract the remaining dynamic clusters and use a tracking algorithm to identify and track the human body dynamic clusters in consecutive frames to ensure the continuity of feature calculation; S5. Temporal feature extraction: Extract the corrected centroid height, velocity, acceleration, and event duration from the tracked human cluster as temporal features. S6. Deep time series model processing: Input the time series features into the pre-trained deep time series network, learn the temporal dependencies of falls, and output anomaly indicators; S7. Posture data fusion: Combine the skeleton key points and posture angles obtained by the visual device with the abnormal indicators through multi-source fusion to obtain comprehensive indicators; S8. Fall alarm determination and output: Determine whether to trigger an alarm based on comprehensive indicators. If triggered, send a reminder and record relevant information.

[0007] S1 data acquisition and preprocessing provides clean and synchronized point cloud data for subsequent processing through operations such as timestamp calibration, coordinate system conversion and noise filtering, solving the problems of poor quality of original point cloud and asynchronous multi-radar data; S2 dynamic clustering adopts density clustering with adaptive neighborhood radius, which can achieve reasonable clustering in both furniture-dense areas and open areas, avoiding the incorrect merging or splitting of static objects (furniture, walls) and human point clouds, and effectively solving the problem that fixed radius clustering is difficult to adapt to environmental density differences; S3 multipath false cluster filtering accurately eliminates false clusters generated by multipath reflection through time continuity, multi-radar field of view verification and reflection intensity fluctuation analysis, reducing misjudgment caused by reflection interference from furniture and other factors; S4 time tracking realizes continuous identification of human targets through Kalman filtering, ensuring the stability of feature extraction, which is convenient for capturing the dynamic process of falling. It provides a continuous data foundation; the S5 time series feature extraction focuses on key time series features such as center of mass height, speed, and acceleration, and can accurately depict the fall process of "rapid falling - touching the ground - lying flat and still", solving the problem of insufficient recognition of slow or atypical fall movements; the S6 deep time series model adopts a pre-trained bidirectional LSTM network, which takes into account real-time performance while learning the temporal dependencies of falls, avoiding the defects of high resource requirements and response delays of complex models; S7 posture data fusion combines visual skeleton information to achieve complementary advantages of radar and vision - radar makes up for the shortcomings of vision in insufficient light or occlusion, and vision supplements radar information under angle restrictions, significantly reducing the probability of daily actions (such as bending over, squatting) being misjudged as falls; S8 alarm judgment and output ensures rapid response and recording of fall events, improving the practicality of the system. It specifically solves the core problems in the background technology: first, through adaptive clustering and multipath filtering, it solves the problem of misjudgment caused by furniture occlusion and multipath reflection; second, through time series feature extraction and deep time series model, it improves the recognition ability of slow or atypical falls and reduces missed detections; third, through multimodal fusion and lightweight model design, it balances detection accuracy and real-time performance, reduces the demand for hardware resources, and solves the problems of complex model response delay and difficulty in deployment. The organic combination of adaptive clustering, multipath dynamic filtering, deep time series learning and multimodal fusion has formed a complete solution suitable for complex home environments, breaking through the limitations of existing technologies in feature extraction depth, environmental adaptability and modal complementarity, and significantly improving the accuracy, robustness and practicality of fall detection.

[0008] Optionally, in step S1, timestamp calibration of the raw point cloud frames achieves time synchronization of multi-radar data; spatial coordinate system conversion unifies the radar point clouds into the world coordinate system via rotation matrices and translation vectors. Timestamp calibration of the raw point cloud frames and spatial coordinate system conversion are fundamental to the entire detection method and directly impact the accuracy and reliability of subsequent processing. Timestamp calibration ensures temporal alignment of point cloud frames acquired by multiple radars by unifying the time base of data collected by each radar. This avoids misalignment in the motion state of dynamic targets (such as humans) due to differences in sampling times between different radars. This provides a temporally consistent data foundation for subsequent steps such as dynamic clustering and time series tracking, addressing issues such as target motion trajectory confusion and feature extraction distortion caused by time asynchrony in multi-source radar data. Spatial coordinate system conversion unifies point cloud data collected by different radars into the world coordinate system via rotation matrices and translation vectors, eliminating spatial coordinate deviations caused by differences in radar installation positions and orientation angles. This enables point cloud data from multiple radars to be fused and analyzed within the same spatial framework, ensuring consistency in the position information of human targets in the fields of view of different radars. This avoids problems such as human point cloud segmentation errors and clustering confusion between static clusters (furniture, walls) and dynamic clusters (human bodies) caused by inconsistent coordinates, and provides spatially consistent data support for the subsequent accurate extraction of human point clouds and the distinction between static and dynamic targets.

[0009] Optionally, in step S1, noise filtering is used to remove outlier noise points from the point cloud. Ground segmentation uses a random sampling consistency algorithm to fit the ground plane equation and remove ground points whose distance to the plane is less than a threshold. In a home environment, the ground is a large static area with a large and densely distributed reflection point cloud. If mixed with the human point cloud, the human point cloud may be "swamped" or mistakenly merged with the ground points during clustering, affecting the accurate extraction of human targets. By accurately segmenting the ground points, the point clouds of static objects such as walls and furniture and dynamic targets such as humans can be focused on. This ensures more accurate delineation of "static clusters" and "dynamic clusters" in subsequent dynamic clustering and avoids interference from ground reflections on human point cloud extraction. By removing outlier noise points from the point cloud, noise filtering effectively eliminates invalid data generated by environmental interference (such as electromagnetic clutter and equipment noise) on the radar signal. The signal-to-noise ratio of the point cloud data is significantly improved, providing a cleaner input for dynamic clustering. Ground segmentation uses the random sampling consensus algorithm (RANSAC) to fit the ground plane equation and eliminate ground points whose distance to the plane is less than a threshold. Its core function is to separate ground and non-ground point clouds.

[0010] Alternatively, in step S2, the adaptive neighborhood radius is determined by calculating the average of the set of distances from each point to its k-th nearest neighbor and multiplying this average by a coefficient α. Static clusters are distinguished from dynamic clusters by comparing the average velocity of the points within the cluster with a velocity threshold. Clusters with average velocities less than the threshold are classified as static, while clusters with average velocities greater than the threshold are classified as dynamic. This adaptive neighborhood radius dynamically adjusts the neighborhood radius by calculating the average of the set of distances from each point to its k-th nearest neighbor and multiplying this average by a coefficient α. In areas with high point cloud density, such as those near dense furniture or walls, the local average nearest neighbor distance is small, and the corresponding neighborhood radius is reduced. This prevents point clouds of human subjects from being mistakenly merged into the same cluster with those of surrounding static objects (such as furniture). In open areas, the local average nearest neighbor distance is large, and the neighborhood radius is correspondingly increased. This prevents the point cloud of the same person from being split into multiple small clusters due to sparseness. This adaptive adjustment overcomes the limitation of a fixed neighborhood radius, which is difficult to account for global density differences. It ensures the rationality of clustering results in different environments and provides more accurate clustering for subsequent target recognition.

[0011] Alternatively, in step S3, the method of filtering out false clusters includes: removing the near Clusters that appear less than the set number of times in a frame are eliminated; clusters that appear only in a single radar are eliminated; clusters whose reflection intensity time series standard deviation exceeds the normal range of human clusters are eliminated. False clusters often appear briefly due to unstable reflection conditions and are difficult to maintain in consecutive frames. However, as a real human being is a continuously moving target, its corresponding dynamic clusters will appear stably in consecutive frames. Clusters that appear fewer than a set number of times within a frame are filtered through temporal continuity, effectively filtering out false targets caused by sudden noise and transient reflection interference. When a real human body is in an overlapping area covered by multiple radars, its point cloud will be captured simultaneously by multiple radars. However, false clusters formed by multipath reflections are limited by the reflection path and can usually only be detected by a single radar. Eliminating clusters that appear only in a single radar reduces misjudgments caused by interference from a single radar signal. The reflection intensity of the human body to the radar signal is relatively stable, while false clusters of multipath reflections are affected by changes in the reflection path, resulting in dramatic fluctuations in reflection intensity and significantly larger temporal standard deviations. Clusters whose reflection intensity temporal standard deviations exceed the normal range for human body clusters are eliminated. Through quantitative signal stability screening, real and false targets are further distinguished at the signal feature level, effectively reducing the interference of multipath effects on detection accuracy.

[0012] Optionally, in step S4, the tracking algorithm is a Kalman filter, and the tracker state includes the three-dimensional coordinates and three-dimensional velocity of the center of mass of the human body cluster; the target association is achieved by comparing the Euclidean distance between the cluster in the new frame and the tracker predicted position in the previous frame with the matching threshold, and the continuous If there is no correlation between the frames, the target tracking is terminated. The introduction of the Kalman filter realizes the dynamic prediction and correction of the motion state of the human cluster. The tracker state quantity includes the three-dimensional coordinates of the center of mass and the three-dimensional velocity, which can not only capture the spatial position of the human body, but also reflect its motion trend, effectively smoothing the noise interference in the point cloud data. Even when the point cloud of some frames is sparse or blocked, the position of the human body can still be inferred based on the historical motion characteristics to ensure the continuity of tracking. The target association method based on Euclidean distance and matching threshold achieves accurate matching of the same human body in consecutive frames by quantifying the difference in spatial position, avoiding confusion in multi-target scenes, and locking a stable target object for subsequent temporal feature extraction. And continuous The design of ending tracking when frames are no longer associated can promptly clear targets that have left the monitoring area, reduce the consumption of ineffective computing resources, and ensure efficient operation of the system.

[0013] Optionally, in step S5, the correction of the center of mass height is calculated by the dot product of the difference vector between the center of mass coordinates and the ground origin coordinates and the ground unit normal vector; the extracted time series features include the duration of the falling stage , in which the falling stage passes through a speed less than and lasts ≥ Frame judgment, the speed at the time of contact is less than And the center of mass height is lower than the standing height threshold. In the actual home environment, the ground may have a slight slope or bulge. Directly using the original z-axis coordinate will cause the height data to be distorted. After correction, it can truly reflect the vertical height change of the human body relative to the ground. In the time series characteristics, the duration of the falling stage The system accurately captures the typical dynamic process of a fall: a fall involves a continuous process of rapid descent followed by ground contact. While normal movements, such as bending and squatting, involve a drop in height, the speed often does not meet the rapid descent threshold, or the duration is insufficient, resulting in a ground contact height not lower than the standing threshold. This multi-dimensional assessment based on speed, height, and duration effectively distinguishes falls from everyday movements.

[0014] Optionally, in step S6, the deep temporal network is a bidirectional LSTM network, with the input being a time-series window feature matrix containing normalized center-of-mass height, velocity, acceleration, body tilt angle, and horizontal movement direction vector, and the output anomaly indicator being the probability of a fall anomaly. Compared to a unidirectional LSTM that only utilizes past information, a bidirectional LSTM can simultaneously combine feature information from both historical and future moments, enabling a more comprehensive capture of the continuous temporal dependencies in the fall process. For example, when identifying slow falls, the essential difference from normal movements can be distinguished by the height change trend of the previous and next frames and the continuous fluctuations in velocity and acceleration. The input time-series window feature matrix contains normalized center-of-mass height, velocity, acceleration, body tilt angle, and horizontal movement direction vector. These features quantify the human motion state from different dimensions: height reflects vertical position changes, velocity and acceleration reflect the intensity of movement, and body tilt angle and horizontal direction vector characterize posture and movement trends. Normalization eliminates the feature scale interference caused by different human bodies, ensuring that the model focuses on motion patterns rather than individual differences.

[0015] Optionally, in step S7, the visual device is an RGB camera, which uses MediaPipe to acquire 3D skeletal keypoints. Posture angles include the body tilt angle calculated based on the shoulder and hip center vectors, as well as the knee height difference. A fusion method combines the fall propensity score derived from visual posture with anomaly indicators for a weighted summation to produce a comprehensive index. The RGB camera acquires 3D skeletal keypoints through MediaPipe, calculates the body tilt angle based on the shoulder and hip center vectors, and directly reflects whether the person is nearly lying flat. The knee height difference is used to distinguish between squatting and falling to the ground. These posture details are difficult for millimeter-wave radar to accurately capture. The weighted summation fusion method organically combines the visually derived fall propensity score with the radar output anomaly indicators, leveraging the complementary strengths of the two modalities. When radar height features are distorted by furniture obstruction, visual posture angles can provide additional verification. When insufficient lighting ambiguities visual skeleton recognition, radar velocity and acceleration, as time series features, provide a reliable basis for judgment. For everyday movements, the fusion method can filter out false positives using the comprehensive index.

[0016] Alternatively, hardware deployment can involve deploying 2-3 millimeter-wave radars indoors, with a 15%-30% overlap between radar coverage areas. These radars are installed at a height of 1.2-1.5 meters, arranged in an equilateral triangle or inverted T-shape, and each radar is calibrated and converted to the world coordinate system. Too few radars can lead to blind spots and make it difficult to handle scenarios obstructed by furniture; too many increase the data processing burden and hardware costs. A configuration of 2-3 radars balances practicality and affordability while ensuring coverage of critical areas. This 15%-30% overlap provides a physical foundation for multi-radar data fusion and cross-validation. Human targets within overlapping areas can be captured simultaneously by multiple radars, facilitating the subsequent filtering of single radar multipath false clusters by identifying the presence of multiple radars. Non-overlapping areas are covered collaboratively by multiple radars, reducing missed detections due to furniture obstruction or blind spots, and improving coverage integrity. The installation height of 1.2m-1.5m is set to fully capture the human body point cloud from the feet to the head, while avoiding the problem of reduced accuracy in near-ground target detection due to installation too high, or the problem of the head point cloud being out of detection range due to installation too low. The placement method of an equilateral triangle or an inverted "T" shape ensures that key areas such as the center and corners of the room are covered by at least two radars through geometric layout optimization, reducing monitoring blind spots in areas prone to occlusion such as densely populated furniture areas and corners. By unifying the spatial coordinate reference of each radar, the spatial deviation of the point cloud caused by differences in installation position and orientation is eliminated, ensuring that the same human target captured by different radars is consistent in coordinates, providing a spatially consistent data foundation for subsequent dynamic clustering and time series tracking.

[0017] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: (1) By automatically calculating the neighborhood radius based on the average distance of the local nearest neighbors of each frame of the point cloud, reasonable clustering results can be obtained in both densely populated areas and open areas. Compared with fixed radius clustering, in high-density areas such as furniture or walls, the human body will not be mistakenly merged with the background, maintaining the integrity of the human cluster; in sparse open areas, the same human body will not be split into multiple small clusters, ensuring the continuity of detection. This significantly reduces the clustering error caused by differences in environmental density and improves the robustness of the system in complex home scenes.

[0018] (2) By combining the triple criteria of the number of frames appearing in the time series, cross-radar visibility, and reflection intensity fluctuation, the instantaneous false clusters generated by multipath reflections are accurately removed; by screening out transient noise artifacts through "stable appearance in the last few frames", false alarms are reduced; by verifying the overlapping fields of view of multiple radars, clusters are retained only when the real human body can be observed from multiple perspectives; based on the standard deviation of the reflection intensity time series, the sharp fluctuations in multipath intensity are filtered out to improve the ability to distinguish false reflections. This effectively reduces the negative impact of multipath effects and surrounding furniture on detection results, and improves detection accuracy.

[0019] (3) By constructing a deep temporal network with bidirectional LSTM as the core, it can simultaneously utilize information such as height, speed, and acceleration at past and future moments to learn the temporal patterns of typical fall processes such as "rapid fall - touchdown - lying flat" at the window level; distinguish between normal short-term falling movements such as bending, sitting, and squatting and real falls, effectively reducing missed and false positives; through large-scale labeled data training, the model has good generalization ability and can adapt to human movements of different heights and different falling methods. The model strikes a balance between accuracy and real-time performance and is suitable for online operation on embedded edge computing platforms.

[0020] (4) The millimeter-wave radar's highly temporal anomaly indicators are combined with visual skeletal posture information in a confidence-weighted fusion. When the millimeter-wave radar's signal is incomplete at certain angles or under occlusion, the visual information can provide supplementary information to reduce missed detections. In low-light conditions or when the human body is partially obscured by furniture, the millimeter-wave radar can still operate stably, compensating for the camera's weaknesses. Different modalities verify each other, significantly reducing the probability of everyday actions such as sitting down and bending over being misjudged as falls. Multimodal fusion makes the overall system more stable and reliable in a variety of complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a millimeter-wave radar human fall detection method based on multi-feature fusion. DETAILED DESCRIPTION

[0022] The present invention will be described in detail below with reference to the accompanying drawings.

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] (1) Hardware deployment module: Two to three millimeter-wave radars are deployed in indoor spaces that need to be monitored (such as living rooms, bedrooms, and corridors). The radars must be positioned to ensure a 15% to 30% overlap in coverage area. Each radar is at the same height, but has a different horizontal viewing angle coverage to avoid blind spots. The three radars are placed in an equilateral triangle or an inverted "T" shape to ensure that the central area of ​​the room is fully covered by at least two radars. The radars are generally installed at a height of about 1.2m-1.5m from the ground to ensure that complete point cloud information from the feet to the head of the human body can be captured simultaneously.

[0025] Each radar exchanges data with the central processing unit (Edge server or embedded device with GPU) via Ethernet or wirelessly to achieve real-time point cloud reporting and command issuance.

[0026] During the deployment phase, it is necessary to first determine the installation location of each radar. , heading angle (yaw angle ) for calibration. Assume that the coordinate system of radar A is consistent with the indoor "world coordinate system" and the north direction, and there is a known rigid transformation between radars B and C and the world coordinate system. 、 ; For the single point cloud collected by radar i , Indicates distance, represents the pitch angle, Represents the azimuth, first converted to a three-dimensional point in the radar's own Cartesian coordinate system, and then through rigid transformation, the transformed radar point cloud is obtained ,in, represents the rotation matrix from the radar i coordinate system to the world coordinate system, Represents the corresponding translation vector.

[0027] (2) Software modules: Figure 1 The flowchart of the millimeter-wave radar human fall detection method based on multi-feature fusion disclosed in this embodiment is shown. The specific steps include: S1. Data collection and preprocessing Get the original point cloud, each frame point cloud Contains several points , through the random sampling consistency algorithm, you and the ground plane equation, and the points with a distance less than the threshold from the plane are regarded as ground points and eliminated, and then noise filtering and point cloud fusion are performed, recorded as ; S2. Dynamic Clustering Specifically include: S21. Calculate the local density of each point based on the counting method with a fixed neighborhood radius: ; Among them, 1() represents the indicator function, Indicates the initial rough neighborhood radius (e.g. 0.2m); however, since the point cloud density near indoor furniture or walls is high, and the density in open areas is low, the fixed It is difficult to take the overall situation into consideration, so adaptive adjustments are required. The specific process is as follows: For each point , first calculate the distance set to the 𝑘th nearest neighbor , and take the average to get ;Definition point The adaptive neighborhood radius is , >0.1, where It is a manually set coefficient and can be set to 1.5-2.0 based on experience to ensure that the neighborhood covers enough points in the cluster; in dense point cloud areas such as furniture or walls, Smaller, Decreases accordingly; in open areas, Larger, increases accordingly; thus ensuring that reasonable local neighborhoods can be obtained in both environments.

[0028] S22, use adaptive radius Recalculate Points The local density of: ； S23, according to the adaptive density of step S22 , for all points according to Sort by size: According to the adaptive density of step S22 , for all points according to Sort by size and get the sequence ; At the same time, define a minimum distance for each point , which represents the distance from the point to the nearest point with higher density than it, expressed as: ; according to and The scatter distribution of , selects several density peak points as cluster centers and is expressed as: , usually take the first M that meet The largest point; for any other point , and assign it to the cluster where the center of the cluster closest to it and with higher density is located, and finally several clusters are obtained , ,…, For each cluster , calculate the average speed of all points in the cluster , set a speed threshold ,like , then the cluster is determined is a static cluster (fixed objects such as furniture, walls, and floors); otherwise, it is a dynamic cluster. The set of all dynamic clusters is .

[0029] S3. Multipath false cluster filtering In indoor multipath environments, millimeter wave signals are reflected multiple times by walls, floors, furniture, etc., resulting in the appearance of false point cloud clusters at locations that did not originally exist. These false clusters often have the following characteristics: The point cloud density is low and the shape is scattered; the temporal features are discontinuous, and the speed information often does not match the real human motion characteristics; when multiple radars are fused, it may only appear in a single radar and cannot be detected by other radar positions.

[0030] Therefore, it is necessary to remove false dynamic clusters before target tracking. The specific steps include: Record each cluster in the near Whether there is continuity within the frame, is the time window length, which is set manually. If a frame appears only 1-2 times, it can be considered as a false cluster and removed. The number of occurrences in frames t-4, t-3, …, t is ,like If the number is less than the set number, the cluster is considered unstable and will be removed; For point clouds that have been jointly registered to the world coordinate system, if a cluster only appears in a single radar perspective, and other radars in its visual overlap area cannot detect the corresponding points, it is likely to be a multipath artifact, which is manifested as: For each point p, check the reflection intensity measured by each corresponding radar. If there is a point cloud output by radar A in the same spatial interval, but there is no corresponding reflection from radars B and C that overlap with A, then the cluster is determined to be a single radar false cluster and is removed. Since the reflection intensity of the real human body is relatively stable at multiple angles, while the reflection intensity of multipath reflection fluctuates violently, the statistical cluster The standard deviation of the reflection intensity of all points in the cluster in recent frames is calculated. If the standard deviation is greater than the normal range of the human body cluster, the cluster may be a multipath artifact and should be removed.

[0031] S4. Timing tracking Extract the remaining dynamic cluster sets as , and conduct dynamic tracking, specifically as follows: For each cluster Calculate its center of mass coordinates and center of mass velocity , define the tracker state ; Tracking via Kalman filter , where F represents the uniform velocity model; providing observations to the Kalman filter , which is the three-dimensional coordinate of the center of mass of the current frame cluster; use the observation matrix to update the Kalman gain K(t) and calculate the new state estimate ; If a cluster in the new frame is at the same position as the tracker predicted in the previous frame If the Euclidean distance of is less than the matching threshold, it is associated with the tracker; otherwise it is determined to be a new target and a new tracker is started. If there is no person associated in the frame, the target is considered to have left the field of view and tracking is terminated. The maximum number of frames allowed to be lost is set manually.

[0032] S5. Temporal feature extraction include: For the tracked Number of human bodies, extract their 3D center of mass coordinates of , and perform height correction for the effect of ground tilt to obtain the corrected straight line coordinates: ,in, Expressed as the ground unit normal vector, Indicates a ground reference point; Calculate the vertical velocity of the center of mass based on the corrected center of mass coordinates , vertical acceleration of the center of mass ; In the rapid falling stage, the speed threshold is set ,when , and the duration is greater than or equal to Frame, judge to enter the falling stage, the time node is recorded as ; Determine the touchdown moment by the height change trend, specifically: , and the center of mass height is lower than the set standing height threshold, it is determined as the touchdown moment ; Therefore, the duration of the falling stage is expressed as .

[0033] S6. Deep Time Series Model Processing For the current time t, create a time series window of length w , for the moment , extract the normalized feature vector ,in represents the normalized centroid height (normalized by subtracting standing height and dividing by height range), , denote the normalized velocity and acceleration, , Respectively represent the body tilt angle and horizontal movement direction vector calculated by fitting the shoulder and hip markers of the radar point cloud; and splice the entire window into a matrix , as the input sequence of the bidirectional LSTM, by collecting the annotated time series data of historical normal behavior and fall time, and constructing the training samples in a sliding window manner ,in Indicates a fall occurred inside the window. Indicates the normal state; the final output layer is a single neuron, which outputs the abnormal probability of falling through the activation function , and use this probability as an indicator of fall abnormality Output to the abnormal indicator fusion model.

[0034] S7, posture data fusion Many common activities (such as bending over to tie shoelaces, picking up objects, squatting, etc.) will also cause a short fall on the millimeter wave point cloud height curve, but the skeletal posture tilt angle will not reach the level of falling. Therefore, an abnormal indicator fusion model is constructed to determine the target activity status; The abnormal indicator fusion model involves deploying an ordinary RGB camera indoors, using MediaPipe to obtain the key points of the human body's 3D skeleton, and obtaining the coordinates of key joints. Calculate the body inclination through the shoulder and hip key points, specifically: let the shoulder center be ksho and the hip center be khip, then the inclination vector is expressed as , do the dot product with the vertical unit vector of the world coordinate system, and we get , and the target tilt angle is expressed as The larger the inclination angle, the closer the body is to lying flat or bending over; calculate the height of the center of the knee relative to the height of the center of the hip The difference between the left knee height and right knee height , then, the height difference between the two knees is expressed as ,like Less than the set threshold , and the duration is less than the set time, it is judged as squatting.

[0035] Calculate the fall tendency score based on the target tilt angle and the knee height difference ,in, Indicates the reference value of the inclination angle. If , then the logarithmic contribution value is 0, if , then the value of this item is 0, represents the Sigmoid function, and Represents weight.

[0036] S8, fall alarm determination and output Fall Proneness Score Abnormal fall indicators The weighted sum is used to obtain the fall threat value. If the fall threat value exceeds the set threshold, a fall alarm is triggered.

[0037] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. The present invention extends to any new features or any new combinations disclosed in this specification, and any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the detailed technical features not disclosed in this embodiment are all prior art, which can be obtained by those skilled in the art from the prior art. For those of ordinary skill in the art, the specific manner in which the terms are used in the embodiments of the present invention can be understood according to the specific circumstances, and the embodiments disclosed herein do not specifically limit this.

Claims

1. A millimeter-wave radar human fall detection method based on multi-feature fusion, characterized in that: The following steps are involved: S1. Data Acquisition and Preprocessing: Obtain raw point cloud frames containing distance, velocity, and reflection intensity from each radar, perform timestamp calibration, and convert spatial coordinates. Perform noise filtering, ground segmentation, and human point cloud extraction on each point cloud frame to obtain clean data. S2. Dynamic clustering: Based on a density clustering algorithm with an adaptive neighborhood radius, the pre-processed point cloud is divided into static clusters and dynamic clusters. S3. Multipath False Cluster Filtering: Filter out false dynamic clusters caused by multipath reflections using temporal features and multi-radar fusion information. S4. Time-series tracking: Extract the remaining dynamic clusters and use a tracking algorithm to identify and track the human body dynamic clusters in consecutive frames to ensure the continuity of feature calculation; S5. Temporal feature extraction: Extract the corrected centroid height, velocity, acceleration, and event duration from the tracked human cluster as temporal features. S6. Deep time series model processing: Input the time series features into the pre-trained deep time series network, learn the temporal dependencies of falls, and output anomaly indicators; S7. Posture data fusion: Combine the skeleton key points and posture angles obtained by the visual device with the abnormal indicators through multi-source fusion to obtain comprehensive indicators; S8. Fall alarm determination and output: Determine whether to trigger an alarm based on comprehensive indicators. If triggered, send a reminder and record relevant information.

2. The method according to claim 1, characterized in that In step S1, the timestamp calibration of the original point cloud frame realizes the time synchronization of multi-radar data; the spatial coordinate system conversion unifies each radar point cloud to the world coordinate system through the rotation matrix and translation vector.

3. The method according to claim 1, characterized in that In step S1, noise filtering is used to remove outlier noise points in the point cloud; ground segmentation uses a random sampling consistency algorithm to fit the ground plane equation and eliminate ground points whose distance to the plane is less than a threshold.

4. The method according to claim 1, wherein In step S2, the adaptive neighborhood radius is determined by calculating the average value of the distance set from each point to its kth nearest neighbor point, and multiplying the average value by the coefficient α; Static clusters and dynamic clusters are distinguished by comparing the average speed of the points in the cluster with the speed threshold. Clusters with an average speed less than the threshold are static clusters, and clusters with an average speed less than the threshold are dynamic clusters.

5. The method according to claim 1, characterized in that In step S3, the method of filtering out false clusters includes: removing the near Clusters that appear less than the set value in a frame are eliminated; clusters that appear only in a single radar are eliminated; clusters whose reflection intensity time series standard deviation exceeds the normal range of human body clusters are eliminated.

6. The method according to claim 1, characterized in that In step S4, the tracking algorithm is a Kalman filter, and the tracker state includes the three-dimensional coordinates and three-dimensional velocity of the center of mass of the human body cluster; the target association is achieved by comparing the Euclidean distance between the cluster in the new frame and the tracker predicted position in the previous frame with the matching threshold, and the continuous If the frames are not associated, the target tracking is terminated.

7. The method according to claim 1, characterized in that In step S5, the correction of the center of mass height is calculated by the dot product of the difference vector between the center of mass coordinates and the ground origin coordinates and the ground unit normal vector; the extracted time series features include the duration of the fall phase , in which the falling stage passes through a speed less than and lasts ≥ Frame judgment, the speed at the time of contact is less than And the center of mass height is lower than the standing height threshold.

8. The method according to claim 1, characterized in that In step S6, the deep time series network is a bidirectional LSTM network, the input is a time series window feature matrix containing normalized center of mass height, velocity, acceleration, body tilt angle and horizontal movement direction vector, and the output abnormality indicator is the abnormal probability of falling.

9. The method according to claim 1, characterized in that In step S7, the visual device is an RGB camera, which obtains the 3D skeleton key points of the human body through MediaPipe; the posture angle includes the body inclination angle calculated based on the center vectors of the shoulders and hips, and the height difference between the two knees; the fusion method is to perform a weighted summation of the fall tendency score obtained by the visual posture and the abnormality index to obtain a comprehensive index.

10. The method according to claim 1, characterized in that During hardware deployment, 2-3 millimeter-wave radars are arranged in an indoor space, with the coverage area overlapping by 15%-30%. The installation height is 1.2m-1.5m, and they are placed in an equilateral triangle or inverted "T" shape. The coordinates of each radar are calibrated and converted to the world coordinate system.

Images