Human body tumble detection method and device

Abstract

The invention discloses a human body tumble detection method and device, and belongs to the technical field of human body tumble detection, and the method comprises the steps: obtaining a radar point cloud time sequence which comprises continuous frame point clouds of all parts of a human body within a set time; performing relative coordinate coding on each frame of point cloud in the radar point cloud time sequence to obtain a spliced feature vector time sequence; and inputting the spliced feature vector time sequence into a trained human body tumble detection model to obtain a human body tumble detection result, and training is semi-supervised learning training based on a dynamic threshold. Relative coordinate coding is realized through construction of a local coordinate system and coordinate conversion of the point cloud, dynamic enhancement of four-limb micro-motion features in the falling process is realized, a better classification effect is shown, and thus the detection accuracy is improved; through semi-supervised learning training based on a dynamic threshold value, the labeling requirement is greatly reduced, and high detection precision in a small amount of training data scene is also realized.

Description

TECHNICAL FIELD

[0001] The present application relates to a human fall detection method and device, and belongs to the technical field of human fall detection. BACKGROUND

[0002] In the prior art, most human fall detection is achieved by processing point cloud data and extracting space-time features, and then combining a classifier to predict whether a human falls. However, the existing methods have the following defects in representing the point cloud coordinate system: Absolute coordinate encoding: directly using an absolute coordinate system. Sensitive to installation position, difficult to directly adapt to changes in the environment in different environments; ignoring the correlation of limb movements, it is difficult to directly extract feature parameters that accurately reflect the changes in human posture from the absolute coordinate system.

[0003] Point cloud voxelization: dividing the space into grids with consistent length, width and height. In the voxelization process, the point cloud information in each voxel is aggregated or simplified, only retaining some statistical features, resulting in the loss of a large amount of detailed information; increasing the dimension and complexity of the data, requiring more storage space to save the voxelized data structure, and the amount of calculation during subsequent calculation and processing is much larger than directly processing the original point cloud data.

[0004] The existing methods have the following defects in extracting space-time features: Common deep learning models are difficult to achieve organic integration of space-time information: CNN-based methods extract spatial features by projecting to a two-dimensional plane, but lose key time dynamic information; LSTM networks can model time sequence dependencies, but are not sensitive enough to point cloud spatial distribution; PointNet series methods face the problem of exponential growth of computational complexity when processing long sequences.

[0005] The existing methods have the following defects in model training: Supervised learning relies on a large amount of labeled data, but the incidence of fall events in real scenes is less than 5%; data labeling work requires a large amount of time cost; semi-supervised methods mostly use fixed threshold to screen pseudo-labels, resulting in a large amount of valid data being discarded incorrectly, and the accuracy is low in complex action recognition scenarios.

[0006] From the above analysis of the existing methods, it can be seen that the existing technology has low accuracy in detecting human falls. SUMMARY

[0007] The present application relates to a human fall detection method and device, and belongs to the technical field of human fall detection.

[0008] To achieve the above purpose, the technical scheme adopted by the present application is as follows: In a first aspect, the present invention provides a method for detecting human falls, comprising: Acquire radar point cloud time series, which includes continuous frame point clouds of various parts of the human body within a set time period. Perform the following operations on each frame of the radar point cloud time series: calculate the coordinates of the centroid of the point cloud, calculate the covariance matrix of the point cloud based on the coordinates of the centroid, perform principal component analysis on all eigenvectors in the covariance matrix, take the directions corresponding to the top three eigenvectors as the three coordinate axes of the local coordinate system, and construct the local coordinate system of the point cloud for that frame with the centroid as the origin of the local coordinate system. Each frame of point cloud is converted from the radar coordinate system to the corresponding local coordinate system to obtain the local point cloud time series. Random perturbations are added to each frame of the point cloud in the local point cloud time series to obtain the perturbed local point cloud time series. Nonlinear mapping is performed on each frame of the point cloud in the time series of the disturbed local point cloud. Then, the point cloud is spliced ​​with the signal strength and centroid of the corresponding frame point cloud to obtain the spliced ​​feature vector of each frame. After normalizing all spliced ​​feature vectors, a spliced ​​feature vector time series is formed. The concatenated feature vector time series is input into the trained human fall detection model to obtain the human fall detection result. The human fall detection model includes a feature embedding module, a Transformer module, a 3D residual network module and a classifier connected in sequence. The training is a semi-supervised learning training based on dynamic threshold.

[0009] Furthermore, the coordinates of the centroid of the point cloud are calculated using the following formula: ; in, Indicates the first The coordinates of the centroid of the point cloud in the frame. N For point cloud points, For the first The first frame in the point cloud The coordinates of the points Indicates the index of the coordinates of a point in the point cloud.

[0010] Furthermore, the covariance matrix of the point cloud calculated based on the centroid coordinates is performed using the following formula: ; in, Indicates the first The covariance matrix of the frame point cloud, Indicates the first The coordinates of the centroid of the point cloud in the frame. N For point cloud points, For the first The first frame in the point cloud The coordinates of the points The index represents the coordinates of a point in the point cloud, and T represents the transpose.

[0011] Furthermore, the phrase "using the directions corresponding to the top three eigenvectors as the three coordinate axes of the local coordinate system" includes: The direction corresponding to the first eigenvector is taken as the X-axis direction of the local coordinate system, the direction corresponding to the second eigenvector is taken as the Y-axis direction of the local coordinate system, and the direction corresponding to the third eigenvector is taken as the Z-axis direction of the local coordinate system.

[0012] Furthermore, the step of converting each frame of point cloud from the radar coordinate system to the corresponding local coordinate system to obtain the local point cloud time series includes: The top three eigenvectors obtained through the principal component analysis are combined into an orthogonal rotation matrix. , , This represents the first eigenvector obtained through the principal component analysis. This represents the second-ranked eigenvector obtained through the principal component analysis. This represents the third-ranked eigenvector obtained through the principal component analysis. Based on the orthogonal rotation matrix, the coordinates of points in the point cloud are transformed from the radar coordinate system to the corresponding local coordinate system using the following formula: ; Where T represents transpose. Indicates the first The first frame in the point cloud The coordinates of a point in the local coordinate system Indicates the first The coordinates of the centroid of the point cloud in the frame. For the first The first frame in the point cloud The coordinates of the points; After converting the coordinates of all points in each frame of the point cloud to the corresponding local coordinate system, a local point cloud time series is obtained.

[0013] Furthermore, the step of adding random perturbations to each frame of the local point cloud time series to obtain a perturbed local point cloud time series includes: The following formula adds random translation perturbation to the coordinates of points in the point cloud in the local coordinate system: ; in, After adding random translation perturbation, the first The first frame in the point cloud The coordinates of a point in the local coordinate system Indicates the first The first frame in the point cloud The coordinates of a point in the local coordinate system This represents the defined translational perturbation vector. This indicates that the mean is 0 and the variance is 0. I is a Gaussian distribution, where I is the identity matrix; The following formula is used to calculate the th... after adding random translational perturbations. The first frame in the point cloud Add random rotation perturbations to the coordinates of each point in the local coordinate system: ; in, This indicates the result after adding random rotation and random translation perturbations. The first frame in the point cloud The coordinates of a point in the local coordinate system This represents the orthogonal rotation matrix formed by the top three eigenvectors obtained through the principal component analysis. The random rotational perturbation is defined, SO(3) represents the special orthogonal group, and T represents the transpose; By adding random translation and random rotation perturbations to the coordinates of points in each frame of the local point cloud time series, a perturbed local point cloud time series is obtained.

[0014] Furthermore, the nonlinear mapping of each frame of the perturbed local point cloud time series is performed using the following expression for a multi-kernel learnable Gaussian kernel function: ; in, This indicates the addition of random perturbations and nonlinear mapping to the th... The first frame in the point cloud The coordinates of a point in the local coordinate system Indicates the set number The first frame in the point cloud Attention weights for the coordinates of a point in the local coordinate system. Indicates the first The weights of each Gaussian kernel, Indicates the first The mean of a Gaussian kernel, Indicates the first The covariance matrix of Gaussian kernels, where K represents the number of Gaussian kernels. This indicates the result after adding random rotation and random translation perturbations. The first frame in the point cloud The coordinates of a point in the local coordinate system, where T represents the transpose and exp represents the natural exponential function.

[0015] Further, the feature embedding module is configured to map the spliced feature vector time sequence to a high-dimensional space to obtain an embedding time sequence; The Transformer module is configured to extract local attention and global attention of the embedding time sequence, and perform gated fusion and MLP expansion on the local attention and the global attention to obtain fused features; The 3D residual network module is configured to perform convolution on the fused features in time, width and height dimensions through a 3D convolution kernel, perform GAP operation, convolution, GELU activation function and again convolution operation on the features obtained by convolution in sequence, calculate channel attention weights through a Sigmoid activation function, multiply the channel attention weights and the features obtained by convolution element by element, perform weighted fusion based on a CA mechanism to obtain weighted fusion features, perform residual connection on the weighted fusion features and the spliced feature vector time sequence, and then perform global pooling to obtain a one-dimensional vector; The classifier is configured to detect whether the human body falls according to the one-dimensional vector, and output a human body fall detection result whether the human body falls within the set time.

[0016] Further, the semi-supervised learning training based on a dynamic threshold value comprises: training data is obtained and divided into a training set and a validation set, part of the training data in the training set is labeled to obtain labeled data, and the remaining training data in the training set is used as unlabeled data; the labeled data is input into the human body fall detection model for preliminary training; the unlabeled data is input into the human body fall detection model after preliminary training for forward propagation to obtain a prediction probability vector of each unlabeled data; if the maximum probability value in the prediction probability vector of the unlabeled data is greater than or equal to a probability threshold value, the maximum probability value index of the unlabeled data is used as a pseudo label, and the pseudo label is labeled on the corresponding unlabeled data; the human body fall detection model is evaluated on the validation set to obtain a validation accuracy of the current human body fall detection model; the labeled data and the unlabeled data with the pseudo label are merged into a new training set, the human body fall detection model is continuously trained on the new training set, the human body fall detection model after each iteration is saved, and the human body fall detection model with the highest validation accuracy is used as the trained human body fall detection model until a preset iteration number is reached; wherein, the probability threshold value is dynamically updated by the following formula: ; wherein, is a probability threshold value at the jth moment, is a probability threshold value at the (j-1)th moment, is a verification accuracy at the jth moment, is a probability threshold value at the initial moment, is a set momentum coefficient, is a set scaling factor, is a verification accuracy at the initial moment.

[0017] In a second aspect, the present application provides a human body fall detection device, comprising: A point cloud acquisition module configured to acquire a radar point cloud time sequence, the radar point cloud time sequence comprising continuous frame point clouds of each part of the human body within a set time; A local coordinate system construction module configured to perform the following operations on each frame point cloud in the radar point cloud time sequence: calculate the coordinates of the centroid of the point cloud, calculate the covariance matrix of the point cloud based on the coordinates of the centroid, perform principal component analysis on all eigenvectors in the covariance matrix to take the directions corresponding to the top three eigenvectors as the directions of the three coordinate axes of the local coordinate system, and take the centroid as the origin of the local coordinate system to construct the local coordinate system of the frame point cloud; A coordinate conversion module configured to convert each frame point cloud from the radar coordinate system to the corresponding local coordinate system to obtain a local point cloud time sequence; A random disturbance addition module configured to add random disturbance to each frame point cloud in the local point cloud time sequence to obtain a disturbed local point cloud time sequence; A mapping and splicing module configured to perform nonlinear mapping on each frame point cloud in the disturbed local point cloud time sequence, then splice the signal intensity and centroid of the corresponding frame point cloud to obtain each frame splicing eigenvector, and normalize all splicing eigenvectors to form a splicing eigenvector time sequence; A human body fall detection module configured to input the splicing eigenvector time sequence into a trained human body fall detection model to obtain a human body fall detection result, the human body fall detection model comprising a feature embedding module, a Transformer module, a 3D residual network module and a classifier connected in sequence, and the training being semi-supervised learning training based on a dynamic threshold.

[0018] Compared with the prior art, the present application has the following beneficial effects: The application provides a human fall detection method and device, which realizes a relative coordinate coding method based on a local coordinate system through construction of the local coordinate system and coordinate conversion of point clouds, realizes dynamic enhancement of limb micro-motion characteristics in a fall process compared with a traditional absolute coordinate method and a point cloud voxelization method, shows better classification effect, and thus improves detection accuracy; compared with a traditional CNN-LSTM hybrid model, the application realizes collaborative modeling of space-time characteristics through a Transformer module and a 3D residual network module, significantly reduces space-time characteristic information loss, and improves detection accuracy; through semi-supervised learning training based on a dynamic threshold, the application greatly reduces labeling requirements, and also realizes high detection precision in a small amount of training data scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of a human fall detection method corresponding to embodiment 1; Figure 2 is a flowchart of a human fall detection method corresponding to embodiment 2; Figure 3 is a flowchart of relative coordinate coding provided by embodiment 2; Figure 4 is a processing flowchart of a human fall detection model provided by embodiment 2; Figure 5 is a semi-supervised learning training flowchart based on a dynamic threshold provided by embodiment 2. DETAILED DESCRIPTION

[0020] The application will be further described below in combination with the drawings, and the following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0021] Embodiment 1

[0022] As shown in Figure 1 , the embodiment provides a human fall detection method, which comprises: obtaining a radar point cloud time sequence, wherein the radar point cloud time sequence comprises continuous frame point clouds of each part of a human body in a set time; performing the following operations on each frame point cloud in the radar point cloud time sequence: calculating a coordinate of a centroid of the point cloud, calculating a covariance matrix of the point cloud based on the coordinate of the centroid, performing principal component analysis on all eigenvectors in the covariance matrix, taking directions corresponding to three eigenvectors with the highest scores as directions of three coordinate axes of a local coordinate system, and taking the centroid as an origin of the local coordinate system to construct a local coordinate system of the frame point cloud; converting each frame point cloud from a radar coordinate system into a corresponding local coordinate system to obtain a local point cloud time sequence; Random perturbations are added to each frame of the point cloud in the local point cloud time series to obtain the perturbed local point cloud time series. Nonlinear mapping is performed on each frame of the point cloud in the time series of the disturbed local point cloud. Then, the point cloud is spliced ​​with the signal strength and centroid of the corresponding frame point cloud to obtain the spliced ​​feature vector of each frame. After normalizing all spliced ​​feature vectors, a spliced ​​feature vector time series is formed. The concatenated feature vector time series is input into the trained human fall detection model to obtain the human fall detection result. The human fall detection model includes a feature embedding module, a Transformer module, a 3D residual network module and a classifier connected in sequence. The training is a semi-supervised learning training based on dynamic threshold.

[0023] This invention achieves a relative coordinate encoding method based on a local coordinate system by constructing a local coordinate system and transforming the coordinates of point clouds. Compared with traditional absolute coordinate methods and point cloud voxelization methods, it achieves dynamic enhancement of the micro-movement features of the limbs during a fall, exhibiting better classification results and thus improving detection accuracy. Compared with traditional CNN-LSTM hybrid models, it achieves collaborative modeling of spatiotemporal features through the Transformer module and the 3D residual network module, significantly reducing the loss of spatiotemporal feature information and improving detection accuracy. Through semi-supervised learning training based on dynamic thresholds, it greatly reduces the labeling requirements and also achieves high detection accuracy in scenarios with limited training data.

[0024] Example 2

[0025] like Figure 2 As shown, this embodiment provides a method for detecting human falls, which specifically includes the following steps: Step S1: Obtain the radar point cloud time series, which includes continuous frame point clouds of various parts of the human body within a set time period.

[0026] The coordinates of the points in the point cloud obtained at this time are in the radar coordinate system.

[0027] For each frame of the radar point cloud time series, perform the following steps S2.

[0028] Step S2, relative coordinate encoding. For example... Figure 3 As shown, the specific implementation of step S2 includes the following steps: Step S2.1: Calculate the coordinates of the centroid of the point cloud using the following formula: ; in, Indicates the first The coordinates of the centroid of the point cloud in the frame. N For point cloud points, For the first coordinates of the i-th point in the i-th frame point cloud, coordinates of the i-th point in the i-th frame point cloud, coordinates of the i-th point in the i-th frame point cloud,

[0029] Step S2.2, calculate the covariance matrix of the point cloud based on the coordinates of the centroid, perform principal component analysis on all eigenvectors in the covariance matrix, take the direction corresponding to the first eigenvector as the X-axis direction of the local coordinate system, take the direction corresponding to the second eigenvector as the Y-axis direction of the local coordinate system, take the direction corresponding to the third eigenvector as the Z-axis direction of the local coordinate system, and take the centroid as the origin of the local coordinate system, to construct the local coordinate system of the frame point cloud.

[0030] wherein the covariance matrix of the point cloud based on the coordinates of the centroid is calculated by the following formula: ; wherein, covariance matrix of the i-th frame point cloud, covariance matrix of the i-th frame point cloud, coordinates of the centroid of the i-th frame point cloud, number of points in the point cloud, N coordinates of the i-th point in the i-th frame point cloud, coordinates of the i-th point in the i-th frame point cloud, coordinates of the i-th point in the i-th frame point cloud, coordinates of the i-th point in the i-th frame point cloud, coordinates of the i-th point in the i-th frame point cloud, and T represents transposition.

[0031] Step S2.3, convert each frame of point cloud from the radar coordinate system to the corresponding local coordinate system to obtain a local point cloud time sequence. The specific implementation of step S2.3 includes the following steps: the first three eigenvectors obtained by principal component analysis form an orthogonal rotation matrix , , the first eigenvector obtained by principal component analysis (whose direction is taken as the X-axis direction of the local coordinate system), the second eigenvector obtained by principal component analysis (whose direction is taken as the Y-axis direction of the local coordinate system), the third eigenvector obtained by principal component analysis (whose direction is taken as the Z-axis direction of the local coordinate system); the directions of the first three eigenvectors represent the three main directions of the point cloud data distribution; based on the orthogonal rotation matrix , convert the coordinates of the points in the point cloud from the radar coordinate system to the corresponding local coordinate system by the following formula: ; wherein T represents transposition, coordinates of the i-th point in the i-th frame point cloud, The first frame in the point cloud The coordinates of a point in the local coordinate system Indicates the first The coordinates of the centroid of the point cloud in the frame; After converting the coordinates of all points in each frame of the point cloud to the corresponding local coordinate system, a local point cloud time series is obtained.

[0032] Step S2.4: Enhance robustness using random perturbations by adding random translational and random rotational perturbations to the coordinates in the local coordinate system. The specific implementation of step S2.4 includes the following steps: The following formula adds random translation perturbation to the coordinates of points in the point cloud in the local coordinate system: ; in, After adding random translation perturbation, the first The first frame in the point cloud The coordinates of a point in the local coordinate system Indicates the first The first frame in the point cloud The coordinates of a point in the local coordinate system This represents the defined translational perturbation vector. This indicates that the mean is 0 and the variance is 0. I is a Gaussian distribution, where I is the identity matrix; The following formula is used to calculate the th... after adding random translational perturbations. The first frame in the point cloud Add random rotation perturbations to the coordinates of each point in the local coordinate system: ; in, This indicates the result after adding random rotation and random translation perturbations. The first frame in the point cloud The coordinates of a point in the local coordinate system This represents the orthogonal rotation matrix formed by the top three eigenvectors obtained through the principal component analysis. This represents the set random rotational perturbation, and SO(3) represents the special orthogonal group. It is a rotational perturbation that follows a special orthogonal group distribution; By adding random translation and random rotation perturbations to the coordinates of points in each frame of the local point cloud time series, a perturbed local point cloud time series is obtained.

[0033] Step S2.5: Design distance-based attention weights to highlight local motion features such as hands and feet that are far from the center of mass.

[0034] Step S2.6: Perform nonlinear mapping on each frame of the point cloud in the time series of the disturbed local point cloud.

[0035] The above nonlinear mapping is performed using the following expression for a multi-kernel learnable Gaussian kernel function: ; in, This indicates the addition of random perturbations and nonlinear mapping to the th... The first frame in the point cloud The coordinates of a point in the local coordinate system Indicates the set number The first frame in the point cloud Attention weights for the coordinates of a point in the local coordinate system. Indicates the first The weights of each Gaussian kernel, Indicates the first The mean of a Gaussian kernel, Indicates the first The covariance matrix of Gaussian kernels, where K represents the number of Gaussian kernels. This indicates the result after adding random rotation and random translation perturbations. The first frame in the point cloud The coordinates of a point in the local coordinate system, where T represents the transpose and exp represents the natural exponential function.

[0036] Step S2.7: The coordinates of the points in each frame point cloud after nonlinear mapping in step S2.6 are spliced ​​with the signal strength and centroid of the corresponding frame point cloud to obtain the spliced ​​feature vector of each frame. After normalizing all spliced ​​feature vectors, a spliced ​​feature vector time series is formed.

[0037] Step S3: Input the spliced ​​feature vector time series obtained in step S2 into the trained human fall detection model to obtain the human fall detection result.

[0038] like Figure 4 As shown, in the human fall detection model, the following operations are performed on the concatenated feature vector time series: Step 1: Map the spliced ​​feature vector time series to a high-dimensional space through the feature embedding layer to complete the feature embedding. The dimension of the high-dimensional space is represented as: time dimension × number of points × number of feature channels. In this embodiment, the number of feature channels is 128.

[0039] Step 2: Pass the embedded feature vector z through three linear transformation layers to obtain the query matrix Q, key matrix K, and value matrix V. These three matrices have the same dimension as the feature vector z and are used to calculate the attention score and obtain the corresponding feature information, respectively.

[0040] Step 3, local attention and global attention calculation is performed by the following formula: ; ; wherein, is the local attention, is the global attention, M is the local mask matrix, is the key vector dimension, softmax represents the softmax function, which is a normalized exponential function that maps real number vectors to probability distributions, and GlobalAttn represents the global attention mechanism.

[0041] Step 4, the fusion gate factor is calculated by concatenating the features by the following formula: ; wherein, G represents the fusion gate factor, Linear represents linear transformation, represents the fusion gate unit.

[0042] According to the gate factor, the local attention and the global attention are weighted and fused to obtain the fusion feature, which is calculated by the following formula: ; wherein, is the fusion feature, and represents the fusion operation.

[0043] Step 5, the fusion feature is extracted using 3D convolution, and the 3D convolution kernel slides in the time, width and height dimensions to obtain the convolution feature.

[0044] Step 6, GAP (Global Average Pooling) operation is performed on the convolution feature to compress the feature of each channel into a single statistical quantity. After convolution, GELU (Gaussian Error Linear Unit) activation function and again convolution operation, the channel attention weight is calculated by Sigmoid activation function .

[0045] Step 7, the channel attention weight vector is multiplied with the convolution feature element by element, and the weighted fusion is performed based on the CA (Coordinate Attention) mechanism.

[0046] ; wherein, GAP(·) represents the global average pooling operation, Conv(·) represents the convolution operation, and represents the element-wise multiplication, represents a channel attention weight, represents a weighted fusion feature.

[0047] Step 8, the weighted fusion feature and the input feature are connected in residual.

[0048] Step 9, a global pooling operation is performed, and a one-dimensional vector obtained by the global pooling operation is input to a fully connected classifier to output a prediction result about whether the human body falls.

[0049] As Figure 5 shown in the embodiment, the training of the human body fall detection model includes the following steps: training data is obtained and divided into a training set and a validation set, part of the training data in the training set is labeled to obtain labeled data, and the remaining training data in the training set is used as unlabeled data; the labeled data is input into the human body fall detection model for preliminary training; the unlabeled data is input into the human body fall detection model after preliminary training for forward propagation to obtain a prediction probability vector of each unlabeled data; if the maximum probability value in the prediction probability vector of the unlabeled data is greater than or equal to a probability threshold, the maximum probability value index of the unlabeled data is used as a pseudo label, and the pseudo label is labeled on the corresponding unlabeled data; the human body fall detection model is evaluated on the validation set to obtain a validation accuracy of the current human body fall detection model; the labeled data and the unlabeled data with the pseudo label are merged into a new training set, the human body fall detection model is continuously trained on the new training set, the human body fall detection model after each iteration is saved, and the human body fall detection model with the highest validation accuracy is used as the trained human body fall detection model until a preset iteration number is reached; wherein the probability threshold is dynamically updated by the following formula: ; wherein, is the probability threshold at the jth moment, is the probability threshold at the (j-1)th moment, is the validation accuracy at the jth moment, is the probability threshold at the initial moment, is a set momentum coefficient, is a set scaling factor, is the validation accuracy at the initial moment.

[0050] Embodiment 3

[0051] The embodiment provides a human body fall detection device, comprising: a point cloud obtaining module configured to obtain a radar point cloud time sequence including continuous frame point clouds of each part of a human body within a set time; a local coordinate system constructing module configured to perform the following operations on each frame point cloud in the radar point cloud time sequence: calculate a coordinate of a centroid of the point cloud, calculate a covariance matrix of the point cloud based on the coordinate of the centroid, perform principal component analysis on all eigenvectors in the covariance matrix to take directions corresponding to the top three eigenvectors as directions of three coordinate axes of a local coordinate system, and take the centroid as an origin of the local coordinate system to construct a local coordinate system of the frame point cloud; a coordinate conversion module configured to convert each frame point cloud from a radar coordinate system to a corresponding local coordinate system to obtain a local point cloud time sequence; a random disturbance adding module configured to add random disturbance to each frame point cloud in the local point cloud time sequence to obtain a disturbed local point cloud time sequence; a mapping and splicing module configured to perform nonlinear mapping on each frame point cloud in the disturbed local point cloud time sequence, then splice the frame point cloud with signal intensity and a centroid of the frame point cloud to obtain a spliced feature vector, and normalize all spliced feature vectors to form a spliced feature vector time sequence; a human fall detection module configured to input the spliced feature vector time sequence into a trained human fall detection model to obtain a human fall detection result, wherein the human fall detection model includes a feature embedding module, a Transformer module, a 3D residual network module, and a classifier connected in sequence, and the training is semi-supervised learning training based on a dynamic threshold.

[0052] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0053] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0054] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0055] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0056] The above only is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the technical field, without departing from the technical principles of the present application, can also make a number of improvements and variations, these improvements and variations should also be considered as the protection scope of the present application.

Claims

1. A human fall detection method, characterized by, The method comprises the following steps: acquire a radar point cloud time sequence, the radar point cloud time sequence comprising continuous frame point clouds of each part of a human body within a set time; for each frame point cloud in the radar point cloud time sequence, calculate the coordinates of the centroid of the point cloud, calculate the covariance matrix of the point cloud based on the coordinates of the centroid, perform principal component analysis on all eigenvectors in the covariance matrix, and take the directions corresponding to the top three eigenvectors as the directions of the three coordinate axes of the local coordinate system, take the centroid as the origin of the local coordinate system, and construct the local coordinate system of the frame point cloud; convert each frame point cloud from the radar coordinate system to the corresponding local coordinate system to obtain a local point cloud time sequence; add random perturbations to each frame point cloud in the local point cloud time sequence to obtain a perturbed local point cloud time sequence; perform nonlinear mapping on each frame point cloud in the perturbed local point cloud time sequence, and then splice the signal intensity and centroid of the corresponding frame point cloud to obtain a frame spliced feature vector, and normalize all spliced feature vectors to form a spliced feature vector time sequence; input the spliced feature vector time sequence into a human fall detection model trained by a dynamic threshold-based semi-supervised learning to obtain a human fall detection result, wherein the human fall detection model comprises a feature embedding module, a Transformer module, a 3D residual network module and a classifier connected in sequence, and the training is a dynamic threshold-based semi-supervised learning training.

2. The human fall detection method according to claim 1, characterized in that, The coordinates of the centroid of the point cloud are calculated by the following formula: ； wherein, denotes the coordinates of the centroid of the frame point cloud, N is the number of point cloud points, is the coordinates of the point in the frame point cloud, denotes the index of the coordinates of a point in the point cloud.

3. The human fall detection method according to claim 1, characterized in that, The covariance matrix of the point cloud based on the coordinates of the centroid is calculated by the following formula: ； wherein, denotes the covariance matrix of the point cloud of the frame, denotes the coordinates of the centroid of the point cloud of the frame, N is the number of points of the point cloud, is the coordinate of the point in the point cloud of the frame, denotes the index of the coordinate of the point in the point cloud, T denotes the transpose.

4. The human fall detection method according to claim 1, characterized in that, The directions corresponding to the top three eigenvectors are taken as the directions of the three coordinate axes of the local coordinate system, which comprises: the direction corresponding to the first eigenvector is taken as the X-axis direction of the local coordinate system, the direction corresponding to the second eigenvector is taken as the Y-axis direction of the local coordinate system, and the direction corresponding to the third eigenvector is taken as the Z-axis direction of the local coordinate system.

5. The human fall detection method according to claim 1, characterized in that, The conversion of each frame point cloud from the radar coordinate system to the corresponding local coordinate system to obtain the local point cloud time sequence comprises: The top three feature vectors obtained by the principal component analysis are composed of an orthogonal rotation matrix , , The first feature vector obtained by the principal component analysis is represented by The second feature vector obtained by the principal component analysis is represented by The third feature vector obtained by the principal component analysis is represented by Based on the orthogonal rotation matrix, the coordinates of the points in the point cloud are converted from the radar coordinate system to the corresponding local coordinate system by the following formula: ； wherein T denotes the transpose, denotes the frame point cloud, coordinate of the frame point cloud, coordinate of the frame point cloud, coordinate of the frame point cloud, After converting the coordinates of all points in each frame point cloud into the corresponding local coordinate system, the local point cloud time sequence is obtained.

6. The human fall detection method according to claim 1, characterized in that, The random perturbation added to the coordinates of the points in the local coordinate system in the point cloud is added by the following formula: After adding random translation perturbations and random rotation perturbations to the coordinates of the points in each frame point cloud in the local point cloud time sequence, the perturbed local point cloud time sequence is obtained. ； wherein, is the coordinate of the i-th point in the frame point cloud in the local coordinate system after adding the random translation disturbance, is the coordinate of the i-th point in the frame point cloud in the local coordinate system after adding the random translation disturbance, is the coordinate of the i-th point in the frame point cloud in the local coordinate system after adding the random translation disturbance, is the coordinate of the i-th point in the frame point cloud in the local coordinate system after adding the random translation disturbance, is the coordinate of the i-th point in the frame point cloud in the local coordinate system after adding the random translation disturbance, is the coordinate of the i-th point in the frame point cloud in the local coordinate system after adding the random translation disturbance, is the set translation disturbance vector, is a Gaussian distribution with mean 0 and variance is a Gaussian distribution with mean 0 and variance The coordinates of the i-th point in the frame point cloud in the local coordinate system are added with a random rotation disturbance by the following formula: The coordinates of the i-th point in the frame point cloud in the local coordinate system are added with a random rotation disturbance by the following formula: The coordinates of the i-th point in the frame point cloud in the local coordinate system are added with a random rotation disturbance by the following formula: ； wherein, represents the i-th point in the frame point cloud after adding random rotation disturbance and random translation disturbance, represents the i-th point in the frame point cloud after adding random rotation disturbance and random translation disturbance, represents the i-th point in the frame point cloud after adding random rotation disturbance and random translation disturbance, represents an orthogonal rotation matrix composed of the top three eigenvectors obtained by the principal component analysis, represents a set random rotation disturbance, SO(3) represents a special orthogonal group, and T represents transposition. The nonlinear mapping of each frame point cloud in the perturbed local point cloud time sequence is performed by the following expression of a multi-kernel learnable Gaussian kernel function:

7. The human fall detection method according to claim 1, characterized in that, The feature embedding module is used to map the spliced feature vector time sequence to a high-dimensional space to obtain an embedded time sequence; ； in, This indicates the addition of random perturbations and nonlinear mapping to the th... The first frame in the point cloud The coordinates of a point in the local coordinate system Indicates the set number The first frame in the point cloud Attention weights for the coordinates of a point in the local coordinate system. Indicates the first The weights of each Gaussian kernel, Indicates the first The mean of a Gaussian kernel, Indicates the first The covariance matrix of Gaussian kernels, where K represents the number of Gaussian kernels. This indicates the result after adding random rotation and random translation perturbations. The first frame in the point cloud The coordinates of a point in the local coordinate system, where T represents the transpose and exp represents the natural exponential function.

8. The human fall detection method according to claim 1, characterized in that, ​ The Transformer module is used to extract local attention and global attention of the embedded time sequence, and gate fusion and MLP expansion are performed on the local attention and the global attention to obtain fusion features; The 3D residual network module performs convolution on the fusion features in the time, width and height dimensions through a 3D convolution kernel, performs GAP operation, convolution, GELU activation function and convolution operation in sequence on the features obtained by convolution, calculates channel attention weights through a Sigmoid activation function, element-wise multiplies the channel attention weights and the features obtained by convolution, performs weighted fusion based on a CA mechanism to obtain weighted fusion features, performs residual connection on the weighted fusion features and the spliced feature vector time sequence, and then performs global pooling to obtain a one-dimensional vector; The classifier is configured to detect whether the human body falls according to the one-dimensional vector, and output a human body fall detection result whether the human body falls within the set time.

9. The human fall detection method according to claim 1, characterized in that, The semi-supervised learning training based on a dynamic threshold value includes: training data is obtained and divided into a training set and a validation set, part of the training data in the training set is labeled to obtain labeled data, and the remaining training data in the training set is used as unlabeled data; the labeled data is input into the human body fall detection model for preliminary training; the unlabeled data is input into the human body fall detection model after preliminary training for forward propagation to obtain a prediction probability vector of each unlabeled data; if the maximum probability value in the prediction probability vector of the unlabeled data is greater than or equal to a probability threshold value, the maximum probability value index of the unlabeled data is used as a pseudo label, and the pseudo label is labeled on the corresponding unlabeled data; the human body fall detection model is evaluated on the validation set to obtain a validation accuracy of the current human body fall detection model; the labeled data and the unlabeled data with the pseudo label are merged into a new training set, the human body fall detection model is continuously trained on the new training set, the human body fall detection model after each iteration is saved, and the human body fall detection model with the highest validation accuracy is used as the trained human body fall detection model until a preset iteration number is reached; wherein the probability threshold value is dynamically updated by the following formula: ； wherein, is a probability threshold value at the jth time, is a probability threshold value at the (j-1)th time, is a verification accuracy at the jth time, is a probability threshold value at an initial time, is a set momentum coefficient, is a set scaling factor, is a verification accuracy at an initial time.

10. A human fall detection apparatus, characterized by, including: a point cloud acquisition module configured to acquire a radar point cloud time sequence, the radar point cloud time sequence including continuous frame point clouds of each part of a human body within a set time; a local coordinate system construction module configured to perform the following operations on each frame of point cloud in the radar point cloud time sequence: calculate the coordinates of the centroid of the point cloud, calculate the covariance matrix of the point cloud based on the coordinates of the centroid, perform principal component analysis on all eigenvectors in the covariance matrix to take the directions corresponding to the top three eigenvectors as the directions of the three coordinate axes of the local coordinate system, and take the centroid as the origin of the local coordinate system to construct the local coordinate system of the frame of point cloud; a coordinate conversion module configured to convert each frame of point cloud from a radar coordinate system to a corresponding local coordinate system to obtain a local point cloud time sequence; The random disturbance adding module is configured to add random disturbance on each frame of point cloud in the local point cloud time sequence to obtain a disturbed local point cloud time sequence; The mapping and splicing module is configured to perform nonlinear mapping on each frame of point cloud in the disturbed local point cloud time sequence, then splice the signal intensity and the centroid of the corresponding frame of point cloud to obtain a spliced feature vector of each frame, and normalize all the spliced feature vectors to form a spliced feature vector time sequence; The human fall detection module is configured to input the spliced feature vector time sequence into a trained human fall detection model to obtain a human fall detection result, wherein the human fall detection model comprises a feature embedding module, a Transformer module, a 3D residual network module and a classifier connected in sequence, and the training is semi-supervised learning training based on a dynamic threshold.

Images