Elderly Fall Risk Warning System Based on Multidimensional Data Fusion

Abstract

The present invention discloses an elderly fall risk warning system based on multi-dimensional data fusion. The system includes a presentation layer, a business layer, a data layer, and a hardware device layer. The presentation layer includes third-party service provider users and institutional administrator users. The third-party service provider users are mainly used for the data viewing page. The service provider understands some physical information of the elderly and the fall risk by viewing the data displayed on the front end. The business layer includes a basic information data management module, a gait analysis module, a posture analysis module, a swing arm balance detection module, and a fall risk assessment module. The data layer includes user information data, distance point cloud data, depth image data, wristwatch sensor data, and result data obtained from model analysis. The hardware device layer mainly includes the hardware devices of this research: lidar, depth lens, smart wristwatch, Raspberry Pi, and server. The system of the present invention is less affected by the environment and has higher accuracy, so it can improve the accuracy of risk assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and in particular, to an elderly fall risk warning system based on multi-dimensional data fusion. Background Art

[0002] With the gradual aggravation of the aging phenomenon in the world, falls have received increasing attention. At present, fall detection, fall risk assessment, and fall prevention have become the focus of research. Fall detection is to detect the fall behavior of the elderly through intelligent devices, and then take corresponding alarm measures, etc. However, the fall event has already occurred, causing harm to the physical, mental, and economic aspects of the elderly. Therefore, how to evaluate the physical state of the elderly and predict the fall risk becomes particularly important. By quantitatively evaluating the physical condition, walking data, etc. of the elderly, predicting the current fall risk of the elderly, and then formulating subsequent corresponding effective prevention, rehabilitation, etc. measures according to the severity of the risk to reduce the occurrence of elderly fall events, which is the most effective means to deal with elderly falls. Therefore, fall risk assessment has become an important research topic at present.

[0003] The main problems existing in current fall risk assessment are as follows: (1) The accuracy of fall risk assessment using single-device data in the real environment is not high. Currently, the research on fall risk assessment is mainly based on gait and posture analysis, etc., and large-scale high-precision measurement systems can only be used in large laboratories; with the development of intelligent devices, cameras, wearable devices, etc. can more conveniently measure the characteristics of the elderly during walking. In order to collect data more accurately, current research attempts to achieve high-precision collection by introducing more devices, but mainly by increasing the number of the same type of devices, such as multi-view cameras, wearable devices for various parts, etc., without considering the fusion analysis of data from multiple devices. (2) There is a lack of research on fall risk assessment of the elderly in the real environment. Although intelligent devices are becoming more and more convenient, the use of these devices still mainly remains in laboratory research. The data collected in the laboratory has certain deviations from the data in daily life due to factors such as the environment and artificial intervention, and cannot fully and truly reflect the behavioral characteristics of the elderly in daily life, and has poor generalization ability in the real environment.

[0004] By analysis, the problems existing in current fall risk assessment can be summarized as: due to being interfered by many external factors such as occlusion in the real environment, there are large errors in data collection by a single device; a large number of intelligent devices will violate the lives of the elderly in the real environment, and there is a lack of a multi-dimensional data fusion perception analysis algorithm with low intrusion on the daily behavior of the elderly. Summary of the Invention

[0005] The technical problem to be solved by the present invention is how to provide an elderly fall risk warning system based on multi-dimensional data fusion that is less affected by the environment, has high accuracy, and accurate early warning.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is: an elderly fall risk warning system based on multi-dimensional data fusion, which is characterized by including: a presentation layer, a service layer, a data layer, and a hardware device layer.

[0007] The presentation layer includes third-party service provider users and institutional administrator users. The third-party service provider users are mainly used for the data viewing page, and the service provider understands some physical information of the elderly and the fall risk by viewing the data displayed on the front end; the institutional administrator part is mainly used for the management of elderly information and the specific content of data display.

[0008] The service layer includes a basic information data management module, a gait analysis module, a posture analysis module, a swing arm balance detection module, and a fall risk assessment module.

[0009] The data layer includes user information data, distance point cloud data, depth image data, wristwatch sensor data, and result data obtained from model analysis, and performs storage and access operations through cloud storage and a Mysql database respectively.

[0010] The hardware device layer mainly includes the hardware devices of this research: lidar, depth camera, smart wristwatch, Raspberry Pi, and server.

[0011] The beneficial effects of adopting the above technical solutions are as follows: The system designs an elderly fall risk assessment model for the gait analysis module, posture analysis module, swing arm balance detection module, and multi-dimensional data fusion module, and evaluates through the elderly fall risk assessment model, which has the advantages of being less affected by the environment, high accuracy, and accurate early warning. In addition, the system has carried out white-box testing and black-box testing, and verified the feasibility of the multi-dimensional data fusion-based elderly fall risk warning system and the above research methods and theories through the operation results of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The following further describes the present invention in detail with reference to the drawings and specific embodiments.

[0013] Figure 1 is the architecture diagram of the system described in the embodiment of the present invention;

[0014] Figure 2 is the functional module diagram of the system described in the embodiment of the present invention;

[0015] Figure 3 is the timing diagram of the gait analysis module in the system described in the embodiment of the present invention;

[0016] Figure 4 It is the gait analysis flowchart in the embodiment of the present invention;

[0017] Figure 5 It is the environmental map drawing flowchart in the embodiment of the present invention;

[0018] Figure 6 It is the schematic diagram of the random forest decision in the embodiment of the present invention;

[0019] Figure 7 It is the Kalman filter tracking result graph in the embodiment of the present invention;

[0020] Figure 8 It is the walking speed evaluation experiment graph in the embodiment of the present invention;

[0021] Figure 9 It is the timing diagram of the posture analysis module in the system described in the embodiment of the present invention;

[0022] Figure 10 It is the posture analysis flowchart in the embodiment of the present invention;

[0023] Figure 11 It is the dataset construction flowchart in the embodiment of the present invention;

[0024] Figure 12 It is the OpenPose network structure diagram in the embodiment of the present invention;

[0025] Figure 13 It is the CNN model network structure diagram in the embodiment of the present invention;

[0026] Figure 14 It is the GRU model network structure diagram in the embodiment of the present invention;

[0027] Figure 15 It is the depth image of the elderly at home in the embodiment of the present invention;

[0028] Figure 16 It is the posture detection result graph in the embodiment of the present invention;

[0029] Figure 17 It is the 3D bone pose graph in the embodiment of the present invention;

[0030] Figure 18 It is the 3D bone pose graph after adaptive view rotation in the embodiment of the present invention;

[0031] Figure 19 It is the timing diagram of the swing arm balance detection module in the embodiment of the present invention;

[0032] Figure 20 It is the experimental environment graph of the walking self-correlation analysis in the embodiment of the present invention;

[0033] Figure 21 It is the waveform diagram of the Y-axis data of the stable walking acceleration in the embodiment of the present invention;

[0034] Figure 22 It is the waveform diagram of the Y-axis data of the acceleration with other behaviors in the embodiment of the present invention;

[0035] Figure 23 It is the flow chart of the walking autocorrelation analysis in the embodiment of the present invention;

[0036] Figure 24 It is the timing diagram of the fall risk assessment module in the embodiment of the present invention;

[0037] Figure 25 It is the network structure diagram of the GRU model in the embodiment of the present invention

[0038] Figure 26 It is the network structure diagram of the DNN model in the embodiment of the present invention;

[0039] Figure 27 It is the system application scenario diagram in the embodiment of the present invention;

[0040] Figure 28 It is the risk report result diagram in the embodiment of the present invention

[0041] Figure 29 It is the bone pose result diagram in the embodiment of the present invention. Detailed implementation manners

[0042] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0043] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar promotions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0044] As Figure 1 shown, the embodiment of the present invention discloses an elderly fall risk warning system based on multi-dimensional data fusion. The system is mainly divided into a presentation layer, a service layer, a data layer, and a hardware device layer.

[0045] 1) The presentation layer of the system is mainly divided into the third-party service provider portal and the institutional administrator portal. The third-party user part mainly includes the data viewing page, and the service provider can understand some physical information of the elderly and the fall risk by viewing the data displayed on the front end; the institutional administrator part mainly includes the management of elderly information and the specific content of data display.

[0046] 2) The business layer of the system mainly includes the user level, gait analysis model, posture analysis model, swing arm balance detection model, multi-dimensional data fusion fall risk assessment model, and data display layer.

[0047] 3) The data layer of the system mainly includes user information data, distance point cloud data, depth image data, wristwatch sensor data, and result data obtained from model analysis, which are stored and retrieved through cloud storage and the Mysql database respectively.

[0048] 4) The hardware device layer of the system mainly includes the hardware devices of this research: lidar, depth camera, smart wristwatch, Raspberry Pi, and server.

[0049] The functional modules of the elderly fall risk warning system based on multi-dimensional data fusion are as Figure 2 shown. This system includes 5 modules: basic information data management module, gait analysis module, posture analysis module, swing arm balance detection module, and fall risk assessment module. The main executor is the institutional administrator.

[0050] (1) The basic information data management module includes: elderly information management module, user information management module, and data visualization management module, which are mainly used to manage basic information and viewable data.

[0051] (2) The gait analysis module includes: point cloud data gait analysis module, walking gait feature extraction module, and walking interval positioning module. This module is mainly used to obtain the data scanned by the lidar and establish a gait analysis model for it, track the walking trajectory, extract walking features according to the tracking of the trajectory, and obtain the walking interval for subsequent positioning of other data.

[0052] (3) The posture analysis module includes: depth image data posture detection module, bone posture perspective rotation module, and walking posture feature extraction module. This module mainly obtains the depth image captured by the depth camera, performs data segmentation through the positioning interval of gait analysis, and then uses the trained posture detection model to obtain the bone posture of the elderly during walking, rotates the perspective of the bone posture, and calculates and extracts the features for subsequent fusion analysis.

[0053] (4) The swing arm balance detection module includes: an automatic correlation coefficient calculation module, which is mainly used to obtain the sensor data collected by the smart watch, perform data positioning segmentation using the positioning interval obtained by gait analysis, and finally calculate the automatic correlation coefficient.

[0054] (5) The fall risk assessment module includes: a fusion feature risk warning module for multi-dimensional data fusion fall risk warning. This module mainly uses the features extracted above for fusion and realizes the prediction and assessment of fall risk through a warning model.

[0055] Gait analysis module:

[0056] The gait analysis module analyzes and processes the point cloud data of the elderly's footsteps collected by the lidar. The timing diagram is as Figure 3 shown. The institutional administrator runs this module, reads the point cloud data from the local file through GetData, calls GetMap to construct an environmental map, then uses Moving_Extra to extract the moving points in the point cloud, calls Clusters to perform clustering to obtain a set of moving points, then uses RF_Recognition to identify the elderly's footsteps, and finally calls Kalman_Track to track the footsteps and calculate the gait features to return for the subsequent fall risk assessment module.

[0057] The method for obtaining gait features through the gait analysis module includes the following steps:

[0058] For the lidar data collected, first establish an environmental map for it, extract the moving points using the environmental map, then cluster these moving points, extract the point set features for the random forest footstep recognition model, and finally track the detected footsteps to obtain the gait features of walking. The overall process is as Figure 4 shown.

[0059] Environmental map drawing:

[0060] The environmental map describes the currently stationary objects around. For each new frame of point cloud data, it can separate the moving point cloud. Since it is a home environment, the environmental map may be different at different time points, so the environmental map needs to be updated according to time. The frame difference method is used to draw the environmental map, and when it is judged that there are no moving objects in the map, the map is updated. The specific algorithm process is as Figure 5 shown.

[0061] Step 1: Initialize the environmental map, read in the data when there is no one at night to construct the initial environmental map;

[0062] Step 2: Read the subsequent n-frame point cloud data, calculate the average value of the distance differences at corresponding angles between two frames through frame difference method, and determine whether there are moving objects in the current environment. If there are no moving objects, execute Step 3. If there are moving objects, repeat Step 2.

[0063] Step 3: Determine whether the pedestrian tracking trajectory in the current Kalman filtering algorithm exists. If the trajectory exists, it indicates that there is a situation where the elderly remain stationary for a long time in the current environment, and repeat Step 2. If the trajectory does not exist, the current environment is all environmental points, and thus calculate the average value of n-frame data to update the environmental map.

[0064] Point cloud feature extraction based on clustering:

[0065] By comparing the newly scanned data with the environmental map, the point cloud data of moving objects is obtained. The original data cannot describe the characteristics of moving objects, and clustering the scattered and disordered points is a necessary means for subsequent processing. In this part, the DBSCAN clustering algorithm is used to cluster these point cloud data and extract the features that can describe the corresponding objects. The distance calculation formula is shown in Equation (2-1):

[0066]

[0067] where P k is the position of the k-th new scan point in the new radar scan period, k = 1, 2,..., N 1 ; C ij is the j-th scan point in the i-th point set, i = 1, 2,..., N 2 ; j = 1, 2,..., N 3 .

[0068] Step 1: Read in the moving point set P, traverse the unlabeled points in P, label them and add them to the new clustering set C. Use formula (2-1) to calculate the distances between other unlabeled points and P, count the points with distances less than ε. If the count exceeds MinPts, add these points to the set N. If it is less than MinPts, do not process.

[0069] Step 2: Traverse the points in the set N, calculate the other points within its ε neighborhood, and add those with more than MinPts to N. Repeat this step until the set N is empty.

[0070] Step 3: Repeat the operations of the above Step 1 and Step 2 for the unlabeled points until each point no longer changes.

[0071] After clustering to obtain the point set, it is necessary to perform target recognition on it. Combining the shape of the footstep point cloud, the following point cloud features are designed:

[0072] Definition 2.1: The size P of the point set n . The number n of points in the point set.

[0073] Definition 2.2: The maximum length F l . Approximate the maximum length of a point cluster as the foot length, and the calculation formula is shown in Equation (2-2):

[0074] F l =|p f -p b | (2-2)

[0075] In the formula, p f , p b are the two points with the maximum distance in the moving direction in the point set P.

[0076] Definition 2.3: The foot arc F c . Calculate the radian of each point on the edge of the point set P and take the average value as the foot arc, and the calculation formula is shown in Equation (2-3):

[0077]

[0078] In the formula, p i , p i-1 are two adjacent points on the edge of the point set P; P c is the centroid of the point set P; n is the number of points on the edge of the point set P.

[0079] Definition 2.4: The foot arc length F a . Calculate the sum of the Euclidean distances between two adjacent points as the foot arc length, and the calculation formula is shown in Equation (2-4):

[0080]

[0081] In the formula, p i , p i-1 are two adjacent points on the edge of the point set P; n is the number of points on the edge of the point set P.

[0082] Definition 2.5: The foot landing area S area . Estimate the area of the point set, and the calculation formula is shown in Equation (2-5):

[0083]

[0084] In the formula, i and j are the x-coordinate values of the two points with the maximum distance in the point set P; y is the y-coordinate value of the points in the point set P.

[0085] Foot recognition based on random forest:

[0086] Footstep recognition mainly distinguishes footsteps from moving objects. Since there are numerous non-footstep objects in daily life and it is impossible to train all of them, and random forest has randomness in feature selection and strong anti-interference ability in dealing with classification tasks, it is more suitable for footsteps recognition. Therefore, this application uses a random forest model to recognize footsteps, taking the above-extracted point set features as input. The specific algorithm model is as Figure 6 shown as follows:

[0087] A random forest consists of multiple decision trees and uses the Gini index as the criterion for feature selection, which represents the probability that a randomly selected sample is misclassified. The smaller the Gini index, the more accurate the classification. Classification is carried out based on this standard, and finally, the optimal classification is determined by voting through multiple decision trees. The calculation formula of the Gini index is shown in Equation (2-6):

[0088]

[0089] where K is the number of categories; p k is the probability that the sample point belongs to the k-th category.

[0090] Finally, after classifying the point set through the random forest, the point cloud set of the footsteps is obtained to complete the footsteps recognition.

[0091] Footstep tracking based on Kalman filter:

[0092] The Kalman filter algorithm is often used in the field of target tracking and can estimate the optimal state from measurement values affected by errors. The lidar selected in this application has certain measurement errors and occlusion phenomena, while the Kalman filter algorithm can handle the occlusion problem to a certain extent through its predicted values. Therefore, this application uses a Kalman filter to track the footsteps and recover the footsteps lost due to occlusion during the tracking process, realizing the tracking of the elderly's footsteps and the extraction of gait features.

[0093] The state prediction equations of the Kalman filter algorithm are shown in Formulas (2-7) and (2-8):

[0094] X k = A k X k-1 + B k u k + w k (2-7)

[0095] z k = H k X k + v k (2-8)

[0096] where X k =(x k yk x' k y' k ) is the centroid state vector of the k-th frame, x k , y k are the position components, x' k , y' k are the velocity components; z k =(x k y k ) is the system measurement value of the k-th frame; A k is the state transition matrix; B k is the control input matrix, which maps the motion measurement value to the state vector; u k is the system control vector of the k-th frame, containing acceleration information; w k is the system noise, whose covariance is Q; H k is the transformation matrix, which maps the state vector to the space of the measurement vector; v k is the observation noise, whose covariance is R.

[0097] Generally, the walking between adjacent frames indoors can be approximated as a uniform linear motion. Therefore, the relationships shown in formulas (2-9), (2-10), (2-11), and (2-12) can be obtained:

[0098] x k =x k-1 +x' k-1 ×Δt (2-9)

[0099] y k =y k-1 +y' k-1 ×Δt (2-10)

[0100] x' k =x' k-1 (2-11)

[0101] y' k =y' k-1 (2-12)

[0102] In the formula, Δt is the time interval; k represents the current time of k.

[0103] Converting it to matrix representation as shown in formulas (2-13) and (2-14):

[0104]

[0105] (x k y k ) T =(1 1 0 0)×(x k y kx′ k y′ k ) T +v k (2 - 14)

[0106] The state transition matrix can be obtained from Equation (2 - 13) and Equation (2 - 7). Meanwhile, B k is a zero matrix; H = (1 1 0 0) can be obtained from Equation (2 - 14) and Equation (2 - 8).

[0107] Since there are errors in both measurement and prediction, it is necessary to calculate the error P existing in the current prediction process, and its calculation formula is shown in Equation (2 - 15):

[0108] P(k|k - 1) = A·P(k - 1|k - 1)·A T +Q (2 - 15)

[0109] Where P(k|k - 1) is the covariance for predicting X(k|k - 1) from X(k - 1|k - 1); P(k - 1|k - 1) is the covariance at time k - 1.

[0110] Combining the predicted state of the system at the current time and the observed state Z(k) obtained from Equation (2 - 7), the optimal estimate at this time is calculated, and the calculation formula is shown in Equation (2 - 16):

[0111] X(k|K) = X(k|K - 1)+Kg(k)(Z(k)-H·X(k|K - 1)) (2 - 16)

[0112] Where Kg(k) is the Kalman gain at time k, and its calculation formula is shown in (2 - 17):

[0113]

[0114] After obtaining the optimal estimate value at time k, finally, the covariance P(k|k) at the current time needs to be updated, and the calculation formula is shown in Equation (2 - 18):

[0115] P(k|k) = (I - Kg(k)·H)•P(k|k - 1) (2 - 18)

[0116] Where I is the identity matrix.

[0117] The specific process of the Kalman filtering algorithm is as follows:[[]]

[0118] Step 1: Calculate the predicted value c at the current time k k ;

[0119] Step 2: Determine whether the observation value at the current moment k exists. If it exists, update the Kalman filter and add the calculated optimal estimate to the tracking step walkset set, and repeat Step1; if it does not exist, proceed to Step3;

[0120] Step3: Use the observation value c k as the optimal estimate to determine whether there are observation values in the next n moments. If none exist, stop the current Kalman filter tracking, indicating the end of walking; if they exist, use the observation value and the predicted value to update the Kalman filter, and add the previously reserved predicted steps to the walkset set, and repeat Step1.

[0121] Thus, the walking trajectory of the elderly is obtained. Combining with the commonly used indicators in gait analysis, the gait characteristics of the elderly are designed as shown in Table 2, which includes the step lengths of the left and right feet, the instantaneous speeds of the left and right feet, and the landing areas during the elderly's walking.

[0122] Table 2 Explanation of gait characteristics

[0123]

[0124] Experimental results and analysis

[0125] The dynamic results of the lidar point cloud data tracked by the Kalman filter are as Figure 7 shown. The black dots in the figure are the point cloud data, the blue dots are the center points of the steps, the black lines represent the step lengths length, and the Kalman filter assigns two independent step tracking orbits to the two feet, and calculates the walking speed and walking step length of this foot according to the corresponding tracking orbit.

[0126] In clinical trials, the habitual gait speed (HGS) is a reliable and useful indicator. The measurement of HGS is easy to perform and does not require a doctor or clinical equipment. Therefore, to verify the accuracy of the results, this application invited 15 elderly people to participate in the evaluation experiment. In the measurement of HGS, distance is an indicator that affects the accuracy of gait speed measurement. According to the literature, HGS over 4 meters is reliable in clinical trials.

[0127] In the experiment, the participants were required to walk a 5.5-meter path at a normal speed and repeat the test 5 times. As Figure 8 shown, the 2D lidar was placed on the ground beside the road, and data was collected when the participants walked. At the same time, a stopwatch was used to time the walking to calculate the actual walking speed.

[0128] Since the walking speed of each step is estimated in gait analysis, the absolute error range, mean absolute error, and error variance are used to evaluate the system, as shown in Table 3. The mean absolute error within all categories is 0.06 m / s, and the maximum error is 0.11 m / s. The slower the walking speed, the more accurate the estimation. Most elderly people walk slowly in life, below 0.60 m / s. Relatively speaking, the average error of 0.06 m / s is small compared to the walking speed. Therefore, the accuracy of gait analysis can be proven.

[0129] Table 3 Mean absolute error and error variance for walking speed evaluation

[0130]

[0131]

[0132] The above steps illustrate the specific algorithm implementation of the walking stability analysis model based on gait analysis. Finally, the correctness of the hypothesis is verified through experimental results, and the accuracy of the gait analysis model is verified through the walking speed evaluation experiment. The walking interval determined by gait analysis will be used for data localization in subsequent posture analysis and swing arm balance analysis. The extracted gait features will be complementarily fused with other features in the subsequent multi-dimensional data fusion risk assessment model.

[0133] Posture Analysis Module

[0134] The posture analysis module analyzes and processes the depth image data of the elderly collected by the depth camera. The timing diagram is as Figure 9 shown. The institutional administrator runs this module, reads the image from the local file through GetData, calls Post_Detect for posture detection to obtain 2D posture data, then calls Depth2_3D to convert the posture data into 3D posture data, uses Draw_Skeleton to draw the unrotated skeleton diagram and returns it. After that, calls Rotate_Skeleton to rotate the skeleton posture diagram, and uses Draw_Skeleton again to draw the rotated skeleton diagram and return it. Finally, calculates the posture features through Calculate_Features and returns them for the subsequent fall risk assessment module.

[0135] The method for extracting posture features through the said posture analysis module is as follows:

[0136] Walking Balance Detection Model Based on Posture Analysis:

[0137] This section will analyze and explain the walking balance detection model based on pose analysis. First, the corresponding skeletal poses are extracted from the collected image data through pose detection, then the perspectives of these skeletal poses are adjusted to make the data more standardized, and finally, pose features describing body balance are designed and calculated. The overall process is as Figure 10 shown.

[0138] Pose detection model based on transfer learning:

[0139] Pose detection is used to extract skeletal information from depth images. In this section, a pose detection model for depth images will be trained through transfer learning of the OpenPose model. First, a dataset of depth images is constructed, and then the model is trained through this dataset.

[0140] (1) Construct the dataset

[0141] In the initial stage of the experiment, depth images and aligned RGB images are collected simultaneously through a depth camera. The skeletal poses are extracted from the RGB images through a pre-trained OpenPose model, and the extracted skeletal poses and the corresponding depth images are combined to form a pose dataset for training a convolutional neural network (CNN) suitable for depth images. The construction process is as Figure 11 shown.

[0142] (2) Transfer learning

[0143] In this application, transfer learning based on parameters is performed on the OpenPose model through fine-tuning, and the pre-trained network parameters are used for initialization. Its network structure is as Figure 12 shown.

[0144] The first half of the network is a feature extraction layer, which extracts features from the input images through multiple layers of convolution and pooling operations. Since depth images are similar to color images, the pre-trained parameters of OpenPose are used for initialization in this part; the second half of the network is divided into two sub-networks, which perform convolution and pooling operations respectively to obtain the position information of the joint points and the correlation information between the joint points. At the same time, the input of each stage is obtained by fusing the results of the previous stage and the original image features to produce more accurate prediction results. The training process of the network is as follows:

[0145] Step 1: Preprocess the depth image. The depth image format is a 16-bit single-channel image. First, convert the depth image from unit_16 to unit_8 data format, and then use the applyColorMap function in the OpenCV library to convert the single-channel data into a 3-channel pseudo-color image.

[0146] Step 2: Construct the network structure and transfer learning. The model extracts features from the image data through a multi-layer convolutional neural network (CNN) and pooling layers, and initializes it with the parameters of the pre-trained feature extraction layer;

[0147] Step 3: Train the model. Use the dataset constructed above to train the model to obtain the joint position information and the association relationship between the joints;

[0148] Step 4: Connect the bones. Connect the bones through the association relationship between the above joints and output the final bone information.

[0149] Bone pose rotation based on adaptive view transformation:

[0150] By referring to the correction algorithm in the image field, fill the 3D bone pose as a pseudo-image and use the convolutional neural network (CNN) to learn the rotation parameters in the spatial domain; at the same time, use the gated recurrent unit (GRU) to learn the parameters of the multi-frame bone data in the time domain, and finally fuse the outputs of the two models to obtain the rotated bone pose.

[0151] The network structure of the CNN model is as Figure 13 shown, and its specific process is as follows:

[0152] Step 1: Data preprocessing. The bone pose obtained in pose detection contains 25 points, and each point is composed of a 3D coordinate, namely its position and depth in the image. Considering the duration of the same action and the image acquisition frequency of the depth camera, the number of frames stitched for each image is set to 30 frames, that is, take 30 frames of bone pose data of the same action and stack them into a matrix of size 30*25*3. If it is less than 30 frames, it is filled with 0.

[0153] Step 2: Construct the network. It consists of 2 convolutional layers, 1 pooling layer and 1 fully connected layer. The convolutional layer performs convolutional operations on the input pseudo-image data. Each convolutional layer is followed by a Batch Normalization (BN) layer for normalization, and the activation function is the Relu() function. The last layer is a fully connected layer that outputs 3D rotation parameters, and uses these rotation parameters to perform rotation transformation on the original input data to obtain the rotated bone pose. The rotation calculation formula is shown in Equation (3-1).

[0154] p′ i =R z,γ R y,β R x,α p i (3-1)

[0155] In the formula, p i is the coordinate of the i-th bone joint pi =(x i , y i , z i ); p i ' is the transformed coordinate of the i-th skeletal joint point; R z,γ , R y,β , R x,α is the transformation matrix, and its calculation formula is shown in Eqs. (3-2), (3-3), and (3-4).

[0156]

[0157]

[0158]

[0159] In the formula, α, β, and γ are the angles of rotation around the x, y, and z axes respectively.

[0160] Step3: Train the network. Calculate the mean square error between the rotated skeletal pose data and the pose data at the frontal view angle as the loss of the network. The optimization function selects the Adam function, the number of training iterations is 50 times, and save the model with the best result in the validation set.

[0161] The network structure of the GRU model is as Figure 14 shown, and its specific process is as follows:

[0162] Step1: Data preprocessing. Convert the skeletal data of each rotated frame into a 1*75 vector. Take 30 frames as the length of the time series, and fill in the insufficient 30 frames with 0s to obtain a matrix of size 30*75 as the input of the network.

[0163] Step2: Build the network. It consists of 2 GRU layers and 1 fully connected layer. The feature dimension of the hidden layer of the GRU is set to 100. The GRU layer obtains the output at each time point, and finally outputs the skeletal pose after occlusion restoration through the fully connected layer.

[0164] Step3: Train the network. The training process is the same as that of the CNN model.

[0165] The skeletal pose obtained by pose detection gets the rotation parameters through the CNN model and performs rotation in the spatial dimension. Then, use the GRU model to restore the occlusion of the rotated pose through the context relationship to obtain the final skeletal pose with perspective transformation and occlusion restoration.

[0166] Walking pose features:

[0167] After obtaining the skeletal pose with a more appropriate perspective, the pose features of the elderly during walking are designed as follows:

[0168] Definition 3.1: Trunk Angle α trunk . It is defined as the angle between the trunk and the horizontal plane, and the calculation formula is shown in Equation (3-5):

[0169]

[0170] Wherein is the normal variable of the horizontal plane p neck is the 3D coordinate of the neck; p mid.hip is the 3D coordinate of the mid-hip.

[0171] Definition 3.2: Forward Flexion Angle α bend . It is defined as the angle of forward flexion of the body, and the calculation formula is shown in Equation (3-6):

[0172]

[0173] Wherein p nose is the 3D coordinate of the nose.

[0174] Definition 3.3: Hip Angle α α.hip . It is defined as the angle of the neck, left and right hips, and left and right knees, and the calculation formula is shown in Equation (3-7):

[0175]

[0176] Wherein p α.hip is the 3D coordinate of the left and right hips, α ∈ {left, right}; p α.knee is the 3D coordinate of the left and right knees.

[0177] Definition 3.4: Shoulder Angle α α.shoulder . It is defined as the angle of the neck, left and right shoulders, and left and right elbows, and the calculation formula is shown in Equation (3-8):

[0178]

[0179] Wherein p α.shoulder is the 3D coordinate of the left and right shoulders; p α.elbow is the 3D coordinate of the left and right elbows.

[0180] Definition 3.5: Knee Angle α α.knee . It is defined as the angle of the left and right hips, left and right knees, and left and right ankles, and the calculation formula is shown in Equation (3-9):

[0181]

[0182] Wherein p α.ankle is the 3D coordinate of the left and right ankles.

[0183] Definition 3.6: Shoulder width d shoulder . It is defined as the distance between the left and right shoulders, used to represent the differences in personal postures. The calculation formula is shown in Equation (3-10):

[0184] d shoulder = |p left.shoulder - p right.shoulder | (3-10)

[0185] In the formula, p left.shoulder is the 3D coordinate of the left shoulder; p right.elbow is the 3D coordinate of the right shoulder.

[0186] Experimental results and analysis:

[0187] The experimental results of the pose detection model based on transfer learning are as Figure 15 , Figure 16 shown. Figure 15 is the depth image of the elderly at home collected, Figure 16 is the result image after pose detection.

[0188] The skeleton pose adaptive view transformation model uses the results of the above pose detection for view transformation. First, it is converted into 3D coordinates, and the results are as Figure 17 shown, where the left arm is lost due to occlusion; after passing through the view transformation model, the results are as Figure 18 shown, and the occluded and lost joint points are predicted and restored.

[0189] This part analyzes the reasons for using transfer learning, and details the process and implementation of transfer learning; then the skeleton pose is transformed in view to make the data more standardized; finally, the transformed pose data is subjected to corresponding feature extraction and used in the subsequent multi-dimensional data fusion risk prediction model in a complementary form with the previously obtained gait features.

[0190] Swing arm balance detection module

[0191] The swing arm balance detection module analyzes and processes the acceleration and gyroscope data of the elderly's arms collected by the smart watch. The time series diagram is as Figure 19 shown. The institutional administrator runs this module, reads the data from the local file through GetData, filters the original data by calling ButterFiler, then calculates the SMV values of the acceleration and gyroscope data through Smv_Filter, then uses Find_Peaks for peak detection, and finally calls Calculate_Self_Correlation to calculate the autocorrelation coefficient of each peak and returns it for the fall risk assessment module.

[0192] The method for obtaining the autocorrelation coefficient through the swing arm balance detection module is as follows:

[0193] Problem Description and Analysis of Swing Arm Balance Detection

[0194] Inertial sensors are commonly used in fields such as fall detection and gait analysis due to factors such as simple equipment and low cost. Especially, wearable sensors can directly capture information of each part of the human body at every moment by being worn at selected positions. Its data mainly consists of acceleration and gyroscope, and is not affected by occlusion.

[0195] Inertial sensors usually have a high acquisition frequency and high sensitivity to sudden behaviors, so they can capture abnormalities occurring in normal regular behaviors. However, due to the difference between daily life and laboratory environment, there is a large amount of interference noise in the trivial behavioral actions of the elderly, which will lead to more redundant data and error data, and more devices need to be combined for joint analysis.

[0196] In response to the above problem description, this part will collect data using a smart watch with relatively low intrusion into the lives of the elderly, and it is not affected by occlusion, making up for the deficiencies of lidar and depth cameras to a certain extent. Therefore, this chapter will locate the walking interval for sensor data and detect the swing arm balance during walking, and extract features describing the swing arm balance of the elderly.

[0197] Problem Analysis

[0198] (1) Hardware Devices and Data Acquisition Methods

[0199] The smart watch selected in this paper is the Huawei Watch2. As Figure 20 shown, it can collect acceleration and gyroscope data, and is worn on the elderly's arm. The three-dimensional acceleration axis and gyroscope frequency are set to 50Hz. To ensure the battery usage duration of the watch, the upload of watch data is restricted. When it is charging and connected to the network, data is uploaded. When upload is not possible, data can be locally saved for 14 days.

[0200] (2) Swing Arm Balance Detection

[0201] Most falls of the elderly occur during walking. Walking requires the coordinated cooperation of various parts of the body. During walking, the arms usually swing with the movement of the feet. Gait analysis focuses on the foot information directly related to walking, and posture detection focuses on the overall body balance information, with less description of the upper limbs. The watch worn on the arm can fill this gap. During normal walking activities, sensor data generally maintains a certain regularity, but encountering other behavioral actions will break this regularity, which may include some behaviors that trigger falls. It is necessary to capture these unbalanced behaviors. Therefore, this paper proposes the concept of self-correlation coefficient to describe the swing arm balance represented by sensor data.

[0202] Based on the above problem description and analysis, the swing arm balance detection algorithm based on walking self - correlation analysis will be studied and analyzed below.

[0203] The information that can be described by the original data of acceleration and gyroscope cannot be directly interpreted. By analyzing the original data, it is found that the data of normal regular behaviors shows certain regularity. When behaviors different from the current ones occur during the process, the data will fluctuate. And these abnormal behaviors may be factors that cause falls during walking. Therefore, it is necessary to analyze the original data to extract the balance and correlation shown by the arms during walking, so as to describe whether the current behavior is abnormal.

[0204] As Figure 21 shown are the data of steady walking measured in the experimental environment for a period of time. Taking the acceleration Y - axis data as an example, it can be seen from the figure that the peaks or valleys during steady walking have certain similarity and balance. The appearance of abnormal behaviors will break this balance. As Figure 22 shown, there are some irregular peaks or valleys in the figure. Therefore, this paper proposes the concept of a self - correlation coefficient, which can describe the correlation between the behavior corresponding to the current peak and the data within a previous period of time, so as to capture the characteristics of unbalanced swing arms during this period of time.

[0205] Swing Arm Balance Detection Model Based on Walking Self - Correlation Analysis

[0206] The self - correlation coefficient calculates the similarity between the current peak and the data within the previous time period, and uses this data to represent the balance of the arm swing during the old person's walking. If the similarity is low, it indicates that the arm movement is abnormal at this time, and vice versa, it belongs to normal behavior. The specific flowchart of its calculation is as follows Figure 23 shown:

[0207] Step1: Read in the data. Both the acceleration and gyroscope data are three - axis data. Calculate its SMV (Signal Magnitude Vector). The calculation formula of SMV is shown in Equation (4 - 1):

[0208]

[0209] where a x , a y , a z are the data of the x, y, and z axes of acceleration or gyroscope.

[0210] Step 2: Find the positions of the peaks through the Peakutils peak detection program;

[0211] Step 3: Search forward from the current peak index moment within the interval [tmin, tmax], and calculate the autocorrelation coefficient R(i) of the current peak i. The calculation formula is shown in Equations (4-2) and (4-3):

[0212]

[0213]

[0214] Where R(i,τ) represents the autocorrelation coefficient of the current peak index i within τ time; tmin, tmax are the ranges of the interval for calculating the autocorrelation coefficient; a(i-k) is the value of SMV at the moment i-k; μ(τ,i) is the mean value of SMV from the moment τ to the moment i; δ(τ,i) is the standard deviation of SMV from the moment τ to the moment i.

[0215] Fall risk assessment module

[0216] The fall risk assessment module performs integrated calculation and analysis on the above gait features, posture features, and autocorrelation coefficients. The timing diagram is as Figure 24 shown. The institutional administrator runs this module, obtains the features calculated by the above module through GetData, then calls Max_Min_Norm to perform normalization operations on these features, and then uses Risk_Model to fuse the features and calculate the fall risk distribution. Finally, it returns the probability of fall risk through Softmax.

[0217] The method for multi-dimensional data feature fusion is as follows:

[0218] Input the gait features, posture features, and autocorrelation coefficients into different GRU models respectively, and perform calculations through attention. The specific network structure is as Figure 25 shown.

[0219] Data preprocessing: Read in three groups of data, namely gait features, posture features, and autocorrelation coefficients, perform normalization processing on each of them respectively. The calculation formula is shown in Equation (1-1), and then cut the data set into training set, validation set, and test set;

[0220]

[0221] Where x min , x max are the maximum and minimum values of the dimension where the variable x is located respectively;

[0222] Construct GRU model and attention mechanism: Input the three groups of data into three different GRU networks respectively. Each GRU model includes two layers of bidirectional BiGRU, and their input layer sizes are 6, 24, 2 respectively, and the output layer size is 50;

[0223] Attention calculation: Calculate the output of each time point of the three GRU networks, and the calculation formula is shown in Equations (1-2), (1-3), and (1-4):

[0224] u = v · tanh(W · h) (1-2)

[0225] att = softmax(u) (1-3)

[0226] out = ∑(att • h) (1-4)

[0227] Where: h is the output of each time point of the GRU network;

[0228] W, v are the parameters of the attention layer;

[0229] att is the probability distribution of the calculated attention;

[0230] out is the output result of the attention layer.

[0231] Perform a concatenation operation on the last layer of the output of the three groups of data as the input vector of the subsequent DNN network.

[0232] The DNN model is used to classify the features extracted and concatenated by the GRU model, and the probability of the elderly falling risk is output. The specific network structure is as Figure 26 shown, where the probability distribution of the data of each fully connected layer is re-normalized by introducing a Batch Normalization (BN) layer to improve the training speed and convergence speed. The specific steps are as follows:

[0233] Construct a DNN model: including 3 hidden layers and 1 output layer; the hidden layers include fully connected layers, and their inputs are 150, 128, and 64 respectively. Then, the BN layer performs batch normalization on the data. Next, the activation function uses the Relu() function. The output layer is a fully connected layer with an input size of 32 and an output of 2;

[0234] Train the network: Train through the constructed training set. The loss function uses the cross-entropy loss function, the optimization function selects the Adam function, and the number of iterative training times is 50 times. Save the model with the best result in the validation set.

[0235] Use the trained model for risk assessment: Input the three groups of data into the trained model, perform a softmax() function transformation on the output of the model, and output the probability of the corresponding fall risk of the current input to achieve the assessment of the fall risk.

[0236] System implementation

[0237] The elderly fall risk warning system based on multi-dimensional data fusion is maintained by pension institutions or community institutions, and the main users are third-party service provider users. By viewing and obtaining the physical data and fall risk of the elderly through the elderly fall risk warning system based on multi-dimensional data fusion, data guidance can be provided for the formulation of subsequent services.

[0238] System application scenarios

[0239] By arranging devices in the elderly's home environment, data in the daily life of the elderly is collected, and the application scenarios are as Figure 27 shown. Among them, the lidar is placed on the corner floor to minimize the interference to the elderly's life; the depth camera is installed on the top of the cabinet; the smart watch is worn on the left arm of the elderly.

[0240] Data analysis module

[0241] After the institutional administrator logs in to the system, the original data can be analyzed. After running the analysis through the system background, the data of the elderly when walking can be viewed: gait characteristics, swing arm balance characteristics, and the posture map at the selected moment. And by running the risk calculation, all the data of the day can be input into the fall risk assessment model to obtain the probability value of the fall risk, and generate the risk report of the elderly on the day, as Figure 28 shown.

[0242] Select a point in the coordinate graph to view the corresponding bone rotation graph at the current moment, and the result is as Figure 29 shown. The left figure is the 3D bone posture graph without rotation, and the right figure is the 3D bone posture graph after rotation.

[0243] Testing of the elderly fall risk warning system based on multi-dimensional data fusion

[0244] This part tests the elderly fall risk warning system based on multi-dimensional data fusion, and completes unit testing and system testing respectively.

[0245] Unit testing

[0246] Unit testing is carried out in the way of white box testing, and each module is tested to verify the logical correctness of the system code. The specific steps are as follows:

[0247] Step 1: The system in this article is implemented using the SpringBoot framework. Import the SpringBoot unit test package and create a test entry program.

[0248] Step 2: Create a Service unit test class and configure the test environment through annotations

[0249] Step 3: Write test cases for testing to verify whether the test results are correct.

[0250] Taking the user's view of the elderly analysis data module as an example, the unit test of its Service layer is carried out, and the test results are shown in Table 4.

[0251] Table 4 Test case table for the data analysis Service class

[0252]

[0253] From the unit test results, it can be seen that the test results of the DataAnalysis module are the same as the expected results, and its logic is correct. Using the same method, the corresponding unit tests are carried out on other modules of the system, and the results prove the usability of the elderly fall risk warning system based on multi-dimensional data fusion.

[0254] Black box testing

[0255] The system uses the method of black box testing to test each function of the elderly fall risk warning system based on multi-dimensional data fusion. Taking the function of user login to view elderly data information as an example for testing, the test results are shown in Table 5.

[0256] Table 5 Test case table for data analysis function

[0257]

[0258] The results of Table 5 show that the test results of the user login to view the elderly data information module are correct, verifying the correctness of its function. Using the same method to test other functional modules of the system, the results show that each functional module of the system meets the expected results.

Claims

1. An elderly fall risk warning system based on multi-dimensional data fusion, characterized in that it includes: a presentation layer, a business layer, a data layer, and a hardware device layer; The business layer includes a basic information data management module, a gait analysis module, a posture analysis module, a swing arm balance detection module, and a fall risk assessment module; The basic information data management module includes: an elderly information management module, a user information management module, and a data visualization management module, which are used to manage basic information and viewable data; The gait analysis module includes: a point cloud data gait analysis module, a walking gait feature extraction module, and a walking interval positioning module, which are used to obtain the data scanned by the lidar and establish a gait analysis model, track the walking trajectory, extract walking features according to the tracking of the trajectory, and obtain the walking interval for subsequent positioning of other data; The posture analysis module includes: an image data posture detection module, a bone posture perspective rotation module, and a walking posture feature extraction module, which are used to obtain the depth image captured by the depth camera, perform data segmentation through the positioning interval of gait analysis, and then use the trained posture detection model to obtain the bone posture of the elderly during walking, rotate the perspective of the bone posture, and calculate and extract the features for subsequent fusion analysis; The swing arm balance detection module includes: an automatic association system calculation module, which is used to obtain the sensor data collected by the smart watch, perform data positioning segmentation using the positioning interval obtained by gait analysis, and finally calculate the automatic association coefficient; the swing arm balance detection module analyzes and processes the acceleration and gyroscope data of the elderly's arm collected by the smart watch; The fall risk assessment module includes: a multi-dimensional data fusion fall risk warning module, which is used to fuse the extracted features and predict and evaluate the fall risk through a warning model; the fall risk assessment module is used to perform multi-dimensional data feature fusion calculation and analysis on gait features, posture features, and self-association coefficients; The self-association coefficient is used to describe the swing arm balance represented by the sensor data, calculate the similarity between the current peak and the data in the previous time period, and use this data to represent the balance of the arm swing during the elderly's walking. If the similarity is low, it means that the arm movement is abnormal at this time, otherwise it belongs to normal behavior. The specific method is as follows: Step1: Read in the data. Both the acceleration and gyroscope data are three-axis data, and calculate their (Signal MagnitudeVector), The calculation formula is as shown in Equation (4-1): （4-1） wherein is the acceleration or gyroscope data of three axes; Step 2: Find the position of the peak through the Peakutils peak detection program; Step 3: Search forward from the current peak index moment in the interval, and calculate the autocorrelation coefficient of the current peak as shown in the calculation formulas of Equation (4-2) and (4-3) as follows: ​ （4-2） （4-3） where represents the current peak index is the autocorrelation coefficient within the time period; is the range of the interval for calculating the autocorrelation coefficient; is the value of at time is the mean value of from time to time ; is the standard deviation of from time to time .

2. The elderly fall risk warning system based on multi-dimensional data fusion according to claim 1, characterized in that: The presentation layer includes third-party service provider users and institutional administrator users. The third-party service provider users are used for the data viewing page. The service provider understands some physical information and fall risk of the elderly by viewing the data displayed on the front end; the institutional administrator part is used for the management of elderly information and the specific content of data display; The data layer includes user information data, distance point cloud data, depth image data, watch sensor data, and result data obtained from model analysis, and performs storage and access operations through cloud storage and the Mysql database respectively; The hardware device layer includes: lidar, depth camera, smartwatch, Raspberry Pi, and server.

3. The elderly fall risk warning system based on multi-dimensional data fusion as claimed in claim 1, characterized in that: The gait analysis module is used to analyze and process the point cloud data of the elderly's footsteps collected by the lidar. The institutional administrator runs this module, reads the point cloud data from the local file through the GetData module, calls the GetMap module to construct the environmental map, then uses the Moving_Extra module to extract the moving points in the point cloud, calls the Clusters module to perform clustering to obtain the set of moving points, and then identifies the elderly's footsteps through the RF_Recognition module. Finally, it calls the Kalman_Track module to track the footsteps and calculate the gait features and returns them for the subsequent fall risk assessment module.

4. The elderly fall risk warning system based on multi-dimensional data fusion as claimed in claim 3, characterized in that The method for obtaining gait features through the gait analysis module includes the following steps: For the lidar data collected, first establish an environmental map for it, extract the moving points using the environmental map, then cluster these moving points, extract the feature of the point set for the random forest footstep recognition model, and finally track the detected footsteps to obtain the gait features of walking. The specific method includes the following steps: Environmental map drawing: Step 1: Initialize the environmental map, read in the data with no one at night, and construct the initial environmental map; Step 2: Read the subsequent frame point cloud data, calculate the mean value of the distance differences of corresponding angles between two frames through frame difference method, and determine whether there are moving objects in the current environment. If there are no moving objects, execute Step 3; if there are moving objects, repeat Step 2; Step 3: Determine whether the pedestrian tracking trajectory in the current Kalman filtering algorithm exists. If the trajectory exists, it indicates that there is a situation where the elderly person remains stationary for a long time in the current environment, and repeat Step 2. If the trajectory does not exist, all points in the current environment are environmental points, and thus calculate the mean value of the frame data to update the environmental map; Point cloud feature extraction based on clustering: By comparing the newly scanned data with the environmental map, the point cloud data of the moving object is obtained; use the DBSCAN clustering algorithm to cluster these point cloud data and extract the features that can describe the corresponding object. The distance calculation formula is shown in Equation (2-1): （2-1） where is the position of the th new scanning point in the new radar scanning period, ; is the th scanning point in the th point set, ; The specific steps are as follows: Step 1: Read in the set of moving points , traverse the unmarked points in and mark them and add them to the new clustering set . Use formula (2-1) to calculate the distances between other unmarked points and , count the points whose distances are less than . If the count exceeds , then add these points to the set . If it is less than , do not process; Step 2: Traverse the set of points, calculate and obtain other points within its neighborhood. If it is greater than , add it to . Repeat this step until the set is empty; Step 3: Repeat the operations of Step 1 and Step 2 for the unlabeled points until each point does not change; After clustering to obtain the point set, it is necessary to perform target recognition on it. Combining the shape of the footstep point cloud, the following point cloud features are designed: Definition 2.1: The size of a point set , the number of points in the point set ; Definition 2.2: Maximum length , approximate the maximum length of a point cluster as the foot length, and the calculation formula is shown in Equation (2-2): （2-2） wherein —— is a point set Two points with the maximum distance in the moving direction in; Definition 2.3: Foot radian , the set of calculation points Calculate the radian of each point on the edge and take the average value as the approximate foot radian. The calculation formula is shown in Equation (2-3): （2-3） wherein is a set of points two edge-adjacent points; is a set of points centroid of; is a set of points number of edge points; Definition 2.4: Foot arc length , the sum of the Euclidean distances between two adjacent points is calculated as the foot arc length, and the calculation formula is shown in Equation (2-4): （2-4） wherein is a set of points two edge - adjacent points; is a set of points the number of edge points; Definition 2.5: Foot landing area , estimate the area of the point set, and the calculation formula is shown in Equation (2-5): （2-5） wherein is the coordinate of the two points with the maximum distance in the point set value; is the coordinate of the points in the point set value; Footstep recognition based on random forest: Footstep recognition is to distinguish the footsteps from the moving objects and use the random forest model to perform footstep recognition. Take the feature of the point set extracted above as the input. The random forest consists of multiple decision trees. Use the Gini index as the criterion for feature selection, which represents the probability that a randomly selected sample is misclassified. The smaller the Gini index, the more accurate the classification. Classify according to this standard, and finally vote through multiple decision trees to determine the optimal classification. The calculation formula of the Gini index is shown in Equation (2-6): （2-6） where is the number of categories; is the probability that the sample point belongs to the th category; Finally, after classifying the point set through the random forest, the point cloud set of the footsteps is obtained to complete the footstep recognition; Footstep tracking based on Kalman filter: Track the footsteps by using the Kalman filter and recover the footsteps lost due to occlusion during the tracking process to achieve the tracking of the elderly's footsteps and the extraction of gait features; The state prediction equations of the Kalman filter algorithm are shown in Formulas (2-7) and (2-8): （2-7） （2-8 ） where is the centroid state vector of the th frame, is the position component, is the velocity component; is the system measurement value of the th frame; is the state transition matrix; is the control input matrix that maps the motion measurement value to the state vector; is the system control vector of the th frame, containing acceleration information; is the system noise, whose covariance is ; is the transformation matrix that maps the state vector into the space of the measurement vector; is the observation noise, whose covariance is ; The movement between adjacent frames indoors can be approximated as uniform linear motion. Therefore, the relationships shown in formulas (2-9), (2-10), (2-11), and (2-12) can be obtained: （2-9） （2-10） （2-11） （2-12） wherein is the time interval; represents the current as moment; Converting it into matrix representation is shown in formulas (2-13) and (2-14): （2-13） （2-14） The state transition matrix can be obtained from Equation (2-13) and Equation (2-7). , and at the same time is a zero matrix; can be obtained from Equation (2-14) and Equation (2-8); Since there are errors in both measurement and prediction, it is necessary to calculate the error existing in the current prediction process , and its calculation formula is shown in Equation (2-15): （2-15） wherein is from predicted covariance is the covariance at the moment; The predicted state and observed state of the system at the current moment obtained by combining formula (2-7) Calculate the optimal estimate at this time, and the calculation formula is shown in formula (2-16): （2-16） In the formula is the Kalman gain at time, and its calculation formula is as shown in (21): （2-17） Obtain After obtaining the optimal estimated value at the moment, finally, the covariance at the current moment needs to be updated, and the calculation formula is shown in Equation (2-18): （2-18） where is the identity matrix; The specific process of the Kalman filtering algorithm is as follows: Step 1: Calculate the predicted value at the current moment ; ; Step 2: Determine the current time Check if the observed value exists. If it exists, update the Kalman filter and add the calculated optimal estimate to the tracking step set, and repeat Step 1; if it does not exist, proceed to Step 3; Step 3: Take the observed value as the optimal estimate to judge whether there is an observed value in the next moments. If there is none, stop the current Kalman filter tracking, indicating the end of walking; if there is, use the observed value and the predicted value to update the Kalman filter, and add the previously reserved predicted footsteps to the set, and repeat Step 1; Thus, the walking trajectory of the elderly is obtained, combined with the commonly used indicators in gait analysis, including the step lengths of the left and right feet, the instantaneous speeds of the left and right feet, and the landing area during the elderly's walking process.

5. The elderly fall risk warning system based on multi-dimensional data fusion as claimed in claim 1, characterized in that: The posture analysis module analyzes and processes the depth image data of the elderly collected by the depth camera. The institutional administrator runs this module, reads the image from the local file through the GetData module, calls the Post_Detect module for posture detection to obtain 2D posture data, then calls the Depth2_3D module to convert the posture data into 3D posture data, uses the Draw_Skeleton module to draw the unrotated skeleton diagram and returns it. After that, calls the Rotate_Skeleton module to rotate the skeleton posture diagram, and uses the Draw_Skeleton module again to draw the rotated skeleton diagram and return it. Finally, calculates the posture features through the Calculate_Features module and returns them for the subsequent fall risk assessment module.

6. The elderly fall risk warning system based on multi-dimensional data fusion as claimed in claim 5, characterized in that The method for extracting posture features through the posture analysis module is as follows: First, perform posture detection on the collected image data to extract the corresponding skeleton postures, then adjust the perspectives of these skeleton postures to make the data more standardized, and finally design and calculate the posture features describing body balance. The specific method is as follows: Posture detection model based on transfer learning: Posture detection is used to extract the skeleton information in the depth image. A posture detection model for depth images is trained through transfer learning of the OpenPose model; first, construct a dataset of depth images, and then train the model through this dataset; (1) Construct the dataset In the initial stage of the experiment, the depth image and the aligned RGB image are simultaneously collected by the depth camera. The skeleton postures are extracted from the RGB image through the pre-trained OpenPose model, and the extracted skeleton postures and the corresponding depth images are combined to form a posture dataset for training the convolutional neural network CNN applicable to depth images; (2) Transfer learning Perform parameter-based transfer learning on the OpenPose model by fine-tuning, and initialize it with the pre-trained network parameters. The first half of the network is the feature extraction layer, which extracts features from the input image through multiple layers of convolution and pooling operations. Since the depth image is similar to the color image, in this part, it is initialized by using the pre-trained parameters of OpenPose. The second half of the network is divided into two sub-networks, which perform convolution and pooling operations respectively to obtain the position information of the joints and the correlation information between the joints. At the same time, the input of each stage is obtained by fusing the results of the previous stage and the original image features to produce more accurate prediction results. The training process of the network is as follows: Step 1: Depth image preprocessing: The depth image is in the format of a 16-bit single-channel image. First, convert the depth image from to data format. Then, use the function in the library to convert the single-channel data into a 3-channel pseudo-color image; Step2: Construct the network structure and transfer learning: The model extracts features from the image data through a multi-layer convolutional neural network and pooling layers, and initializes it with the parameters of the pre-trained feature extraction layer; Step3: Train the model: Use the constructed dataset above to train the model to obtain the position information of the joints and the correlation relationship between the joints; Step4: Connect the bones: Connect the bones through the above correlation relationship between the joints and output the final bone information; Bone pose rotation based on adaptive perspective transformation: Through the correction algorithm in the image field, fill the 3D bone pose into a pseudo-image and use the convolutional neural network CNN to learn the rotation parameters in the spatial domain; at the same time, use the gated recurrent unit GRU to learn the parameters of the multi-frame bone data in the time domain. Finally, fuse the outputs of the two models to obtain the rotated bone pose; The specific process of the network of the CNN model is as follows: Step1: Data preprocessing: The skeletal poses obtained in pose detection contain 25 points, each point consisting of a 3D coordinate, namely its position and depth in the image; Considering the duration of the same behavior and the image acquisition frequency of the depth camera, the number of frames stitched for each image is set to 30 frames, that is, 30-frame skeletal pose data of the same behavior are taken and stacked into a matrix of size If the number of frames is less than 30, it is padded with 0; Step 2: Build the network: It consists of 2 convolutional layers, 1 pooling layer, and 1 fully connected layer; the convolutional layers perform convolutional operations on the input pseudo-image data, and each convolutional layer is followed by a Batch Normalization (BN) layer for normalization. The activation function is function. The last layer is a fully connected layer that outputs 3D rotation parameters. These rotation parameters are used to perform a rotation transformation on the original input data to obtain the rotated skeletal pose. The rotation calculation formula is shown in Equation (3-1); （3-1） where is the coordinate of the th skeletal joint point ; is the transformed coordinate of the th skeletal joint point; the transformation matrix is calculated as shown in Equations (3-2), (3-3), and (3-4). （3-2） （3-3） （3-4） wherein are respectively the angles of rotation about axes; Step 3: Train the network: Calculate the mean squared error between the rotated skeleton pose data and the pose data at the frontal view angle as the loss of the network, and select the function. The number of training iterations is 50, and save the model with the best result in the validation set; The network processing process of the GRU model is as follows: Step1: Data preprocessing: Convert the skeletal data of each rotated frame into vectors; Take 30 frames as the length of the time series, and fill in the frames less than 30 with 0s to obtain a matrix of size as the input of the network; Step2: Construct the network: It consists of 2 layers of GRU and 1 layer of fully connected layer. The feature dimension of the hidden layer of GRU is set to 100. The GRU layer obtains the output at each time point, and finally outputs the occluded and restored bone pose through the fully connected layer; Step3: Train the network: The training process is the same as that of the CNN model; The bone pose obtained by pose detection obtains the rotation parameters through the CNN model and performs rotation in the spatial dimension. Then use the GRU model to occlude and restore the rotated pose through the context relationship to obtain the final bone pose with perspective transformation and occlusion restoration; Walking posture Features: After obtaining a more appropriate perspective bone pose, the posture features of the elderly during walking are designed as follows: Definition 3.1: Trunk Angle ; defined as the angle between the trunk and the horizontal plane, and the calculation formula is shown in Equation (3-5): （3-5） wherein is the normal variable of the horizontal plane ; is the 3D coordinate of the neck; is the 3D coordinate of the mid-hip; Definition 3.2: Forward flexion angle ; defined as the angle of forward flexion of the body, and the calculation formula is shown in Equation (3-6): （3-6） In the formula is the 3D coordinate of the nose; Definition 3.3: Hip Angle ; defined as the angle formed by the neck, the left and right hips, and the left and right knees, and the calculation formula is as shown in Equation (3-7): （3-7） wherein are the 3D coordinates of the left and right buttocks, ; are the 3D coordinates of the left and right knees; Definition 3.4: Shoulder Angle ; defined as the angle formed by the neck, the left and right shoulders, and the left and right elbows, and the calculation formula is shown in Equation (3-8): （3-8） wherein are the 3D coordinates of the left and right shoulders; are the 3D coordinates of the left and right elbows; Definition 3.5: Knee Angle ; defined as the angle formed by the left and right hips, the left and right knees, and the left and right ankles, and the calculation formula is as shown in Equation (3-9): （3-9） Formula are the 3D coordinates of the left and right ankles; Definition 3.6: Shoulder width ; It is defined as the distance between the left and right shoulders and is used to represent the differences in personal postures. The calculation formula is shown in Equation (3-10): （3-10） wherein is the 3D coordinate of the left shoulder; is the 3D coordinate of the right shoulder.

7. The elderly fall risk warning system based on multi-dimensional data fusion according to claim 1, characterized in that: The swing arm balance detection module analyzes and processes the acceleration and gyroscope data of the elderly's arm collected by the smart watch; the institutional administrator runs this module, reads data from the local file through the GetData module, calls the ButterFiler module to filter the original data, and then calculates the values of the acceleration and gyroscope data through the Smv_Filter module. After that, the Find_Peaks module is used for peak detection, and finally the Calculate_Self_Correlation module is called to calculate the self-correlation coefficient of each peak and return it to the fall risk assessment module.

8. The elderly fall risk warning system based on multi-dimensional data fusion according to claim 1, characterized in that: The fall risk assessment module is used to perform multi-dimensional data feature fusion calculation and analysis on gait features, posture features, and self-correlation coefficients. The institutional administrator runs this module, obtains the features calculated by the above module through the GetData module, then calls the Max_Min_Norm module to perform normalization operations on these features, and then uses the Risk_Model module to fuse the features and calculate the fall risk distribution. Finally, the probability of fall risk is returned through the Softmax module.

9. The elderly fall risk warning system based on multi-dimensional data fusion as described in claim 8, characterized in that The gait feature, posture feature, and self-correlation coefficient are respectively input into different GRU models, and through the calculation of attention, the method for multi-dimensional data feature fusion is as follows: Data preprocessing: Read in three groups of data, namely gait features, posture features, and self-correlation coefficients, and perform normalization processing on each of them. The calculation formula is as shown in Equation (1-1). Then, the data set is cut and processed into a training set, a validation set, and a test set; （1-1） where are respectively the maximum and minimum values of the variables in the dimension where they are located; Constructing the GRU model and the attention mechanism: Input the three groups of data into three different GRU networks respectively. Each GRU model includes two layers of bidirectional BiGRU, and their input layer sizes are 6, 24, and 2 respectively, and the output layer size is 50; Attention calculation: Calculate the output of each time point of the three GRU networks. The calculation formulas are as shown in Equations (1-2), (1-3), and (1-4): （1-2） （1-3） （1-4） Wherein: is the output of the GRU network at each time point; are the parameters of the attention layer; is the calculated probability distribution of attention; is the output result of the attention layer. The last layers of the three groups of data outputs are concatenated as the input vector of the subsequent DNN network.

Images