Elderly fall risk assessment method based on multidimensional data fusion

Abstract

The present invention discloses a method for assessing the risk of elderly people falling based on multidimensional data fusion, comprising the following steps: constructing a data set required for the elderly people falling risk assessment of a DNN model; inputting gait features, posture features and autocorrelation coefficients into different gate control unit GRU models respectively and realizing multidimensional data feature fusion through attention calculation; using the DNN model to classify the features extracted and spliced ​​by the GRU model, and outputting the probability of the elderly people falling risk. The method of the present invention is less affected by the environment and has high precision, so it can improve the accuracy of risk assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and in particular to a method for assessing the risk of falling of the elderly based on multi-dimensional data fusion. Background Art

[0002] As the aging of the world population becomes more serious, falls are receiving more and more attention. At present, fall detection, fall risk assessment and fall prevention have become the focus of research. Fall detection is to detect the elderly's falling behavior through smart devices, and then take corresponding alarm measures, but the fall has already occurred, which has caused physical, psychological and economic damage to the elderly. Therefore, it is particularly important to evaluate the physical condition of the elderly and predict the risk of falls. Through the corresponding quantitative assessment of the elderly's physical and walking data, the current fall risk of the elderly is predicted, so that the severity of the risk can be used to formulate subsequent effective prevention and rehabilitation measures for the elderly to reduce the occurrence of falls, and pave the way for subsequent research on fall prevention. It is the most effective means to deal with falls in the elderly. Therefore, fall risk assessment has become an important research topic at present.

[0003] The main problems in fall risk assessment at present are: (1) The accuracy of fall risk assessment using data from a single device in a real environment is not high. At present, the research on fall risk assessment is mainly based on gait and posture analysis, and large-scale high-precision measurement systems can only be used in large laboratories. With the development of smart devices, cameras, wearable devices, etc. can more conveniently measure the characteristics of the elderly's walking process. In order to collect data more accurately, current research introduces more devices to achieve high-precision collection, but the main reason is the increase in the number of similar devices, such as multi-view cameras, wearable devices for various parts, etc., without considering the fusion analysis of data from multiple devices. (2) There is a lack of research on fall risk assessment for the elderly in a real environment. Although smart devices are becoming more convenient, the use of these devices is still mainly in laboratory research. The data collected in the laboratory is limited by factors such as the environment and human intervention, and there is a certain deviation from the data in daily life. It cannot fully and truly reflect the behavioral characteristics of the elderly in daily life, and has poor generalization ability in a real environment.

[0004] Through analysis, the current problems in fall risk assessment can be summarized as follows: due to interference from many external factors such as occlusion in the real environment, there are large errors in data collected by a single device; a large number of smart devices in the real environment will infringe on the lives of the elderly, and there is a lack of perception analysis algorithms that integrate multi-dimensional data with low intrusion into the daily behavior of the elderly. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to provide a fall risk assessment method for the elderly based on multidimensional data fusion which is less affected by the environment and has higher accuracy.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for assessing the risk of falling of the elderly based on multi-dimensional data fusion, which is characterized by comprising the following steps:

[0007] Construct the data set required for DNN model fall risk assessment for the elderly;

[0008] The gait features, posture features and autocorrelation coefficients are input into different gate control unit GRU models respectively and multi-dimensional data feature fusion is realized through attention calculation;

[0009] The DNN model is used to classify the features extracted and concatenated by the GRU model, and the probability of the elderly person’s fall risk is output.

[0010] The beneficial effects of adopting the above technical solution are: the method proposes the use of the GRU model and the attention mechanism to fuse these features, solving the problem that data of different dimensions and frequencies are difficult to fuse; then the fused features are used to implement the assessment of the fall risk of the elderly through the DNN model, and finally, through experiments, a comparative analysis is carried out with models such as the hidden Markov model, and it is concluded that the use of the GRU model and the introduction of the attention mechanism is better in the fall risk assessment of multi-dimensional data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0012] Figure 1 is a flow chart of the method described in an embodiment of the present invention;

[0013] Figure 2 This is a GRU model network structure diagram in the method described in the embodiment of the present invention

[0014] Figure 3 This is a DNN model network structure diagram in the method described in the embodiment of the present invention

[0015] Figure 4 is a gait analysis flow chart in an embodiment of the present invention;

[0016] Figure 5 is a flow chart of environmental mapping in an embodiment of the present invention;

[0017] Figure 6 is a schematic diagram of random forest decision making in an embodiment of the present invention;

[0018] Figure 7 This is a Kalman filter tracking result diagram in an embodiment of the present invention;

[0019] Figure 8 is a walking speed evaluation experiment diagram in an embodiment of the present invention;

[0020] Fig. 9 is a flow chart of posture analysis in an embodiment of the present invention;

[0021] Fig.10 is a flow chart of constructing a data set in an embodiment of the present invention;

[0022] Fig.11 This is a diagram of the OpenPose network structure in an embodiment of the present invention;

[0023] Fig.12 It is a network structure diagram of a CNN model in an embodiment of the present invention;

[0024] Fig.13 It is a network structure diagram of a GRU model in an embodiment of the present invention;

[0025] Fig.14 is a depth image of an elderly person at home in an embodiment of the present invention;

[0026] Fig.15 is a graph of posture detection results in an embodiment of the present invention;

[0027] Fig.16 is a 3D skeleton posture diagram in an embodiment of the present invention;

[0028] Fig.17 is a 3D skeleton posture diagram after adaptive view rotation in an embodiment of the present invention;

[0029] Fig.18 This is a diagram of the walking self-correlation analysis experiment environment in an embodiment of the present invention;

[0030] Fig.19 : is a Y-axis data waveform diagram of steady walking acceleration in an embodiment of the present invention;

[0031] Fig. 20 is a waveform diagram of acceleration Y-axis data in which other behaviors exist in the embodiment of the present invention;

[0032] Fig.21 This is a flow chart of walking self-correlation analysis in an embodiment of the present invention;

[0033] Fig. 22 This is the Y-axis data waveform of the acceleration of steady walking in the embodiment of the present invention

[0034] Fig.23 This is the acceleration Y-axis data waveform diagram of other behaviors in the embodiment of the present invention

[0035] Fig.24MGAD model training process diagram according to an embodiment of the present invention

[0036] Fig.25a This is a comparison chart of the accuracy of the four models in the embodiment of the present invention.

[0037] Fig.25b This is a comparison chart of the accuracy of the four models in the embodiment of the present invention.

[0038] Fig.25c is a comparison chart of recall rates of four models in an embodiment of the present invention;

[0039] Fig.25d 4 is a comparison chart of F values ​​of four models in the embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0042] like Figure 1 As shown, the embodiment of the present invention discloses a method for assessing the risk of falling of the elderly based on multi-dimensional data fusion, comprising the following steps:

[0043] Construct the data set required for DNN model fall risk assessment for the elderly;

[0044] The gait features, posture features and autocorrelation coefficients are input into different gate control unit GRU models respectively and multi-dimensional data feature fusion is realized through attention calculation;

[0045] The DNN model is used to classify the features extracted and concatenated by the GRU model, and the probability of the elderly person’s fall risk is output.

[0046] Build the dataset:

[0047] The original data is first used to extract gait features through a gait analysis model, and the data is segmented by time intervals to obtain the various time periods when the elderly walk; then posture analysis and arm swing balance detection are used to segment the corresponding data according to the time period and process the extracted features; after obtaining 3 groups of data for each walking time period, they are labeled. Through the consultation of professionals and the medical history of the elderly, this application will invite the elderly subjects to be divided into two groups: healthy people with no history of falls and those who suffer from or have suffered from diseases such as Parkinson's disease and cerebral thrombosis. The two groups of data are labeled as 0 and 1 respectively. This data set will be used for the following fall risk assessment model.

[0048] The data set is divided into training set, validation set and test set. The specific description is shown in Table 1:

[0049] Table 1 Dataset description

[0050]

[0051] Multidimensional data feature fusion:

[0052] The gait features, posture features and autocorrelation coefficients are input into different GRU models respectively, and then calculated by attention. The specific network structure is as follows Figure 2 shown.

[0053] Step 1: Data preprocessing. Read in three sets of data: gait features, posture features, and autocorrelation coefficients, and normalize them respectively. The calculation formula is shown in formula (1-1). Then the data set is cut into training set, validation set, and test set.

[0054]

[0055] Where x min ,x max are the maximum and minimum values ​​of the dimension where the variable x is located.

[0056] Step 2: Construct GRU model and attention mechanism. The three sets of data are input into three different GRU networks. Each GRU model consists of two layers of bidirectional GRU (BiGRU), with input layer sizes of 6, 24, and 2, and output layer size of 50.

[0057] Step 3: Attention calculation. Calculate the output of each time point of the three GRU networks. The calculation formulas are shown in equations (1-2), (1-3), and (1-4):

[0058] u=v·tanh(W·h) (1-2)

[0059] att=softmax(u) (1-3)

[0060] out=∑(att·h) (1-4)

[0061] Where h is the output of the GRU network at each time point; W, v are the parameters of the attention layer; att is the calculated probability distribution of attention; out is the output result of the attention layer. The last layer of the three sets of data output is concatenated as the input vector of the subsequent DNN network.

[0062] Fall risk assessment based on DNN model:

[0063] Use the DNN model to classify the features extracted and concatenated by the GRU model and output the probability of fall risk. The specific network structure is as follows: Figure 3 As shown in the figure, the Batch Normalization (BN) layer is introduced to renormalize the probability distribution of the data in each fully connected layer, thereby improving the training speed and convergence speed.

[0064] Step 1: Build a DNN model, which consists of 3 hidden layers and 1 output layer. The hidden layer consists of a fully connected layer, whose inputs are 150, 128, and 64 respectively. Then the BN layer performs batch normalization on the data, and the activation function uses the Relu() function. The output layer is a fully connected layer with an input size of 32 and an output of 2.

[0065] Step 2: Train the network. Use the training set constructed in the previous section for training. Use the cross entropy loss function as the loss function and the Adam function as the optimization function. Iterate the training 50 times and save the model with the best result in the validation set.

[0066] Step 3: Use the trained model for risk assessment. Input the three sets of data into the trained model, transform the model output with the softmax() function, and output the probability of fall risk corresponding to the current input to achieve the assessment of fall risk.

[0067] The following is a detailed description of the above steps combined with specific methods

[0068] Walking stability analysis model based on gait analysis:

[0069] For the collected LiDAR data, we first build an environment map, use the environment map to extract moving points, then cluster these moving points, extract point set features for the random forest footstep recognition model, and finally track the detected footsteps to obtain the walking gait features. The overall process is as follows: Figure 4 shown.

[0070] Environment Mapping:

[0071] The environment map describes the stationary objects around the current environment. For each frame of new point cloud data, the moving point cloud can be separated. Because it is a home environment, the environment map at different time points may be different, so the environment map needs to be updated according to time. The frame difference method is used to draw the environment map, and the map is updated when it is determined that there are no moving objects in the map. The specific algorithm flow is as follows: Figure 5 shown.

[0072] Step 1: Initialize the environment map and read the nighttime unmanned data to build the initial environment map;

[0073] Step 2: Read the subsequent n frames of point cloud data, calculate the mean of the distance difference of the corresponding angles between the two frames by frame difference method, and determine whether there is a moving object in the current environment. If there is no moving object, execute Step 3; if there is a moving object, repeat Step 2.

[0074] Step 3: Determine whether the pedestrian tracking track in the current Kalman filter algorithm exists. If the track exists, it means that there is an elderly person who has been stationary for a long time in the current environment. Repeat Step 2. If the track does not exist, the current environment is all environmental points, so as to calculate the mean of n frames of data and update the environmental map.

[0075] Point cloud feature extraction based on clustering:

[0076] By comparing the newly scanned data with the environment map, the point cloud data of the moving object is obtained. The original data cannot describe the characteristics of the moving object, and clustering the scattered and messy points is a necessary means for subsequent processing. This section uses the DBSCAN clustering algorithm to cluster these point cloud data and extract features that can describe the corresponding objects. The distance calculation formula is shown in formula (2-1):

[0077]

[0078] Where P k is the position of the kth new scanning point in the new radar scanning cycle, k = 1, 2, ..., N1; C ij is the jth scanning point in the i-th point set, i = 1, 2, ..., N2; j = 1, 2, ..., N3.

[0079] Step 1: Read the moving point set P, traverse the unmarked points in P, mark them and add them to the new cluster set C, use formula (2-1) to calculate the distance between other unmarked points and P, count the points with a distance less than ε, if it exceeds MinPts, add these points to the set N, if it is less than MinPts, do not process them.

[0080] Step 2: Traverse the points in set N, calculate other points in its ε neighborhood, add them to N if they are larger than MinPts, and repeat this step until set N is empty.

[0081] Step 3: Repeat the above Step 1 and Step 2 operations for the unmarked points until each point does not change.

[0082] After clustering the point set, it is necessary to identify the target. Combined with the shape of the footstep point cloud, the following point cloud features are designed:

[0083] Definition 2.1: The size P of a point set n . The number of points n in the point set.

[0084] Definition 2.2: Maximum length F l The maximum length of a point cluster is approximated as the foot length, and the calculation formula is shown in formula (2-2):

[0085] F l =|p f -p b | (2-2)

[0086] Where p f ,p b are the two points in the point set P with the maximum distance in the moving direction.

[0087] Definition 2.3: Foot arc F c . Calculate the arc of each point on the edge of the point set P and take the average to approximate the arc of the foot. The calculation formula is shown in formula (2-3):

[0088]

[0089] Where p i ,p i-1 are two points adjacent to each other on the edge of the point set P; c is the centroid of the point set P; n is the number of edge points of the point set P.

[0090] Definition 2.4: Foot arc length F a The sum of the Euclidean distances between two adjacent points is calculated as the arc length, and the calculation formula is shown in formula (2-4):

[0091]

[0092] Where p i ,p i-1 are two adjacent points on the edge of the point set P; n is the number of edge points in the point set P.

[0093] Definition 2.5: Foot landing area S area. Estimate the area of ​​the point set, and the calculation formula is shown in formula (2-5):

[0094]

[0095] Where i and j are the x-coordinate values ​​of the two points with the maximum distance in the point set P; y is the y-coordinate value of the point in the point set P.

[0096] Footstep recognition based on random forest:

[0097] Footstep recognition is mainly to distinguish footsteps from moving objects. Since there are many kinds of non-footstep objects in daily life, it is impossible to train all of them. Random forest has random feature selection in classification tasks and strong anti-interference ability, which is more suitable for footstep recognition. Therefore, this application uses the random forest model to identify footsteps, taking the above extracted point set features as input. The specific algorithm model is as follows Figure 6 As shown:

[0098] Random forest is composed of multiple decision trees. It uses the Gini index as the criterion for feature selection, which indicates the probability of a randomly selected sample being misclassified. The smaller the Gini index, the more accurate the classification. Classification is performed based on this standard, and finally multiple decision trees vote to determine the optimal classification. The calculation formula of the Gini index is shown in formula (2-6):

[0099]

[0100] Where K is the number of categories; p k is the probability that the sample point belongs to the kth class.

[0101] Finally, after classifying the point set through random forest, the point cloud set of the footsteps is obtained to complete the footstep recognition.

[0102] Footstep tracking based on Kalman filter:

[0103] The Kalman filter algorithm is commonly used in the field of target tracking, and can estimate the optimal state from the measurement value affected by the error. The laser radar selected in this application has certain measurement errors and occlusion, and the Kalman filter algorithm can handle the occlusion problem to a certain extent through its predicted value. Therefore, this application uses the Kalman filter to track the footsteps and recover the footsteps lost due to occlusion during the tracking process, so as to achieve the tracking of the footsteps of the elderly and the extraction of gait features.

[0104] The state prediction equation of the Kalman filter algorithm is shown in formulas (2-7) and (2-8):

[0105] X k =A k X k-1 +Bk u k +w k (2-7)

[0106] z k =H k X k +v k (2-8)

[0107] Where X k =(x k y k x′ k y′ k ) is the center of mass state vector of the kth frame, x k ,y k is the position component, x′ k ,y′ k is the velocity component; z k =(x k y k ) is the system measurement value of the kth frame; A k is the state transfer matrix; B k is the control input matrix that maps motion measurements to the state vector; u k is the system control vector of the kth frame, including acceleration information; w k is the system noise, whose covariance is Q; H k is the transformation matrix, which maps the state vector to the space of the measurement vector; v k is the observation noise, and its covariance is R.

[0108] In general, walking between adjacent frames indoors can be approximated as uniform linear motion, so we can obtain the relationships shown in formulas (2-9), (2-10), (2-11), and (2-12):

[0109] x k =x k-1 +x′ k-1 ×Δt (2-9)

[0110] y k =y k-1 +y′ k-1 ×Δt (2-10)

[0111] x′ k = x′ k-1 (2-11)

[0112] y′ k =y′ k-1 (2-12)

[0113] Where Δt is the time interval; k means the current time is k.

[0114] Convert it into matrix representation as shown in formulas (2-13) and (2-14):

[0115]

[0116] (x k y k ) T =(1 1 0 0)×(x k y k x′ k y′ k ) T +v k (2-14)

[0117] The state transfer matrix can be obtained by using formula (2-13) and formula (2-7): At the same time B k is a zero matrix; through formula (2-14) and formula (2-8), we can get H = (1 1 0 0).

[0118] Since both measurement and prediction have errors, it is necessary to calculate the error P in the current prediction process. The calculation formula is shown in formula (2-15):

[0119] P(k|k-1)=A·P(k-1|k-1)·A T +Q (2-15)

[0120] Where P(k|k-1) is the covariance of predicting X(k|k-1) from X(k-1|k-1); P(k-1|k-1) is the covariance at time k-1.

[0121] Combine the predicted state and observed state Z(k) of the system at the current moment obtained by formula (2-7) to calculate the optimal estimate at this time. The calculation formula is shown in formula (2-16):

[0122] X(k|K)=X(k|K-1)+Kg(k)(Z(k)-H·X(k|K-1)) (2-16)

[0123] Where Kg(k) is the Kalman gain at time k, and its calculation formula is shown in (2-17):

[0124]

[0125] After obtaining the optimal estimate at time k, the covariance P(k|k) at the current time needs to be updated. The calculation formula is shown in formula (2-18):

[0126] P(k|k)=(I-Kg(k)·H)·P(k|k-1) (2-18)

[0127] Where I is the unit matrix.

[0128] The specific process of the Kalman filter algorithm is as follows:

[0129] Step 1: Calculate the predicted value c of k at the current time k ;

[0130] Step 2: Determine whether the observation value of k at the current moment exists. If so, update the Kalman filter and add the calculated optimal estimate to the tracked footsteps walkset set, and repeat Step 1; if not, proceed to Step 3;

[0131] Step 3: The observed value c k As the optimal estimate, determine whether there are observation values ​​in the next n moments. If there are no observation values, stop the current Kalman filter tracking, indicating the end of the walk. If there are observation values ​​and predicted values, use the observed values ​​and predicted values ​​to update the Kalman filter, add the previously retained predicted steps to the walkset set, and repeat Step 1.

[0132] So far, the walking trajectory of the elderly is combined with the commonly used indicators in gait analysis. The gait characteristics of the elderly are designed as shown in Table 2, including the step length of the left and right feet, the instantaneous speed of the left and right feet, and the landing area during the elderly's walking.

[0133] Table 2 Description of gait characteristics

[0134]

[0135] Experimental results and analysis

[0136] The dynamic results of laser radar point cloud data tracking through Kalman filtering are as follows: Figure 7 As shown in the figure, the black dots are point cloud data, the blue dots are the center points of the steps, and the black lines are the step lengths. The Kalman filter assigns two independent footstep tracking tracks to the two feet, and calculates the walking speed and walking step length of the foot according to the corresponding tracking tracks.

[0137] In clinical trials, habitual gait speed (HGS) is a reliable and useful indicator. HGS is easy to measure and does not require doctors or clinical equipment. Therefore, in order to verify the accuracy of the results, this application invited 15 elderly people to participate in the evaluation experiment. In HGS measurement, distance is an indicator that affects the accuracy of gait speed measurement. According to the literature, HGS over 4 meters is reliable in clinical trials.

[0138] In the experiment, participants were asked to walk a 5.5-meter path at a normal speed and repeat the test 5 times, e.g. Figure 8 As shown in the figure, a 2D LiDAR is placed on the ground beside the road to collect data when the participants walk. A stopwatch is used to time the walking to calculate the actual walking speed.

[0139] Since the walking speed of each step is estimated in gait analysis, the absolute error range, mean absolute error and error variance are used to evaluate the system, as shown in Table 3. The mean absolute error in all categories is 0.06 m / s, and the highest error is 0.11 m / s. The slower the walking speed, the more accurate the estimation. Most elderly people walk slowly in life, less than 0.60 m / s. The average error of 0.06 m / s is relatively small compared to the walking speed, so the accuracy of gait analysis can be proved.

[0140] Table 3 Mean absolute error and error variance of walking speed evaluation

[0141]

[0142]

[0143] The above steps illustrate the specific algorithm implementation of the walking stability analysis model based on gait analysis. Finally, the correctness of the hypothesis was verified through experimental results, and the accuracy of the gait analysis model was verified through walking speed evaluation experiments. The walking interval determined by gait analysis will be used for data positioning in subsequent posture analysis and arm swing balance analysis. The extracted gait features will be used in a complementary fusion with other features in the subsequent multidimensional data fusion risk assessment model.

[0144] Walking balance detection model based on posture analysis:

[0145] This section will analyze and explain the walking balance detection model based on posture analysis. First, posture detection is performed on the collected image data to extract the corresponding bone postures, and then the perspective of these bone postures is adjusted to make the data more standardized. Finally, the posture features that describe the body balance are designed and calculated. The overall process is as follows: Fig. 9 shown.

[0146] Posture detection model based on transfer learning:

[0147] Pose detection is used to extract skeleton information from depth images. This section will train a pose detection model for depth images by performing transfer learning on the OpenPose model. First, a dataset of depth images is constructed, and then the model is trained using the dataset.

[0148] (1) Building a dataset

[0149] At the beginning of the experiment, the depth image and the aligned RGB image are collected by the depth camera at the same time. The skeleton posture of the RGB image is extracted by the pre-trained OpenPose model. The extracted skeleton posture and the corresponding depth image are combined into a posture dataset for training the convolutional neural network (CNN) suitable for depth images. The construction process is as follows: Fig.10 shown.

[0150] (2) Transfer Learning

[0151] This application uses fine-tuning to perform parameter-based transfer learning on the OpenPose model, using pre-trained network parameters for initialization. The network structure is as follows: Fig.11 shown.

[0152] The first half of the network is the feature extraction layer, which extracts features from the input image through multi-layer convolution and pooling operations. Since the depth image is similar to the color image, this part is initialized by using the pre-trained parameters of OpenPose; the second half of the network is divided into two sub-networks, which perform convolution and pooling operations to obtain the position information of the joints and the association information between the joints. At the same time, the input of each stage is obtained by fusing the results of the previous stage with the original image features to produce more accurate prediction results. The network training process is as follows:

[0153] Step 1: Depth image preprocessing. The depth image format is a 16-bit single-channel image. First, convert the depth image from unit_16 to unit_8 data format, and then use the applyColorMap function in the OpenCV library to convert the single-channel data into a 3-channel pseudo-color image.

[0154] Step 2: Build network structure and transfer learning. The model extracts features from image data through multi-layer convolutional neural network (CNN) and pooling layer, and initializes it using the parameters of the pre-trained feature extraction layer;

[0155] Step 3: Train the model. Use the above-constructed data set to train the model and obtain the joint point location information and the relationship between the joint points;

[0156] Step 4: Connect the bones. Connect the bones through the association relationship between the above joint points and output the final bone information.

[0157] Skeletal pose rotation based on adaptive viewpoint transformation:

[0158] By drawing on the correction algorithm in the image field, the 3D skeleton posture is filled with a pseudo image and the convolutional neural network (CNN) is used to learn the rotation parameters in the spatial domain; at the same time, the gated recurrent unit (GRU) is used to learn the parameters in the temporal domain for multi-frame skeleton data, and finally the outputs of the two models are fused to obtain the rotated skeleton posture.

[0159] The network structure of the CNN model is as follows Fig.12 As shown, the specific process is as follows:

[0160] Step 1: Data preprocessing. The skeleton posture obtained in posture detection contains 25 points, each of which consists of a 3D coordinate, which is its position and depth in the image. Considering the duration of the same behavior and the image acquisition frequency of the depth camera, the number of frames for each image splicing is set to 30 frames, that is, 30 frames of skeleton posture data of the same behavior are taken and stacked into a matrix of size 30*25*3. If there are less than 30 frames, they are filled with 0.

[0161] Step 2: Build the network. It consists of 2 convolutional layers, 1 pooling layer and 1 fully connected layer. The convolutional layer performs convolution operation on the input pseudo image data. Each convolutional layer contains a Batch Normalization (BN) layer for normalization. The activation function is the Relu() function. The last layer is a fully connected layer that outputs 3D rotation parameters. The rotation parameters are used to rotate the original input data to obtain the rotated skeleton posture. The rotation calculation formula is shown in formula (3-1).

[0162] p′ i =R z,γ R y,β R x,α p i (3-1)

[0163] Where p i is the coordinate p of the i-th bone joint point i =(x i ,y i ,z i ); p′ i is the transformed coordinate of the i-th bone joint point; R z,γ ,R y,β ,R x,α is the transformation matrix, and its calculation formula is shown in equations (3-2), (3-3), and (3-4).

[0164]

[0165]

[0166]

[0167] Where α, β, and γ are the angles of rotation around the x, y, and z axes, respectively.

[0168] Step 3: Train the network. The mean square error between the rotated skeleton posture data and the posture data at the frontal angle is calculated as the network loss. The Adam function is used as the optimization function. The number of training iterations is 50, and the model with the best result is saved in the validation set.

[0169] The network structure of the GRU model is as follows Fig.13 As shown, the specific process is as follows:

[0170] Step 1: Data preprocessing. Convert the rotated skeleton data of each frame into a 1*75 vector. Take 30 frames as the length of the time series, and fill the frames less than 30 with 0 to obtain a matrix of 30*75 as the input of the network.

[0171] Step 2: Build a network. It consists of 2 GRU layers and 1 fully connected layer. The hidden layer feature dimension of GRU is set to 100. The GRU layer obtains the output at each time point, and finally outputs the skeleton posture after occlusion restoration through the fully connected layer.

[0172] Step 3: Train the network. The training process is the same as that of the CNN model.

[0173] The skeleton posture obtained by posture detection obtains the rotation parameters through the CNN model and rotates the spatial dimension. Then, the GRU model is used to restore the rotated posture through occlusion through the context relationship to obtain the final skeleton posture with perspective conversion and occlusion restoration.

[0174] Walking posture characteristics:

[0175] After obtaining the skeleton posture with a more appropriate viewing angle, the posture feature design of the elderly during walking is as follows:

[0176] Definition 3.1: Trunk angle a trunk . It is defined as the angle between the torso and the horizontal plane, and the calculation formula is shown in formula (3-5):

[0177]

[0178] In the formula is the normal variable of the horizontal plane p neck is the 3D coordinate of the neck; p mid.hip is the 3D coordinate of the mid-hip.

[0179] Definition 3.2: Angle of flexion a bend . It is defined as the angle of body flexion, and the calculation formula is shown in formula (3-6):

[0180]

[0181] Where p nose is the 3D coordinate of the nose.

[0182] Definition 3.3: Hip angle a α.hip . Defined as the angle between the neck, left and right hips, and left and right knees, the calculation formula is shown in formula (3-7):

[0183]

[0184] Where p α.hip are the 3D coordinates of the left and right hips, α∈{left,right}; p α.knee are the 3D coordinates of the left and right knees.

[0185] Definition 3.4: Shoulder Angle a α.shoulder . It is defined as the angle between the neck, left and right shoulders, and left and right elbows. The calculation formula is shown in formula (3-8):

[0186]

[0187] Where p α.shoulder is the 3D coordinate of the left and right shoulders; p α.elbow are the 3D coordinates of the left and right elbows.

[0188] Definition 3.5: Knee angle a α.knee It is defined as the angles of the left and right hips, left and right knees, and left and right ankles. The calculation formula is shown in formula (3-9):

[0189]

[0190] Where p α.ankle are the 3D coordinates of the left and right ankles.

[0191] Definition 3.6: Shoulder Width d shoulder It is defined as the distance between the left and right shoulders, and is used to indicate the difference in individual body shape. The calculation formula is shown in formula (3-10):

[0192] d shoulder =|p left.shoulder -p right.shoulder | (3-10)

[0193] Where p left.shoulder is the 3D coordinate of the left shoulder; p right.elbow is the 3D coordinate of the right shoulder.

[0194] Experimental results and analysis:

[0195] The experimental results of the posture detection model based on transfer learning are as follows Fig.14 , Fig.15 As shown, Fig.14 The depth image of the elderly at home is collected. Fig.15 This is the result diagram after posture detection.

[0196] The skeleton posture adaptive perspective conversion model uses the above posture detection results to convert the perspective. First, it is converted into 3D coordinates. The result is as follows Fig.16 As shown in the figure, the left arm is lost due to occlusion; after the perspective conversion model, the result is as follows Fig.17 As shown, the occluded and lost joint points are predicted and restored.

[0197] This section analyzes the reasons for adopting transfer learning and explains in detail the process and implementation of transfer learning; then the skeletal posture is transformed in perspective to make the data more standardized; finally, the converted posture data is subjected to corresponding feature extraction and used in a complementary form with the previously obtained gait features in the subsequent multidimensional data fusion risk prediction model.

[0198] Problem description and analysis of swing arm balance detection

[0199] Inertial sensors are commonly used in fields such as fall detection and gait analysis due to their simple equipment and low cost. Wearable sensors, in particular, can directly capture information about various parts of the human body at every moment by being worn in selected locations. Their data mainly consists of acceleration and gyroscopes and will not be affected by occlusion.

[0200] Inertial sensors usually have a high acquisition frequency and are highly sensitive to sudden behaviors, so they can capture anomalies that occur in normal regular behaviors. However, because daily life is different from the laboratory environment, the trivial behaviors of the elderly contain a lot of interference noise, which will lead to more redundant data and erroneous data, requiring the combination of more equipment for joint analysis.

[0201] In response to the above problem description, this section will use smart watches that are less invasive to the elderly's lives to collect data, and they will not be affected by occlusion, which to a certain extent makes up for the shortcomings of lidar and depth lenses. Therefore, this chapter will locate the walking interval based on the sensor data and detect the balance of arm swing during walking, and extract the features that describe the balance of the elderly's arm swing.

[0202] Problem Analysis

[0203] (1) Hardware equipment and data collection method

[0204] The smartwatch used in this article is Huawei Watch2. Fig.18As shown, the acceleration and gyroscope data can be collected. When worn on the elderly's arm, the three-dimensional acceleration axis and gyroscope frequency are set to 50Hz. In order to ensure the battery life of the watch, the data upload of the watch is limited. The data is uploaded when it is in charging state and connected to the network. If the upload cannot be performed, the data can be saved locally for 14 days.

[0205] (2) Swing arm balance test

[0206] Most falls of the elderly occur during walking, and walking requires coordination of various parts of the body. When walking, the arms usually swing with the movement of the feet. Gait analysis focuses on the footsteps information directly related to walking, and posture detection focuses on the overall balance information of the body, with less description of the upper limbs. The wristwatch worn on the arm can fill this gap. In normal walking activities, sensor data generally maintains a certain regularity, but encountering other behaviors will break this regularity, which may include some behaviors that cause falls. These unbalanced behaviors need to be captured. Therefore, this paper proposes the concept of autocorrelation coefficient to describe the swing balance of the sensor data.

[0207] Based on the above problem description and analysis, the arm swing balance detection algorithm based on walking autocorrelation analysis will be studied and analyzed below.

[0208] The information described by the raw data of the acceleration and gyroscope cannot be directly interpreted. By analyzing the raw data, it is found that the data performance of normal regular behaviors has certain regularity. When the behavior different from the current one appears in the process, the data will fluctuate. These abnormal behaviors may be the factors that cause falls during walking. Therefore, it is necessary to analyze the raw data to extract the balance and correlation of the arms when walking, so as to describe whether the current behavior is abnormal.

[0209] like Fig.19 The figure shows the data of steady walking over a period of time measured in an experimental environment. Taking the acceleration Y-axis data as an example, it can be concluded from the figure that the peaks or troughs of steady walking have certain similarities and balance. The emergence of abnormal behavior will break this balance, such as Fig. 20 As shown in the figure, there are some irregular peaks or troughs. Therefore, this paper proposes the concept of an autocorrelation coefficient, which can describe the correlation between the behavior corresponding to the current peak and the data in the previous period, thereby capturing the characteristics of the swing imbalance that occurs during this period.

[0210] Arm swing balance detection model based on walking autocorrelation analysis

[0211] The autocorrelation coefficient calculates the similarity between the current peak and the data in the previous time period. This data is used to represent the balance of the elderly's arm swing during walking. If the similarity is low, it means that the arm movement is abnormal at this time, otherwise it is normal behavior. The specific flow chart of its calculation is as follows Fig.21 As shown:

[0212] Step 1: Read the data. Both the acceleration and gyroscope data are three-axis data. Calculate their SMV (Signal Magnitude Vector). The calculation formula of SMV is shown in formula (4-1):

[0213]

[0214] Where a x ,a y ,a z The data of the acceleration or gyroscope x, y, and z axes.

[0215] Step 2: Find the location of the peak using the Peakutils peak detection program;

[0216] Step 3: Search the [tmin, tmax] interval forward from the current peak index time and calculate the autocorrelation coefficient R(i) of the current peak i. The calculation formula is shown in formulas (4-2) and (4-3):

[0217]

[0218]

[0219] Where R(i,τ) represents the autocorrelation coefficient of the current peak index i within τ time; tmin, tmax are the ranges for calculating the autocorrelation coefficient interval; a(ik) is the value of the SMV at time ik; μ(τ,i) is the mean SMV from time τ to time i; δ(τ,i) is the standard deviation of the SMV from time τ to time i.

[0220] Experimental results and analysis

[0221] The acceleration data and gyroscope data collected by the smart watch are analyzed through autocorrelation to obtain the autocorrelation coefficient within the walking time interval. Taking the acceleration data as an example, the results of normal and stable walking are as follows: Fig. 22 As shown in the figure, the value at the top of each peak in the figure is the autocorrelation coefficient corresponding to the peak. It can be concluded that the autocorrelation coefficient is higher during stable walking, indicating that the similarity between the previous and subsequent behaviors is higher.

[0222] The autocorrelation coefficient of acceleration with abnormal behavior is as follows: Fig.23 As shown, Fig. 22Compared with the autocorrelation coefficient, it is significantly lower, indicating that its behavior has large fluctuations and uncertainties.

[0223] This section starts with the analysis of the advantages and disadvantages of wearable sensors, elaborating on the disadvantages of highly invasive devices and the shortcomings in gait analysis and posture analysis, and introducing less invasive, unobstructed smart watches to collect acceleration and gyroscope data. Based on the analysis of the original data, the autocorrelation coefficient is proposed and calculated to characterize the balance of the elderly's arm swing during walking, reflecting whether the elderly's arm movements are abnormal. Since this feature will not be affected by environmental occlusion, it will be complementary and integrated with gait features and posture features for subsequent analysis.

[0224] Experimental results and analysis of the risk assessment method

[0225] Experimental evaluation indicators:

[0226] This application abstracts fall risk assessment into a classification problem and uses common classification evaluation indicators to evaluate the experimental results, including accuracy, precision, recall, and F1-score. The calculation formulas are shown in (5-5), (5-6), (5-7), and (5-8).

[0227]

[0228]

[0229]

[0230]

[0231] Where TP is the number of correctly predicted positive samples; FP is the number of incorrectly predicted positive samples; TN is the number of correctly predicted negative samples; and FN is the number of incorrectly predicted negative samples.

[0232] Experimental results:

[0233] The training process of MultipleGRU combined with Attention and DNN (MGAD) model is as follows Fig.24 As shown in the figure, the blue solid line is the network loss during training, and the orange dotted line is the accuracy in the validation set during training.

[0234] In addition, this application also built a MultipleGRU-DNN (MGD) model without the Attention mechanism, downsampled the data and directly fused it with a single GRU-Attention-DNN (GAD) model and a Hidden Markov Model (HMM), and compared them in the test set. The comparison results are as follows Figure 25a-25d shown.

[0235] From the figure, we can conclude that the MGAD model constructed in this paper has higher accuracy, recall rate, and F value than other models. The GAD model that directly downsamples and fuses the data also performs well. In addition, the HMM and MGD models without Attention network perform poorly. By analyzing the experimental results, we can conclude that in the processing of long-term sequences, the Attention mechanism can capture more important information, while the non-Attention mechanism or traditional machine learning algorithms cannot process ultra-long time series data well.

[0236] Experimental results show that the MGAD model constructed in this paper can better assess the risk of falls based on multi-dimensional time characteristics of different frequencies.

[0237] This application analyzes the advantages and disadvantages of gait features, posture features, and arm swing balance features, and proposes to use redundant complementary methods to fuse features. It then analyzes common data fusion algorithms, and based on the characteristics of the above features, proposes to use the GRU model and attention mechanism to fuse these features, solving the problem that data of different dimensions and frequencies are difficult to fuse. The fused features are then used through the DNN model to implement the assessment of the risk of falls for the elderly. Finally, a comparative analysis is conducted through experiments with models such as the Hidden Markov Model, and it is concluded that the use of the GRU model and the introduction of the attention mechanism is more effective in the assessment of fall risk in multi-dimensional data fusion.

Claims

1. A method for assessing the risk of falls in the elderly based on multidimensional data fusion, characterized in that The steps include: Construct the data set required for DNN model fall risk assessment for the elderly; The gait features, posture features and autocorrelation coefficients are respectively input into different gate control unit GRU models and multi-dimensional data feature fusion is realized through attention calculation; the autocorrelation coefficient is used to describe the arm swing balance represented by the sensor data, calculate the similarity between the current peak and the data in the previous time period, and use the data to represent the balance of the arm swing during the elderly's walking process; The DNN model is used to classify the features extracted and concatenated by the GRU model, and the probability of the elderly person’s fall risk is output.

2. The method for assessing the risk of falling of the elderly based on multidimensional data fusion as claimed in claim 1, characterized in that The method for constructing the data set required for DNN model fall risk assessment for the elderly is as follows: The original data is firstly extracted with gait analysis model, and then the data is segmented by time interval to obtain the time periods of the elderly's walking; then the posture analysis and arm swing balance detection are used to segment the corresponding data according to the time period and process the extracted features; After obtaining three sets of data, namely, gait characteristics, posture characteristics and autocorrelation coefficient of each walking time period, they are labeled.

3. The method for assessing the risk of falling of the elderly based on multidimensional data fusion as claimed in claim 1, characterized in that The method for multidimensional data feature fusion is as follows: Data preprocessing: read in three sets of data: gait features, posture features, and autocorrelation coefficients, and normalize them respectively. The calculation formula is shown in formula (1-1). Then, the data set is divided into training set, validation set, and test set. （1-1） In the formula Variables The maximum and minimum values ​​of the dimension; Construct GRU model and attention mechanism: Input the three sets of data into three different GRU networks respectively. Each GRU model includes two layers of bidirectional BiGRU, with input layer sizes of 6, 24, and 2 respectively, and output layer size of 50; Attention calculation: The output of each time point of the three GRU networks is calculated. The calculation formulas are shown in equations (1-2), (1-3), and (1-4): （1-2） （1-3） （1-4） Where: is the output of the GRU network at each time point; is the parameter of the attention layer; is the calculated probability distribution of attention; is the output result of the attention layer; The last layer of the three sets of data outputs is concatenated and used as the input vector of the subsequent DNN network.

4. The method for assessing the risk of falling of the elderly based on multidimensional data fusion as claimed in claim 1, characterized in that The method of using the DNN model to classify the features extracted and concatenated by the GRU model and outputting the probability of the risk of falling of the elderly is as follows: Construct a DNN model: including 3 hidden layers and 1 output layer; the hidden layer includes a fully connected layer, whose inputs are 150, 128, and 64 respectively, followed by a BN layer to batch normalize the data, and then the activation function uses Function, the output layer is a fully connected layer with an input size of 32 and an output of 2; Training network: training is performed using the constructed training set. The loss function uses the cross entropy loss function and the optimization function uses function, iterates training for 50 times, and saves the model with the best result in the validation set; Use the trained model for risk assessment: Input the three sets of data into the trained model and Function transformation outputs the probability of falling risk corresponding to the current input to achieve the assessment of falling risk.

5. The method for assessing the risk of falling of the elderly based on multidimensional data fusion as claimed in claim 1, characterized in that The method for gait characterization comprises the following steps: Environment Mapping: Step 1: Initialize the environment map, read in the nighttime unmanned data, and build the initial environment map; Step 2: Read the follow-up Frame point cloud data, calculate the mean of the distance difference of the corresponding angles between the two frames by frame difference method, and judge whether there is a moving object in the current environment. If there is no moving object, execute Step 3, if there is a moving object, repeat Step 2; Step 3: Determine whether the pedestrian tracking track in the current Kalman filter algorithm exists. If the track exists, it means that there is an elderly person who has been stationary for a long time in the current environment. Repeat Step 2. If the track does not exist, the current environment is all environmental points, so as to calculate Update the environment map with the mean of frame data; Point cloud feature extraction based on clustering: By comparing the newly scanned data with the environment map, the point cloud data of the moving object is obtained; the DBSCAN clustering algorithm is used to cluster these point cloud data and extract features that can describe the corresponding objects. The distance calculation formula is shown in formula (2-1): .......................（2-1） In the formula The first The position of the new scanning point, ; For the The first Scan points, ; Specifically include the following steps: Step 1: Read in the moving point set , traverse The unlabeled points in the cluster are labeled and added to the new cluster set. In the equation (2-1), calculate the other unmarked points and The distance is less than If the point count exceeds Then add these points to the set , if less than No treatment; Step 2: Traverse the collection The points in Other points in the neighborhood are greater than Add it Repeat this step until the set is empty; Step 3: Repeat the above Step 1 and Step 2 operations for the unmarked points until each point does not change; After clustering the point set, it is necessary to identify the target. Combined with the shape of the footstep point cloud, the following point cloud features are designed: Definition 2.1: Size of a Point Set , the number of points in the point set ; Definition 2.2: Maximum length , the maximum length of a point cluster is approximated as the foot length, and the calculation formula is shown in formula (2-2): .......................（2-2） In the formula ——is a set of points The two points with the largest distance in the moving direction; Definition 2.3: Foot arc , calculation point set The arc of each point on the edge is taken as the average to approximate the arc of the foot. The calculation formula is shown in formula (2-3): .......................（2-3） In the formula A set of points Two points adjacent to each other on the edge; A set of points The centroid of A set of points The number of edge points; Definition 2.4: Foot arc length , the sum of the Euclidean distances between two adjacent points is calculated as the arc length, and the calculation formula is shown in formula (2-4): .......................（2-4） In the formula A set of points Two points adjacent to each other on the edge; A set of points The number of edge points; Definition 2.5: Foot landing area , estimate the area of ​​the point set, the calculation formula is shown in formula (2-5): .......................（2-5） In the formula A set of points The coordinates of the two points with the maximum distance in value; A set of points Coordinates of the midpoint value; Footstep recognition based on random forest: Footstep recognition is to distinguish footsteps from moving objects. The random forest model is used to recognize footsteps. The above-extracted point set features are used as input. The random forest is composed of multiple decision trees. The Gini index is used as the criterion for feature selection, which indicates the probability that the randomly selected sample is misclassified. The smaller the Gini index, the more accurate the classification. Classification is performed based on this standard. Finally, multiple decision trees are voted to determine the optimal classification. The calculation formula of the Gini index is shown in formula (2-6): .......................（2-6） In the formula is the number of categories; The sample point belongs to Probability of class; Finally, after classifying the point set through random forest, the point cloud set of the footsteps is obtained to complete the footstep recognition; Footstep tracking based on Kalman filter: By using the Kalman filter to track footsteps and recover footsteps lost due to occlusion during the tracking process, the footsteps of the elderly can be tracked and gait features extracted; The state prediction equation of the Kalman filter algorithm is shown in formulas (2-7) and (2-8): （2-7） （2-8 ） In the formula For the The center-of-mass state vector of the frame, is the position component, is the velocity component; For the System measurements of frames; is the state transfer matrix; To control the input matrix, the motion measurements are mapped onto the state vector; For the The system control vector of the frame, including acceleration information; is the system noise, and its covariance is ; is the transformation matrix, which maps the state vector into the space of the measurement vector; is the observation noise, and its covariance is ; Walking between adjacent frames indoors can be approximated as uniform linear motion, so we can obtain the relationships shown in formulas (2-9), (2-10), (2-11), and (2-12): .......................（2-9） .......................（2-10） .......................（2-11） .......................（2-12） In the formula is the time interval; Indicates that the current time; Convert it into matrix representation as shown in formulas (2-13) and (2-14): （2-13） （2-14） The state transfer matrix can be obtained by using formula (2-13) and formula (2-7) ,at the same time is a zero matrix; through formula (2-14) and formula (2-8), we can get ; Since both measurement and prediction have errors, it is necessary to calculate the error in the current prediction process. , and its calculation formula is shown in formula (2-15): .......................（2-15） In the formula For predict The covariance of for covariance of moments; Combined with formula (2-7), the predicted state and observed state of the system at the current moment are obtained Calculate the optimal estimate at this time, the calculation formula is shown in formula (2-16): .......................（2-16） In the formula for The Kalman gain at time , is calculated as shown in (21): .......................（2-17） get After obtaining the optimal estimate of the time, we need to calculate the covariance at the current time. Update, the calculation formula is shown in formula (2-18): .......................（2-18） In the formula is the identity matrix; The specific process of the Kalman filter algorithm is as follows: Step 1: Calculate the current time The predicted value of ; Step 2: Determine the current time If the observed value exists, the Kalman filter is updated and the calculated optimal estimate is added to the tracking step. In the set, repeat Step 1; if it does not exist, proceed to Step 3; Step 3: Observation value As the best estimate, the next Whether there is an observation value at the moment, if there is none, the current Kalman filter tracking is stopped, indicating the end of walking; if there is, the Kalman filter is updated using the observation value and the predicted value, and the previously retained predicted steps are added to In the collection, repeat Step 1; At this point, the walking trajectory of the elderly is combined with the commonly used indicators in gait analysis, including the step length of the left and right feet, the instantaneous speed of the left and right feet, and the landing area during the elderly's walking.

6. The method for assessing the risk of falling of the elderly based on multidimensional data fusion as claimed in claim 1, characterized in that The method for extracting posture features is as follows: Posture detection model based on transfer learning: Pose detection is used to extract skeleton information from depth images. The OpenPose model is used to transfer learning to train a pose detection model for depth images. First, a dataset of depth images is constructed, and then the model is trained using the dataset. (1) Building a dataset At the beginning of the experiment, the depth image and the aligned RGB image are collected by the depth camera at the same time. The skeleton posture of the RGB image is extracted by the pre-trained OpenPose model. The extracted skeleton posture and the corresponding depth image are combined into a posture dataset for training the convolutional neural network CNN suitable for depth images. (2) Transfer Learning The OpenPose model is fine-tuned for parameter-based transfer learning, and is initialized using pre-trained network parameters. The first half of the network is the feature extraction layer, which extracts features from the input image through multi-layer convolution and pooling operations. Since the depth image is similar to the color image, this part is initialized using the pre-trained parameters of OpenPose; the second half of the network is divided into two sub-networks, which perform convolution and pooling operations to obtain the position information of the joints and the association information between the joints. At the same time, the input of each stage is obtained by fusing the results of the previous stage with the original image features to produce more accurate prediction results; the network training process is as follows: Step 1: Depth image preprocessing: The depth image format is a 16-bit single-channel image. First, the depth image is converted from Convert to Data format, then use it for single channel data In the library The function converts to a 3-channel pseudo-color image; Step 2: Build network structure and transfer learning: The model extracts features from image data through multi-layer convolutional neural networks and pooling layers, and initializes them using the parameters of the pre-trained feature extraction layer; Step 3: Training model: Use the above-constructed data set to train the model to obtain the position information of the joint points and the relationship between the joint points; Step 4: Connect bones: Connect bones through the association relationship between the above joint points and output the final bone information; Skeletal pose rotation based on adaptive viewpoint transformation: Through the correction algorithm in the image field, the 3D skeleton posture is filled into a pseudo image and the convolutional neural network CNN is used to learn the rotation parameters in the spatial domain; at the same time, the gated recurrent unit GRU is used to learn the parameters in the temporal domain for the multi-frame skeleton data, and finally the outputs of the two models are fused to obtain the rotated skeleton posture; The specific process of the CNN model network is as follows: Step 1: Data preprocessing: The skeleton posture obtained in posture detection contains 25 points, each of which consists of a 3D coordinate, which is its position and depth in the image. Considering the duration of the same behavior and the image acquisition frequency of the depth camera, the number of frames for each image splicing is set to 30 frames, that is, 30 frames of skeleton posture data of the same behavior are taken and stacked into a size of If the matrix is ​​less than 30 frames, it will be filled with 0; Step 2: Build a network: It consists of 2 convolutional layers, 1 pooling layer and 1 fully connected layer. The convolutional layer performs convolution operations on the input pseudo image data. Each convolutional layer contains a Batch Normalization (BN) layer for normalization. The activation function is Function, the last layer is the fully connected layer that outputs the 3D rotation parameters, which are used to rotate the original input data to obtain the rotated bone posture. The rotation calculation formula is shown in formula (3-1); .......................（3-1） In the formula For the The coordinates of the bone joints ; For the The transformed coordinates of the bone joint points; is the transformation matrix, and its calculation formula is shown in equations (3-2), (3-3), and (3-4); .......................（3-2） .......................（3-3） .......................（3-4） In the formula Respectively around Angle of axis rotation; Step 3: Train the network, calculate the mean square error between the rotated skeleton posture data and the posture data at the frontal angle as the network loss, and use the optimization function function, the number of training iterations is 50, and the model with the best result in the validation set is saved; The network processing flow of the GRU model is as follows: Step 1: Data preprocessing: Convert the rotated skeleton data of each frame into ; take 30 frames as the length of the time series, and fill it with 0 if it is less than 30 frames, so the size is The matrix is ​​used as the input of the network; Step 2: Build a network: It consists of 2 layers of GRU and 1 layer of fully connected layer. The hidden layer feature dimension of GRU is set to 100. The GRU layer obtains the output at each time point, and finally outputs the skeleton posture after occlusion restoration through the fully connected layer; Step 3: Train the network: The training process is consistent with the training of the CNN model; The skeleton posture obtained by posture detection is rotated by the CNN model to obtain the rotation parameters and rotate the spatial dimension. Then, the GRU model is used to restore the rotated posture through occlusion through the context relationship to obtain the final skeleton posture with perspective conversion and occlusion restoration. Walking posture feature: After obtaining the skeleton posture with a more appropriate viewing angle, the posture feature design of the elderly during walking is as follows: Definition 3.1: Trunk Angle ; defined as the angle between the torso and the horizontal plane, the calculation formula is shown in formula (3-5): .......................（3-5） In the formula is the normal variable of the horizontal plane ; is the 3D coordinate of the neck; is the 3D coordinate of the mid-buttocks; Definition 3.2: Flexion Angle ; defined as the angle of forward bending of the body, the calculation formula is shown in formula (3-6): .......................（3-6） In the formula is the 3D coordinate of the nose; Definition 3.3: Hip Angle ; It is defined as the angle between the neck, left and right hips, and left and right knees. The calculation formula is shown in formula (3-7): .......................（3-7） In the formula are the 3D coordinates of the left and right hips, ; are the 3D coordinates of the left and right knees; Definition 3.4: Shoulder Angle ; It is defined as the angle between the neck, left and right shoulders, and left and right elbows. The calculation formula is shown in formula (3-8): .......................（3-8） In the formula are the 3D coordinates of the left and right shoulders; are the 3D coordinates of the left and right elbows; Definition 3.5: Knee Angle ; defined as the angles of the left and right hips, left and right knees, and left and right ankles, the calculation formula is shown in formula (3-9): .......................（3-9） In the formula are the 3D coordinates of the left and right ankles; Definition 3.6: Shoulder Width ; defined as the distance between the left and right shoulders, used to represent the difference in individual body shape. The calculation formula is shown in formula (3-10): .......................（3-10） In the formula is the 3D coordinate of the left shoulder; is the 3D coordinate of the right shoulder.

7. The method for assessing the risk of falling of the elderly based on multidimensional data fusion as claimed in claim 1, characterized in that The specific implementation method of the autocorrelation coefficient comprises the following steps: Step 1: Read the data. Both the acceleration and gyroscope data are three-axis data. Calculate their (Signal MagnitudeVector), The calculation formula is shown in formula (4-1): .......................（4-1） In the formula is the accelerometer or gyroscope Three-axis data; Step 2: Find the location of the peak using the Peakutils peak detection program; Step 3: Search forward from the current peak index time Interval, calculate the current peak The autocorrelation coefficient , the calculation formula is shown in formula (4-2) and (4-3): （4-2） .......................（4-3） In the formula Indicates the current peak index exist Autocorrelation coefficient in time; The range of the interval for calculating the autocorrelation coefficient; for Moment The value of For Time has come Moment Mean; For Time has come Moment Standard deviation.

Images