Precise prediction and rapid emergency system for falling of old people

Abstract

The invention belongs to the field of computer vision, and discloses an accurate prediction and rapid emergency system for tumble of old people, and the system is characterized in that a frame selection module carries out feature extraction on each group of video frames; obtaining representative frames of each group of video frames; the skeleton graph module adopts an OpenPose model to carry out attitude estimation on the representative frame to generate a skeleton graph; the SPD manifold module is used for processing the skeleton image into a one-dimensional feature sequence vt; and the position sensing Transform model is used for sensing time sequence features in the one-dimensional feature sequence, capturing a long-distance dependency relationship in the time sequence features by using a self-attention mechanism, and performing probability prediction of the falling action through a full connection layer and a Softmax layer. According to the system disclosed by the invention, the SPD manifold and the Position-Aware Transform model are combined, so that the time sequence dependence in human body actions is effectively captured, and the accuracy of tumble prediction is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically is a video monitoring system for predicting and alarming falls of the elderly by combining a symmetric positive definite manifold (SPD manifold for short) and a position-aware Transformer network. Background of the Invention

[0003] Falling is one of the most common accidents among the elderly, which often leads to serious health problems and even endangers life. Therefore, how to timely and accurately predict the elderly's falling movements so as to take quick rescue measures has become the key to improving the health protection level of the elderly.

[0004] Current fall prediction methods have many limitations in capturing the geometric structure and temporal dependencies of data, resulting in low prediction accuracy, especially in the modeling of complex action sequences. Skeletal data has natural geometric characteristics, but traditional methods are easily affected by deformation when processing these data and cannot effectively capture the complex structure in the data. Symmetric positive definite manifold (SPD manifold) is a mathematical tool for describing the geometric structure of symmetric positive definite matrices, and is often used in geometric analysis of high-dimensional data. With its geometric advantages in processing symmetric positive definite matrices, SPD manifold can accurately describe the geometric relationship in high-dimensional data, and has a significant effect on improving the accuracy and stability of fall prediction. At the same time, Transformer can analyze the temporal characteristics of the elderly's skeletal data through the self-attention mechanism, and effectively capture long-term dependencies. However, the traditional Transformer itself does not have temporal information, so the present invention introduces position encoding to assign position information to each input feature vector. Through this position encoding, Position-Aware Transformer can more accurately capture the changing trend of the elderly's movements during the fall process, and then better identify the temporal changes in skeletal data, providing higher accuracy for fall prediction.

[0005] At present, the prediction of elderly falls mainly faces the following engineering problems:

[0006] (1) Insufficient prediction accuracy: Traditional fall prediction methods (such as CNN-based models) have difficulty capturing the geometric characteristics of the human body when dealing with complex movements. These methods usually rely on raw sensor data or only use simple feature extraction, and cannot effectively analyze the spatial and motion relationships between human joints. In particular, the prediction effect is poor in complex scenes, resulting in low accuracy in fall action prediction.

[0007] (2) Limited ability to model temporal dependencies: Traditional RNN or LSTM models are prone to gradient vanishing or exploding problems when processing long time series, resulting in limited ability of the model to capture long-term dependencies. Fall prediction usually requires identifying key features in long time series, and existing models do not perform well in this regard.

[0008] (3) Long response time: The existing fall prediction and alarm system has a slow response speed, making it difficult to respond quickly when a fall occurs, delaying the timely protection of the safety of the elderly.

[0009] (4) Insufficient modeling capabilities for complex motion sequences: Traditional methods have difficulty capturing the dynamic changes in complex human motion sequences, especially when the motion changes involve multiple joints. Summary of the invention

[0010] The technical problem to be solved by the present invention is the problems of insufficient prediction accuracy, limited temporal dependency modeling capability, long response time, and insufficient modeling capability for complex action sequences in fall action prediction. The present invention proposes a fall prediction method, comprising the following steps:

[0011] Step 1, group the video in chronological order, with J frames as one group, and a total of Q groups of video frames;

[0012] The feature vector of each frame in each group of video frames is extracted, and the feature vector of the jth frame in the qth group is represented by γ qj , j ranges from 1 to J, and q ranges from 1 to Q;

[0013] Step 2, selecting one frame from each group of video frames as a representative frame of the group of video frames;

[0014] Step 3, generating a skeleton graph from the representative frame;

[0015] Step 4: Process the skeleton graph into a one-dimensional feature sequence and input it into the position-aware Transformer model to predict falls.

[0016] Furthermore, in step 2, one frame is selected from each group of video frames as a representative frame of the group of video frames, specifically: an average feature vector of each group of video frames is calculated, and then the cosine similarity between the average feature vector of each group of video frames and each feature vector of a subsequent group of video frames is calculated, and the feature vector with the highest similarity to the average feature vector is selected, and the video frame corresponding to the feature vector is used as the representative frame of the subsequent group of video frames.

[0017] The average eigenvector of the qth group of eigenvectors is denoted as γ avgq ;

[0018]

[0019] Calculate the average eigenvector γ of the qth group avgq The cosine similarity between each feature vector in the q+1th group is calculated as:

[0020]

[0021] γ (q+1)j represents the jth eigenvector in the q+1 group;

[0022] The feature vector with the highest similarity to the average feature vector of the qth group is selected from the q+1th group, and the video frame corresponding to the feature vector is used as the representative frame of the q+1th group of video frames; the representative frame of each group of video frames is selected using the same method.

[0023] Furthermore, in step 3, the representative frame is used to generate a skeleton graph, and the specific steps are as follows:

[0024] Step 3.1: Input the representative frame into the pose estimation model OpenPose, and the pose estimation model OpenPose outputs a heat map of each joint Where k represents the joint point number, and t represents the time number of the video frame;

[0025] Heatmap The elements in are denoted as Represents pixel points (s i ,s j ) belongs to the confidence of joint point k;

[0026] In the heat map Find the coordinates of the pixel with the highest confidence in the formula as follows:

[0027]

[0028] represents the pixel coordinates of the kth joint at time t, and the value range of k is 1 to K;

[0029] The skeleton graph consists of the three-dimensional coordinates of K joint points:

[0030]

[0031] The skeleton diagram of the entire time series is represented as:

[0032] {P t |t=1,2,…,t max}

[0033] Furthermore, in step 4, the skeleton graph is processed into a one-dimensional feature sequence. The specific steps are as follows:

[0034] Step 4.1, for each time step t, calculate the skeleton graph P t The covariance matrix C of the joints t , the formula is as follows:

[0035]

[0036] is the mean vector of all relevant nodes at time step t, that is:

[0037]

[0038] Step 4.2: transform the covariance matrix C t Transform to linear space through logarithmic mapping;

[0039] Step 4.2.1, for the covariance matrix C t Perform eigenvalue decomposition:

[0040]

[0041] Among them, U t is the eigenvector matrix, Λ t is a diagonal matrix containing the eigenvalues ​​of the covariance matrix;

[0042] Step 4.2.2, for the eigenvalue matrix Λ t Take the logarithm of each eigenvalue in and get the logarithmic eigenvalue matrix log(Λ t ):

[0043] log(Λ t )=diag(log(λ t1 ),log(λ t2 ),…,log(λ tn ))

[0044] Among them, λ tn The nth eigenvalue of the covariance matrix at time step t reflects the geometric changes of the human joint points in that specific direction.

[0045] Then, the covariance matrix after logarithmic mapping is reconstructed using the eigenvector matrix;

[0046]

[0047] Step 4.2.3, transform the covariance matrix log(C t ) is expanded into a one-dimensional feature vector v t , the vector v of all time steps t Arrange them in order to form a time series feature sequence: {v1,v2,…,v T}.

[0048] Furthermore, in step 4, the one-dimensional feature sequence v t Input into the position-aware Transformer model to predict falls, which includes the following steps:

[0049] Step 4.3, assign position information to the time series feature sequence of each time step t, and obtain the time series feature sequence H with position information input ;

[0050] H input =v t +PE

[0051] Step 4.4: transform the time series feature sequence H input , linear transformation generates query matrix Q, key matrix K and value matrix V; the self-attention mechanism is used to capture the dependencies between time steps, expressed as:

[0052]

[0053] in, is the scaling factor;

[0054] Step 4.5, use a multi-head attention mechanism to capture different feature relationships;

[0055] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0056] head1=Attention(Q i ,K i ,V i ) is the output of the i-th attention head, and h represents the number of attention heads;

[0057] is the weight matrix for linear transformation, d model Represents the dimension of the input feature vector, that is, the length of the feature vector at each time step, which represents the feature dimension of each time step in this patent; d k Represents the feature segmentation and attention score calculation.

[0058] Step 4.6: The features obtained by the multi-head attention mechanism are input into the feed-forward neural network FFN. FFN consists of two linear transformation layers and an activation function, expressed as

[0059] Y=FFN(x)=ReLU(xW1+b1)W2+b2

[0060] x refers to the output feature vector after the multi-head attention mechanism;

[0061] is the linear transformation matrix;

[0062] is the bias vector;

[0063] d ff is the hidden layer dimension of the feedforward neural network.

[0064] Step 4.7: The output Y of the feedforward neural network FFN is further input into the fully connected layer, and the features are mapped to the action category space through linear transformation to obtain the scores of different action categories, which are marked as Y. t ; Then, the Softmax layer converts the category scores into probability distributions, which are used to predict the probability of actions at the current or future time steps, including standing, losing balance, and falling.

[0065] The present invention also proposes an elderly fall prediction and automatic alarm system based on SPD manifold and Position-Aware Transformer, which mainly includes a frame selection module, a skeleton graph module, an SPD manifold module, a Position-Aware Transformer module and an early warning module. Q groups of video frames are input into the frame selection module;

[0066] The frame selection module uses ResNet-18 to extract features from each group of video frames; calculates the average feature vector of each group of video frames; performs cosine similarity calculation on the average feature vector of each group of video frames and each feature vector of a subsequent group of video frames, selects the feature vector with the highest similarity to the average feature vector, and the video frame corresponding to the feature vector is the representative frame of the group of video frames; the skeleton graph module uses the OpenPose model to estimate the posture of the representative frame and generate a skeleton graph; the SPD manifold module is used to process the skeleton graph into a one-dimensional feature sequence v t ; The position-aware Transformer model is used to perceive the one-dimensional feature sequence v t The self-attention mechanism is used to capture the long-distance dependencies in the temporal features and the probability of falling action is predicted through the fully connected layer and the Softmax layer.

[0067] The frame selection module first processes the video of the elderly falling down frame by frame, extracts the features of each video frame using ResNet-18, and then selects the key frames related to the action through cosine similarity. In practical applications, due to the instability of monitoring equipment or human operation, there may be jitter and noise in the captured image frames, and these invalid frames will interfere with the accuracy of fall prediction. The frame selection module effectively filters out these invalid frames caused by jitter through feature screening, ensuring that only key frames related to the action are retained. In addition, in high-frame rate videos, if feature extraction and analysis are performed on each frame, the amount of data will be very large, resulting in longer calculation time and thus reducing the system response speed. By screening key frames, the frame selection module reduces the redundancy of data processing, greatly optimizes the utilization of computing resources, and significantly shortens the response time of the system. Therefore, the frame selection module effectively solves the two problems of insufficient prediction accuracy and long response time in traditional prediction methods.

[0068] Skeleton graph module: Use the OpenPose model to estimate the posture of the selected elderly fall images, extract the joint information to generate a skeleton graph, capture the posture changes, and provide basic data for fall prediction. When the elderly fall, the posture changes of the body are complex, involving the movement of multiple joints. Traditional feature extraction methods are difficult to accurately describe these complex motion trajectories. The skeleton graph module extracts the key joints of the human body (such as shoulders, elbows, knees, etc.) through the OpenPose model and generates a skeleton graph to effectively capture the dynamic changes of these joints, provide accurate posture information for the fall prediction model, and enhance the model's modeling ability for complex action sequences. Secondly, in actual applications, there may be a lot of background noise (such as noise from home appliances) in the surveillance video to interfere with the recognition of fall actions. The skeleton graph module focuses on the posture changes of the human joints, only pays attention to the parts related to the action, filters out background interference information, and thus improves the system's accurate capture of fall actions. The skeleton graph module mainly solves the two problems of insufficient prediction accuracy and insufficient modeling ability for complex action sequences.

[0069] SPD manifold module: The geometric information of human joints is converted into a one-dimensional feature sequence suitable for deep learning model processing through the calculation of covariance matrix, logarithmic mapping and vectorization steps. Traditional fall prediction methods have limitations in capturing the geometric characteristics of complex human movements, especially multi-joint movements during falls. The SPD manifold describes the geometric relationship of human joints by constructing the covariance matrix of human skeletal joints, providing high-precision geometric information input for subsequent models, and significantly improving the recognition accuracy of fall movements. Secondly, the SPD manifold module introduces logarithmic mapping and vectorization steps. By performing eigenvalue decomposition, logarithmic mapping, and vectorization on the covariance matrix, the original nonlinear geometric data is successfully converted into a one-dimensional feature sequence in linear space, providing rich geometric features for time series modeling. The SPD manifold module mainly solves the two problems of insufficient prediction accuracy and insufficient modeling capabilities for complex action sequences.

[0070] The Position-Aware Transformer module accurately captures the long-distance dependencies in the skeleton data through the self-attention mechanism and time series modeling, realizes the prediction of the elderly's falling actions, and outputs the probability of falling through the Softmax layer. Falling of the elderly is usually a process with a long time span and involves multiple continuous actions. When processing long time series, traditional RNN or LSTM models are prone to gradient vanishing or explosion problems, and it is difficult to effectively capture long-term time series dependencies. In contrast, the Position-Aware Transformer introduces position encoding to give position information to each input feature, thereby more accurately capturing the entire process from losing balance to falling. Through the self-attention mechanism, the Position-Aware Transformer can capture the long-term dependencies in human body movements and significantly improve the accuracy of prediction. In addition, the falling process involves dynamic changes of multiple joints, and traditional methods are difficult to accurately capture these complex interactions. Through the multi-head self-attention mechanism, this module can simultaneously focus on the features of different time points in the action sequence when processing time series data, ensuring that the model recognizes the changes of each joint point during the fall process, thereby accurately predicting the occurrence of falls. Position-Aware Transformer mainly solves two problems of traditional models: limited ability to model temporal dependencies and insufficient modeling of complex action sequences, providing a more accurate solution for fall prediction.

[0071] Problems and causes addressed by the frame selection module: In actual scenarios, monitoring equipment may shake due to wind, unstable brackets, or human operation, resulting in noise and interference information in the video frames. The frame selection module can filter out invalid frames caused by shaking by calculating the similarity of feature vectors, and only retain key frames related to actions, thereby improving data quality. In addition, the processing of each frame in a high-frame rate video will generate a large amount of data, resulting in a huge amount of calculation and a slow response speed. The frame selection module reduces the redundant data processed by screening key frames, greatly optimizes the utilization of computing resources, and shortens the processing time of the system. Therefore, the frame selection module effectively solves the two problems of insufficient prediction accuracy and long response time in traditional prediction methods.

[0072] Problems and reasons addressed by the skeleton graph module: In actual applications, surveillance videos often contain background noise, such as furniture, other people's activities, etc. The skeleton graph module only focuses on the posture changes of the human body's joints and ignores background interference information, thereby improving the accuracy of fall action recognition. The fall of the elderly involves the movement of multiple joints, and the posture changes are complex. Traditional feature extraction methods are difficult to accurately describe these complex motion trajectories. The skeleton graph module extracts the key joints of the human body through the OpenPose model and generates a skeleton graph, which can effectively capture these dynamic changes and provide accurate posture information for the fall prediction model. The skeleton graph module mainly solves two problems: insufficient prediction accuracy and insufficient modeling capabilities for complex action sequences.

[0073] The SPD manifold module calculates the geometric associations between human joints through covariance matrix, logarithmic mapping and vectorization processing, and maps the geometric relationship between human joints from nonlinear manifold to linear space. Through these steps, the originally complex geometric information is expanded into a one-dimensional feature sequence suitable for the input of the Position-Aware Transformer model. This process not only effectively retains the spatial geometric relationship between joints, but also provides rich geometric and temporal features for subsequent temporal modeling, which helps to more accurately predict the fall movements of the elderly.

[0074] Problems and reasons addressed by the SPD manifold module: Traditional fall prediction methods have limitations in capturing the geometric characteristics of complex human movements, especially the modeling of multi-joint movements. The SPD manifold module accurately captures the geometric relationships between joints by constructing the covariance matrix of human skeletal joints. These geometric information can better reflect the physical changes of the elderly before they fall, such as the slight movements of joints such as the hip and knee, thereby improving the accuracy of prediction. Especially in practical applications, when the elderly begin to lose balance, the SPD manifold module can identify these subtle changes in movement and make more accurate fall predictions. In addition, the fall process involves dynamic changes in multiple joints, and traditional methods find it difficult to accurately model these complex action sequences. The SPD manifold module captures the geometric relationship between joints, converts it into a one-dimensional feature sequence, and combines it with time series modeling to accurately identify the dynamic changes of different joints during the fall process. For example, during the fall process, the different movement speeds and directions of joints such as arms, legs, and shoulders are accurately modeled, thereby enhancing the ability to recognize complex movements. The SPD manifold module mainly solves two problems: insufficient prediction accuracy and insufficient modeling capabilities for complex action sequences.

[0075] The Position-Aware Transformer module uses the self-attention mechanism and time series modeling to analyze the time series features in the elderly's skeletal data and effectively capture long-term dependencies. At the same time, the module introduces position encoding to give position information to each input feature vector, thereby more accurately capturing movement changes, especially the posture features at various stages of the elderly's fall.

[0076] The Position-Aware Transformer encoder is able to model the joint features of each time step and capture long-distance dependencies in the sequence with the help of the self-attention mechanism. By introducing position encoding, the model can perceive the temporal information in the sequence, so as to more accurately analyze the entire process of the elderly from standing to losing balance to falling. This time series modeling capability significantly improves the accuracy of fall prediction. The Position-Aware Transformer module maps the output features to the probability distribution of action categories through the fully connected layer and the Softmax layer to predict the actions of the elderly in the current or future time steps (such as standing, losing balance, falling, etc.). This prediction method can not only identify the current actions of the elderly in real time, but also provide early warning signals for falls.

[0077] Falling of the elderly often involves a series of continuous actions from standing to losing balance and then falling, with a long time span. When processing long time series, traditional RNN or LSTM models are prone to gradient vanishing or gradient exploding problems, and it is difficult to effectively capture long-range dependencies in the sequence. In contrast, the Position-Aware Transformer module can simultaneously focus on the features of different time steps in the sequence through the self-attention mechanism, and is particularly good at capturing long-range dependencies in long time series. For example, the module can identify subtle posture changes when the elderly initially lose balance, and then accurately infer the subsequent falling behavior. In addition, the falling action involves dynamic changes in multiple joints, and traditional methods are difficult to accurately model these complex interactions. The Position-Aware Transformer module uses a multi-head self-attention mechanism to simultaneously focus on the changes in key joints at different time points when processing time series data, and capture these complex action sequences. For example, the model can simultaneously identify the changes in the joints of the elderly's shoulders, knees, etc. during the fall process, thereby more comprehensively modeling the entire fall action. By introducing position encoding and multi-head self-attention mechanism, the Position-Aware Transformer module effectively solves two problems of traditional models: limited ability to model temporal dependencies and insufficient modeling of complex action sequences, providing higher accuracy and stability for fall prediction for the elderly.

[0078] Furthermore, the fall prediction system of the present invention also includes an early warning module; the early warning module receives the prediction result of the position-aware Transformer module, and the prediction result is standing, losing balance or falling; when the prediction result of the position-aware Transformer module is falling, the early warning module issues a notification.

[0079] The early warning module automatically sends notifications to emergency contacts and medical services, and provides location information and health data to ensure that the elderly can get help in time. In traditional systems, the response time from fall prediction to alarm is long, and timely help cannot be provided at critical moments. The early warning module is closely integrated with fall prediction. Once the predicted result is a fall, an alarm is immediately sent to the preset emergency contact or medical service. This greatly shortens the time from prediction to alarm, ensuring that the elderly can get rescued quickly after a fall occurs. The early warning module mainly solves the problem of long response time; the system also supports multi-channel notifications, continuous monitoring and anti-false alarm mechanisms to ensure that the elderly get timely help in the critical time after a fall and reduce the false alarm rate.

[0080] Beneficial effect: A fall prediction method and system of the present invention solves the problems existing in the prior art of insufficient prediction accuracy, limited modeling capability of time-dependent modeling, long response time, and insufficient modeling capability for complex action sequences;

[0081] (1) This paper combines the SPD manifold and the Position-Aware Transformer model. The SPD manifold accurately captures the geometric structure of human motion and expands it into a one-dimensional feature sequence suitable for the Position-Aware Transformer model input. Subsequently, the Position-Aware Transformer model is used for temporal modeling, which effectively captures the temporal dependencies in human motion and greatly improves the accuracy of fall prediction, especially when processing complex motion sequences.

[0082] (2) The Position-Aware Transformer model is used to replace the traditional time series model. The model introduces position encoding and assigns position information to each input feature, enabling the model to perceive the time order in the input data. Combined with the multi-head self-attention mechanism, the Position-Aware Transformer can simultaneously focus on the dependencies between different time steps in a long sequence and accurately capture long-distance temporal dependencies. It can more accurately predict falling actions and solve the shortcomings of traditional models in long time series modeling.

[0083] (3) Through modular design, efficient feature processing, and real-time alarm mechanism, the response time from prediction to alarm is significantly shortened, enabling rapid response to elderly falls.

[0084] (4) The present invention combines the skeleton graph module with the SPD manifold to accurately capture the geometric relationship between the joints of the human body. With the help of the self-attention mechanism of the Position-Aware Transformer, it is possible to capture the complex dynamic changes between time steps, thereby achieving modeling of complex action sequences. This multi-module collaborative work enables the system to accurately identify and predict continuous action changes during a fall. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 It is a flow chart of fall prediction and emergency treatment based on SPD manifold and Position-Aware Transformer of the present invention;

[0086] Figure 2 It is a flow chart of frame selection based on cosine similarity of the present invention;

[0087] Figure 3 It is a flow chart of skeleton graph generation based on OpenPose of the present invention;

[0088] Figure 4 It is a flow chart based on SPD manifold feature extraction of the present invention;

[0089] Figure 5 It is a flow chart of fall prediction based on time series features of Position-Aware Transformer of the present invention; DETAILED DESCRIPTION

[0090] The content of the present invention is further explained below in conjunction with the accompanying drawings and specific embodiments.

[0091] Based on existing artificial intelligence and computer vision technologies, the present invention proposes a solution to the four problems of existing video action prediction systems, namely, insufficient prediction accuracy, limited temporal dependency modeling capability, long response time, and insufficient modeling capability for complex action sequences. The solution can effectively solve the above problems and significantly improve the accuracy of the prediction of the elderly falling action in the video.

[0092] The present invention is a method for predicting and automatically alarming elderly people's falls based on the combination of SPD manifold and Position-Aware Transformer, which mainly includes a frame selection module, a skeleton graph module, an SPD manifold module, a Position-Aware Transformer module and an alarm module. The overall process is as follows Figure 1 shown.

[0093] The main innovation of the present invention lies in the combined application of frame selection module, SPD manifold and Position-Aware Transformer, which effectively improves the accuracy and response speed of predicting the falling action of the elderly in the video.

[0094] Specifically, taking a 3-second video of a fall as an example, the video frame rate is 30 frames / second, with a total of 90 frames, representing 90 time steps t. The frame selection module is responsible for processing the input elderly fall action video frame by frame. First, ResNet-18 is used to extract features for each frame. Then, the extracted features are grouped into groups of five frames in chronological order. Next, the average feature vector of each group of five frames is calculated. The first group of selected frames is more special. The average feature vector of the first group of 5 frames is calculated. The cosine similarity between each frame in the first group and the average feature vector of the group is calculated. The frame with the highest similarity is selected as the representative frame of the group. Then, the average feature vector of the first group is calculated one by one with the feature vectors of the subsequent 5 frames. The most similar frame is selected as the representative frame of the next group, and so on, to ensure that the representative frame of each group can reflect the main features. The skeleton graph module uses the OpenPose model to estimate the posture of the selected representative frame images, extract joint point information and generate a skeleton graph to capture posture changes and provide basic data for fall prediction. The SPD manifold module converts the spatial position relationship and motion trajectory between human joints in the skeleton diagram into a one-dimensional feature sequence suitable for processing by the Position-Aware Transformer model by calculating the covariance matrix, logarithmic mapping and vectorization steps. The Position-Aware Transformer performs temporal modeling on the one-dimensional feature sequence output by the SPD manifold module. By introducing position encoding, each input feature is given time step information, so that the Position-Aware Transformer model can better perceive the temporal features in the sequence. Subsequently, the self-attention mechanism is used to capture the long-distance dependencies in the temporal features and the probability of falling actions is predicted through the fully connected layer and the Softmax layer. When the early warning module predicts the risk of falling of the elderly, it automatically sends an emergency notification to ensure timely intervention and reduce the risk of false alarms.

[0095] Among them, the frame selection module is used to process the input video of the elderly falling down frame by frame and perform feature extraction, and group the processed video frames into groups of five frames, and use cosine similarity to select the frame with the largest similarity and save it, such as Figure 2 As shown, the specific steps are as follows:

[0096] Step 1: Process the input 3s video frame by frame to obtain video frame I m =(I m1 ,I m2 ,I m3 ,…I mn), n is the total number of frames, and n = 90. When preprocessing the video frames, the input action image is resized to 256*256 pixels according to the range of human body movements in the original image, and a 224*224 pixel area is cropped in the center of the image to retain the main content of the image while ensuring the consistency of the size of the input image; the image is normalized, that is, each channel of the image is normalized according to the given mean and standard deviation.

[0097] Step 2: Input the preprocessed frame into ResNet-18 and use the ResNet-18 deep residual network for feature extraction;

[0098] The present invention uses a ResNet-18 deep residual network model, which is consistent with the classic, He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.

[0099] Assume that the input image of the elderly falling down is I m1 , this is an RGB color image of size 224*224*3, where 224*224 is the width and height of the image, and 3 is the number of channels. The ResNet-18 model initially performs a convolution operation to extract the original features of the image, where the convolution kernel size is 7*7, the stride is 2, and the final output is:

[0100] X conv1 =RELU(I m1 *W conv1 +b conv1 )

[0101] Among them, I m1 is the input image, W conv1 is the first layer convolution weight, b conv1 For bias.

[0102] In the first convolution layer of ResNet-18, 64 7×7 convolution kernels are used, so the output feature map has 64 channels. The feature map refers to the output image generated by the convolution operation. It can be regarded as a spatial representation of the features or patterns extracted from the input image after the convolution kernel is processed. Each convolution kernel extracts a set of features from the input image, and the number of output feature maps is the number of convolution kernels (64), that is, the output size is 112*112*64.

[0103] Then, after a pooling layer with a step size of 2, the final output is X pool1 , the size is 56*56*64.

[0104] In ResNet-18, each residual block contains two 3×3 convolutional layers. The first layer transforms and extracts input features, and the second layer extracts and enhances features in depth. In the residual block, the input is directly added to the output to form a skip connection. This operation allows the network to directly learn the residual, that is, the difference between the input and output, thereby more effectively training deep networks.

[0105] In the residual block, the input is X pool1 , the output is X res1 , the mathematical representation of the residual block is:

[0106] X res1 =F(X pool1 ,W ii )+X pool1

[0107] Where F represents the combination of two convolutional layer operations in the residual block, and W i Represents the weight parameter in the residual block. After the last residual block, the output feature map of the convolution layer is retained as the key feature. These feature maps contain rich semantic information and can accurately represent the action features in the video frame. After performing global average pooling on these feature maps, a feature vector of fixed length is obtained. The formula is as follows:

[0108]

[0109] H and W: the height and width of the feature map, indicating the size of each feature map;

[0110] f i : represents the pixel index of the feature map in the height direction (from 1 to H), that is, the vertical pixel position of the feature map;

[0111] f j : represents the pixel index of the feature map in the width direction (from 1 to W), that is, the horizontal pixel position of the feature map;

[0112] f k : represents a specific channel index or feature map index, indicating the kth feature map currently being processed;

[0113] The pixel value of the kth channel at position (i, j) in the feature map;

[0114] Normalize the sum result with the number of pixels H*W and calculate the average value of the kth channel;

[0115] γ: The final feature vector, which is the average value of each channel γ k composition;

[0116] This formula calculates the average value of all pixels on a specific channel (the kth feature map). The i and j in the formula traverse each pixel (row and column) of the feature map, calculate the average value of all pixels, and finally obtain a scalar value γ of the kth feature map. k Through this global average pooling operation, each feature map can be compressed into one value. The final generated feature vector is:

[0117] γ=[γ1,γ2,γ3,…γ c ]

[0118] Here c represents the number of feature maps generated after each frame of video is processed by ResNet-18, that is, the number of feature map channels obtained after each frame of image is processed by multiple convolution kernels. The purpose of this operation is to compress the high-dimensional feature map into a feature vector of fixed length through global average pooling, which is convenient for the subsequent selection of key frames through cosine similarity.

[0119] The output vectors of the global average pooling layer are input to the fully connected layer. The fully connected layer further compresses these vectors into a fixed-length feature vector for subsequent similarity calculation.

[0120] Step 3: The processed frames of the elderly falling video are grouped at equal intervals, with 5 frames in each group, resulting in 18 groups of video clips. Each time 5 consecutive frames are processed, the feature vectors of the five frames in each group are averaged to obtain the average feature vector of the group. Then, the cosine similarity between this average feature vector and the feature vectors of the next five frames is calculated. By calculating these similarities, the frame that is most similar to the average feature vector is found and selected as the representative frame.

[0121] For example, suppose the feature vectors of the five frames in the first group are γ 11 ,γ 12 ,γ 13 ,γ 14 ,γ 15 , its average eigenvector is:

[0122]

[0123] Next, calculate each frame from the 1st to the 5th frame and their average feature vector γ avg1 The cosine similarity between the two frames is calculated, and then the frame with the highest similarity is selected as the representative frame. Assuming that the calculation shows that the third frame γ3 has the highest similarity, then γ3 will become the representative frame of the first to fifth frames.

[0124] Next, calculate the first set of average eigenvectors γ avg1 and the feature vectors γ6,γ7,γ8,γ9,γ 10 , the cosine similarity between them. The calculation formula of cosine similarity is:

[0125]

[0126] Among them, ||γ j || and ||γ avg1 || are the modulus of the vector, indicating the length of the vector.

[0127] γ j represents the feature vector of the jth frame, j∈[6,10], and the cosine similarity represents the cosine value of the angle between two vectors in the vector space, and the value range is [-1,1]. When the similarity is 1, it means that the two vectors are exactly the same; when the similarity is 0, it means that the two vectors are orthogonal; when the similarity is -1, it means that the two vectors are in completely opposite directions.

[0128] Assuming that by calculating the similarity, it is found that γ6 has the highest similarity (for example, the similarity is 0.98), then the system will select the sixth frame γ6 as the representative frame of the second group of video frames.

[0129] Through the above steps, the system can effectively filter out the most representative frames and provide high-quality input data for the subsequent prediction of elderly falling movements. The extraction of feature vectors and similarity calculation provide the basis for the entire system, ensuring that elderly falling events can be accurately and reliably predicted and predicted.

[0130] like Figure 3 As shown, the pose estimation model OpenPose is used to extract joint coordinates of representative frame images in the video and construct a skeleton graph;

[0131] Step 4: First, input each frame of the elderly falling image into the pose estimation model OpenPose. The model outputs a heat map (also called a confidence map) for each joint. Heatmap It is generated for each frame, and each frame will have an independent joint point heat map. Where k represents the joint point number (such as shoulder, elbow, knee, etc.), t represents the time number, that is, the time when this frame of image occurs. Each heat map is a two-dimensional matrix where each element Represents pixel points (s i ,s j ) belongs to the joint point k. Next, use the max operation to add the confidence map Find the pixel coordinates with the highest confidence. The formula is as follows:

[0132]

[0133] in:

[0134] s i : Indicates in the confidence map The vertical axis in is the number of pixel rows from top to bottom;

[0135] s j : Indicates in the confidence map The horizontal axis in is the number of pixel columns from left to right;

[0136] Represented in the confidence map In the figure, the pixel (s i ,s j )’s confidence value;

[0137] max is an operator that performs a Traverse all pixels in and find The largest index (s i ,s j ), and use the index as the coordinate of the joint point

[0138] For example, in a certain frame of image, we need to determine the position of the elderly person’s shoulders. OpenPose generates a confidence map of the shoulder joint. This confidence map is a two-dimensional matrix, each element Indicates the pixel position (s i ,s j ) belongs to the confidence of the shoulder joint point, and the confidence map will be traversed through the max operation For each pixel in The maximum position (s i ,s j ) Find and convert the coordinates of the pixel point And use this position as the coordinates of the shoulder joint point The three-dimensional coordinates of the shoulder joint point.

[0139] in Represents the coordinates of the kth joint point in time frame t, and t represents the time frame number of the joint point.

[0140] If the confidence If the coordinate exceeds a preset threshold θ (e.g., 50%), the coordinate is retained; otherwise, it is marked as missing:

[0141]

[0142] For example, when predicting the joint coordinates of the elderly's shoulders, elbows, and knees, if the confidence of a certain joint is lower than a threshold (e.g., 50%), the joint coordinate will be marked as missing, otherwise the coordinate will be retained.

[0143] Step 5: Get the joint coordinates at each time t Finally, the joints are connected according to the human body structure to form a skeleton diagram P t , k ranges from 1 to K, and there are K joints in total. The skeleton graph consists of the three-dimensional coordinates of K joints:

[0144]

[0145] in, represents the 3D coordinates of the kth joint in time frame t. The connections between joints (e.g., shoulder to elbow, elbow to wrist, etc.) are defined as edges. As time frame t passes, the changes of joints and edges are recorded through the skeleton graph P t Perform dynamic description to reflect the changing process of human body posture at different time points.

[0146] Skeleton diagram of the entire time series:

[0147] {P t |t=1,2,…,t max}

[0148] It is used for subsequent feature extraction and fall action prediction to help identify the posture changes of the elderly. The core of this step is to simplify complex image data, extract the coordinates of human joints, and generate a skeleton diagram to more efficiently analyze the posture changes of the elderly. In the fall prediction process, the skeleton diagram retains the position information of key parts of the human body, such as shoulders, hips, knees, and ankles. By observing the position changes of these joints, it is possible to analyze whether the elderly are standing, losing balance, or about to fall.

[0149] like Figure 4 As shown in the figure, the accuracy of fall prediction is further improved by using SPD manifold to capture the geometric relationship and motion pattern between human joints.

[0150] Step 6: At each time step t, calculate the covariance matrix C using the formula t , to capture the geometric relationship and movement pattern between human joints. When the elderly stand up, lose balance, and fall, the covariance between different joints (such as knee, hip, and shoulder) will show different patterns.

[0151] Through the joint point data obtained at each time step, the system constructs a covariance matrix, which is used to describe the geometric relationship between the joint points of the human body. For example, the relative position between the shoulder and the knee, the relationship between the hip and the ankle, etc., this information is crucial to identifying movement changes.

[0152] The formula for calculating the covariance matrix is ​​as follows

[0153]

[0154] in:

[0155] represents the three-dimensional coordinates of the kth joint point in time frame t;

[0156] is the mean vector of all relevant nodes at time step t, that is:

[0157]

[0158] Through the covariance matrix, the system can capture the changes in the geometric relationships between joint points. These changes can reflect the movement patterns of the elderly in different states (such as standing, losing balance or falling), providing important features for subsequent fall prediction.

[0159] Step 7: Since the covariance matrix C t It is a symmetric positive definite matrix (SPD matrix) located on a nonlinear manifold. In order to facilitate the subsequent deep learning model processing, it needs to be converted into a linear space through logarithmic mapping. The core of this step is to simplify the original geometric data to make it easier to model while preserving the relative geometric relationship between the joint points.

[0160] The specific steps are as follows:

[0161] 7.1 Eigenvalue Decomposition: Covariance Matrix C t Perform eigenvalue decomposition:

[0162]

[0163] Among them, U t is the eigenvector matrix, Λ t is a diagonal matrix containing the eigenvalues ​​of the covariance matrix.

[0164] 7.2 Logarithmic Mapping: Eigenvalue Matrix Λ t Take the logarithm of each eigenvalue in and get the logarithmic eigenvalue matrix log(Λ t ):

[0165] log(Λ t )=diag(log(λ t1 ),log(λt2 ),…,log(λ tn ))

[0166] Then, the eigenvector matrix is ​​used to reconstruct the logarithmic mapped covariance matrix:

[0167]

[0168] Through logarithmic mapping, the original geometric information is converted into linear data that is convenient for time series modeling, while retaining the relative geometric relationship between joint points. In the process of fall prediction, logarithmic mapping can not only retain the geometric information of the elderly's posture changes, but also simplify this information, making their posture changes before falling (such as shoulder sinking, body tilt, etc.) clearer, which helps to better extract and process these key features.

[0169] Step 8: Next, the present invention converts the logarithmically mapped covariance matrix log(C t ) is vectorized to fit the input of the Position-Aware Transformer model. Specifically, the matrix log(C t ) is expanded into a one-dimensional feature vector v t , the vector v of all time steps t Arrange them in order to form a time series feature sequence: {v1,v2,…,v T}

[0170] This feature sequence captures the geometric changes of joints in the entire time series and can effectively describe the posture changes of the elderly during the fall. Using the self-attention mechanism of the Position-Aware Transformer model, these features can be further analyzed to capture the long-term dependencies in the action sequence.

[0171] The covariance matrix processed by logarithmic mapping is a symmetric matrix. Through vectorization operations, the originally complex geometric relationship is simplified into a numerical feature sequence, making the subsequent time series modeling more efficient. For fall prediction, the vectorization result of each time step contains the geometric features of the elderly's current posture. As time goes by, the system gradually captures the characteristics of posture changes, providing an important basis for accurately identifying fall actions.

[0172] To more clearly demonstrate how the covariance matrix and its logarithmic mapping and vectorization can help capture posture changes during falls in the elderly, the following is a specific application example.

[0173] Assume that there is a sequence of skeleton graphs of an elderly person from standing to falling. The system captures the geometric features and motion patterns of joints through the following three key state changes:

[0174] 1. Standing (stable state):

[0175] When the elderly are in a standing state, the system calculates the covariance matrix C at each time step t To capture the geometric relationship between joint points. The covariance matrix C t and its logarithmic mapping log(C t ) shows a relatively stable pattern, indicating that the relative position of the joint points changes little. After vectorization, the generated feature vector v t It also shows stable geometric features, and the system judges that the elderly are still in a state of balance.

[0176] 2. Loss of balance (transition state):

[0177] As the elderly gradually lose balance, the center of gravity of the body begins to shift, and the joints of the legs and upper body begin to move significantly. At this time, the covariance matrix C t This reflects that the geometric relationship between the joints has changed significantly, especially the relative position of the leg and hip joints has changed significantly. The logarithmic mapping of the covariance matrix log(C t ) also shows corresponding changes. After vectorization, the generated feature vector v t This change was captured, showing that older people were losing their balance.

[0178] 3. Fall (critical condition):

[0179] When the elderly fall, the movement of the body joints increases, especially the relative positions of the shoulders, knees and hips change significantly. t and its logarithmic mapping log(C t ) shows a sharp change in geometric pattern, showing violent movements between joint points. The vectorized feature vector v t By accurately capturing these drastic change characteristics, the system can accurately predict and warn of falling actions based on these characteristics.

[0180] like Figure 5 As shown in the figure, the Position-Aware Transformer module performs time series modeling on the input time series feature vector and makes predictions through the fully connected layer and the Softmax layer. Position-Aware Transformer is a deep learning model based on the self-attention mechanism. It is particularly good at processing time series data and can effectively capture long-term dependencies. In the prediction of falls in the elderly, Position-Aware Transformer analyzes the skeleton data sequence of continuous time steps to capture the trend of movement changes and thus predict possible falls.

[0181] The Transformer model used in this paper is consistent with the classic model, referring to the paper "Attention is all you need"[J](Advances in Neural Information Processing Systems, 2017, 30) published by Vaswani A, Shazeer N, Parmar N et al. in 2017. In the fall prediction process, each time step t generates a corresponding bone data vector v t , which is the vectorized result of the processed joint point covariance matrix, and contains the geometric relationship of the joint points at each time step. Assuming there are T time steps, the input sequence of the system is {v1,v2,…,v T}, where each v t Represents the skeleton feature vector at time step t, representing the geometric relationship of the joint points.

[0182] Step 9: Since the Position-Aware Transformer model itself does not have timing information, position encoding is used to give position information to each time step t. The position encoding vector is generated by sine and cosine functions:

[0183]

[0184] in:

[0185] t is the time step, which refers to a moment in the action sequence;

[0186] i is the index of the feature dimension;

[0187] d model is the vector dimension.

[0188] Position Encoding Matrix Added to the input feature sequence, we get the input sequence H with position information input .

[0189] H input =v t +PE

[0190] In the elderly fall prediction example, through the above operations, the vector sequences of all time steps are input into the Position-Aware Transformer model, and the sequence information is introduced through position encoding. These features will be used as input to the Transformer encoder for time series modeling and subsequent action prediction. When the elderly are likely to fall, position encoding helps the model identify features at different time steps, allowing the Position-Aware Transformer to effectively capture changes in skeletal posture during the fall, thereby improving the accuracy of the prediction.

[0191] In the prediction of falls among the elderly, this process helps the model capture posture changes at different moments and enhances the understanding of geometric information in action sequences.

[0192] Step 10: Timing Modeling

[0193] Step 10.1. Linear transformation

[0194] First, the input features Generate the query matrix Q, key matrix K and value matrix V through linear transformation:

[0195] Q=H input W Q

[0196] K=H input W K

[0197] V=H input W V

[0198] in:

[0199] is a learnable weight matrix;

[0200] d k is the dimension of the attention head.

[0201] Through the above steps, the vector sequences of all time steps are input into the Position-Aware Transformer model and the features are converted into matrices suitable for Position-Aware Transformer input. These matrices will be used in the subsequent attention mechanism to help the model capture the changes in the skeletal posture of the elderly during the fall.

[0202] Step 10.2 Attention Mechanism

[0203] The dependency between time steps is captured by calculating the similarity between the query and the key and normalized using the Softmax function:

[0204]

[0205] in:

[0206] is a scaling factor to prevent the inner product value from being too large.

[0207] The softmax function is used to normalize the attention scores so that their sum is 1. In the prediction of elderly people falling, it is assumed that a node represents a joint of the elderly (such as the shoulder). By calculating the similarity between the query matrix Q and the key matrix K at each time step, the relative importance of different nodes in the time step is determined. After normalization using the Softmax function, the obtained attention score represents the importance of each node. The attention mechanism can effectively capture the changes in the key joint points of the elderly during the fall process, thereby improving the accuracy of action prediction.

[0208] Step 10.3 Multi-head attention: In order to capture different feature relationships, the model uses a multi-head attention mechanism:

[0209] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0210] in:

[0211] head1=Attention(Q i ,K i ,V i ) is the output of the i-th attention head;

[0212] is the weight matrix used for the linear transformation.

[0213] This mechanism allows the model to focus on different positions in the sequence at the same time, thereby capturing rich information in the feature sequence. When an elderly person falls, different attention heads can focus on changes in key nodes such as shoulders, elbows, and knees, respectively, thereby capturing the characteristics of the falling action more comprehensively.

[0214] Through the self-attention mechanism, Position-Aware Transformer can capture the dependencies between different time steps. For example, in the prediction of an elderly person falling, the system can identify the changes in joints such as shoulders and hips at different times, thereby discovering the early signals of the elderly person losing balance from standing up.

[0215] 10.4 Feedforward Neural Networks

[0216] After the multi-head attention mechanism, each encoder layer also includes a feed-forward neural network (FFN) for further feature processing. The FFN consists of two linear transformation layers and an activation function:

[0217] Y=FFN(x)=ReLU(xW1+b1)W2+b2

[0218] in:

[0219] x refers to the output vector after the multi-head attention mechanism;

[0220] is the linear transformation matrix;

[0221] is the bias vector;

[0222] d ff is the hidden layer dimension of the feedforward neural network.

[0223] In the prediction of elderly people falling, FFN further improves the expressiveness of features and enhances the nonlinear modeling ability of the model through nonlinear transformation and linear transformation. For example, after the multi-head attention mechanism processes the input features, FFN extracts deeper feature information through a series of linear and nonlinear transformations. These enhanced features help to more accurately capture the changes in the skeletal posture of the elderly during the fall, thereby optimizing the accuracy of the prediction of the elderly's fall action.

[0224] Step 11: Prediction of the elderly falling down

[0225] The output feature Y of the Position-Aware Transformer encoder is input into the fully connected layer, and the feature is mapped to the action category space through linear transformation to obtain the scores of different action categories, which are marked as Y t The Softmax layer then transforms these class scores into probability distributions for predicting the actions that the elderly may perform in the current or future time steps (e.g., “stand up,” “lose balance,” or “fall down”).

[0226] Through the above steps, the output features of the Position-Aware Transformer encoder are mapped to the action category space through the fully connected layer to obtain the category score. The Softmax layer converts the category score into a probability distribution through the Softmax function and selects the category with the highest probability as the final action prediction result.

[0227] P(Action t )=softmax(Y t )

[0228] For fall prediction, the model outputs the probability of an action that the elderly may perform (e.g., standing up, losing balance, falling, etc.) in the current or future time steps.

[0229] When the model predicts a high probability of a fall, the system automatically sends out an early warning signal to ensure that the elderly can get timely help. At this point, the system not only predicts the fall, but also needs to quickly pass this information to the relevant parties.

[0230] Step 12: Working mechanism of the early warning module

[0231] When the model predicts a high probability of a fall, the system will automatically send out an early warning signal to ensure that the elderly can get timely help. At this point, the system not only predicts the fall, but also needs to quickly pass this information to the relevant parties.

[0232] In the early warning module, the system will respond accordingly based on the prediction results of the Position-Aware Transformer model. The specific process is as follows:

[0233] (1) Fall prediction: If the model concludes through time series modeling analysis that the elderly are more likely to fall, the system will determine that the elderly are at risk of falling and enter the early warning mode.

[0234] (2) Notification mechanism: Once the early warning mode is activated, the system will automatically call the emergency contact number (such as a relative's number) set by the elderly person in advance to inform the family of the elderly person's condition as soon as possible. At the same time, the system will also call the 120 emergency number and provide location information and related health data to ensure that medical services can respond in a timely manner.

[0235] (3) Multi-channel notification: The early warning module not only notifies through phone calls, but also sends SMS or App push notifications to pre-set contacts and relevant medical service agencies. The SMS or push content will include the time and location of the fall and the health status of the elderly, so that family members or medical staff can respond quickly.

[0236] (4) Continuous monitoring: Even if the early warning mode is activated, the system will continue to monitor the elderly’s condition. If the model predicts that the elderly have successfully regained their balance or stopped dangerous movements, the system will send an update to inform the relevant parties of the improvement of the elderly’s condition and avoid unnecessary emergency intervention.

[0237] This design ensures that in the event of an emergency where an elderly person falls, the system can quickly and accurately send out warning signals and promptly contact family members and medical service agencies. Through the warning module, the elderly can receive timely intervention and help within the golden time after a fall occurs, reducing the risk of serious injury caused by a fall.

[0238] Based on the existing artificial intelligence and computer vision technologies, this paper proposes solutions to the four main problems existing in the existing elderly fall video action prediction. The solutions corresponding to each problem are as follows:

[0239] (1) Insufficient prediction accuracy: Traditional fall prediction methods (such as those based on CNN or RNN models) have limitations in capturing the geometric characteristics of complex human movements. They usually rely on raw sensor data or simple feature extraction, resulting in low prediction accuracy for fall movements, especially in complex scenarios. The present invention uses SPD manifolds to capture the geometric structure of human movements and calculates the geometric correlation between human joints through covariance matrices, logarithmic mapping, and vectorization processing. Subsequently, these geometric information are converted into a one-dimensional feature sequence suitable for processing by the Position-Aware Transformer model to improve the accuracy of fall prediction, which is particularly prominent when processing complex action sequences.

[0240] (2) Limited temporal dependency modeling capabilities: Traditional RNN or LSTM models are prone to gradient vanishing or gradient exploding problems when processing long time series, resulting in insufficient ability to model long-range dependencies in long time series. Fall prediction requires identifying key features in long time series, but existing models perform poorly in this regard. To this end, the present invention adopts the Position-Aware Transformer model to replace the traditional temporal model. The model introduces position encoding so that each input feature vector contains position information, and combines the self-attention mechanism to analyze the temporal characteristics of the elderly's skeletal data and accurately capture long-term dependencies. In addition, by using the multi-head attention mechanism, the model can identify changes in key joint points in different time steps, thereby more accurately predicting falling actions.

[0241] (3) Long response time: The existing system has a slow prediction and alarm response speed, and cannot respond quickly when a fall occurs, which delays the protection of the safety of the elderly. The present invention improves the response speed of the system through modular design and efficient feature processing flow. Specifically, the frame selection module pre-processes the video frames to reduce redundant data, and combines the rapid response of the early warning module to ensure that the elderly can get timely help within the critical time after the fall.

[0242] (4) Insufficient modeling capabilities for complex motion sequences: Traditional methods are difficult to capture the characteristics of complex human motion sequences, especially those involving dynamic changes in multiple joints. The present invention accurately captures the geometric relationship between human joints by combining the skeleton graph module with the SPD manifold. The SPD manifold uses the calculation of the covariance matrix, logarithmic mapping, and vectorization steps to convert the geometric information of human joints into a one-dimensional feature sequence suitable for processing by the Position-Aware Transformer model. Subsequently, the Position-Aware Transformer model captures the complex dynamic changes between different time steps by introducing position encoding and self-attention mechanisms, thereby modeling complex motion sequences. This multi-module collaboration enables the system to better understand and predict the continuous motion changes during the fall of the elderly.

[0243] In summary, the present invention solves the shortcomings of existing fall prediction technologies in terms of accuracy, timing modeling, response speed, and complex action sequence modeling, and provides a more accurate and efficient solution for predicting the fall actions of the elderly.

Claims

1. A fall prediction method, characterized in that: The steps include: Step 1, group the video in chronological order, with J frames as one group, and a total of Q groups of video frames; The feature vector of each frame in each group of video frames is extracted, and the feature vector of the jth frame in the qth group is represented by γ qj , j ranges from 1 to J, and q ranges from 1 to Q; Step 2, selecting one frame from each group of video frames as a representative frame of the group of video frames; Step 3, generating a skeleton graph from the representative frame; Step 4: Process the skeleton graph into a one-dimensional feature sequence and input it into the position-aware Transformer model to predict falls.

2. A fall prediction method according to claim 1, characterized in that: In step 1, ResNet-18 is used to extract feature vectors for each frame of image.

3. A fall prediction method according to claim 1, characterized in that: In step 2, one frame is selected from each group of video frames as a representative frame of the group of video frames, specifically: an average feature vector of each group of video frames is calculated, and then the cosine similarity between the average feature vector of each group of video frames and each feature vector of a subsequent group of video frames is calculated, and the feature vector with the highest similarity to the average feature vector is selected, and the corresponding video frame is the representative frame of the subsequent group of video frames; For the first group of video frames, the average feature vector of the first group of video frames is calculated, and then the cosine similarity between the average feature vector and each feature vector of the first group of video frames is calculated, and the feature vector with the highest similarity to the average feature vector of the first group of video frames is selected, and the corresponding video frame is the representative frame of the first group of video frames.

4. A fall prediction method according to claim 3, characterized in that: The average eigenvector of the qth group of eigenvectors is denoted as γ avgq ; Calculate the average eigenvector γ of the qth group avgq The cosine similarity between each feature vector in the q+1th group is calculated as: γ (q+1)j represents the jth eigenvector in the q+1 group; The feature vector with the highest similarity to the average feature vector of the qth group is selected from the q+1th group, and the video frame corresponding to the feature vector is used as the representative frame of the q+1th group of video frames; the representative frame of each group of video frames is selected using the same method.

5. A fall prediction method according to claim 3, characterized in that: In step 3, the representative frame is used to generate a skeleton graph, and the specific steps are as follows: Step 3.1: Input the representative frame into the pose estimation model OpenPose, and the pose estimation model OpenPose outputs a heat map of each joint Among them, k represents the number of the joint point, and t represents the time step; Heatmap The elements in are denoted as Represents pixel points (s i ,s j ) belongs to the confidence of joint point k; In the heat map Find the coordinates of the pixel with the highest confidence in the formula as follows: Represents the pixel coordinates of the kth joint at time t, and the value range of k is 1 to K; The skeleton graph consists of the three-dimensional coordinates of K joint points:

6. A fall prediction method according to claim 3, characterized in that: In step 4, the skeleton graph is processed into a one-dimensional feature sequence. The specific steps are as follows: Step 4.1, for each time step t, calculate the skeleton graph P t The covariance matrix C of the joints t , the formula is as follows: is the mean vector of all relevant nodes at time step t, that is: Step 4.2: transform the covariance matrix C t Transform to linear space through logarithmic mapping; Step 4.2.1, for the covariance matrix C t Perform eigenvalue decomposition: Among them, U t is the eigenvector matrix, Λ t is a diagonal matrix containing the eigenvalues ​​of the covariance matrix; Step 4.2.2, for the eigenvalue matrix Λ t Take the logarithm of each eigenvalue in and get the logarithmic eigenvalue matrix log(Λ t ): log(Λ t )=diag(log(λ t1 ),log(λ t2 ),…,log(λ tn )) Among them, λ tn Refers to the covariance matrix C at time step t t The nth eigenvalue of ; Then, the covariance matrix after logarithmic mapping is reconstructed using the eigenvector matrix; Step 4.2.3, transform the covariance matrix log(C t ) is expanded into a one-dimensional feature vector v t , the vector v of all time steps t Arrange them in order to form a time series feature sequence: {v1,v2,…,v T }.

7. A fall prediction method according to claim 1, characterized in that: In step 4, the one-dimensional feature sequence v t Input into the position-aware Transformer model to predict falls, which includes the following steps: Step 4.3, assign position information to the time series feature sequence of each time step t, and obtain the time series feature sequence H with position information input ; H input =v t +PE Step 4.4: transform the time series feature sequence H input , linear transformation generates query matrix Q, key matrix K and value matrix V; the self-attention mechanism is used to capture the dependencies between time steps, expressed as: in, is the scaling factor; Step 4.5, use a multi-head attention mechanism to capture different feature relationships; MultiHead(Q,K,V)=Concat(head1,…,head h )W O head1=Attention(Q i ,K i ,V i ) is the output of the i-th attention head, and h represents the number of attention heads; is the weight matrix used for linear transformation, the feature dimension used in each attention head, d model Represents the feature dimension of each time step; d k Represents the function used to segment input features and calculate attention scores; Step 4.6, the features obtained by the multi-head attention mechanism are input into the feedforward neural network FFN. FFN includes two linear transformation layers and an activation function, which is expressed as: Y=FFN(x)=ReLU(xW1+b1)W2+b2 x refers to the output feature vector after the multi-head attention mechanism; is the linear transformation matrix; is the bias vector; d ff is the hidden layer dimension of the feedforward neural network; Step 4.7: The output Y of the feedforward neural network FFN is further input into the fully connected layer, and the features are mapped to the action category space through linear transformation to obtain the scores of different action categories, which are marked as Y. t ; Then, the Softmax layer converts the category scores into probability distributions, which are used to predict the probability of actions at the current or future time steps, including standing, losing balance, and falling.

8. A fall prediction method according to claim 7, characterized in that: The position encoding vector PE is generated by sine and cosine functions in: t is the time step, which refers to a moment in the action sequence; i is the index of the feature dimension; d model is the vector dimension.

9. A fall prediction system constructed based on the fall prediction method according to claim 8, characterized in that: Including frame selection module, skeleton graph module, SPD manifold module, and position-aware Transformer module; Inputting Q groups of video frames into the frame selection module; The frame selection module uses ResNet-18 to extract features from each group of video frames; and calculates the average feature vector of each group of video frames; Calculate the cosine similarity between the average feature vector of each group of video frames and each feature vector of a subsequent group of video frames, select the feature vector with the highest similarity to the average feature vector, and the video frame corresponding to the feature vector is the representative frame of the group of video frames; The skeleton graph module uses the OpenPose model to estimate the pose of the representative frame and generate a skeleton graph; The SPD manifold module is used to process the skeleton graph into a one-dimensional feature sequence v t ; The position-aware Transformer model is used to perceive the temporal features in the one-dimensional feature sequence, capture the long-distance dependencies in the temporal features using a self-attention mechanism, and predict the probability of falling actions through a fully connected layer and a Softmax layer.

10. The fall prediction system according to claim 9, characterized in that: It also includes an early warning module; The warning module receives the prediction result of the position perception Transformer module, wherein the prediction result is standing, losing balance or falling; When the prediction result of the location-aware Transformer module is a fall, the early warning module issues a notification.

Images