A fall early warning method and system for the elderly based on non-complete sequence and physical check

Abstract

The application discloses an old people fall early warning method and system based on non-complete sequence and physical checking, and the method comprises the following steps: obtaining the reference standing height of a person to be detected; buffering the video stream containing the person to be detected to a ring buffer with a capacity of N frames frame by frame, updating the buffer and triggering detection after a new frame is entered; performing skeleton key point extraction and normalization on each frame of image to obtain a skeleton sequence; cutting a sub-sequence from the skeleton sequence as a truncated early skeleton sub-sequence and resampling to obtain a standardized early sequence; inputting the standardized early sequence into a trained physical perception double-flow graph convolution network to obtain an initial fall probability of the person to be detected; calculating physical state space features, and combining the initial fall probability to perform hierarchical physical constraint posterior checking to obtain a frame determination result; and issuing a fall warning information when a plurality of continuous frames are determined as falling. The application can warn before falling occurs, and can also maintain high recall rate and high accuracy.

Description

Technical Field

[0001] This invention relates to the field of fall detection technology, and in particular to a method and system for early warning of falls in the elderly based on incomplete sequences and physical verification. Background Technology

[0002] With the increasing trend of population aging, falls have become one of the major risks threatening the lives and health of the elderly. Statistics show that falls are the leading cause of accidental death among people aged 65 and over. Therefore, timely and accurate detection of falls among the elderly is of great significance for ensuring their safety and gaining valuable emergency medical time.

[0003] Although existing fall detection schemes based on graph convolutional networks have made some progress in skeletal temporal analysis, they all have the following common limitations: (1) Inability to provide early warning: Most existing schemes require waiting for the action to be completed before making a judgment, and cannot issue a warning during the fall process; (2) Lack of physical constraint posterior mechanism: Existing schemes either use physical features directly as network input features or use multi-stream networks to process the input of the same modality separately, without proposing a mechanism for multi-level posterior verification based on the physical state space after the network output; (3) No systematic edge deployment scheme: Existing schemes do not provide a quantitative deployment method for real-time inference on resource-constrained edge intelligent computing devices.

[0004] Therefore, there is an urgent need for a fall detection method that can provide early warnings before a fall occurs, while maintaining high recall and high precision. Summary of the Invention

[0005] This invention provides a method and system for early warning of falls in the elderly based on incomplete sequences and physical verification, in order to solve the technical problems of existing technologies that cannot achieve early warning and have a high false alarm rate.

[0006] In a first aspect, the present invention provides a method for early warning of falls in the elderly based on incomplete sequences and physical verification, comprising:

[0007] S1. Obtain a natural standing image of the person under test during the initial stage of monitoring, extract the human skeletal representation to generate skeletal vector data containing joint node coordinate components, and calculate the reference standing height.

[0008] S2. Obtain the video stream containing the person to be tested, and cache it frame by frame as a continuous video segment of length N in a circular buffer with a capacity of N frames; after the buffer is full for the first time, update the segment and trigger a fall detection process every time a new frame is entered.

[0009] S3. Extract skeletal key points from each frame of the video clip to obtain the skeletal key point coordinate data corresponding to each frame. Normalize the skeletal key point coordinate data of all frames to generate the skeletal sequence corresponding to the video clip.

[0010] S4. Extract the first γ proportion of the subsequence from the bone sequence as the truncated early bone subsequence, and resample the truncated early bone subsequence into a standardized early sequence of a preset fixed length.

[0011] S5. A physical perception dual-flow graph convolutional network is constructed and trained using standardized early sequence inputs to obtain the initial fall probability of the test subject experiencing a fall event. The physical perception dual-flow graph convolutional network includes a posture flow and a motion flow set in parallel. The posture flow extracts spatial structure features using the coordinates of skeletal key points as input, and the motion flow extracts kinematic features using velocity and acceleration features calculated based on standardized early sequences as input.

[0012] S6. Based on the complete skeletal sequence corresponding to the video segment, calculate the physical state spatial characteristics of the person under test;

[0013] S7. Based on the initial fall probability and physical state space characteristics, perform hierarchical physical constraint posterior verification to obtain the judgment result of each frame of image;

[0014] S8. When the judgment results of multiple consecutive frames are all fall events, issue a fall warning message.

[0015] Furthermore, the specific process of S1 is as follows:

[0016] S11. Acquire multiple frames of images of the person under test in a natural standing position during the initial stage of monitoring;

[0017] S12. Use the pose estimation algorithm to extract 33 key points of the human body in each frame image, and map them into the skeletal key points and corresponding three-dimensional coordinates of the standard 25-joint skeleton topology according to the preset mapping rules.

[0018] S13. Calculate the height of the subject in each frame of the image based on the skeletal key points and their corresponding 3D coordinates. The calculation formula is as follows:

[0019]

[0020] in, Indicates body height; Indicates average ankle height coordinate; Indicates the vertex joint coordinate; and These represent the left and right ankle joints, respectively. coordinate;

[0021] S14. Sort the human height values ​​of each frame in descending order and take the average of the first K maximum values ​​as the reference standing height.

[0022] Furthermore, the normalization process in S3 specifically includes:

[0023] S31. Using the center coordinates of the hip joint in the first frame of the sequence as the origin, perform translation and normalization on the coordinates of all joints in all frames.

[0024] S32. Take the downward direction of the y-axis as positive, and invert the non-downward coordinates to unify the coordinate system into the image coordinate system.

[0025] Furthermore, in S4, the preset proportional parameter The value is 0.6, and the formula for calculating the number of frames truncated from the early skeletal subsequence is:

[0026] ;

[0027] in, To truncate the number of frames in the early skeletal subsequences; This represents the total number of frames in the video clip. This is the preset cutoff ratio parameter;

[0028] The truncated early sequence is resampled to a preset fixed length using a time-indexed resampling method to obtain a standardized early sequence.

[0029] Furthermore, the specific process by which the physical perception dual-flow graph convolutional network in S5 predicts the initial fall probability is as follows:

[0030] In the pose flow, the coordinates of the skeleton key points are taken as input, and spatial structure features are extracted through three layers of adaptive graph convolutional layers. After global average pooling, a high-dimensional spatial feature vector is output.

[0031] The velocity and acceleration features calculated from the standardized early sequence in the motion flow are concatenated, and kinematic features are extracted through three layers of adaptive graph convolutional layers. After global average pooling, a high-dimensional kinematic feature vector is output.

[0032] The high-dimensional spatial feature vector and the high-dimensional kinematic feature vector are concatenated in the channel dimension to form a comprehensive feature vector, and the initial fall probability is output through a two-layer fully connected network and the Softmax function.

[0033] Furthermore, during the training phase of the physical perception dual-flow graph convolutional network, the skeletal sequence corresponding to each training sample is truncated and resampled, and the resulting standardized early sequence is used as the network input for that training sample. A weighted random sampler is used to construct training batches, where the weight of each training sample is determined according to the number of samples of the category to which the sample belongs in the training set, and the smaller the number of samples of a category, the greater the corresponding weight. A weighted cross-entropy loss function is used to update the network parameters, and the weighted cross-entropy loss function assigns a greater loss weight to samples whose true label is the fall category than to samples whose true label is the normal category.

[0034] Furthermore, the construction process of the adaptive graph convolutional layer is as follows:

[0035] S711. Construct a 25×25 physical adjacency matrix based on the connection relationships of the human skeleton. ;

[0036] S712. Perform symmetric normalization on the adjacency matrix. The calculation formula is as follows:

[0037] ;

[0038] in, This represents the normalized adjacency matrix; Degree matrix;

[0039] S713. Introduce a learnable parameter matrix with the same shape as the physical adjacency matrix, and calculate the final adjacency matrix:

[0040] ;

[0041] in, Represents the final adjacency matrix; This represents the normalized adjacency matrix; Represents the learnable parameter matrix

[0042] S714. In a single operation of the adaptive graph convolutional layer, the feature vectors at each time step and at each skeletal keypoint in the input of this layer are first linearly mapped by a convolution with a kernel size of 1×1 to obtain the convolutional feature tensor used for aggregation. Then, the Einstein summation convention is used to weight and aggregate the final adjacency matrix and the convolutional feature tensor in the dimension of the skeletal keypoints to obtain the feature fusion information between adjacent skeletal keypoints as the output of this layer. Among them, the input of the first layer of the pose flow and motion flow is the tensor obtained by organizing the skeletal keypoint coordinates and the tensor obtained by concatenating and organizing the velocity features and acceleration features, respectively. The input of the second and third layers is the output of the previous adaptive graph convolutional layer.

[0043] Furthermore, the physical state space features include at least the final attitude tilt angle, final center of gravity height, height drop, minimum descent velocity, and the height of the last frame of the normalized early sequence;

[0044] The final tilt angle is calculated by taking the vector from the center of the shoulder joint to the center of the hip joint in the last frame and finding the angle with the vertical direction. The formula is as follows:

[0045] ;

[0046] ;

[0047] ;

[0048] in, Indicates the coordinates of the central joint of the shoulder; Represents the coordinates of the center of the hip joint; This represents the vector pointing from the hip joint to the center joint of the shoulder; ,in for The horizontal component; for vertical component for The depth component; To prevent division by zero of constants; The final tilt angle;

[0049] The final center of gravity height is taken as the normalized value of the shoulder center joint in the last frame. coordinate;

[0050] The height drop is calculated by subtracting the shoulder center height from the shoulder center height in the last frame. The formula is as follows:

[0051] ;

[0052] in, Indicates the magnitude of the drop; This indicates the central joint of the shoulder in the first frame. coordinate; This indicates the central joint of the shoulder in the last frame. coordinate;

[0053] The minimum descent rate is determined by performing frame-by-frame difference on the vertical coordinates (y) of at least two key joints on the complete skeletal sequence, obtaining the minimum value of the difference sequence for each joint, and taking the minimum of the minimum values; the at least two key joints include at least the center of the hip joint and the center of the shoulder joint.

[0054] The shoulder center height of the last frame of the normalized early sequence is: the shoulder center height (i.e., the y-coordinate value) of the last frame when the normalized early sequence is arranged in chronological order.

[0055] Furthermore, the process of obtaining the determination result of each frame image by performing hierarchical physical constraint posterior verification in S7 is specifically as follows:

[0056] S91, First layer of protective verification: If the initial fall probability is greater than the preset threshold and the physical state space characteristics meet the preset protection conditions, then it is directly determined as a fall;

[0057] S92, Second-level rejection check: If the initial fall probability is greater than the preset threshold but does not meet the protection conditions, then determine whether the physical state space characteristics meet the preset rejection rules. If they do, then the judgment result will be corrected to normal.

[0058] S93, Third-level rescue verification: If the initial fall probability is less than or equal to the preset threshold, determine whether the physical state space characteristics meet the preset rescue rules. If they do, the judgment result will be corrected to a fall.

[0059] Secondly, the present invention provides an early warning system for falls in the elderly based on incomplete sequences and physical verification, the system being used to perform the method described above, including:

[0060] Height calibration module: used to acquire natural standing images of the person under test in the initial stage of monitoring, extract human skeletal representation to generate skeletal vector data containing joint node coordinate components, and calculate reference standing height;

[0061] Data acquisition module: It is used to receive video streams containing the person to be tested frame by frame according to the video acquisition frame rate, and buffer them in a circular buffer with a capacity of N frames to obtain continuous video segments of length N; after the buffer is full for the first time, the segment is updated and a fall detection process is triggered every time a new frame is received.

[0062] Skeletal key point extraction module: used to extract skeletal key points from each frame of the video clip, obtain the skeletal key point coordinate data corresponding to each frame, and normalize the skeletal key point coordinate data of all frames to generate the skeletal sequence corresponding to the video clip.

[0063] Golden Window Truncation Module: Used to truncate the front end of a skeletal sequence. The proportionate subsequence is used as the truncated early skeletal subsequence, and the truncated early skeletal subsequence is resampled into a standardized early sequence of a preset fixed length;

[0064] Initial fall probability calculation module: This module is used to construct and train a physical perception dual-flow graph convolutional network based on standardized early sequence inputs to obtain the initial fall probability of the test subject experiencing a fall event. The physical perception dual-flow graph convolutional network includes a posture flow and a motion flow set in parallel. The posture flow extracts spatial structure features using the coordinates of skeletal key points as input, while the motion flow extracts kinematic features using velocity and acceleration features calculated based on standardized early sequences as input.

[0065] Physical state calculation module: used to calculate the physical state spatial characteristics of the person under test based on the complete skeleton sequence corresponding to the video segment;

[0066] The hierarchical verification module is used to: perform hierarchical physical constraint posterior verification based on the initial fall probability and physical state space characteristics to obtain the judgment result of each frame of image;

[0067] Warning output module: Used to issue a fall warning message when the judgment results of multiple consecutive frames of images are all fall events.

[0068] Furthermore, the early warning output module includes a state machine management unit and an early warning output unit:

[0069] The state machine management unit introduces a three-state state machine to manage the lifecycle of fall events. The states include a normal state, a falling state, and a fallen state. When the judgment result of three consecutive frames is a fall, the state machine switches from the normal state to the falling state and records the warning time. When the judgment result of three consecutive frames is normal and the posture has stabilized, the state machine switches from the falling state to the fallen state and calculates the early warning duration.

[0070] The early warning output unit is used to generate alarm information and push it to the target device; wherein, the alarm information includes the identifier of the acquisition device of the monitoring video stream, the alarm information generation time, the detection confidence level, and the alarm image list.

[0071] The initial fall probability calculation module includes a neural network processing unit. A physical perception dual-flow graph convolutional network is deployed in the neural network processing unit as an integer quantized neural network model and outputs the initial fall probability. The hierarchical verification module is executed by the central processing unit and obtains the judgment result. The acquisition of video segments, the generation of skeleton sequences, the calculation of initial fall probability and the generation of judgment results are all completed locally, without sending the image data corresponding to the video segments and the coordinate data corresponding to the skeleton sequences to the cloud server.

[0072] This invention proposes a method and system for early warning of falls in the elderly based on incomplete sequences and physical verification. Compared with existing technologies, the method has the following advantages:

[0073] First, the golden window truncation early warning mechanism: This invention proposes to use only the first 60% of the frames of the action sequence in both the model training and inference stages, without relying on the complete action sequence or the final state features of the action for detection. Through truncation training, the model learns only the features of the imbalance stage and the early stage of the fall. During inference, a sliding window is maintained to trigger truncation in real time, enabling an early warning to be issued before the fall process is completed. The sliding window in the inference stage is a circular buffer of length N frames, while a complete fall action usually lasts for much longer than N frames. When enough frames of fall imbalance and early fall have accumulated in the buffer, the earliest γ-proportion frames enqueued in the buffer are sent to the physical perception dual-flow graph convolutional network. At this time, the human landing stage has not yet occurred, and the system can output a fall judgment. Compared with the post-hoc method of waiting for the fall to be completely completed before detection, the entire fall action is detected about 40%–60% earlier.

[0074] Second, the heterogeneous dual-stream design of the physical perception dual-stream graph convolutional network: The two streams in this invention are completely heterogeneous in physical semantics - the attitude stream models the spatial structure with skeleton coordinates (3 channels) as input, and the motion stream models the kinematic features with velocity and acceleration (6 channels, i.e. first-order and second-order differences) as input. The features of the two streams are spliced ​​and fused for classification.

[0075] Third, the hierarchical physical constraint posterior decision mechanism (HPCPD): The physical constraints of this invention occur after the output of the neural network. Through three layers of serial posterior decision-making—protective verification, rejection verification, and rescue verification—the accuracy is significantly improved (from 67.9% to 90.67%) without changing the network structure. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a flowchart illustrating the method for early warning of falls in the elderly based on incomplete sequences and physical verification provided in an embodiment of the present invention.

[0078] Figure 2 This is a schematic diagram of 33 key points and 25 joints of the human body provided in this embodiment of the invention: Figure 2 (a) is a schematic diagram of 33 key points of the human body. Figure 2 (b) Several key conversion formulas for the process of converting 33 key points of the human body into 25 joint bones; Figure 2 (c) is a schematic diagram of the 25 joint bones;

[0079] Figure 3 This is a schematic diagram of the physical perception dual-flow graph convolutional network architecture provided in an embodiment of the present invention;

[0080] Figure 4 This is a physical connection diagram and a schematic diagram of the matrix structure of the human skeleton adjacency matrix provided in an embodiment of the present invention; wherein, Figure 4 (a) shows the topology of the 25-joint skeleton. Figure 4 (b) is a schematic diagram of the corresponding 25×25 adjacency matrix. Figure 4 (c) is a schematic diagram of the normalized adjacency matrix;

[0081] Figure 5 This is a schematic diagram of the training process of the physical perception dual-flow graph convolutional network provided in an embodiment of the present invention;

[0082] Figure 6 This is a schematic diagram of the decision-making process of the hierarchical physical constraint posterior decision-making mechanism provided in the embodiments of the present invention;

[0083] Figure 7 This is a schematic diagram of the state machine management unit performing three-state state machine transitions provided in an embodiment of the present invention;

[0084] Figure 8 This is a performance comparison chart provided in an embodiment of the present invention;

[0085] Figure 9 This is a warning effect diagram for daytime bright light scenarios provided by the present invention; wherein, Figure 9 (a) is an early warning map. Figure 9 (b) is a diagram of the stable state after the fall. Figure 9 (c) is the second early warning map. Figure 9 (d) shows the second diagram of the stable state after the fall;

[0086] Figure 10 This is an illustration of the early warning effect in a dark, no-light scenario provided by the present invention; wherein, Figure 10 (a) is an early warning map. Figure 10 (b) is a diagram of the stable state after the fall. Figure 10 (c) is the second early warning map. Figure 10 (d) is the second diagram of the stable state after the fall. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0088] Example 1

[0089] like Figure 1 As shown, this embodiment provides a method for early warning of falls in the elderly based on incomplete sequences and physical verification, including:

[0090] S1. Obtain a natural standing image of the person to be tested during the initial stage of monitoring, extract human skeletal representation from the image, generate skeletal vector data containing joint node coordinate components (excluding facial texture information that can individually identify an individual), and calculate a reference standing height from the skeletal vector data.

[0091] S11. Acquire multiple frames of images of the person under test in a natural standing position during the initial monitoring phase. Specifically, when the system starts or the person under test first enters the monitoring area, a continuous series of multiple frames of images of them in a natural standing position are acquired via a camera. The number of frames is not limited; in this embodiment, the number of frames is 30. The acquisition of these multiple frames is part of the human posture calibration sampling during the monitoring initialization phase, used to generate a reference standing height. This is functionally independent of the N-frame circular buffer used for fall detection in S2. The circular buffer is used for fall detection in the continuous video stream only after the reference height calibration is completed.

[0092] S12. Extract 33 key points of the human body in each frame of the image using a pose estimation algorithm, and map them to skeletal key points of a standard 25-joint skeleton topology according to a preset mapping rule, outputting the 3D coordinates of each skeletal key point. Specifically, the MediaPipe pose estimation algorithm is used to extract 33 key points of the human body in each frame of the image, such as... Figure 2 As shown in (a), these keypoints are mapped to a standard 25-joint skeleton format. The definition of 25 joints is as follows: Figure 2 As shown in (c), the specific components include: 0-center of the hip joint, 1-base of the spine, 2-neck, 3-top of the head, 4-left shoulder, 5-left elbow, 6-left wrist, 7-left fingertip, 8-right shoulder, 9-right elbow, 10-right wrist, 11-right fingertip, 12-left hip, 13-left knee, 14-left ankle, 15-left toe, 16-right hip, 17-right knee, 18-right ankle, 19-right toe, 20-center of the shoulder, 21-base of left fingers, 22-left fingers, 23-base of right fingers, 24-right thumb. The fusion calculation process is as follows: Figure 2 As shown in (b). More specifically, the mapping rules for mapping multiple keypoints to a standard 25-joint skeleton format are shown in Table 1:

[0093]

[0094] The above mapping rules ensure that keypoints obtained from different pose estimation libraries can be uniformly converted into a standard 25-joint format, guaranteeing the consistency of data format between edge deployment and offline training, and improving the accuracy of model inference. A schematic diagram of the keypoint-to-skeleton mapping is shown below. Figure 2 As shown.

[0095] S13. Calculate the height of the subject in each frame of the image based on skeletal key points. The calculation formula is as follows:

[0096] ;

[0097] ;

[0098] in, Indicates body height; Indicates average ankle height coordinate; Indicates the vertex joint coordinate; and These represent the left and right ankle joints, respectively. Coordinates. In the coordinate system output by the attitude estimation library, The downward axis is considered positive, and its value range is normalized to 0 to 1. Therefore, when standing, the ankle... Coordinates greater than head coordinate, The calculation result is positive. To prevent outliers, the calculation result is rounded down to the nearest whole number. .

[0099] S14. Arrange the human height values ​​of each frame in descending order, and take the average of the first K largest values ​​as the reference standing height. In this specific implementation, the human height values ​​of 30 frames are arranged in descending order, and the average of the first 10 largest values ​​is taken as the reference standing height. In the first 30 frames, there may be cases where the elderly person is not fully upright or there is noise. Taking the largest values ​​can more accurately estimate the body height when fully standing.

[0100] S2. Receive the video stream containing the person under test frame by frame according to the video acquisition frame rate, and buffer it in a circular buffer with a capacity of N frames to obtain a continuous video segment of length N. After the buffer is full for the first time, each new frame updates the segment and triggers a fall detection process. In this specific implementation, the video stream of the monitored area is continuously acquired by a camera, with a frame rate set to 30fps and a resolution of 640×480; N is 50, meaning the system maintains a circular buffer with a capacity of 50 frames (approximately 1.67 seconds). After the buffer is full for the first time (50 frames), it enters a continuous detection state: each time a new image is received, the oldest frame is popped out, the current frame is pushed in, and a complete detection process from S3 to S8 is immediately triggered, achieving real-time rolling detection triggered once per frame.

[0101] S3. Extract skeletal keypoints from each frame of the video clip to obtain the skeletal keypoint coordinate data corresponding to each frame. Normalize the skeletal keypoint coordinate data of all frames to generate the skeletal sequence corresponding to the video clip.

[0102] In practice, for each frame of the video clip, the skeletal key point extraction operation in S1 is repeated to obtain the skeletal key point coordinate data corresponding to each frame. Finally, an array of shape (3, total number of frames, 25, number of people) is returned, where 3 represents the three coordinate dimensions of x, y, and z, 25 represents the number of joints, and the number of people is at most 2. This constitutes the original skeletal sequence.

[0103] Normalization is performed:

[0104] S31. Using the coordinates of the hip joint center (index 0) of the first frame of the sequence as the origin, translate and normalize the coordinates of all joints in all frames, that is, subtract the coordinates of the hip joint in the first frame from each joint coordinate to eliminate the influence of absolute position.

[0105] S32. Take the downward direction of the y-axis as positive, and invert the non-downward coordinates to unify the coordinate system into the image coordinate system.

[0106] More specifically, the vertical axis is processed differently based on the data source, ensuring that the skeletal sequence input to the physical perception dual-flow graph convolutional network is consistent with the coordinate axis convention used during model training:

[0107] Model training phase: When reading sequences from offline skeletal data files such as NTU RGB+D, this embodiment only performs the translation normalization described in S31 (and can be combined with preprocessing such as taking the maximum value of each joint in the human dimension and fusing when there are multiple people), without inverting the y-axis coordinate, that is, keeping the coordinate axis agreed upon by the dataset file into the network training.

[0108] Real-time inference stage: When the skeletal keypoints come from an image-based pose estimation library (such as MediaPipe, where the y-axis is usually positive downwards in the image plane), after completing the translation normalization described in S31, the y-coordinates of all frames and all joints are reversed (equivalent to multiplying the y-component by -1) so that the vertical direction is consistent with the axis used in the training data, so that it can be used in conjunction with the trained model and the thresholds and rules in the hierarchical physical constraint posterior decision.

[0109] Finally, when performing offline evaluation using data from the same source as the training (e.g., directly reading NTU skeleton files without MediaPipe), if the data axes are already consistent with the training axes, there is no need to perform the above y-inversion again. This yields the skeleton sequence used for network input and physics rule calculation.

[0110] S4. Extract the preceding part from the skeletal sequence. The proportionate subsequence is used as the truncated early skeletal subsequence, and the truncated early skeletal subsequence is resampled into a standardized early sequence of a preset fixed length. This truncation operation is performed during both the model training and real-time inference phases, forcing the model to output prediction results only based on the early stages of the action sequence, thereby achieving early warning capability before the fall process is completed. Here, γ is a preset truncation proportion parameter, and γ < 1. The value of γ enables the system to issue a warning before the fall action is completely finished, achieving early detection of fall events. Maintaining consistency between the model training and real-time inference phases forces the model to learn features from the early stages of the fall (the imbalance phase and the early falling phase), rather than relying on the final posture information after the fall is completed. This achieves an early warning effect, issuing a warning before the fall is finished. During real-time inference, the skeletal sequence comes from a circular buffer of length N frames. The subsequence of the first γ proportion is taken from the earliest consecutive max(5,N×γ) frames enqueued in the buffer in chronological order. The selected subsequence corresponds to the kinematic segment accumulated first in the current observation window in temporal semantics. Since the actual duration of the fall is usually much longer than the duration covered by the N-frame buffer, when the system detects the trigger, the landing phase of the fall has not yet occurred, achieving an early warning effect, issuing a warning while the fall is still in progress.

[0111] Specifically, the preset proportional parameters The value is 0.6, and the formula for calculating the number of frames truncated from the early skeletal subsequence is:

[0112] ;

[0113] in, To truncate the number of frames in the early skeletal subsequences; This represents the total number of frames in the video clip. This is a preset cutoff ratio parameter. In this embodiment, With 50 frames, the number of frames in the truncated subsequence is calculated to be 30. This subsequence is the truncated early skeletal subsequence.

[0114] The truncated early sequence is resampled to a preset fixed length using a time-indexed resampling (nearest neighbor frame) method to obtain a standardized early sequence. In this specific implementation, the preset fixed length is set to 50 frames (this value is not limited and can be adjusted according to actual application needs). Then, the 30 frames of the truncated early skeletal subsequence are resampled to 50 frames using the time-indexed resampling (nearest neighbor frame) method, thus obtaining the standardized early sequence.

[0115] It is important to note that the N=50-frame circular buffer in the real-time inference phase is a sliding observation window that extends backward by approximately 1.67 seconds (at 30 frames / second) from the current moment, rather than representing the entirety of a single fall. Statistical analysis of fall actions in the NTU RGB+D dataset shows that a complete fall (from the start of imbalance to full landing and stabilization) typically lasts between 90 and 200 frames (approximately 3 to 7 seconds), far exceeding the buffer length.

[0116] The early warning mechanism in this embodiment works as follows: Before a fall occurs, the buffer contains skeletal frames of normal movements (walking, standing, etc.); as a fall occurs, fall frames gradually enter the buffer and push out earlier normal frames; when enough fall imbalance and early fall frames have accumulated in the buffer, the physical perception dual-flow graph convolutional network detection is triggered. At this time, the 30 earliest frames enqueued in the buffer in chronological order (i.e., the earliest accumulated fall frames) correspond to the imbalance stage and early fall stage of the fall; the landing stage of the fall has not yet occurred, and the human body is still in the process of falling.

[0117] The aforementioned inference truncation and training truncation employ the same truncation ratio γ and the same temporal resampling strategy in engineering implementation (in this embodiment, the first γ-ratio subsequence is truncated and resampled to a preset fixed length). Translation and normalization are performed using the hip joint of the first frame of each sequence segment as the origin, ensuring that the incomplete sequences fed into the network during the inference phase maintain consistency with those during the training phase in terms of preprocessing links and key hyperparameters. During the training phase, the first γ-ratio frames of offline samples (such as NTU RGB+D) correspond to the early dynamic processes unfolding over time within the clip from the start of the action annotation (primarily focusing on the early stages of imbalance and fall). During the real-time inference phase, the earliest enqueued consecutive frames are retrieved from the circular buffer. In the typical scenario where fall-related kinematic features dominate within the current N-frame window and the system triggers detection accordingly, this subsequence corresponds temporally to the early stages of the process and the non-final landing phase emphasized during training. This facilitates the reuse of early dynamic representations learned by the model within the golden window. It should be noted that when the buffer still contains many pre-fall background action frames, the earliest enqueued γ-proportion subsequence may not temporally correspond one-to-one with the first γ-proportion frames from the action annotation starting point in the offline clip. As the sliding window continues to update, and combined with the joint determination of network output and hierarchical physical constraint posterior decisions, this embodiment can still maintain an effective response to early fall dynamic features in practical applications.

[0118] This mechanism allows this embodiment to issue an early warning on average about 60% earlier in the entire fall process under bright daylight (see Table 5), and about 50% earlier in the dark at night (see Table 6). When the warning is issued, the human body is usually still in the process of falling and has not yet fully landed.

[0119] S5. A physical perception dual-flow graph convolutional network is constructed and trained using standardized early sequence inputs to obtain the initial fall probability of the test subject experiencing a fall event; wherein, the physical perception dual-flow graph convolutional network includes a posture flow and a motion flow set in parallel, the posture flow extracts spatial structure features using skeletal keypoint coordinates as input, and the motion flow extracts kinematic features using velocity features and acceleration features calculated based on standardized early sequences as input.

[0120] In the specific implementation of this embodiment, as follows: Figure 3 As shown, the pose flow uses 3-channel skeleton keypoint coordinates. As input, the shape of the pose flow is (3, 50, 25). First, it is batch normalized (BN), and then spatial structure features are extracted through three layers of adaptive graph convolutional layers (channel number 3→64→128→256). After global average pooling, a 256-dimensional high-dimensional spatial feature vector is output.

[0121] In the motion stream, the early standardized sequence is subjected to first and second difference to obtain velocity and acceleration features. The velocity and acceleration features are concatenated to obtain a 6-channel input (6, 50, 25). Kinematic features are extracted through three layers of adaptive graph convolutional layers (channel number 6→64→128→256). After global average pooling, a 256-dimensional high-dimensional kinematic feature vector is output.

[0122] More specifically, the acquisition of velocity features: for the first time dimension... frame( The velocity is defined as the current frame coordinates minus the previous frame coordinates, i.e.:

[0123]

[0124] For frame 0, the velocity is set to zero vector (zero-padding strategy), i.e. .

[0125] Acquisition of acceleration features: For the first acceleration feature in the time dimension frame( Acceleration is defined as the current frame velocity minus the previous frame velocity, i.e.:

[0126]

[0127] For frame 0, the acceleration is set to a zero vector (zero-fill strategy), i.e. Through the above differential calculation, the original 3-channel coordinates are expanded into 6-channel kinematic features (velocity x, velocity y, velocity z, acceleration x, acceleration y, acceleration), which are used as inputs to the motion flow.

[0128] The high-dimensional spatial feature vector and the high-dimensional kinematic feature vector are concatenated in the channel dimension to form a 512-dimensional comprehensive feature vector. The initial fall probability is output through a two-layer fully connected network (512→256→2) and the Softmax function (that is, the final output is the classification result corresponding to the two categories of "normal" and "fall".

[0129] The construction process of the adaptive graph convolutional layer is as follows:

[0130] S511. Construct a 25×25 physical adjacency matrix based on the connection relationships of the human skeleton. .

[0131] Based on the 25-joint index defined in S1, the bone connection edges are defined as follows: (0,1), (1,20), (20,2), (2,3), (20,4), (4,5), (5,6), (6,7), (7,21), (20,8), (8,9), (9,10), (10,11), (11,23), (0,12), (12,13), (13,14), (14,15), (0,16), (16,17), (17,18), (18,19). For example... Figure 4 The diagram shows the physical connection relationships and matrix structure of the human skeleton adjacency matrix. Figure 4 (a) shows a topological diagram of the 25-joint skeleton, connecting adjacent joints with lines (such as the hip joint connecting the spine, left and right legs, etc.), which visually demonstrates the physical structure of the human skeleton. Figure 4(b) is the corresponding 25×25 adjacency matrix. A value of 1 in the matrix indicates a physical connection between two joints, while a value of 0 indicates no direct connection. This adjacency matrix, after symmetric normalization, is used for information aggregation in graph convolutional networks, enabling the features of adjacent joints to be mutually transferred and fused, thereby better capturing the spatial structural features of human posture. The normalized adjacency matrix is ​​shown below. Figure 4 As shown in (c).

[0132] S512. To prevent eigenvalue divergence during graph convolution, the adjacency matrix is ​​symmetrically normalized. The calculation formula is as follows:

[0133] ;

[0134] in, This represents the normalized adjacency matrix; The degree matrix represents the sum of the elements in each row plus a minimum value. (Prevent division by zero)

[0135] S513. Based on the physical adjacency matrix, introduce a learnable parameter matrix with the same shape as the physical adjacency matrix, initialized to small values ​​close to zero (uniformly distributed in...). arrive (between), the final adjacency matrix is ​​calculated:

[0136] ;

[0137] in, Represents the final adjacency matrix; This represents the normalized adjacency matrix; This represents the learnable parameter matrix, initialized with small values ​​close to zero (uniformly distributed on...). arrive The graph convolution behavior of the network at the beginning is approximately dependent on the topology of the human physical skeleton, and the dynamic dependencies between joints are gradually learned as training progresses.

[0138] S514. In a single operation of the adaptive graph convolutional layer, the feature vectors at each time step and at each skeletal keypoint in the input of this layer are first linearly mapped by a convolution with a kernel size of 1×1 to obtain a convolutional feature tensor for aggregation. Then, the final adjacency matrix and the convolutional feature tensor are weighted and aggregated in the dimension of the skeletal keypoints using the Einstein summation convention to obtain the feature fusion information between adjacent skeletal keypoints as the output of this layer. Among them, the input of the first layer of the pose flow and the motion flow is the tensor obtained by organizing the skeletal keypoint coordinates and the tensor obtained by concatenating and organizing the velocity feature and the acceleration feature, respectively. The input of the second and third layers is the output of the previous adaptive graph convolutional layer.

[0139] Analysis of numerous fall videos reveals that falls typically unfold in three phases: the imbalance phase (first 30% of the time): the body's center of gravity begins to shift, and uncoordinated movements begin; posture changes are minimal, but speed and acceleration become abnormal. The descent phase (30%-70% of the time): the body drops rapidly, speed and acceleration reach their peak, and the torso begins to tilt noticeably. The landing phase (last 30% of the time): the body contacts the ground, and posture stabilizes, typically resulting in a horizontal lying position. Existing fall detection schemes largely rely on the final posture features of the landing phase (e.g., lying completely flat), thus only detecting falls after they have occurred and failing to provide early warnings. To achieve early fall prediction (rather than post-fall prediction), the training process of the physical perception dual-flow graph convolutional network utilizes only the first 60% of the action sequence (referred to as the "golden window"), forcing the model to learn features from the early imbalance and descent phases of the fall, rather than relying on the final posture information of the landing phase. In the real-time inference phase, the aforementioned three-stage structure also forms the physical basis for early warning: after the fall begins, the skeletal kinematic characteristics of the imbalance phase and the early stage of the fall (continuous shift of the center of gravity, abnormal increase in joint acceleration) first enter and fill the N-frame circular buffer; when the earliest accumulated first γ-proportion frames (30 frames in this embodiment) in the buffer have fully reflected these early characteristics, the model can output a fall determination—at this time, the landing phase (i.e., the final state of the fall) has not yet occurred in reality. This is the core physical mechanism of the incomplete sequence early warning of this invention: the judgment is based on the early dynamic characteristics of the fall that has already occurred, rather than the final static posture after the fall is completed.

[0140] During the model training phase, for each training sample, steps S2 to S4 are first executed to obtain its corresponding standardized early sequence. Simultaneously, the real-world behavior labels for the complete video segment corresponding to that sample are obtained. The input is fed into a physics-aware dual-flow graph convolutional network, and the network outputs the fall prediction probability. By minimizing and The network parameters are updated using a loss function (such as cross-entropy loss). The training process of a Physics-aware dual-flow graph convolutional network is as follows: Figure 5As shown. During training, the AdamW optimizer with weight decay (learning rate 0.002, weight decay 0.001 in this embodiment) and a cosine annealing learning rate scheduler (period equal to the number of training epochs) are used to achieve a stable model training process. For each training epoch, the training data loader is traversed, and forward propagation, loss calculation, backpropagation, and parameter update are performed for each batch. In this specific implementation, every two epochs, evaluation is performed on the test set, and recall, precision, and F1 score are calculated at different thresholds. The threshold with the highest F1 score while ensuring recall is greater than 88% is selected as the optimal threshold. When the evaluation results on the test set reach the target (recall greater than 90% and precision greater than 80%), the model training is considered complete.

[0141] The model employs a cost-sensitive loss function, which, in addition to the standard cross-entropy loss, modifies the loss function for positive samples (falling, Higher weights are assigned to mitigate the class imbalance between fallen samples and normal samples. The cost-sensitive loss function is calculated as follows:

[0142]

[0143] in, This represents the total number of samples within the batch. For the model to the first The output of each sample (Softmax probability for the two classes: falling / normal); This is a real label; For sample weights, when (When falling) ,when (Normal) time ; This is the standard multi-class cross-entropy loss function. By setting the weight of positive samples to 1.5, a larger gradient is applied to the fall samples during loss backpropagation, enabling the model to effectively learn fall features on imbalanced datasets.

[0144] Implement a weighted random sampler. The weighted random sampler and the AdamW optimizer described above are completely independent mechanisms, addressing class imbalance at the data sampling and parameter update levels, respectively. The weighted random sampler works as follows: First, calculate the number of samples in each class in the training set, and use the reciprocal of the frequency as the sampling weight for each sample, i.e.:

[0145]

[0146] in, For the first The sampling weights of each sample; The sample number in the training set to be assigned sampling weights; To iterate through all training samples in the training set, the sample indexes are defined, with values ​​ranging from 1 to 2. ,in The total number of samples in the training set; For the first The class labels of each training sample. For the first The class labels of each training sample; set Indicates the training set and the first Index of samples with the same category label The set that constitutes; The number of elements in the set is the number of elements in the set, which is related to the number of elements in the first set. The total number of training samples of the same class. When constructing a training batch through random sampling according to the above weights, the probability of a sample being selected is related to its weight. This is proportional to the expected frequency of minority class samples such as those that fall, thus mitigating the adverse effects of class imbalance on training.

[0147] The innovation of this invention lies in the simultaneous use of WeightedRandomSampler (data sampling layer) and cost-sensitive loss function (loss calculation layer, positive sample weights). The three-layer collaborative strategy of the AdamW optimizer (parameter update layer) and the AdamW optimizer (parameter update layer) achieves effective learning of minority class fall samples in an extremely imbalanced fall detection task.

[0148] This invention relies solely on skeletal keypoint data for both streams, without requiring video images. It achieves a heterogeneous design in the dimension of physical features: the posture stream models the spatial structure (the positional topology of each joint in the human body), and the motion stream models the kinematic state (joint velocity and acceleration). The two streams are independent and complementary in physical semantics. The concatenated feature vector simultaneously encodes information in two dimensions: "where is the person" and "how is the person moving." It is highly sensitive to kinematic anomalies (rapid velocity changes, acceleration peaks) in the early stages of a fall.

[0149] S6. Based on the complete skeletal sequence corresponding to the video segment, calculate the physical state spatial features of the person under test. The physical state spatial features include at least the final posture tilt angle, final center of gravity height, height drop, minimum descent velocity, and shoulder center height in the last frame of the early sequence.

[0150] Final pose tilt angle: Calculate the angle between the vector from the shoulder center joint (index 20) to the hip joint (index 0) and the vertical direction in the last frame. Specific calculation method:

[0151] ;

[0152] ;

[0153] ;

[0154] in, Represents the coordinates of the central shoulder joint (key point indexed as 20 in the 25-joint system); Represents the coordinates of the center of the hip joint (key point with index 0); This represents the vector pointing from the hip joint to the center of the shoulder (index 20); ,in, for The horizontal component (left and right directions within the camera image plane); for The vertical component (this invention uses a coordinate system where the y-axis is positive downwards, when standing) When the human body is lying horizontally , The smaller the value, the closer the body is to horizontal. for The depth component (camera optical axis direction); To prevent the minimum value of division by zero; The final tilt angle (unit: degrees) is when the human body is upright. When the human body lies horizontally .

[0155] Final shoulder center height: Take the center height of the shoulder joint in the last frame. Coordinates (normalized relative values). This value reflects the final position of the body's center of gravity; the smaller the value, the lower the body.

[0156] Drop height: The difference between the shoulder center height in the first frame and the shoulder center height in the last frame. The calculation formula is as follows:

[0157] ;

[0158] in, Indicates the magnitude of the drop; This indicates the central joint of the shoulder in the first frame. coordinate; This indicates the central joint of the shoulder in the last frame. Coordinates. This feature is used to determine whether there is a significant drop in altitude during the entire movement.

[0159] Minimum descent velocity: In the complete skeletal sequence (and, if necessary, in the standardized early sequence), the vertical coordinates (y) of at least two key joints are differencing frame by frame. The minimum descent velocity is taken as the minimum value among the minimum differences in the joint difference sequences. In this embodiment, the minimum descent velocity is taken as the minimum value among the minimum differences in the frame-by-frame (y) between the hip joint center (index 0) and the joint at index 20 (shoulder center). The smaller this descent velocity value (the larger the negative value), the more intense the downward motion, which is a key indicator for judging the impact force of a fall.

[0160] Shoulder center height in the last frame of the standardized early sequence: This is the (y) coordinate of the shoulder center joint in the last frame of the standardized early sequence in the time dimension (consistent with the network input and the last frame of the early tensor used in the rejection rule of this embodiment). This feature is used in the rejection rule to determine whether the human shoulder is still in a high position at the end of the early observation time axis after resampling.

[0161] It also includes the minimum shoulder center height: the minimum y-coordinate of the shoulder center joint across all frames in the complete sequence. This feature is used in rescue rules to capture situations where the shoulder center reaches a very low position during a fall (even if there is some rebound at the end).

[0162] S7. Based on the initial fall probability and the physical state space characteristics, perform hierarchical physical constraint posterior verification to obtain the judgment result of each frame image.

[0163] The process of obtaining the judgment result for each frame of image by performing hierarchical physical constraint posterior verification is as follows:

[0164] S71, First layer of protective verification: If the initial fall probability is greater than the preset threshold (0.4 in this embodiment) and the physical state space characteristics meet the preset protection conditions, it is directly determined to be a fall; In the specific implementation of this embodiment, the preset protection conditions are that the final posture is obviously horizontal (i.e., the final posture tilt angle is >60 degrees) and obviously lower (i.e., the final shoulder center height is <0.08), and there is obvious height drop (i.e., height drop amount is >0.15) or obvious impact (i.e., the minimum descent speed of the complete sequence is <-0.03).

[0165] S72, Second-level rejection check: If the initial fall probability is greater than the preset threshold but does not meet the protection conditions, then determine whether the physical state space characteristics meet the preset rejection rules. If they do, then the judgment result will be corrected to normal.

[0166] In the specific implementation of this embodiment, the preset veto rules are shown in Table 2:

[0167]

[0168] The purpose of this layer is to filter false positives from the Physical Aware Dual Flow Graph Convolutional Network. If any of the above rejection rules are met, the Physical Aware Dual Flow Graph Convolutional Network's alarm is rejected, and the output judgment result is 0 (normal); otherwise, the model's alarm result is maintained, and the output judgment result is 1 (fall). Through multi-layered physical rules, various false alarm patterns are accurately identified, thereby significantly improving the accuracy (90.67%).

[0169] S73, Third-level rescue verification: If the initial fall probability is less than or equal to the preset threshold, determine whether the physical state space characteristics meet the preset rescue rules. If they do, the judgment result will be corrected to a fall.

[0170] In the specific implementation of this embodiment, the preset rescue rules are shown in Table 3:

[0171]

[0172] Note: The minimum descent velocity mentioned in Table 3 above refers to the minimum of the frame-by-frame difference between the hip joint center (index 0) and the shoulder center (index 20).

[0173] The purpose of this layer is to capture true falls (false negatives) missed by the Physically Aware Dual Flow Graph Convolutional Network (HPCPD). If any of the above rescue rules are met, a fall is detected, the output result is 1 (fall), and the triggered rule number is recorded; otherwise, the model remains inactive, and the output result is 0 (normal). By capturing true falls missed by the HPCPD through the rescue verification layer, especially those falls whose features are not obvious in the early stages but become obvious in later stages, the recall rate is further improved. The complete decision-making process of the Hierarchical Physically Constrained Posterior Decision Mechanism (HPCPD) is as follows: Figure 6 As shown.

[0174] Priority levels: First-level protective check > Second-level rejection check > Third-level rescue check.

[0175] More specifically, the process for obtaining the preset threshold for comparing the initial fall probabilities is as follows:

[0176] Calculate true positives based on the final determination results and the true labels. (A real fall is judged as a fall), false positive (Normal diagnosis: fall) False negative (Falling is judged as normal), then the recall, precision, and F1 composite score are calculated:

[0177]

[0178]

[0179]

[0180] Within a fall probability threshold range of 0.15 to 0.80, a scan was performed with a step size of 0.05. S7 was repeated for each threshold, and the calculation was performed to find the optimal threshold that simultaneously satisfies a recall rate greater than or equal to 99% and a precision rate greater than or equal to 90%. Ultimately, 0.40 was determined as the optimal threshold, achieving excellent performance with a recall rate of 99.27%, a precision rate of 90.67%, and an F1 composite score of 94.77.

[0181] S8. When the determination result of multiple consecutive frames is a fall event, a fall warning message is issued. In specific implementation of this embodiment, a fall warning message is issued when the determination result of 3 consecutive frames is a fall event.

[0182] Example 2

[0183] This embodiment provides an early warning system for falls in the elderly based on incomplete sequences and physical verification. The system is used to execute the method described above, including:

[0184] The height calibration module is used to acquire images of the subject standing naturally during the initial monitoring phase, extract human skeletal representations to generate skeletal vector data containing joint node coordinate components, and calculate a reference standing height. Specifically, it is used to acquire multiple consecutive frames of visible light or infrared images during the monitoring initialization phase (when the subject is standing naturally and the system has not yet entered or fully relied on the fall detection logic). For each frame, a human posture estimation algorithm is called to obtain human key points, which are then converted into a 25-joint skeletal format consistent with subsequent fall detection using a unified mapping rule. The single-frame body height scalar is calculated from the y-coordinates of the head joint and both ankle joints (or their equivalent vertical direction definition). After the accumulated frame count reaches a preset value, several larger values ​​of the body height scalar sequence are selected in descending order and averaged to obtain the reference standing height. This reference standing height is used to compare the body height within the current observation window with the standing baseline (e.g., to calculate the relative fall ratio) to suppress false alarms such as posture resembling a fall but with no significant decrease in overall height.

[0185] Data acquisition module: Receives video streams containing the person to be tested frame by frame according to the video acquisition frame rate, and buffers them in a circular buffer with a capacity of N frames to obtain continuous video segments of length N; after the buffer is full for the first time, each new frame updates the segment and triggers a fall detection process; where N is the preset buffer length; where the video segment contains N consecutive frames; in specific implementation of this embodiment, a visible light camera and an infrared camera can be used to acquire the video stream (such as an 850nm dual-pass camera with a light sensor, which automatically switches to a visible light camera in strong light environments and to an 850nm narrowband infrared camera in low light environments to support monitoring in low light environments such as night).

[0186] Skeletal key point extraction module: used to extract skeletal key points from each frame of the video clip, obtain the skeletal key point coordinate data corresponding to each frame, and normalize the skeletal key point coordinate data of all frames to generate the skeletal sequence corresponding to the video clip.

[0187] Golden Window Truncation Module: Extracts the preceding part of the skeletal sequence The proportion of the subsequence is used as the truncated early skeletal subsequence, and the truncated early skeletal subsequence is resampled into a standardized early sequence of a preset fixed length; the truncation operation is performed in both the model training stage and the real-time inference stage, forcing the model to output prediction results only based on the early stage of the action sequence, thereby achieving the early warning capability of issuing a warning before the fall process is completed; where γ is a preset truncation proportion parameter and γ<1.

[0188] Initial fall probability module: This module is used to construct and train a physical perception dual-flow graph convolutional network based on standardized early sequence inputs to obtain the initial fall probability of a test subject experiencing a fall event. The physical perception dual-flow graph convolutional network includes a posture flow and a motion flow set in parallel. The posture flow extracts spatial structure features using skeletal keypoint coordinates as input, and the motion flow extracts kinematic features using velocity and acceleration features calculated based on standardized early sequences as input.

[0189] Physical state calculation module: used to calculate the physical state spatial characteristics of the person under test based on the complete skeleton sequence corresponding to the video segment;

[0190] Layered verification module: Based on the initial fall probability and the physical state space characteristics, it performs layered physical constraint posterior verification to obtain the judgment result of each frame of image;

[0191] Warning output module: Used to issue a fall warning message when the judgment results of multiple consecutive frames of images are all fall events.

[0192] Furthermore, the early warning output module includes a state machine management unit and an early warning output unit:

[0193] The state machine management unit introduces a three-state state machine to manage the lifecycle of fall events. The states include a normal state, a falling state, and a fallen state. When the judgment result of three consecutive frames is a fall, the state machine switches from the normal state to the falling state and records the warning time. When the judgment result of three consecutive frames is normal and the posture has stabilized, the state machine switches from the falling state to the fallen state and calculates the early warning duration.

[0194] The warning output unit is used to generate alarm information and push it to the target device; wherein, the alarm information includes the identification of the acquisition device of the monitoring video stream, the alarm information generation time, the detection confidence level, and the list of alarm images.

[0195] Furthermore, in this embodiment, the end-side real-time deployment and model quantization are as follows:

[0196] Step 1: Model export. Export the trained physical perception two-stream graph convolutional network model from the PyTorch format to the Open Neural Network Exchange format (ONNX), and verify the correctness of the export with an example input tensor (shape (1,3,50,25)).

[0197] Step 2: INT8 quantization. Use the RKNN toolchain supporting RK3568 to perform 8-bit integer (INT8) quantization on the ONNX model. The quantization process uses a mixed sample of falls and normal actions as the calibration data set. The volume of the quantized model is compressed to about 1 / 4 of the original floating-point model, and the inference speed is increased by about 3 times. After verification, the F1 comprehensive score drops by no more than 2% compared with the floating-point model (the actual drop is about 0.5%), meeting the accuracy requirements.

[0198] Step 3: NPU inference integration. On the end-side intelligent computing device (such as the RK3568 AI processor), the physical perception two-stream graph convolutional network inference task is scheduled to be executed on the NPU hardware, and the computationally intensive matrix operations are completed by the dedicated hardware circuit of the NPU; the conditional judgment logic of the hierarchical physical constraint posterior decision mechanism (HPCPD) is executed on the CPU, and the two cooperate in parallel to achieve real-time detection at 30 frames per second. The end-side deployment architecture of the present invention realizes the true "all processing on the local device and zero data transmission": the video frames collected by the camera complete the entire process of pose estimation, skeleton extraction, GCN inference, HPCPD decision-making, and state machine management within the end-side device, without uploading the video frames, skeleton data, or detection results to the cloud server, which not only eliminates the privacy risk of video surveillance data leakage but also avoids the costs and network latency of cloud computing, enabling the system to operate reliably in a network-free environment.

[0199] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0200] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the controller described in any of the foregoing embodiments, such as the controller's hard drive or memory. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the controller. Further, the readable storage medium can include both the controller's internal storage unit and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0201] Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0203] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0204] To further illustrate the effects of the technology presented in this application and to verify the effectiveness of the invention, comparative experiments were conducted using the NTURGB+D public dataset. The test set included 274 fall samples and a corresponding number of normal action samples (including walking, sitting, standing, bending over, kneeling, and other actions easily confused with falls). The comparison scheme included:

[0205] Baseline Scheme 1: Pre-detection pure physical perception dual-flow graph convolutional network baseline

[0206] Solution Description: This solution uses the same golden window truncation (first 60% of frames) and physical-aware dual-flow graph convolutional network architecture as the present invention, but does not use a hierarchical physical constraint posterior decision mechanism. It relies solely on the output of the physical-aware dual-flow graph convolutional network for decision-making (without using HPCPD).

[0207] Training configuration: 40 training epochs, using cost-sensitive loss and weighted random sampling.

[0208] Feature analysis: Due to the lack of physical constraints, the accuracy rate is low (it is difficult to distinguish normal movements such as squatting and bending over), and in the early stages, it is easy to miss or falsely report due to the lack of obvious features.

[0209] Baseline Scenario 2: Post-hoc detection of traditional baselines

[0210] Solution Description: Use the complete action sequence (100% frames) to perform detection based on the final pose features after the action ends;

[0211] Detection logic: When the final posture angle is close to horizontal (lying flat), it is determined as a fall;

[0212] Feature analysis: The recall rate is relatively high (because the final state of a fall is usually lying flat), but it cannot provide early warning; the false alarm rate for normal lying-flat actions such as sleeping, lying down, and resting is extremely high, and the accuracy rate is very low.

[0213] The solution of this invention: a physical perception dual-flow graph convolutional network + hierarchical physical constraint posterior decision (HPCPD)

[0214] Solution Description: Use a golden window truncation (first 60% of frames) + a physical awareness dual-flow graph convolutional network + three layers of physical checks (protection layer + rejection layer + rescue layer).

[0215] Fall probability determination threshold: 0.40 (the optimal value determined through threshold scanning);

[0216] Rescue rule trigger statistics: Rescue rule 1 was triggered 7 times, rescue rule 4 was triggered 4 times, and rescue rule 5 was triggered 5 times.

[0217] The performance comparison results of the three schemes are shown in Table 4 below:

[0218]

[0219] The following conclusions can be drawn from the above experimental results:

[0220] a. Recall rate comparison: The proposed solution (99.27%) is significantly better than the baseline of pre-detection pure physical perception dual-flow graph convolutional network (92.0%) and post-detection baseline (95.26%). The rescue verification layer successfully captured 16 real fall samples that were missed by the physical perception dual-flow graph convolutional network (rescue rule 1 + rescue rule 4 + rescue rule 5 were triggered a total of 16 times), with only 2 false negatives (missed reports).

[0221] b. Accuracy Comparison: The proposed solution (90.67%) significantly outperforms the baseline of the pre-detection pure physical perception dual-flow graph convolutional network (67.9%, an improvement of 22.77 percentage points) and the post-detection baseline (34.34%, an improvement of 56.33 percentage points). The rejection verification layer effectively filters out normal actions such as bending over, squatting, and sitting down, which are easily misjudged by the physical perception dual-flow graph convolutional network.

[0222] c. F1 overall score comparison: The proposed solution (94.77) is 16.67 and 44.29 higher than the baseline of the pre-detection pure physical perception dual-flow graph convolutional network (78.1) and the post-detection baseline (50.48), respectively, and the overall performance is significantly improved.

[0223] d. Early warning capability: While maintaining extremely high detection accuracy, this invention can complete the early warning using only 60% of the frames before the action, which saves about 40% of the early warning time compared to the post-detection baseline.

[0224] In summary, the experimental results demonstrate that this invention, through the deep integration of a physically-aware dual-flow graph convolutional network and a hierarchical physical constraint posterior decision mechanism, successfully resolves the contradiction between "early warning" and "high-precision detection," achieving a breakthrough in simultaneously meeting the stringent objectives of recall ≥99% and precision ≥90%. For detailed performance comparisons, please refer to [link to performance comparison]. Figure 7 .

[0225] In addition, verification was also conducted in bright daylight environments and dark nighttime scenarios, as detailed below:

[0226] The verification results under high daylight intensity are shown in Table 5. The data in the table show that the early warning system of the present invention can detect the fall trend 2 to 5 seconds in most events, demonstrating significant early detection capability and facilitating early notification and handling. Throughout the entire process from "starting to fall" to "stabilizing the fall", the system can issue an early warning on average in about 64% of the time period (calculated by averaging the fall probabilities of the network output in Table 6), demonstrating excellent early detection coverage.

[0227]

[0228]

[0229] The verification results in a dark, nighttime scenario are shown in Table 6. The data in the table indicate that the early warning system of this invention can detect fall trends 0.3 to 2.9 seconds in advance in most events in a dark, nighttime scenario, demonstrating significant early detection capability. Throughout the entire process from "starting to fall" to "stabilizing the fall," the system can issue an early warning in approximately 50% of the time periods (calculated by averaging the fall probabilities from the network output in Table 6), demonstrating good early detection coverage.

[0230]

[0231]

[0232] It should be noted that: Tables 5 and 6 above contain a small number of test cases with extremely rapid descent speed and very short descent duration (such as events 2, 5, and 6 in Table 5). For special cases where the total duration is shorter than the buffer capacity (i.e., the fall process is extremely rapid), this system, relying on the high sensitivity of the physical perception dual-flow graph convolutional network to extreme kinematic features (sharp acceleration peaks) and the physical rules of the HPCPD layer as a fallback, can still issue an early warning before landing stabilization, thus constituting a useful supplement to the main mechanism.

[0233] Based on the above test results and actual verification results, it is fully demonstrated that, compared with general fall detection solutions, the technical solution of this application has the following specific advantages:

[0234] (1) Achieving early prediction of fall detection: By using only 60% of the frames before the fall process for training and inference, the model is forced to learn the features of the early stage of the fall, thus achieving early warning capability (the early warning time usually accounts for 50%-60% of the entire fall process), which buys valuable time for timely rescue. At the same time, through the physical perception dual-flow graph convolutional network (simultaneously modeling spatial structure and kinematic features) and the hierarchical physical constraint posterior decision mechanism (physical perception dual-flow graph convolutional network + three-layer physical verification hybrid decision), it achieves excellent performance of 99.27% ​​recall, 90.67% precision and 94.77 F1 score in the early prediction scenario, which is significantly better than the post-detection baseline (95.26% recall, only 34.34% precision) and the pure physical perception dual-flow graph convolutional network baseline (approximately 92% recall, approximately 67.9% precision, and approximately 78.1 F1 score).

[0235] (2) Efficient Decision-Making Fusion of Physical Rules and Physically Perceived Dual-Stream Graph Convolutional Network: Based on the physical laws of human movement, a human posture-kinematic state space theory is proposed, mapping the human state to a physical state space composed of four dimensions: posture tilt angle, center of gravity height, descent speed, and drop height, and dividing it into safe zone, danger zone, and fuzzy zone. Based on this theoretical framework, a hierarchical physical constraint posterior decision mechanism (HPCPD) is designed. Through a three-layer progressive decision-making process of protective verification layer, rejection verification layer, and rescue verification layer, a deep fusion of the physically perceived dual-stream graph convolutional network and physical rules is achieved. This method significantly improves precision (90.67%) while maintaining a high recall rate (99.27%), effectively solving the problem of false alarms for normal actions such as bending over, kneeling, and sitting down quickly by the pure physically perceived dual-stream graph convolutional network method, as well as the problem that post-event detection methods cannot predict in advance.

[0236] (3) All-weather applicability independent of image color: Based on the detection of key points of the human skeleton, it does not rely on the color information of the image. It uses a visible light camera and an infrared camera, and uses a light sensor to detect the ambient light brightness. The visible light camera is used during the day or when the light is sufficient, and it automatically switches to a narrowband 850nm infrared camera at night or when the light is dim. This design not only increases the spatiotemporal applicability of the system (it can work 24 hours a day), but also protects personal privacy in dark privacy scenarios such as at night (the infrared image does not show the detailed features of the human body, but only the outline).

[0237] (4) Privacy Protection Features: The fall detection process of this invention relies solely on the three-dimensional coordinate sequence of key points on the human skeleton, without processing color information, texture information, facial features, or clothing features of the video image. Specifically, the MediaPipePose algorithm completes pose estimation locally on the edge device, transmitting only the coordinate data of 33 key points to the downstream fall detection module. Subsequent skeleton sequence preprocessing, physical perception dual-flow graph convolutional network inference, and hierarchical physical constraint posterior decision mechanism are all based on this coordinate data. Regardless of whether it is in daytime visible light mode or nighttime infrared mode, the system always processes only the abstract coordinates representing the human motion posture, rather than the image content that can identify the human face. All processing is completed locally on the edge intelligent computing device, without uploading any video frames or skeleton data to the cloud server, fundamentally eliminating the risk of leakage of human face data in video surveillance. This makes this invention particularly suitable for all-weather, seamless deployment in privacy-sensitive locations such as nursing homes, hospitals, and homes.

Claims

1. A method for early warning of falls in the elderly based on incomplete sequences and physical verification, characterized in that, include: S1. Obtain a natural standing image of the person under test during the initial stage of monitoring, extract the human skeletal representation to generate skeletal vector data containing joint node coordinate components, and calculate the reference standing height. S2. Receive the video stream containing the person to be tested frame by frame according to the video acquisition frame rate, and buffer it in a circular buffer with a capacity of N frames to obtain a continuous video segment of length N; after the buffer is full for the first time, update the segment and trigger a fall detection process every time a new frame is received. S3. Extract skeletal key points from each frame of the video clip to obtain the skeletal key point coordinate data corresponding to each frame. Normalize the skeletal key point coordinate data of all frames to generate the skeletal sequence corresponding to the video clip. S4. Extract the first γ proportion subsequence from the skeletal sequence as the truncated early skeletal subsequence, and resample the truncated early skeletal subsequence into a standardized early sequence of a preset fixed length; the same γ and the same resampling rules are used for truncation and resampling during the training and inference phases. S5. Input the standardized early sequence into the trained physical perception dual-flow graph convolutional network to obtain the initial fall probability of the test subject experiencing a fall event; wherein, the physical perception dual-flow graph convolutional network includes parallel posture flow and motion flow; the posture flow extracts spatial structure features with the coordinates of skeletal key points as input, and the motion flow extracts kinematic features with the velocity and acceleration features calculated based on the standardized early sequence as input. S6. Based on the complete skeletal sequence corresponding to the video segment, calculate the physical state spatial characteristics of the person under test; S7. Based on the initial fall probability and physical state space characteristics, perform hierarchical physical constraint posterior verification to obtain the judgment result of each frame of image; S8. When the judgment results of multiple consecutive frames are all fall events, issue a fall warning message.

2. The method according to claim 1, characterized in that, The specific process of S1 is as follows: S11. Acquire multiple frames of images of the person under test in a natural standing position during the initial stage of monitoring; S12. Use the pose estimation algorithm to extract 33 key points of the human body in each frame image, and map them to the skeletal key points of the standard 25-joint skeleton topology according to the preset mapping rules, and output the three-dimensional coordinates corresponding to the skeletal key points. S13. Calculate the height of the subject in each frame of the image based on the three-dimensional coordinates corresponding to the skeletal key points. The calculation formula is as follows: ； in, Indicates body height; Indicates average ankle height coordinate; Indicates the vertex joint coordinate; and These represent the left and right ankle joints, respectively. coordinate; S14. Sort the human height values ​​of each frame in descending order and take the average of the first K maximum values ​​as the reference standing height.

3. The method according to claim 1, characterized in that, The normalization process in S3 specifically includes: S31. Using the center coordinates of the hip joint in the first frame of the sequence as the origin, perform translation and normalization on the coordinates of all joints in all frames. S32. Take the downward direction of the y-axis as positive, and invert the non-downward coordinates to unify the coordinate system into the image coordinate system.

4. The method according to claim 1, characterized in that, In S4, the preset proportional parameter The value is 0.6, and the formula for calculating the number of frames truncated from the early skeletal subsequence is: ； in, To truncate the number of frames in the early skeletal subsequences; This represents the total number of frames in the video clip. The preset truncation ratio parameter is used; the truncated early sequence is resampled to a preset fixed length using a time-indexed resampling method to obtain a standardized early sequence.

5. The method according to claim 1, characterized in that, The specific process by which the physical perception dual-flow graph convolutional network in S5 predicts the initial fall probability is as follows: In the pose flow, the coordinates of the skeleton key points are taken as input, and spatial structure features are extracted through three layers of adaptive graph convolutional layers. After global average pooling, a high-dimensional spatial feature vector is output. The velocity and acceleration features calculated from the standardized early sequence in the motion flow are concatenated, and kinematic features are extracted through three layers of adaptive graph convolutional layers. After global average pooling, a high-dimensional kinematic feature vector is output. The high-dimensional spatial feature vector and the high-dimensional kinematic feature vector are concatenated in the channel dimension to form a comprehensive feature vector, and the initial fall probability is output through a two-layer fully connected network and the Softmax function.

6. The method according to claim 1, characterized in that, During the training phase of the physical perception dual-flow graph convolutional network, the skeletal sequence corresponding to each training sample is truncated and resampled, and the resulting standardized early sequence is used as the network input for that training sample. A weighted random sampler is used to construct training batches, where the weight of each training sample is determined according to the number of samples of the category to which the sample belongs in the training set, and the smaller the number of samples of a category, the greater the corresponding weight. The weighted cross-entropy loss function is used to update the network parameters, and the weighted cross-entropy loss function assigns a greater loss weight to samples whose true label is the fall class than to samples whose true label is the normal class.

7. The method according to claim 5, characterized in that, The construction process of the adaptive graph convolutional layer is as follows: S711. Construct a 25×25 physical adjacency matrix based on the connection relationships of the human skeleton. ; S712. Perform symmetric normalization on the adjacency matrix. The calculation formula is as follows: ； in, This represents the normalized adjacency matrix; Degree matrix; S713. Introduce a learnable parameter matrix with the same shape as the physical adjacency matrix, and calculate the final adjacency matrix: ； Represents the final adjacency matrix; This represents the normalized adjacency matrix; Represents the learnable parameter matrix; S714. In a single operation of the adaptive graph convolutional layer, the feature vectors at each time step and at each skeletal keypoint in the input of this layer are first linearly mapped by a convolution with a kernel size of 1×1 to obtain the convolutional feature tensor used for aggregation. Then, the Einstein summation convention is used to weight and aggregate the final adjacency matrix and the convolutional feature tensor in the dimension of the skeletal keypoints to obtain the feature fusion information between adjacent skeletal keypoints as the output of this layer. Among them, the input of the first layer of the pose flow and motion flow is the tensor obtained by organizing the skeletal keypoint coordinates and the tensor obtained by concatenating and organizing the velocity features and acceleration features, respectively. The input of the second and third layers is the output of the previous adaptive graph convolutional layer.

8. The method according to claim 1, characterized in that, The physical state space features include at least the final attitude tilt angle, final center of gravity height, height drop, minimum descent velocity, and the height of the last frame of the normalized early sequence; The final tilt angle is calculated by taking the vector from the center of the shoulder joint to the center of the hip joint in the last frame and finding the angle with the vertical direction. The formula is as follows: ； ； ； in, Indicates the coordinates of the central joint of the shoulder; Represents the coordinates of the center of the hip joint; This represents the vector pointing from the hip joint to the center joint of the shoulder; for The horizontal component; for The vertical component; for The depth component; To prevent division by zero of constants; The final tilt angle; The final center of gravity height is taken as the normalized value of the shoulder center joint in the last frame. coordinate; The height drop is calculated by subtracting the shoulder center height from the shoulder center height in the last frame. The formula is as follows: ； in, Indicates the magnitude of the drop; This indicates the central joint of the shoulder in the first frame. coordinate; This indicates the central joint of the shoulder in the last frame. coordinate; The minimum descent rate is determined by performing frame-by-frame difference on the vertical coordinate values ​​of at least two key joints on the complete skeletal sequence, obtaining the minimum value of each joint difference sequence, and taking the smallest of the minimum values; the at least two key joints include at least the center of the hip joint and the center of the shoulder joint. The shoulder center height in the last frame of the normalized early sequence is the shoulder center height in the last frame when the normalized early sequence is arranged in chronological order, i.e., the y-coordinate value of the shoulder center.

9. The method according to claim 1, characterized in that, The process of obtaining the determination result of each frame image by performing hierarchical physical constraint posterior verification in S7 is as follows: S91, First layer of protective verification: If the initial fall probability is greater than the preset threshold and the physical state space characteristics meet the preset protection conditions, then it is directly determined as a fall; S92, Second-level rejection check: If the initial fall probability is greater than the preset threshold but does not meet the protection conditions, then determine whether the physical state space characteristics meet the preset rejection rules. If they do, then the judgment result will be corrected to normal. S93, Third-level rescue verification: If the initial fall probability is less than or equal to the preset threshold, determine whether the physical state space characteristics meet the preset rescue rules. If they do, the judgment result will be corrected to a fall.

10. A fall early warning system for the elderly based on incomplete sequences and physical verification, the system being used to execute the method according to any one of claims 1-9, characterized in that, include: Height calibration module: used to acquire natural standing images of the person under test in the initial stage of monitoring, extract human skeletal representation to generate skeletal vector data containing joint node coordinate components, and calculate reference standing height; Data acquisition module: used to acquire video streams containing the person to be tested, and caches consecutive video segments of length N frame by frame in a circular buffer with a capacity of N frames; after the buffer is full for the first time, the segment is updated and a fall detection process is triggered every time a new frame is entered; Skeletal key point extraction module: used to extract skeletal key points from each frame of the video clip, obtain the skeletal key point coordinate data corresponding to each frame, and normalize the skeletal key point coordinate data of all frames to generate the skeletal sequence corresponding to the video clip. Golden Window Truncation Module: Used to truncate the front end of a bone sequence. The proportionate subsequence is used as the truncated early skeletal subsequence, and the truncated early skeletal subsequence is resampled into a standardized early sequence of a preset fixed length; Initial fall probability calculation module: This module is used to input the standardized early sequence into a trained physical perception dual-flow graph convolutional network to obtain the initial fall probability of the test subject experiencing a fall event. The physical perception dual-flow graph convolutional network includes a posture flow and a motion flow set in parallel. The posture flow extracts spatial structure features with the coordinates of skeletal key points as input, and the motion flow extracts kinematic features with velocity and acceleration features calculated based on the standardized early sequence as input. Physical state calculation module: used to calculate the physical state spatial characteristics of the person under test based on the complete skeleton sequence corresponding to the video segment; The hierarchical verification module is used to: perform hierarchical physical constraint posterior verification based on the initial fall probability and physical state space characteristics to obtain the judgment result of each frame of image; Warning output module: Used to issue a fall warning message when the judgment results of multiple consecutive frames of images are all fall events.

11. The system according to claim 10, characterized in that, The early warning output module includes a state machine management unit and an early warning output unit: The state machine management unit introduces a three-state state machine to manage the lifecycle of fall events, including the normal state, the state in the process of falling, and the state after falling. When three consecutive frames indicate a fall, the state machine switches from the normal state to the falling state and records the warning time. When the judgment result of three consecutive frames is normal and the posture is stable, the state machine switches from the falling state to the fallen state and calculates the early warning duration. The early warning output unit is used to generate alarm information and push it to the target device; the alarm information includes the identification of the acquisition device of the monitoring video stream, the alarm information generation time, the detection confidence level, and the list of alarm images.

12. The system according to claim 10, characterized in that, The initial fall probability calculation module includes a neural network processing unit. The physical perception dual-flow graph convolutional network is deployed in the neural network processing unit with an integer-quantized neural network model and outputs the initial fall probability. The layered verification module is executed by the central processing unit and obtains the judgment result; the acquisition of video segments, the generation of skeletal sequences, the calculation of the initial fall probability, and the generation of judgment results are all completed locally, without sending the image data corresponding to the video segments and the coordinate data corresponding to the skeletal sequences to the cloud server.

Images