Multi-frame feature fusion ai fall monitoring method based on visual sensor

Abstract

The application relates to the technical field of computer vision, and particularly discloses a multi-frame feature fusion AI fall monitoring method based on a visual sensor, which comprises the following steps: acquiring human body images at a set frame rate and buffering the human body images to form a time sequence image sequence in a first-in-first-out strategy; detecting a human body region by a visual AI acceleration chip and normalizing a dynamic region of interest to obtain posture features, contour features and motion features in parallel; and performing Kalman smoothing on a human body center of mass and a trunk inclination angle; weighting the three types of features, splicing the weighted features according to dimensions to form a weighted fusion feature vector, transmitting the weighted fusion feature vector to a main control microcontroller through a high-speed serial interface, inputting a time sequence model according to a sliding window, guiding attention by an impact moment at which a vertical acceleration peak value is located, frame by frame outputting a fall probability, and determining that a fall has occurred and issuing an alarm when the fall probability exceeds a set threshold and a plurality of continuous frames meet the threshold. The application can suppress single-frame interference and distinguish between daily actions close to falls, and real-time processing is realized by means of heterogeneous double-chip cooperation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an AI fall detection method based on multi-frame feature fusion using visual sensors. Background Technology

[0002] Computer vision technology involves using image sensors to acquire scene images and performing preprocessing, target detection, feature extraction, and state judgment. Typically, a camera device acquires continuous images, which are then cached and processed by a chip. Based on the recognition results, signals are transmitted to a terminal or alarm device via wired or wireless links. This process focuses on acquiring, transmitting, recognizing, and judging the human body and its position, posture, contour, and movement state, and is applied in security monitoring and human status monitoring. Traditional fall detection methods involve acquiring and recognizing human posture and movement to determine if a person has fallen. Typically, a vision sensor acquires human images at a set frame rate, a detection program locates the area where the person is located, and extracts features representing the person's posture, contour, and movement. These extracted features are compared with preset conditions; if the features meet the set judgment criteria, a fall is determined, and the judgment result is transmitted to a local alarm device via an interface or pushed to a remote terminal via a wireless link.

[0003] The aforementioned traditional fall detection methods mostly use single-frame images as the processing unit and combine them with static threshold judgment. The inter-frame temporal information is not utilized, and they are easily interfered with by sudden changes in posture, occlusion, or noise. Everyday actions that are close to falling, such as sitting down, lying down, or bending over, are difficult to distinguish from real falls. When the human body is occluded and key points are distorted, all kinds of features are still accepted equally, and multi-dimensional information such as posture, contour, and motion lacks coordinated representation. Moreover, the acquisition, detection, feature extraction, and judgment are carried out serially by a single processing unit, making it difficult to balance computing power and real-time performance. Summary of the Invention

[0004] The purpose of this invention is to provide an AI fall detection method based on multi-frame feature fusion using visual sensors, in order to solve the problems of traditional fall detection methods, such as relying on single-frame images and single-class features combined with static threshold judgment, not utilizing inter-frame temporal information, being easily interfered with by occlusion and noise due to sudden changes in posture, difficulty in distinguishing between daily actions that are close to falling, such as sitting down, lying down, and bending over, and real falls, still accepting features equally when the human body is occluded and key points are distorted, lack of collaborative representation of multi-dimensional information, and difficulty in balancing computing power and real-time performance in serial processing by a single processing unit.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a multi-frame feature fusion AI fall detection method based on visual sensors, comprising the following steps: Human images are captured by a visual sensor at a set frame rate, and the most recent frames are cached using a first-in-first-out strategy to form a time-series image sequence. The visual AI acceleration chip detects the human body region in the time-series image sequence, expands, fills in, scales, and normalizes the human body region to obtain the dynamic region of interest. Pose features, contour features, and motion features are extracted in parallel from the dynamic region of interest. Kalman smoothing is then applied to the human centroid image coordinates, the vertical height of the human centroid, and the tilt angle of the human torso. The smoothing results are used to construct the pose features and the motion features. The posture features, contour features and motion features are weighted and then concatenated by dimension to form a weighted fusion feature vector, which is transmitted to the main microcontroller via a high-speed serial interface. When the confidence of key point detection is normal, the weight of posture features is greater than the weight of contour features and the weight of contour features is greater than the weight of motion features. The main control microcontroller takes multiple consecutive frames of the weighted fused feature vectors according to the sliding window to form a time-series input sequence and inputs it into the time-series model. Under the guidance of the impact moment, the time-series model weights attention to each frame and outputs the fall probability frame by frame. The impact moment is the frame where the vertical acceleration peak of the smoothed human body center of mass vertical height sequence is located. When the probability of falling exceeds a set threshold and is met for multiple consecutive frames, it is determined to be a fall, and the local alarm device is activated and the alarm is pushed to the remote terminal via a wireless link; otherwise, data collection and determination continue.

[0006] Furthermore, the steps for constituting the temporal image sequence include: The human images acquired by the visual sensor at a set frame rate are sequentially written into a first-in-first-out queue. When the first-in-first-out queue is full of the set capacity, each time a newly acquired human body image is written, the earliest written frame is removed. The time-series image sequence is formed by taking each frame of image in the first-in-first-out queue and arranging them according to the acquisition time sequence.

[0007] Furthermore, the step of defining the dynamic region of interest includes: The boundary of the human body region is expanded outward according to a set expansion ratio to obtain the expanded region; When the extended region exceeds the boundary of the image frame, the excess portion is padded with boundary pixels. The expanded area after padding is scaled to a set pixel size, and each pixel value is normalized.

[0008] Furthermore, the steps of extracting pose features, contour features, and motion features in parallel include: the visual AI acceleration chip extracts the coordinates of key human body points and the angular relationships between key points within the dynamic region of interest to form pose features; Extract the aspect ratio and shape description of the human body within the dynamic region of interest to form contour features; The motion feature of frame t is composed of the displacement of the human body between frame t and frame t-1 in the time-series image sequence and the inter-frame change in the vertical height of the human body's centroid after Kalman smoothing. The motion feature of the first frame in the time-series image sequence is set to a default value of zero.

[0009] Furthermore, the steps of performing Kalman smoothing on the coordinates of the human centroid image, the vertical height of the human centroid, and the tilt angle of the human torso include: A uniform motion model consisting of position and velocity is established for the coordinates of the human body's center of mass, the vertical height of the human body's center of mass, and the tilt angle of the human torso. The position of the current frame is predicted from the position and velocity of the previous frame, and then the prediction is corrected by the observation value of the current frame. The correction is determined by the uncertainty of the prediction and the uncertainty of the observation. When the confidence level of human keypoint detection in the current frame is lower than the set condition, the observed value of the current frame is not used, and only the predicted value is used as the smoothing result of the current frame.

[0010] Furthermore, the weights of the pose feature, the contour feature, and the motion feature are each adjusted by their respective reliability: The reliability of the attitude flow is obtained by aggregating the confidence scores of each human key point detection within the dynamic region of interest; The reliability of the contour flow is determined by whether the dynamic region of interest is close to the edge of the image frame and whether the human body is truncated; The reliability of motion flow is determined by whether the human body displacement in adjacent frames is within the physically reasonable range of human body movement; The weights of the posture features, contour features, and motion features are adjusted according to the reliability of posture flow, contour flow, and motion flow. When the detection confidence of human key points decreases due to occlusion, the weight of the posture features is reduced accordingly.

[0011] Furthermore, the time-series model is a long short-term memory network model with fixed-point quantization, and the step of outputting the fall probability frame by frame includes: The time-series input sequence is input into the long short-term memory network model step by step to obtain a hidden state sequence composed of hidden states step by step. The vertical acceleration sequence is obtained from the vertical height sequence of the human body's center of mass after Kalman smoothing, and the frame where the peak of the vertical acceleration occurs is determined as the impact moment; Prior weights are obtained from the proximity of each time step to the impact moment. Content scores are obtained from the hidden states of each time step through scoring transformation. The prior weights and content scores are combined and normalized, and then the hidden state sequence is weighted and summed to obtain the sequence representation vector. After fully connected mapping and normalization, the fall probability of the current frame is obtained.

[0012] Furthermore, the steps to determine a fall and trigger an alarm notification include: The fall probability of each frame is compared with a set threshold. When the fall probability is greater than the set threshold, it is recorded as a fall frame. When the fall probability is not greater than the set threshold, it is recorded as a non-fall frame. When the number of consecutive frames recorded as falling frames reaches the set number, the candidate falls; if the set number of frames is not reached, the non-fall judgment is maintained and collection and judgment continue. After entering the candidate fall, the frame within the set time window is taken to verify the final state of the human body. When the three conditions are met simultaneously, the human body's center of mass is at a low position close to the ground, the human body remains still, and the human body outline is close to horizontal, it is finally determined to be a fall; otherwise, the candidate fall is cancelled. When a fall is detected, the main control microcontroller drives the local alarm device to sound an alarm via the local interface and pushes the fall information to the remote terminal via the wireless link.

[0013] Furthermore, it also includes steps for processing invalid frames: When detecting human body regions in the time-series image sequence, image frames in which no human body regions are detected are recorded as invalid frames, and invalid frames are not included in the formation of the weighted fusion feature vector; When the number of consecutive invalid frames reaches a set number, the cached weighted fusion feature vector is cleared, and image frames are re-acquired to form the time-series image sequence. After the human body region is detected again, feature extraction and subsequent determination of the dynamically interested region are resumed.

[0014] Furthermore, the step of the visual AI acceleration chip and the main control microcontroller cooperating via the high-speed serial interface includes: The visual AI acceleration chip performs human region detection and parallel extraction of pose features, contour features and motion features, and sends the weighted fusion feature vector to the main control microcontroller via the high-speed serial interface; The main control microcontroller performs Kalman smoothing, weighted fusion, temporal model inference scheduling, and fall detection on the coordinates of the human body's center of mass, the vertical height of the human body's center of mass, and the tilt angle of the human torso. While the visual AI acceleration chip extracts features of the current frame, the main control microcontroller performs reasoning and judgment on the weighted fusion feature vector of the previous frame.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this invention, a temporal image sequence is constructed by caching multiple frames and processing continuous frames, transforming the judgment criterion from an instantaneous state to a continuous temporal state. A uniform motion model is established using the coordinates of the human centroid image, the vertical height of the human centroid, and the tilt angle of the human torso, and Kalman smoothing is applied. When the confidence of keypoint detection is too low, only the predicted value is used for compensation, suppressing distortion caused by keypoint jitter and short-term occlusion. Posture features, contour features, and motion features are weighted separately and then stitched together by dimension, retaining the three types of feature information and adjusting the weights according to the reliability of each flow. When occlusion occurs, the weights are allocated to more reliable contour and motion features. The impact moment is determined by the peak vertical acceleration and the attention of the temporal model is guided, focusing the model on the fall occurrence window, thus distinguishing between a real fall and everyday actions that are close to a fall, such as quickly sitting down, lying down, or bending over. Continuous frame confirmation combined with final state verification eliminates everyday actions that are close to a fall state for multiple consecutive frames. The visual AI acceleration chip and the main control microcontroller work separately and collaborate in a frame-level pipeline, balancing computing power and real-time processing. Attached Figure Description

[0016] Figure 1 This is the main flowchart of the fall detection method of the present invention; Figure 2 This is a schematic diagram of the temporal image sequence caching and motion feature frame alignment of the present invention; Figure 3 This is a schematic diagram of Kalman smoothing and occlusion compensation for kinematic state quantities in this invention; Figure 4 This is a schematic diagram illustrating the weighted fusion feature vector constructed by weighted concatenation of multiple types of features according to the present invention; Figure 5 This is a schematic diagram of the long short-term memory network data flow for guiding attention at the moment of impact in this invention; Figure 6 This is a schematic diagram illustrating the collaborative determination of continuous frame confirmation and final state verification in this invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0019] Please see Figures 1 to 6This embodiment provides a multi-frame feature fusion AI fall detection method based on a visual sensor. This method operates on a visual monitoring device installed in a monitoring room, continuously imaging the human body entering the frame and determining frame-by-frame whether a fall has occurred. In practice, implementing this system in elderly care or home settings encounters two types of situations that existing processing mechanisms struggle to handle. One type involves elderly individuals quickly sitting down on a low stool, lying down, or bending over to pick something up: their center of gravity drops rapidly, their torso lowers, and their silhouette changes from tall and thin to short and wide. If only single-frame features are considered and compared to a fixed threshold, these actions appear almost identical to a genuine fall, easily leading to false alarms. The other type involves the lower body being obscured by a sofa or bed edge, causing a sharp drop in confidence in leg keypoint detection and coordinate drift. If the posture is fed into the judgment with the same weight regardless of feature reliability, the results will be distorted. This embodiment addresses these two scenarios by introducing a temporal image sequence at the acquisition end, multi-feature fusion and kinematic smoothing adjusted according to confidence level at the feature acquisition end, and temporal modeling and final state verification guided by fall impact at the decision-making end, changing the decision criterion from isolated single-frame states to a coherent temporal process. The main process of this method includes six steps, S1 to S6.

[0020] S1 is for image acquisition and temporal buffering. The vision sensor continuously images the monitored area at a set frame rate, outputting one frame of human image at each acquisition moment. The acquisition frame rate is selected within the range supported by the original disclosure to ensure that human movement is continuous and differential between adjacent frames. The acquired human images are sequentially written to a first-in-first-out queue, the queue capacity of which is set according to the number of most recent frames buffered. When the queue is not full, only writing is done, no removal; once it is full, each new frame that arrives simultaneously pushes out the oldest frame, so that the queue always contains a number of frames from the most recent period, arranged in the acquisition sequence, and is continuously updated.

[0021] The currently sorted frames in the queue constitute the temporal image sequence, which serves as the common data source for subsequent motion difference and temporal inference. Here, we need to handle a boundary case during the startup phase: when the device is first powered on and the sensor is cold-starting, the queue hasn't accumulated enough frames, motion features cannot be calculated, and the temporal model doesn't have a sufficiently long input. At this point, we only perform data acquisition and buffering, and only proceed after the number of frames in the queue reaches the set capacity, avoiding the need to forcibly construct a fragmented temporal input when historical frames are insufficient.

[0022] S2 is for human detection and dynamic region of interest generation. The visual AI acceleration chip performs human detection on image frames in the temporal image sequence and outputs the human region containing the human body. This step first determines whether a human body has been detected: if not, the frame is marked as an invalid frame and does not participate in the construction of the weighted fusion feature vector; if the number of consecutive invalid frames reaches a set number, it means that the person has left the frame or has been completely occluded for a long time, so the cached weighted fusion feature vector is cleared and the process returns to S1 to reassemble the sequence. Subsequent processing resumes only after a human body is detected again. The purpose of this branch is to prevent the short-term entry and exit of human bodies from interfering with the temporal determination.

[0023] After detecting a human body, the boundary of the human body region is expanded outward by a set expansion ratio, resulting in a slightly larger extended area. This is to prevent the loss of edge information due to arms and legs extending beyond the original frame during a fall. If the expanded area exceeds the image frame boundary, the excess portion is padded with boundary pixels to maintain a regular shape. The padded area is then scaled to a set pixel size and the pixel values ​​are normalized to obtain a dynamically sized region of interest (ROI). The ROI moves as the person moves within the image, ensuring that subsequent feature extraction always targets the area containing the human body, rather than the entire image.

[0024] S3 performs parallel feature extraction across multiple categories and Kalman smoothing on the kinematic state variables. The visual AI acceleration chip extracts the coordinates of key points on the human body within the dynamic region of interest. This type of pose model provides the detection confidence score for each key point along with the coordinates; this confidence score is a ready-made output in the pipeline, which will be used later in this invention to measure the reliability of the pose. Pose features are constructed from the coordinates of each key point and the angular relationships between them (e.g., the angle between the torso and the vertical direction, the angle between the hip and knee lines), used to characterize the relative positions of different parts of the human body and the tilt of the torso. Simultaneously, the aspect ratio and shape descriptor of the human body within the dynamic region of interest are extracted to form contour features. The change in the contour from vertical length to horizontal width as a person moves from standing to falling is reflected in these contour features.

[0025] Motion features are differential components, which can only be calculated from two adjacent frames. Therefore, the motion features of frame t are taken from the pair between frame t and frame (t-1): they consist of the displacement of the human body and the change in the center of mass between the two frames, used to characterize the speed and direction of the motion. The first frame in the sequence has no preceding frame to pair with, so its motion features are set to the default zero value, which does not affect subsequent frames. With this arrangement, each frame in the sequence corresponds to a set of pose features, contour features, and motion features.

[0026] This step involves a crucial but often overlooked process: the kinematic state variables—centroid image coordinates, centroid height, and torso tilt angle—are directly read from the detection results of each frame. The detection itself is noisy, and keypoints may occasionally jump or experience sudden coordinate shifts due to brief occlusion. Therefore, the main control microcontroller performs Kalman filtering on each of these kinematic state variables. The filter constructs a uniform motion model for each state variable, consisting of position and velocity: first, the position and velocity of the previous frame are used to estimate and predict the position of the current frame; then, the observed values ​​detected in the current frame are used to correct this prediction. The magnitude of the correction is determined by both the uncertainty of the prediction and the uncertainty of the observation; the more reliable the observation, the greater the correction. The smoothed centroid height obtained here is also used to calculate the centroid change in the motion features, ensuring that the difference component is not skewed by frame-by-frame jitter.

[0027] Kalman filtering serves two purposes here. First, it smooths out the jitter in the kinematic states frame by frame, resulting in smooth centroid and tilt trajectories. This smoothed centroid height sequence is then used by S5 to find the impact moment. Second, when a key point in a frame has low confidence due to occlusion, making observation unreliable, observation is temporarily omitted, and only the predicted value is used to push forward. This compensates for short-term data loss, preventing one or two bad frames from skewing the entire trajectory. It's important to note that Kalman filtering deals with signal-level issues like "whether the value of this frame is reliable and stable," which differs from the Long Short-Term Memory (LSTM) network model in S5, which handles event-level issues like "whether these dozens of frames form a single fall." The two are complementary, not redundant. After this step, pose features, contour features, and motion features are extracted in parallel on the visual AI acceleration chip, independent of each other's output. The centroid and tilt components use the smoothed Kalman values.

[0028] S4 represents weighted fusion, resulting in a weighted fused feature vector. As a basic implementation method, the main microcontroller assigns a preset weight to each of the posture, contour, and motion features. When the confidence level for keypoint detection is normal, the weight of the posture feature is greater than that of the contour feature, and the weight of the contour feature is greater than that of the motion feature, reflecting that posture contributes relatively more to fall detection under normal circumstances. During fusion, the three feature sets are not simply added together, as this would result in mismatched dimensions and meanings, and information would be lost. Instead, each feature type is multiplied by its own weight, and then concatenated end-to-end according to the original dimensional order to obtain a weighted fused feature vector. Each of the three feature types occupies its own position in the vector and is retained.

[0029] The occlusion scenario mentioned earlier illustrates a shortcoming of simply assigning weights based on a fixed size relationship: it assumes that the three feature streams are always equally reliable, but in reality, the reliability of each feature stream fluctuates in each frame. Therefore, as a further implementation method, the main control microcontroller can dynamically adjust the base weights according to the reliability of each stream. The reliability of the pose stream is obtained by aggregating the detection confidence of each keypoint in the dynamic region of interest. When occlusion occurs, the keypoint confidence drops, and the reliability of the pose stream decreases accordingly. The reliability of the contour stream depends on whether the dynamic region of interest is close to the edge of the image and whether the human body is obviously truncated; if truncated, it decreases. The reliability of the motion stream depends on whether the displacement of adjacent frames falls within the physically reasonable range of normal human movement; if there are unreasonable large jumps (often false detections), it decreases. Based on the base weights where the pose weight is greater than the contour weight and the contour weight is greater than the motion weight when the keypoint detection confidence is normal, the base weights are adjusted according to the reliability of the three streams: the weights of streams with high reliability are increased, and the weights of streams with low reliability are decreased, and then the weighted stitching is performed. In this way, when the pose becomes unreliable due to occlusion, the pose weight is reduced accordingly, and the weight is automatically allocated to the relatively more reliable contour and motion, so that the judgment is not skewed by the bad pose.

[0030] S5 is the temporal model inference, outputting frame-by-frame fall probability. In this embodiment, the temporal model specifically adopts a long short-term memory network model with fixed-point quantization. The inference here is performed in a sliding window manner: for each new frame, the weighted fused feature vectors of the most recent consecutive frames up to the current frame are taken, arranged into a temporal input sequence according to the acquisition time sequence, and sent to the model. The model gives a fall probability belonging to the current frame based on this. The window moves forward frame by frame as acquisition progresses, and inference is performed frame by frame, thus forming a frame-by-frame fall probability sequence in time. The data flows within the model as follows: the input layer receives the weighted fused feature vector of each frame within the window, and its dimension is consistent with the length of the weighted fused feature vector; at each time step, the memory unit simultaneously obtains the weighted fused feature vector of the current time step, the hidden state passed down from the previous time step, and the unit state. Within a cell, the forget gate determines how much of the cell state from the previous time step is retained, while the input gate determines how much of the currently generated candidate state is written. Candidate states are generated jointly by the current time step input and the previous hidden state. The cell state is updated under the combined action of the forget gate and the input gate. Finally, the output gate determines how much of the updated cell state becomes the hidden state for the current time step. Moving through the window step by step yields a sequence of hidden states, with each time step corresponding to a hidden state vector.

[0031] The conventional approach is to directly classify the hidden state of the last frame of the window. However, the hidden states of a quick sit-down and a true fall are very similar, making it difficult to distinguish them. This invention adds an impact-guided attention layer on top of the hidden state sequence: first, the vertical descent velocity of each frame is calculated over time from the centroid height sequence after S3 Kalman smoothing; then, the change in velocity is calculated to obtain the vertical acceleration sequence. A true fall will show a steep acceleration peak at the moment of impact; this frame is the impact moment. Sitting, lying down, and bending over are controlled decelerations with gentle acceleration changes, and no such steep peak is found. The attention weights are determined by two parts: one is the proximity of each time step within the window to the impact moment, with higher prior weights given to frames closer to the impact frame; the other is the content score obtained by a scoring transformation of the hidden states at each time step. After combining and normalizing these two parts, a weighted summation of the hidden state sequence is obtained to obtain a sequence representation vector, allowing the model's attention to naturally focus on the window position where the fall occurs. The sequence representation vector is then mapped and normalized to a range of zero to one through a fully connected layer, which gives the fall probability of the current frame. The weights and activations of the entire network are fixed-point quantized to accommodate the computational resources of the main microcontroller.

[0032] This section clarifies how the model is trained and how it effectively distinguishes fall-like actions to avoid creating a black box. The training samples not only collect video clips of real falls but also specifically collect video clips of everyday actions prone to false alarms, such as sitting, lying down, bending over, squatting, and normal walking. All of these are processed using the same pipeline as S1 to S4 to extract weighted fusion feature vector sequences, and the centroid height sequence is recorded simultaneously to indicate the impact time. Each sample is labeled as either a fall or not. During training, the temporal input sequence of each sample is fed into the model, and the fall probability output for that sample is obtained using the aforementioned sliding window and interlayer flow. The loss statement then applies to the relationship between this fall probability output and the fall / non-fall label for that sample. The cost-sensitive classification loss employed is based on a cross-entropy form of classification loss. First, the classification loss for each sample segment is calculated from the fall probability and label. Then, cost coefficients are applied to samples according to their error type: a false positive cost coefficient greater than one is applied to non-fall samples judged as falls, a false negative cost coefficient greater than one is applied to fall samples judged as non-falls, and a false negative cost coefficient greater than one is applied to missed errors. The cost coefficient for correctly judged samples is set to one, ensuring that false positives and false negatives contribute more to the loss than ordinary misclassifications. The total loss is obtained by averaging the above cost-sensitive classification losses for a batch of samples. The parameters are repeatedly updated in the direction that decreases the total loss. By collecting a sufficient number and variety of fall-related action samples, the model learns the discriminative cue of the impact moment until the fall recognition performance on validation samples no longer improves, at which point training stops. The trained model is then deployed to the main control microcontroller for inference after fixed-point quantization. During inference, the fall probability is output frame-by-frame according to the aforementioned sliding window and inter-layer flow.

[0033] S6 is for fall detection and alarm push notification. First, a frame-by-frame comparison is performed: the fall probability of each frame is compared to a set threshold. Frames with a probability greater than the threshold are recorded as fall frames, and those with a probability less than the threshold are recorded as non-fall frames. Next, continuous frame confirmation is performed: the number of consecutive frames recorded as fall frames is counted. Only frames that reach the set number of frames are considered candidate fall frames; otherwise, the frame remains as non-fall and returns to S1 to continue sampling and judgment. Continuous frame confirmation can filter out occasional probability spikes in single frames, but it cannot prevent actions like an elderly person sitting or lying down steadily, which can appear as a fall for multiple consecutive frames.

[0034] Therefore, as a further implementation method, after confirming consecutive frames, an additional final state verification is added: frames within a short time window after a candidate fall are taken to verify whether the final state of the human body matches the appearance of a real fall. The final state is considered from three aspects: first, whether the center of mass height falls to a low position close to the ground; second, whether the center of mass remains basically still, that is, the displacement of the center of mass in adjacent frames is calculated from the Kalman-smoothed center of mass image coordinates within this time window. When the displacement of the center of mass in each adjacent frame is less than a set displacement, it is considered that the human body remains basically still; third, whether the human body outline is close to horizontal. Only when all three conditions are met is it finally confirmed as a fall. Using this verification to review the previous two difficult scenarios: after bending over to pick something up, the person will stand up again, and the final state center of mass returns to a high position, which is rejected; when sitting down, the final state center of mass stops at the height of the seat, and the person is still semi-upright, which is also rejected; only a real fall will have both the impact captured by S5 and the final state of lying flat on the ground.

[0035] Once a fall is confirmed, the main microcontroller drives the local alarm device to issue an audible and visual alarm via the local interface. Simultaneously, it pushes the fall information to a remote terminal via a wireless link. The fall information includes the time of the fall and the frame of image at which the fall was determined, allowing the remote terminal to know the time and location of the fall. If the wireless push fails, the main microcontroller temporarily stores the fall information and retransmits it at set time intervals until it is successfully transmitted or the set number of retries is reached. During the retransmission period, the local alarm device remains on, ensuring that even if remote communication experiences a temporary problem, the room will not miss any alerts.

[0036] Looking at S1 through S6 together, this embodiment completes human region detection and parallel extraction of three-way features (pose, contour, and motion) on the visual AI acceleration chip. The assembled weighted fused feature vector is sent to the main control microcontroller via a high-speed serial interface. The main control microcontroller then performs Kalman smoothing of kinematic state variables, weighted fusion, inference scheduling of the long short-term memory network model with impact-guided attention, and determination of consecutive frames and final states. Because the two chips operate independently yet are connected by a high-speed serial interface, while the visual AI acceleration chip extracts features of the current frame, the main control microcontroller can fuse and determine the weighted fused feature vector of the previous frame. Both sides proceed in parallel at the frame level pipeline, separating computationally intensive tasks and logical determination tasks onto appropriate carriers, thus keeping up with the frame-by-frame processing pace. Each of the above steps has a clear source of input and a clear destination of output, and they are sequentially connected without skipping steps.

[0037] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-frame feature fusion AI fall detection method based on visual sensors, characterized in that, include: Human images are captured by a visual sensor at a set frame rate, and the most recent frames are cached using a first-in-first-out strategy to form a time-series image sequence. The visual AI acceleration chip detects the human body region in the time-series image sequence, expands, fills in, scales, and normalizes the human body region to obtain the dynamic region of interest. Pose features, contour features, and motion features are extracted in parallel from the dynamic region of interest. Kalman smoothing is then applied to the human centroid image coordinates, the vertical height of the human centroid, and the tilt angle of the human torso. The smoothing results are used to construct the pose features and the motion features. The posture features, contour features and motion features are weighted and then concatenated by dimension to form a weighted fusion feature vector, which is transmitted to the main microcontroller via a high-speed serial interface. When the confidence of key point detection is normal, the weight of posture features is greater than the weight of contour features and the weight of contour features is greater than the weight of motion features. The main control microcontroller takes multiple consecutive frames of the weighted fused feature vectors according to the sliding window to form a time-series input sequence and inputs it into the time-series model. Under the guidance of the impact moment, the time-series model weights attention to each frame and outputs the fall probability frame by frame. The impact moment is the frame where the vertical acceleration peak of the smoothed human body center of mass vertical height sequence is located. When the probability of falling exceeds a set threshold and is met for multiple consecutive frames, it is determined to be a fall, and the local alarm device is activated and the alarm is pushed to the remote terminal via a wireless link; otherwise, data collection and determination continue.

2. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The steps for the time-series image sequence include: The human body images acquired by the visual sensor at a set frame rate are sequentially written into a first-in-first-out queue. When the first-in-first-out queue is full of the set capacity, each time a newly acquired human body image is written, the earliest written frame is removed. The time-series image sequence is formed by taking each frame of image in the first-in-first-out queue and arranging them according to the acquisition time sequence.

3. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The steps for defining the dynamic region of interest include: The boundary of the human body region is expanded outward according to a set expansion ratio to obtain the expanded region; When the extended region exceeds the boundary of the image frame, the excess portion is padded with boundary pixels. The expanded area after padding is scaled to a set pixel size, and each pixel value is normalized.

4. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The steps for extracting pose features, contour features, and motion features in parallel include: The visual AI acceleration chip extracts the coordinates of key human body points and the angular relationships between key points within the dynamic region of interest to form posture features. Extract the aspect ratio and shape description of the human body within the dynamic region of interest to form contour features; The motion feature of frame t is composed of the displacement of the human body between frame t and frame t-1 in the time-series image sequence and the inter-frame change in the vertical height of the human body's centroid after Kalman smoothing. The motion feature of the first frame in the time-series image sequence is set to a default value of zero.

5. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The steps for performing Kalman smoothing on the coordinates of the human centroid image, the vertical height of the human centroid, and the tilt angle of the human torso include: A uniform motion model consisting of position and velocity is established for the coordinates of the human body's center of mass, the vertical height of the human body's center of mass, and the tilt angle of the human torso. The position of the current frame is predicted from the position and velocity of the previous frame, and then the prediction is corrected by the observation value of the current frame. The correction is determined by the uncertainty of the prediction and the uncertainty of the observation. When the confidence level of human keypoint detection in the current frame is lower than the set condition, the observed value of the current frame is not used, and only the predicted value is used as the smoothing result of the current frame.

6. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The weights of the pose feature, the contour feature, and the motion feature are each adjusted by their respective reliability: The reliability of the attitude flow is obtained by aggregating the confidence scores of each human key point detection within the dynamic region of interest; The reliability of the contour flow is determined by whether the dynamic region of interest is close to the edge of the image frame and whether the human body is truncated; The reliability of motion flow is determined by whether the human body displacement in adjacent frames is within the physically reasonable range of human body movement; The weights of the posture features, contour features, and motion features are adjusted according to the reliability of posture flow, contour flow, and motion flow. When the detection confidence of human key points decreases due to occlusion, the weight of the posture features is reduced accordingly.

7. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The time-series model is a long short-term memory network model with fixed-point quantization. The steps for outputting the fall probability frame by frame include: The time-series input sequence is input into the long short-term memory network model step by step to obtain a hidden state sequence composed of hidden states step by step. The vertical acceleration sequence is obtained from the vertical height sequence of the human body's center of mass after Kalman smoothing, and the frame where the peak of the vertical acceleration occurs is determined as the impact moment; Prior weights are obtained from the proximity of each time step to the impact moment. Content scores are obtained from the hidden states of each time step through scoring transformation. The prior weights and content scores are combined and normalized, and then the hidden state sequence is weighted and summed to obtain the sequence representation vector. After fully connected mapping and normalization, the fall probability of the current frame is obtained.

8. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The steps to determine a fall and trigger an alarm include: The fall probability of each frame is compared with a set threshold. When the fall probability is greater than the set threshold, it is recorded as a fall frame. When the fall probability is not greater than the set threshold, it is recorded as a non-fall frame. When the number of consecutive frames recorded as falling frames reaches the set number, the candidate falls; if the set number of frames is not reached, the non-fall judgment is maintained and collection and judgment continue. After entering the candidate fall, the frame within the set time window is taken to verify the final state of the human body. When the three conditions are met simultaneously, the human body's center of mass is at a low position close to the ground, the human body remains still, and the human body outline is close to horizontal, it is finally determined to be a fall; otherwise, the candidate fall is cancelled. When a fall is detected, the main control microcontroller drives the local alarm device to sound an alarm via the local interface and pushes the fall information to the remote terminal via the wireless link.

9. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, It also includes steps for handling invalid frames: When detecting human body regions in the time-series image sequence, image frames in which no human body regions are detected are recorded as invalid frames, and invalid frames are not included in the formation of the weighted fusion feature vector; When the number of consecutive invalid frames reaches a set number, the cached weighted fusion feature vector is cleared, and image frames are re-acquired to form the time-series image sequence. After the human body region is detected again, feature extraction and subsequent determination of the dynamically interested region are resumed.

10. The AI ​​fall detection method based on multi-frame feature fusion using a visual sensor according to claim 1, characterized in that, The steps for the visual AI acceleration chip and the main control microcontroller to coordinate via the high-speed serial interface include: The visual AI acceleration chip performs human region detection and parallel extraction of pose features, contour features and motion features, and sends the weighted fusion feature vector to the main control microcontroller via the high-speed serial interface; The main control microcontroller performs Kalman smoothing, weighted fusion, temporal model inference scheduling, and fall determination on the coordinates of the human body's center of mass, the vertical height of the human body's center of mass, and the tilt angle of the human torso. While the visual AI acceleration chip extracts features of the current frame, the main control microcontroller performs reasoning and judgment on the weighted fusion feature vector of the previous frame.

Images