A privacy-enhanced fall detection method, device, and medium based on machine vision

Abstract

The application discloses a privacy-enhanced fall detection method and device based on machine vision and a medium, and relates to the field of artificial intelligence.The method comprises the following steps: obtaining a time window with a preset time length from a received sensor data stream as a current time window; extracting human skeleton key point coordinate information from each frame of sensor data in the current time window to generate a skeleton coordinate sequence, and discarding the corresponding original sensor data frame; obtaining a fall confidence through a pre-trained spatio-temporal graph convolution network based on the skeleton coordinate sequence; when the fall confidence is greater than or equal to a first preset threshold, taking the current time as a still timing starting time; continuously obtaining the displacement rate of the human skeleton key points, and recording the cumulative time length during which the displacement rate is continuously lower than a second preset threshold; and if the cumulative time length reaches a third preset threshold, a fall event is confirmed.The application can realize low false alarm rate fall detection without collecting recognizable human faces and physical features.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a privacy-enhancing fall detection method, device, and medium based on machine vision. Background Technology

[0002] In elderly care facilities and home-based care settings, deploying visual sensors for fall detection has become a common industry practice to ensure the safety of senior citizens. However, these scenarios involve the collection and processing of seniors' personal biometric information, which is subject to strict constraints from privacy protection regulations and ethical requirements.

[0003] In existing technologies, one mainstream solution uses a regular RGB camera to capture a panoramic view of the human body, and then uses computer vision algorithms to segment the human posture from the entire image. Although this solution has mature algorithms and high detection accuracy, its front end directly captures a complete image containing clear facial and physical features. This image faces the risk of leakage during transmission and storage, making it difficult to pass privacy compliance reviews. Furthermore, another approach, to avoid privacy controversies, attempts to use low-resolution infrared sensors or depth cameras. While the original images do not have clear facial recognition capabilities, they also lose rich texture and appearance features. As a result, existing behavior recognition algorithms based on this hardware can only perform simple area intrusion or motion detection, and cannot distinguish different behavior patterns from temporal movement patterns, such as "suddenly squatting down to pick up an object" and "standing still after falling," leading to frequent false alarms. Furthermore, due to the inherent defects of the two types of solutions mentioned above, elderly care institutions face a dilemma: if they adopt the high-precision RGB solution, they will face the red line of privacy compliance; furthermore, if they adopt privacy-friendly sensors, the monitoring reliability is insufficient, and caregivers still need to rely on visual inspection or reviewing blurry videos to verify abnormalities, rendering automatic early warning ineffective. Summary of the Invention

[0004] This invention provides a privacy-enhancing fall detection method, device, and medium based on machine vision. The technical problem it aims to solve is: how to provide a fall detection scheme with a low false alarm rate without collecting identifiable facial and physical features.

[0005] In a first aspect, the present invention provides a privacy-enhanced fall detection method based on machine vision, comprising: In response to the received sensor data stream, sensor data within a preset time window is obtained from the sensor data stream, and the preset time window is used as the current time window; Extract the coordinate information of key points of the human skeleton from each frame of sensor data within the current time window, generate the skeleton coordinate sequence corresponding to the current time window, and discard the corresponding original sensor data frames; Based on the skeleton coordinate sequence, inference is performed through a pre-trained spatiotemporal graph convolutional network to obtain the fall confidence of the current time window; Determine whether the fall confidence level is greater than or equal to a first preset threshold; If the fall confidence level is greater than or equal to the first preset threshold, then the current moment is taken as the start moment of the stationary timing. Starting from the moment of the static timing, the displacement rate of key points of the human skeleton in each frame of sensor data is continuously acquired; Record the cumulative duration for which the displacement rate remains below the second preset threshold; If the cumulative duration reaches the third preset threshold, a fall event is confirmed.

[0006] Optionally, the step of extracting coordinate information of key points of the human skeleton from each frame of sensor data within the current time window and generating a skeleton coordinate sequence corresponding to the current time window includes: Perform human contour detection on each frame of sensor data within the current time window to obtain the human contour region; Obtain the sensor type corresponding to the sensor data stream; Select a background segmentation strategy that matches the sensor type; Based on the selected background segmentation strategy, background segmentation is performed on the human body contour region to obtain the foreground human body contour. Based on the foreground human body contour, extract the two-dimensional coordinates and depth values ​​of a preset number of human skeleton key points in the image plane, and use them as the initial coordinate data of each human skeleton key point. The initial coordinate data of each key point of the human skeleton are scored with confidence to obtain the confidence score value of each key point of the human skeleton. The human skeleton key points whose confidence scores are lower than the fourth preset threshold are marked as low-confidence human skeleton key points, and the low-confidence human skeleton key points are removed from all the current human skeleton key points. For the low-confidence human skeleton key points that are removed, interpolation is performed based on the coordinate data of the adjacent human skeleton key points of the low-confidence human skeleton key points in the current frame and multiple frames before and after the current frame to generate alternative coordinate data for the low-confidence human skeleton key points, and the alternative coordinate data is added to the human skeleton key points. Based on the coordinate data of all key points of the human skeleton after the supplementation, single-frame skeleton coordinate data corresponding to each frame of sensor data within the current time window is formed. The skeleton coordinate data of multiple consecutive frames within the current time window are arranged according to the acquisition time sequence to generate the skeleton coordinate sequence corresponding to the current time window.

[0007] Optionally, the step of performing background segmentation on the human body contour region based on the selected background segmentation strategy to obtain the foreground human body contour includes: If the sensor type is infrared thermal imaging, then obtain the thermal radiation intensity value of each pixel in the sensor data; Based on the distribution of thermal radiation intensity within the human body contour area, calculate the statistical characteristics of thermal radiation intensity within the human body contour area, including the mean thermal radiation intensity and the standard deviation of thermal radiation intensity. Based on the statistical characteristics of the thermal radiation intensity, a dynamic segmentation threshold matching the sensor type is determined. The dynamic segmentation threshold is the weighted sum of the mean thermal radiation intensity and the standard deviation of the thermal radiation intensity. Pixels with thermal radiation intensity values ​​lower than the dynamic segmentation threshold within the human body contour area are identified as background pixels and removed to obtain the foreground human body contour. If the sensor type is time-of-flight depth type, then obtain the depth value of each pixel in the sensor data; Histogram statistics are performed on all depth values ​​within the human body contour area, and the depth layer corresponding to the depth value with the highest frequency in the histogram is determined as the background depth layer. Pixels whose depth values ​​within the human body contour area belong to the background depth layer are identified as background pixels and removed to obtain the foreground human body contour.

[0008] Optionally, the step of obtaining the fall confidence score for the current time window based on the skeleton coordinate sequence through a pre-trained spatiotemporal graph convolutional network includes: Normalization processing is performed on the coordinates of each human skeleton key point in the skeleton coordinate sequence to align the coordinates of each human skeleton key point to a unified standard pose space, thus obtaining a normalized skeleton coordinate sequence. Obtain the predefined natural connectivity topology of the human skeleton from the pre-trained spatiotemporal graph convolutional network, and construct a spatial adjacency matrix based on the natural connectivity topology of the human skeleton. The normalized skeleton coordinate sequence is input into the pre-trained spatiotemporal graph convolutional network, which includes a graph convolutional layer, a temporal convolutional layer, an attention layer, and a fully connected layer. The attention layer is set after the graph convolutional layer and the temporal convolutional layer, and before the fully connected layer. The spatial dependency relationships between key points of each human skeleton are extracted by the graph convolutional layer based on the spatial adjacency matrix to obtain a spatial feature map; The spatial feature map is convolved along the time dimension by the temporal convolutional layer to extract the temporal motion features between frames within the current time window, thus obtaining the temporal feature map. The attention layer weights the temporal feature map along the channel dimension and the spatial dimension respectively to obtain the enhanced temporal feature map. The enhanced temporal feature map is input into the fully connected layer for mapping, and the fall confidence of the current time window is output.

[0009] Optionally, the step of continuously acquiring the displacement rate of key points of the human skeleton in each frame of sensor data from the start of the stationary timing includes: All key points of the human skeleton are divided into the trunk key point group and the limb key point group according to the human body parts. The trunk key point group includes the human skeleton key points at the shoulders, hips and spine, and the limb key point group includes the human skeleton key points at the elbows, wrists, knees and ankles. Starting from the moment of the static timing, the coordinate change of the same human skeleton key point in consecutive frames is obtained as the displacement of the human skeleton key point. Based on the displacement and the inter-frame time interval, calculate the inter-frame displacement rate of each human skeleton key point between adjacent frames. Based on the inter-frame displacement rate of all human skeleton key points in the trunk key point group, the average displacement rate within the trunk key point group is calculated to obtain the average displacement rate of the trunk. Based on the inter-frame displacement rate of all human skeleton key points in the limb key point group, the average displacement rate within the limb key point group is calculated to obtain the average displacement rate of the limbs. The average displacement rate of the torso and the average displacement rate of the limbs are weighted and summed, wherein the weight of the average displacement rate of the torso is greater than the weight of the average displacement rate of the limbs, to obtain the weighted average displacement rate of the current frame, and the weighted average displacement rate is used as the displacement rate.

[0010] Optionally, the method further includes: Obtain historical behavior time-series data of the target object within the initial time period; Unsupervised clustering is performed on the historical behavior time series data to establish an individualized behavior baseline model for the target object; The real-time behavioral features of the target object are obtained, and the real-time behavioral features are input into the individualized behavioral baseline model to calculate the probability density value corresponding to the real-time behavioral features. Determine whether the probability density value is less than a fifth preset threshold; If the probability density value is less than the fifth preset threshold, a behavior deviation flag is generated; In response to the generation of the behavior deviation flag, a deviation duration timer is started to record the duration of the behavior deviation flag; Determine whether the cumulative duration of the deviation duration timer is greater than or equal to a sixth preset threshold; If the cumulative duration of the deviation duration timer is greater than or equal to the sixth preset threshold, then a behavior deviation event is confirmed. If the cumulative duration of the deviation duration timer is less than the sixth preset threshold, the behavior deviation event will not be confirmed until the cumulative duration reaches the sixth preset threshold or the behavior deviation flag is removed. In response to the confirmation of the behavioral deviation event, the individualized behavioral baseline model is updated based on the real-time behavioral characteristics at the current moment.

[0011] Optionally, obtaining the historical behavior time-series data of the target object within the initial time period includes: During the initial time period, the bed rest status and bed alighting status of the target object are continuously monitored, and a status timestamp log is generated. Extract the target object's first time out of bed each day, number of times it gets out of bed at night, and duration of each toilet visit from the status timestamp log during the initial period. Within each unit time period of the initial time period, the spatial coordinates of each key point of the human skeleton of the target object in the bedridden state are obtained, and the cumulative value of the position change of each key point of the human skeleton relative to the previous sampling time is calculated. The cumulative value is used as the total amount of bedridden activity within the unit time period. Divide the total amount of bed rest activities within the unit time period by the duration of the unit time period to obtain the average range of bed rest activities within the unit time period. The first time of getting out of bed each day, the number of times getting out of bed at night, the duration of each toilet visit, and the average range of bed activity within a unit time period are used as the historical behavior time series data.

[0012] Optionally, the method further includes: Monitor the confirmation status of the fall event and the confirmation status of the behavioral deviation event; If the fall event is confirmed, a Level 1 warning signal is generated, and an emergency push process is triggered. In response to the emergency push process being triggered, a drive signal is sent to the audible and visual alarm terminal, and an emergency pop-up signal and an SMS signal are simultaneously pushed to the pre-associated mobile terminal. After the first-level warning signal is generated, monitor whether the confirmation status of the fall event remains valid. If the fall event is not cancelled within the preset seventh preset time period, the drive signal, the emergency pop-up signal and the SMS signal will be sent repeatedly every seventh preset time period until the fall event is manually cancelled. If the fall event is not confirmed but the behavioral deviation event is confirmed, a second-level warning signal is generated, and the priority of the second-level warning signal is lower than that of the first-level warning signal. In response to the generation of the second-level warning signal, a text alert signal is pushed to the application message queue of the pre-associated mobile terminal; After the second-level warning signal is generated, it is determined whether the same type of text reminder signal has been generated within a preset eighth preset time period; If a text reminder signal of the same behavior type has been generated within the eighth preset time period, then the push of the current text reminder signal will be suppressed. If no text reminder signal of the same behavior type has been generated within the eighth preset time period, then the current text reminder signal can be pushed.

[0013] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0015] This invention provides a privacy-enhancing fall detection method, device, and medium based on machine vision. By extracting the coordinate information of key points on the human skeleton and immediately discarding the corresponding original sensor data frames, the original sensor image data is deleted immediately after extraction, preventing its retention or transmission. This avoids the collection of identifiable facial and physical features from the data source, thus satisfying privacy protection requirements while achieving detection functionality. Furthermore, a pre-trained spatiotemporal graph convolutional network is used to infer the skeleton coordinate sequence within the current time window. Spatial dependencies are extracted using the natural connectivity topology of the human skeleton, and inter-frame temporal motion features are captured. This quantifies the degree of matching between the current posture change and the fall action pattern, outputting a fall confidence score as a quantitative basis for subsequent judgment. Further, by determining whether the fall confidence score reaches a first preset threshold, non-fall posture changes below this threshold are excluded. A static timing is only initiated when the confidence score reaches the threshold, achieving preliminary screening of suspected fall actions. Furthermore, after the start of the static timing, the preliminary screening results are further verified by continuously monitoring the cumulative duration for which the displacement rate is below the second preset threshold. This is achieved by utilizing the sustained stillness characteristic of a person who cannot get up after a real fall. The fall event is only confirmed when the cumulative duration reaches the third preset threshold, thus eliminating false alarms caused by instantaneous actions such as quickly crouching down to pick up an object. It is evident that the above features sequentially perform privacy desensitization processing, suspected fall action identification, instantaneous action exclusion, and sustained stillness verification, collaboratively achieving the technical objective of providing a low-false-alarm-rate fall detection solution without collecting identifiable facial and physical features. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic flowchart of a privacy-enhancing fall detection method based on machine vision provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Please see Figure 1 This invention provides a privacy-enhancing fall detection method based on machine vision. This method can be applied to scenarios such as elderly care institutions and home-based elderly care, automatically monitoring and issuing warnings for fall behavior in the elderly through visual sensors. Furthermore, in such scenarios, deploying visual sensors for fall behavior monitoring has become a common industry practice. However, ordinary RGB cameras directly capture complete images containing clear facial and physical features, which face serious privacy leakage risks during transmission and storage, making it difficult to pass privacy compliance reviews. Furthermore, while low-resolution infrared sensors or depth cameras lack clear facial recognition capabilities, their original images also lose rich texture and appearance features. Existing behavior recognition algorithms based on this hardware cannot distinguish between different behavior patterns, such as quickly squatting to pick up an object and being unable to get up after a real fall, from temporal movement patterns, resulting in frequent false alarms. Therefore, this embodiment achieves privacy enhancement by extracting key points of the human skeleton from sensor data and immediately discarding the original image data. It also utilizes a two-stage decision mechanism combining short-window spatiotemporal graph convolutional network inference and long-term static verification to achieve low false alarm rate fall detection. Specifically, the method includes the following steps: S1, in response to the received sensor data stream, obtain sensor data within a preset time window from the sensor data stream, and use the preset time window as the current time window.

[0024] In practice, the sensor data stream is a data sequence continuously collected and transmitted in real time by a visual sensor deployed within the monitoring area. This visual sensor can be an infrared thermal imaging sensor or a time-of-flight depth sensor. Further, the infrared thermal imaging sensor collects the thermal radiation intensity distribution data of each pixel, reflecting the temperature distribution of the surface of the object being measured. The human body region is represented as a continuous area with significantly higher thermal radiation intensity than the background. Further, the time-of-flight depth sensor calculates the distance from each pixel to the sensor by measuring the flight time of a light pulse from emission to reflection. This distance value is the depth value. The human body region is represented as a connected area with depth values ​​within a specific continuous range and exhibiting human shape characteristics. Further, the sensor data stream is continuously generated at a fixed sampling frame rate, with each sampling moment corresponding to one frame of data. The frame rate is, for example, 30 frames per second, meaning the time interval between each frame is approximately 33.3 milliseconds. Further, upon receiving the sensor data stream, a continuous data frame is extracted from the data stream according to a preset duration as a basic processing unit. This preset duration can be set according to the specific application scenario and hardware performance, for example, set to 1 second. Using a preset time window as the current time window ensures that all subsequent processing steps are based on the same time reference, guaranteeing the synchronization and consistency of data processing. For example, in practical implementation, a circular buffer can be maintained. This buffer stores sensor data frames within the most recent preset time window according to a first-in, first-out (FIFO) principle. Whenever a new sensor data frame is received from the data stream, it is pushed to the end of the buffer, while older frames at the beginning of the buffer that exceed the preset time window are removed. This ensures that the buffer always contains all data frames within the preset time window, and the data stored in this buffer is the sensor data within the current time window.

[0025] S2, extract the coordinate information of key points of the human skeleton from each frame of sensor data within the current time window, generate the skeleton coordinate sequence corresponding to the current time window, and discard the corresponding original sensor data frame.

[0026] In specific implementation, key points of the human skeleton refer to the joints with clear anatomical significance in the human posture, which together constitute the topological structure of the human skeleton. For example, in this embodiment, key points of the human skeleton include at least joints at the head, shoulders, elbows, wrists, hips, knees, and ankles. The preset number can be set according to specific application scenarios and accuracy requirements, such as 14 or 18 key points. Further, each skeleton key point is represented by three-dimensional coordinates, which include the horizontal coordinates, vertical coordinates, and depth value of the key point in the image plane. The horizontal and vertical coordinates represent the pixel position of the key point in the two-dimensional image, and the depth value represents the distance from the key point to the sensor plane. Further, for infrared thermal imaging sensors, the depth value can be obtained through multi-view stereo matching, or by jointly calibrating the infrared thermal imaging sensor and the depth sensor and obtaining the corresponding depth value from the depth sensor; this invention does not specifically limit this. Further, for time-of-flight depth sensors, the depth value is directly extracted from the depth data output by the sensor.

[0027] Furthermore, during the extraction process, human detection and pose estimation algorithms are performed on each frame of sensor data. The human detection algorithm is used to locate the area where the human body is located in the sensor data, and the pose estimation algorithm is used to locate the precise positions of each skeletal key point within that area. Further, for infrared thermal imaging data, the characteristic that the thermal radiation intensity of the human body area is significantly higher than that of the background area can be used for human detection; for time-of-flight depth data, the characteristic that the depth value of the human body area is within a specific continuous range and has human shape features can be used for human detection. Further, the pose estimation algorithm can employ a deep convolutional neural network based on heatmap regression. This network takes the sensor image as input and outputs a confidence heatmap of each key point; the peak position in the heatmap is the predicted position of the corresponding key point.

[0028] Furthermore, after extraction is complete, the corresponding original sensor data frame is immediately discarded. This means the original sensor image data is only temporarily stored in memory during skeleton coordinate extraction and is permanently deleted afterward, without being written to any persistent storage device or transmitted to other devices over the network. This discarding operation ensures that no copy of the original sensor image data remains in the entire system, fundamentally eliminating the possibility of leakage of original images containing facial and physical features.

[0029] Furthermore, the skeleton coordinate data of multiple consecutive frames within the current time window are arranged according to the acquisition sequence to obtain the skeleton coordinate sequence corresponding to the current time window. This skeleton coordinate sequence is a three-dimensional tensor. Its first dimension is the number of frames, which is equal to the total number of frames within the current time window, i.e., the preset duration multiplied by the sampling frame rate. For example, when the preset duration is 1 second and the sampling frame rate is 30 frames per second, the number of frames is 30. The second dimension is the number of keypoints, which is a preset number, such as 14. The third dimension is the number of coordinate dimensions, which is 3, corresponding to the horizontal coordinate, vertical coordinate, and depth value, respectively. Each element in this three-dimensional tensor stores the value of a certain coordinate dimension of a certain keypoint in a certain frame.

[0030] In some preferred embodiments, the step of extracting coordinate information of key human skeleton points from each frame of sensor data within the current time window and generating a skeleton coordinate sequence corresponding to the current time window includes: performing human contour detection on each frame of sensor data within the current time window to obtain a human contour region; obtaining the sensor type corresponding to the sensor data stream; selecting a background segmentation strategy matching the sensor type; performing background segmentation on the human contour region based on the selected background segmentation strategy to obtain a foreground human contour; extracting the two-dimensional coordinates and depth values ​​of a preset number of key human skeleton points in the image plane based on the foreground human contour as initial coordinate data for each key human skeleton point; and performing confidence scoring on the initial coordinate data of each key human skeleton point to obtain a confidence score for each key human skeleton point. Value; mark the human skeleton key points with confidence scores lower than the fourth preset threshold as low-confidence human skeleton key points, and remove the low-confidence human skeleton key points from the current set of all human skeleton key points; for the removed low-confidence human skeleton key points, interpolate the coordinate data of the adjacent human skeleton key points in the current frame and the frames before and after the current frame to generate alternative coordinate data for the low-confidence human skeleton key points, and add the alternative coordinate data to the human skeleton key points; based on the coordinate data of all human skeleton key points after addition, form single-frame skeleton coordinate data corresponding to each frame of sensor data in the current time window; arrange the single-frame skeleton coordinate data of multiple consecutive frames in the current time window according to the acquisition time sequence to generate the skeleton coordinate sequence corresponding to the current time window.

[0031] In practical implementation, different sensor types exhibit different data characteristics during sensor data acquisition. Infrared thermal imaging sensors acquire thermal radiation intensity distribution data, where the human body region is characterized by a continuous area with significantly higher thermal radiation intensity than the background. Time-of-flight depth sensors acquire depth value data for each pixel, where the human body region is characterized by a connected area with depth values ​​within a specific continuous range and exhibiting human shape features. Therefore, this embodiment selects an appropriate background segmentation strategy based on the sensor type to obtain the foreground human body contour before skeleton extraction. Furthermore, it performs quality screening and missing point compensation on the extracted skeleton key points to ensure the integrity and accuracy of the skeleton coordinate sequence, providing high-quality input data for the subsequent spatiotemporal graph convolutional network. The specific implementation method is as follows: First, human contour detection is performed on each frame of sensor data within the current time window to obtain the human contour region. Human contour detection refers to image processing technology that identifies the region where a human body is located in sensor data. For infrared thermal imaging data, human contour detection can utilize the characteristic that the thermal radiation intensity of the human body region is significantly higher than that of the background region, and is achieved using a threshold-based segmentation method. Specifically, the thermal radiation intensity histogram of the entire frame image can be calculated, the optimal segmentation threshold can be determined using Otsu's method, pixels with thermal radiation intensity higher than this threshold are marked as foreground candidate regions, and then connected component analysis is used to filter out connected regions that conform to the size and shape characteristics of the human body as the human contour region. Furthermore, for time-of-flight depth data, human contour detection can utilize the characteristic that the depth value of the human body region is within a specific range, and is achieved using a depth threshold-based segmentation method. Specifically, a depth value range can be preset according to the sensor installation height and the spatial range of the monitoring area, pixels with depth values ​​falling within this range are marked as foreground candidate regions, and then connected component analysis and shape matching are used to filter out the human contour region.

[0032] Furthermore, the sensor type corresponding to the sensor data stream is obtained. The sensor type identifies the source device type of the current sensor data stream, including infrared thermal imaging type and time-of-flight depth type. The sensor type can be determined by reading the sensor device's model information or metadata fields in the data stream. Furthermore, during the system initialization phase, each data acquisition channel records its associated sensor type information upon registration, and this type information can be obtained from the channel configuration when processing each frame of data.

[0033] Furthermore, a background segmentation strategy matching the sensor type is selected based on the sensor type. Different sensor types have different image characteristics. Infrared thermal imaging data reflects the distribution of thermal radiation intensity, and there is a distinguishable difference between background temperature and human body temperature. However, the thermal radiation intensity of the background environment may change over time, for example, during day-night cycles or weather changes, the background temperature will change significantly, and a fixed segmentation threshold is difficult to adapt to such dynamic changes. Furthermore, time-of-flight depth data reflects distance information, and there is a distinguishable difference between background distance and human body distance. However, the background depth layer may vary with the sensor's installation location and the spatial layout of the monitoring area. Different installation locations will produce different background depth distributions, making it impossible to pre-set a fixed background depth value. Therefore, different background segmentation strategies need to be selected based on the sensor type to adapt to the distribution characteristics of different data types.

[0034] Furthermore, based on the selected background segmentation strategy, background segmentation is performed on the human body contour region to obtain the foreground human body contour. Background segmentation is the process of removing background pixels from the human body contour region, retaining only the foreground pixels belonging to the human body. The foreground human body contour obtained after background segmentation is a set of human body pixels after removing background interference, providing clean input data for subsequent skeleton keypoint extraction.

[0035] Furthermore, based on the foreground human body contour, a preset number of skeletal key points are extracted in the image plane along with their two-dimensional coordinates and depth values, serving as the initial coordinate data for each skeletal key point. The preset number can be set according to specific application scenarios and accuracy requirements, for example, 14 or 18 key points, covering the main joint positions of the human body. The positions of each skeletal key point are located from the foreground human body contour using a human pose estimation algorithm. Further, the human pose estimation algorithm can employ a deep learning-based heatmap regression method. This method takes the foreground human body contour image as input and outputs a confidence heatmap of each key point through a deep convolutional neural network. The peak position in the heatmap is the predicted position of the corresponding key point. Further, each key point is represented by a triple containing horizontal coordinates, vertical coordinates, and a depth value, where the horizontal and vertical coordinates represent the pixel position of the key point in the image plane, and the depth value represents the distance from the key point to the sensor plane. Furthermore, for infrared thermal imaging sensors, depth values ​​can be obtained through multi-view stereo matching methods, or by jointly calibrating the infrared thermal imaging sensor and the depth sensor and then obtaining the corresponding depth value from the depth sensor; for time-of-flight depth sensors, depth values ​​are directly extracted from the depth data output by the sensor.

[0036] Furthermore, a confidence score is applied to the initial coordinate data of each human skeleton keypoint to obtain a confidence score value for each keypoint. The confidence score is a numerical value output by the human pose estimation algorithm for each detected keypoint, representing the reliability of the keypoint's localization result. In the heatmap regression method, the confidence score for each keypoint can be extracted from its corresponding heatmap. Specifically, the response value at the peak position in the heatmap is used as the confidence score for that keypoint, reflecting the network's certainty in locating the keypoint. The confidence score ranges from 0 to 1; a higher score indicates a more reliable localization result and higher network certainty in predicting its location; a lower score indicates a less reliable localization result, potentially due to factors such as occlusion, lighting variations, sensor noise, or limitations of the pose estimation algorithm.

[0037] Further, key points of the human skeleton with a confidence score lower than a fourth preset threshold are marked as low-confidence key points of the human skeleton, and these low-confidence key points are removed from the current set of all key points of the human skeleton. Further, the fourth preset threshold is a pre-set confidence threshold value used to determine the reliability of key points. The setting of this fourth preset threshold needs to balance the relationship between accuracy and completeness: the higher the threshold, the more key points are removed, resulting in higher quality but fewer remaining skeleton key points; the lower the threshold, the fewer key points are removed, resulting in higher completeness of the skeleton key points but potentially including unreliable positioning results. Further, in this embodiment, the fourth preset threshold can be set to 0.5, meaning that key points with a confidence score lower than 0.5 are considered unreliable in their positioning and need to be removed. The removal operation refers to removing key points with a confidence score lower than the fourth preset threshold from the set of skeleton key points in the current frame, temporarily reducing the number of skeleton key points in that frame to a preset number. While removing key points with low confidence, the identity of the removed key points is recorded so that they can be interpolated and completed later.

[0038] Furthermore, for the eliminated low-confidence human skeleton keypoints, interpolation is performed based on the coordinate data of adjacent human skeleton keypoints in the current frame and multiple frames before and after the current frame to generate alternative coordinate data for the low-confidence human skeleton keypoints, and this alternative coordinate data is then added to the human skeleton keypoints. Further, interpolation is a numerical calculation method that uses known data points to estimate unknown data points. In this embodiment, interpolation needs to comprehensively utilize information from both the spatial and temporal domains. Further, in the spatial domain, keypoints directly adjacent to the eliminated keypoints in the human skeleton topology are used as references. For example, for the eliminated elbow keypoint, its adjacent keypoints include shoulder and wrist keypoints. Using the coordinates of these two adjacent keypoints in the current frame, the position of the elbow keypoint can be estimated through linear interpolation or bone length constraints. Further, in the temporal domain, the historical position information of the eliminated keypoints in multiple frames before and after the current frame is used to estimate the position of the keypoint in the current frame through linear interpolation or spline interpolation. The spatial domain interpolation results and the temporal domain interpolation results are weighted and fused to obtain the final alternative coordinate data. After the infeasibility operation is completed, the number of all human skeleton keypoints in the current frame is restored to the preset number, forming a complete set of skeleton keypoints without any missing points.

[0039] Furthermore, based on the coordinate data of all the key points of the human skeleton after supplementation, single-frame skeleton coordinate data corresponding to each frame of sensor data within the current time window is formed. The single-frame skeleton coordinate data is a set of coordinates for all preset number of skeleton key points in the current frame. Its data structure is a matrix with dimensions of the number of key points and the number of coordinate dimensions. The number of key points is a preset number, and the number of coordinate dimensions is 3, corresponding to the horizontal coordinate, vertical coordinate, and depth value, respectively. Each row in the matrix corresponds to one skeleton key point, and each column corresponds to one coordinate dimension.

[0040] Furthermore, the single-frame skeleton coordinate data of multiple consecutive frames within the current time window are arranged according to the acquisition sequence to generate the skeleton coordinate sequence corresponding to the current time window. The single-frame skeleton coordinate data corresponding to each frame within the current time window are arranged sequentially in chronological order to form a three-dimensional tensor with dimensions of frame number, keypoint number, and coordinate dimension. This tensor is the skeleton coordinate sequence corresponding to the current time window and can be directly used as the inference input for subsequent spatiotemporal graph convolutional networks. Further, the first dimension of this three-dimensional tensor is the time dimension, corresponding to the number of frames within the current time window; the second dimension is the spatial dimension, corresponding to the number of skeleton keypoints; and the third dimension is the feature dimension, corresponding to the three features: horizontal coordinates, vertical coordinates, and depth value.

[0041] This embodiment improves the accuracy and robustness of foreground human contour extraction by selecting a matching background segmentation strategy based on sensor type, enabling the background segmentation operation to adapt to the different distribution characteristics of infrared thermal imaging data and time-of-flight depth data. Furthermore, by performing confidence scoring on the extracted human skeleton key points and removing low-confidence key points with confidence scores below a fourth preset threshold from the current skeleton key point set, the quality of the skeleton data input to subsequent processing stages is ensured, avoiding the negative impact of low-quality key points on fall reasoning accuracy. Further, by interpolating the coordinates of adjacent key points in the human skeleton topology in the current frame and the coordinates of the same key point in multiple frames before and after the current frame to generate replacement coordinate data for the removed low-confidence key points, the integrity and continuity of the skeleton key point set are maintained. This avoids skeleton sequence interruptions due to missing key points, ensuring the integrity of the skeleton coordinate sequence in both time and space dimensions, and providing continuous and complete input data for the spatiotemporal graph convolutional network. Furthermore, this combination of confidence scoring and interpolation completion can effectively address common interference factors in real-world scenarios, such as sensor noise and attitude occlusion, thereby improving the stability and reliability of the system in complex environments.

[0042] In some preferred embodiments, performing background segmentation on the human body contour region based on the selected background segmentation strategy to obtain the foreground human body contour includes: if the sensor type is infrared thermal imaging, acquiring the thermal radiation intensity value of each pixel in the sensor data; calculating the thermal radiation intensity statistical characteristics of the human body contour region based on the thermal radiation intensity distribution within the human body contour region, the thermal radiation intensity statistical characteristics including the mean thermal radiation intensity and the standard deviation of thermal radiation intensity; and determining a dynamic segmentation threshold matching the sensor type based on the thermal radiation intensity statistical characteristics, the dynamic segmentation threshold being the thermal radiation intensity... The weighted sum of the mean intensity and the standard deviation of the thermal radiation intensity; pixels with thermal radiation intensity values ​​below the dynamic segmentation threshold within the human body contour area are identified as background pixels and removed to obtain the foreground human body contour; if the sensor type is time-of-flight depth type, the depth value of each pixel in the sensor data is obtained; histogram statistics are performed on all depth values ​​within the human body contour area, and the depth layer corresponding to the depth value with the highest frequency in the histogram is determined as the background depth layer; pixels within the human body contour area whose depth values ​​belong to the background depth layer are identified as background pixels and removed to obtain the foreground human body contour.

[0043] In practice, before extracting key points of the human skeleton from sensor data, the foreground human region needs to be separated from the original sensor image; this separation process is called background segmentation. The accuracy of background segmentation directly affects the accuracy of subsequent skeleton key point extraction. Furthermore, infrared thermal imaging sensors collect the thermal radiation intensity values ​​of each pixel. The thermal radiation intensity of the human region is usually higher than that of the background environment, but the thermal radiation intensity of the background environment may change over time. For example, in summer when the ambient temperature is high, the thermal radiation intensity of the background may rise to near the human body temperature, resulting in a decrease in the contrast between the human body and the background; in winter when the ambient temperature is low, the thermal radiation intensity of the background is significantly lower than the human body temperature, resulting in a higher contrast between the human body and the background. Furthermore, time-of-flight depth sensors collect the depth values ​​of each pixel. The background depth layer depends on the sensor's installation location and the spatial layout of the monitoring area. For example, when the sensor is installed in a corner, the background depth layer is the depth value of each wall surface; when the sensor is installed on the ceiling, the background depth layer is the depth value of the ground. Therefore, this embodiment designs adaptive background segmentation methods for the above two sensor types, enabling background segmentation to dynamically determine segmentation parameters based on the actual distribution of the current frame data, adapting to changes in data characteristics under different scenarios. The specific implementation is as follows: If the sensor type is infrared thermal imaging, the following steps are performed first: Obtain the thermal radiation intensity value of each pixel in the sensor data. The raw data collected by the infrared thermal imaging sensor is the thermal radiation intensity value corresponding to each pixel, which reflects the temperature distribution of the surface of the object being measured. Further, the thermal radiation intensity value is usually represented in grayscale value form; the higher the grayscale value, the higher the surface temperature of the object corresponding to that pixel. Further, traverse all pixels in the sensor data, extract the thermal radiation intensity value of each pixel, and form a two-dimensional thermal radiation intensity matrix with the same size as the image.

[0044] Further, based on the distribution of thermal radiation intensity within the human body contour region, statistical characteristics of the thermal radiation intensity of the human body contour region are calculated. These statistical characteristics include the mean thermal radiation intensity and the standard deviation of the thermal radiation intensity. Further, the human body contour region is a region containing the human body pre-located using a human detection algorithm. The pixels within this region include human body pixels and some background pixels. All pixels within the human body contour region are traversed, and the thermal radiation intensity value of each pixel is extracted. The arithmetic mean of these values ​​is calculated to obtain the mean thermal radiation intensity, which reflects the central trend of thermal radiation intensity within the human body contour region. Further, the dispersion of the thermal radiation intensity value of each pixel within the human body contour region relative to this mean is calculated to obtain the standard deviation of the thermal radiation intensity. This standard deviation reflects the degree of fluctuation in thermal radiation intensity within the human body contour region; the greater the fluctuation, the more significant the difference in thermal radiation intensity between human body pixels and background pixels within the human body contour region.

[0045] Further, based on the statistical characteristics of the thermal radiation intensity, a dynamic segmentation threshold matching the sensor type is determined. This dynamic segmentation threshold is a weighted sum of the mean thermal radiation intensity and the standard deviation of the thermal radiation intensity. Further, the dynamic segmentation threshold is a segmentation threshold dynamically calculated based on the thermal radiation intensity distribution of the human body contour region in the current frame. This threshold is not a pre-set fixed value but is calculated in real-time based on the actual distribution of data in each frame. Further, the mean thermal radiation intensity and the standard deviation of the thermal radiation intensity are multiplied by their respective weighting coefficients and then added together to obtain the dynamic segmentation threshold. For example, the dynamic segmentation threshold can be expressed as the mean thermal radiation intensity plus a preset weighting coefficient multiplied by the standard deviation of the thermal radiation intensity. The value of the weighting coefficient can be adjusted according to the actual scene; the larger the value, the higher the segmentation threshold, and more pixels are judged as background; the smaller the value, the lower the segmentation threshold, and more pixels are judged as foreground. Further, in this embodiment, the weighting coefficient can be set to 0.5. Through this dynamic calculation method, the segmentation threshold can be adaptively adjusted according to the changes in the thermal radiation intensity distribution of the human body contour area. When the ambient temperature rises and the background thermal radiation intensity increases, the mean and standard deviation of thermal radiation intensity change accordingly, and the dynamic segmentation threshold is also adjusted accordingly, maintaining effective distinction between the human body and the background.

[0046] Further, pixels within the human body contour region whose thermal radiation intensity values ​​are lower than the dynamic segmentation threshold are identified as background pixels and removed to obtain the foreground human body contour. The thermal radiation intensity of the human body region is typically higher than that of the background region; therefore, pixels with thermal radiation intensity values ​​lower than the dynamic segmentation threshold are determined to be background pixels, which may correspond to environmental objects such as the ground or walls. These background pixels are then removed from the human body contour region, and the remaining pixels with thermal radiation intensity values ​​higher than or equal to the dynamic segmentation threshold constitute the foreground human body contour. This foreground human body contour includes the main areas of the human body, providing an accurate input region for subsequent skeletal keypoint extraction.

[0047] Further, if the sensor type is time-of-flight depth, the following steps are performed first: Obtain the depth value of each pixel in the sensor data. A time-of-flight depth sensor calculates the distance from each pixel to the sensor by measuring the flight time of a light pulse from emission to reflection; this distance value is the depth value. The unit of depth value is typically millimeters or centimeters. Traverse all pixels in the sensor data, extract the depth value of each pixel, and form a two-dimensional depth matrix with the same size as the image.

[0048] Furthermore, histogram statistics are performed on all depth values ​​within the human contour region to determine the depth layer corresponding to the most frequent depth value in the histogram as the background depth layer. Histogram statistics divide the depth value range into several consecutive depth layers and count the number of pixels in each depth layer. Each depth layer corresponds to a depth value range; for example, if the depth range is 0 to 10 meters, it can be divided into 100 depth layers, each with a width of 0.1 meters. Further, all pixels within the human contour region are traversed, and each pixel is assigned to its corresponding depth layer based on its depth value, counting the number of pixels in each depth layer. The depth layer with the highest frequency corresponds to the depth value that accounts for the largest proportion within the human contour region. This depth layer typically corresponds to the depth value of the background region, as the background region usually occupies a large area in the image. Further, the human contour region is the area where the human body is located in the original sensor image. The periphery of this region usually contains some background pixels, which often occupy the largest proportion in the depth histogram. The depth value range corresponding to this most frequent depth layer is determined as the background depth layer.

[0049] Further, pixels within the human body contour region whose depth values ​​belong to the background depth layer are identified as background pixels and removed to obtain the foreground human body contour. Pixels within the human body contour region whose depth values ​​fall within the background depth layer are determined to be background pixels and removed; the remaining pixels whose depth values ​​do not belong to the background depth layer constitute the foreground human body contour. Furthermore, in time-of-flight depth data, the depth values ​​of the human body region typically differ from those of the background region; therefore, pixels not belonging to the background depth layer primarily correspond to the human body region.

[0050] This embodiment targets infrared thermal imaging sensors. By calculating the mean and standard deviation of thermal radiation intensity within the human contour region, a dynamic segmentation threshold is determined based on these statistical characteristics. This allows the segmentation threshold to adaptively adjust with the dynamic changes in thermal radiation intensity distribution, avoiding segmentation failure caused by fixed thresholds when ambient temperature changes. Furthermore, for time-of-flight depth sensors, histogram statistics are performed on the depth values ​​within the human contour region. The depth layer with the highest frequency is identified as the background depth layer and removed, fully utilizing the characteristic that the background region usually occupies the largest proportion, achieving adaptive background recognition and removal. Moreover, the dynamic segmentation methods adapted to each sensor type fully utilize the physical characteristics of different types of sensor data, ensuring high accuracy and robustness in foreground human contour extraction under different sensor data conditions, providing a reliable data foundation for the accurate extraction of subsequent skeletal key points.

[0051] S3. Based on the skeleton coordinate sequence, inference is performed through a pre-trained spatiotemporal graph convolutional network to obtain the fall confidence of the current time window.

[0052] In specific implementation, the Spatiotemporal Graph Convolutional Network (SGMCN) is a deep learning network architecture used to process graph-structured temporal data. This network is specifically designed for processing temporal data organized in a graph structure, such as human skeleton sequences. In the SGMCN, the human skeleton is modeled as a graph, where key points of the skeleton serve as nodes and natural connections of the skeleton serve as edges. The network extracts spatial dependencies between key points through graph convolution operations and extracts temporal motion features between frames through temporal convolution operations, ultimately combining spatial and temporal information to output a classification result. Further, in this embodiment, the output of the SGMCN is a single numerical value, namely the fall confidence score. This confidence score is a probability value ranging from 0 to 1, representing the degree of matching between the human posture change pattern and the fall action pattern within the current time window. Furthermore, the closer the confidence score is to 1, the closer the current posture change is to a fall action; further, the closer the confidence score is to 0, the lower the similarity between the current posture change and the fall action. Furthermore, pre-training refers to training the network parameters in advance on a large dataset of skeleton sequences labeled with fall actions and everyday actions, enabling the network to distinguish between fall actions and everyday actions. This pre-training can be performed on general public datasets or fine-tuned on customized datasets for specific scenarios; this invention is not specifically limited to these.

[0053] In some preferred embodiments, the step of obtaining the fall confidence score of the current time window based on the skeleton coordinate sequence through inference using a pre-trained spatiotemporal graph convolutional network includes: normalizing the coordinates of each human skeleton key point in the skeleton coordinate sequence, aligning the coordinates of each human skeleton key point to a unified standard pose space, and obtaining a normalized skeleton coordinate sequence; obtaining the predefined natural connection topology of the human skeleton in the pre-trained spatiotemporal graph convolutional network, and constructing a spatial adjacency matrix based on the natural connection topology of the human skeleton; and inputting the normalized skeleton coordinate sequence into the pre-trained spatiotemporal graph convolutional network, which includes graph convolutional layers, temporal convolutional layers, etc. An attention layer and a fully connected layer are configured, with the attention layer positioned after the graph convolutional layer and the temporal convolutional layer, and before the fully connected layer. The graph convolutional layer extracts spatial dependencies between key points of the human skeleton based on the spatial adjacency matrix, obtaining a spatial feature map. The temporal convolutional layer performs convolution operations along the temporal dimension on the spatial feature map to extract temporal motion features between frames within the current time window, obtaining a temporal feature map. The attention layer weights the temporal feature map along both the channel and spatial dimensions, obtaining an enhanced temporal feature map. The enhanced temporal feature map is then input into the fully connected layer for mapping, outputting the fall confidence score for the current time window.

[0054] In specific implementation, the spatiotemporal graph convolutional network (SPCRN) is a deep learning architecture specifically designed for processing graph-structured temporal data. It models the human skeleton as a graph structure, uses graph convolution operations to extract spatial dependencies between key points of the skeleton, and uses temporal convolution operations to extract temporal motion features between frames, thereby achieving end-to-end recognition of human movement patterns. In this embodiment, the input to the SPCRN is a normalized sequence of skeleton coordinates, and the output is a fall confidence score. Further, before inputting the data into the network, the skeleton coordinate data needs to be normalized to eliminate the influence of individual body size differences and sensor installation location differences on the data value range. Further, within the network, spatial dependencies are extracted through graph convolutional layers, temporal motion features are extracted through temporal convolutional layers, key features are enhanced through attention layers to improve discriminative ability, and finally, high-dimensional features are mapped to a single fall confidence score value through fully connected layers. The specific implementation is as follows: First, the coordinates of each key point in the skeleton coordinate sequence are normalized to align them to a unified standard pose space, resulting in a normalized skeleton coordinate sequence. The purpose of normalization is to eliminate the influence of differences in height and body shape among individuals, as well as differences in sensor installation height and angle, on the numerical range of the skeleton coordinate data. Differences in height and body shape among elderly individuals lead to variations in the absolute coordinate positions of key points in the same pose; different sensor installation positions also result in different positions and scales of the skeleton coordinates of the same person in the image coordinate system. Without normalization, the spatiotemporal graph convolutional network would need to learn these differences unrelated to pose discrimination, increasing the learning difficulty and reducing generalization ability. Furthermore, the normalization process is as follows: First, a reference key point is determined in the skeleton coordinate sequence, for example, the center of the hip is used as the reference origin. The coordinates of all skeleton key points are subtracted from the coordinates of the reference origin to achieve coordinate translation and alignment. Then, a reference scale is calculated, such as the torso length, i.e., the distance from the center of the shoulder to the center of the hip. The coordinates of all skeleton key points are divided by this reference scale to achieve coordinate scale normalization. Rotation alignment can also be performed as needed, for example, rotating the torso direction to the vertical direction to eliminate rotational differences caused by the sensor installation angle. Furthermore, through the above normalization process, all skeleton coordinate sequences are transformed into a unified standard attitude space coordinate system, eliminating individual and installation differences unrelated to attitude discrimination.

[0055] Further, the predefined natural connectivity topology of the human skeleton in the pre-trained spatiotemporal graph convolutional network is obtained, and a spatial adjacency matrix is ​​constructed based on the natural connectivity topology. The natural connectivity topology of the human skeleton refers to the skeletal connections defined according to the anatomical structure of the human body. For example, there is a skeletal connection between shoulder keypoints and elbow keypoints, between elbow keypoints and wrist keypoints, between hip keypoints and knee keypoints, and between knee keypoints and ankle keypoints. Further, in the spatiotemporal graph convolutional network, these connections are modeled as edges between nodes in the graph. Further, the spatial adjacency matrix is ​​a two-dimensional square matrix used to describe the connectivity between nodes in the graph structure. The rows and columns of the matrix correspond to the indices of each human skeleton keypoint, and the value of each element in the matrix indicates whether there is a skeletal connection between two corresponding keypoints; if a connection exists, the value is 1, and if no connection exists, the value is 0. Furthermore, the spatial adjacency matrix is ​​the computational basis for graph convolution operations. It determines which key points in the graph convolutional layer can directly perform feature aggregation and information transfer. According to this matrix, each key point only collects information from its neighboring key points, which conforms to the physical constraints of human posture.

[0056] Further, the normalized skeleton coordinate sequence is input into the pre-trained spatiotemporal graph convolutional network, which includes graph convolutional layers, temporal convolutional layers, attention layers, and fully connected layers. The attention layer is placed after the graph convolutional layers and the temporal convolutional layers, and before the fully connected layers. The structure of the spatiotemporal graph convolutional network is organized in the following order: first, spatial dependencies are extracted through graph convolutional layers; then, temporal motion features are extracted through temporal convolutional layers; then, key features are enhanced through attention layers; and finally, the features are mapped to classification results through fully connected layers.

[0057] Furthermore, the spatial dependencies between key points of the human skeleton are extracted using the graph convolutional layer based on the spatial adjacency matrix, resulting in a spatial feature map. The graph convolutional layer is one of the core components of the spatiotemporal graph convolutional network, and its function is to perform convolution operations on graph-structured data. The basic principle of graph convolution is as follows: for each key point node, its own features and the features of all its neighboring nodes are weighted and aggregated according to the adjacency relationship defined by the spatial adjacency matrix, and then the output features of that node are generated through linear transformation and activation functions. By stacking multiple layers of graph convolutions, the network can progressively expand its receptive field, enabling the output features of each key point to fuse information from key points at greater distances, ultimately capturing the global spatial dependencies between all key points in the human skeleton. The output of the graph convolutional layer is a spatial feature map, the dimensions of which are the number of frames, the number of feature channels, and the number of key points. The number of feature channels is determined by the network structure, for example, it can be set to 64 channels or 128 channels.

[0058] Furthermore, the spatial feature map is convolved along the time dimension by the temporal convolutional layer to extract the temporal motion features between frames within the current time window, resulting in a temporal feature map. The temporal convolutional layer is another core component of the spatiotemporal graph convolutional network, its function being to extract dynamically changing features along the time dimension. The basic principle of temporal convolution is: a one-dimensional convolutional kernel is applied to the temporal dimension of the spatial feature map. The kernel slides along the time axis, performing convolution calculations on the same key points and feature channels of consecutive frames, capturing the change patterns between frames. Different sized convolutional kernels can capture motion patterns at different time scales; for example, small-sized kernels can capture rapid pose changes, while large-sized kernels can capture slow, continuous motion. Through temporal convolution, the network can identify the temporal evolution process from a standing posture to a falling posture. The output of the temporal convolutional layer is a temporal feature map, which contains both spatial dependency information and temporal motion information.

[0059] Furthermore, the attention layer weights the temporal feature map along both the channel and spatial dimensions to obtain an enhanced temporal feature map. The attention layer is an adaptive feature recalibration module that, without increasing the number of network parameters, enhances the feature map by calculating the importance weights of each channel and spatial location, enabling the network to focus on the features most discriminative for fall detection. The attention layer contains two parallel branches: a channel attention branch and a spatial attention branch. In the channel attention branch, the temporal feature map is first subjected to global average pooling and global max pooling to obtain global description vectors along the channel dimension. These two description vectors are then mapped to channel weight vectors using a shared multilayer perceptron. An activation function normalizes the weights to between 0 and 1. Finally, the original feature map is multiplied channel by channel weight vectors to achieve feature enhancement along the channel dimension. In the spatial attention branch, the temporal feature map is first subjected to average pooling and max pooling along the channel dimension, resulting in two spatial attention maps. These two maps are concatenated and then passed through a convolutional layer to generate a spatial weight map. An activation function is used to normalize the weights to between 0 and 1. Finally, the original feature map and the spatial weight map are multiplied element-wise to achieve feature enhancement in the spatial dimension. Furthermore, channel attention enables the network to identify which feature channels contribute most to fall detection, while spatial attention enables the network to identify which key point locations contribute most to fall detection. Through the joint enhancement of channel and spatial attention, the network can focus on the key features that are most discriminative for fall detection.

[0060] Furthermore, the enhanced temporal feature map is input into the fully connected layer for mapping, outputting the fall confidence score for the current time window. The fully connected layer is a common linear mapping layer in deep learning networks. It flattens the input high-dimensional features into a one-dimensional feature vector and maps this one-dimensional feature vector to an output vector of a preset dimension through a linear transformation. In this embodiment, the output dimension of the fully connected layer is 1, i.e., a single numerical value. After processing by the activation function, this value is compressed to between 0 and 1, serving as the fall confidence score output for the current time window.

[0061] This embodiment normalizes the skeleton coordinate sequence, aligning the coordinates of key points of each human skeleton to a unified standard pose space. This eliminates the influence of differences in individual height, body shape, and sensor installation location on the skeleton coordinate data, enabling the spatiotemporal graph convolutional network to have consistent recognition capabilities for data from different individuals and under different installation conditions, thus improving the network's generalization ability. Furthermore, by constructing a spatial adjacency matrix based on the natural connection topology of the human skeleton, the graph convolutional layers can extract the spatial dependencies between key points according to the human anatomical structure, conforming to the physical constraints of human posture and avoiding invalid information transmission between unconnected key points. Furthermore, the combination of graph convolutional layers and temporal convolutional layers simultaneously captures the spatial structural information and temporal dynamic information of the skeleton, enabling the network to distinguish subtle differences between fall actions and everyday actions. The combined use of spatiotemporal features improves the accuracy of action recognition. Furthermore, by setting an attention layer after the graph convolutional layer and the temporal convolutional layer and before the fully connected layer, the temporal feature map is weighted along the channel dimension and the spatial dimension respectively, enabling the network to adaptively focus on the feature channels and key point locations that are most discriminative for fall detection. The dual enhancement mechanism of channel attention and spatial attention improves the discriminative power of feature representation, further improving the accuracy and robustness of fall confidence inference.

[0062] Furthermore, it should be noted that the spatiotemporal graph convolutional network is a deep learning architecture based on graph neural networks, used to process human skeleton sequence data. This network includes graph convolutional layers, temporal convolutional layers, attention layers, and fully connected layers. The graph convolutional layers are used to extract spatial dependencies between key points of each human skeleton based on a spatial adjacency matrix, which is constructed according to the natural connection topology of the human skeleton. The temporal convolutional layers are used to perform one-dimensional convolution operations along the temporal dimension on the spatial feature map output by the graph convolutional layers to extract temporal motion features between frames. The attention layer is used to weight the temporal feature map along both the channel and spatial dimensions to enhance the features most discriminative for fall detection. The fully connected layers are used to map the enhanced features to fall confidence scores. Furthermore, the network employs end-to-end supervised learning during training, using the cross-entropy loss function as the optimization objective, updating network parameters using a stochastic gradient descent optimizer or an adaptive moment estimation optimizer, and adjusting the learning rate using a piecewise constant decay strategy. Furthermore, the training data for the network comes from publicly available skeleton action recognition datasets, including the NTU RGB+D dataset, the NTU RGB+D 120 dataset, and a self-built fall action dataset. Further, before training, the skeleton sequence samples in the training dataset undergo coordinate normalization, temporal dimension unification, and data augmentation preprocessing. Further, in an optional implementation, a transfer learning strategy is employed, pre-training on a large-scale public dataset followed by fine-tuning on the self-built fall action dataset, enabling the network to adapt to the fall action recognition requirements of specific application scenarios.

[0063] Furthermore, it should be noted that the self-built fall action dataset is constructed as follows: In elderly care institutions or simulated elderly care scenarios, infrared thermal imaging sensors and time-of-flight depth sensors are deployed according to a preset acquisition protocol to collect sensor data streams containing fall actions and daily activities. The acquisition protocol specifies the sensor's installation height, installation angle, monitoring area range, and sampling frame rate. Further, for each acquisition, the action type label, action start time, and action end time are recorded. The action type label includes at least two categories: fall actions and daily activities. Daily activities include walking, sitting down, standing up, bending over to pick up an object, squatting quickly, lying down, and turning over. Further, the fall actions include different fall directions such as falling forward from a standing position, falling backward, falling sideways, and falling from a sitting position. After data acquisition is completed, a pose estimation algorithm is used to extract the three-dimensional coordinate sequence of key points of the human skeleton from the acquired sensor data, forming a skeleton sequence sample. Furthermore, for data acquired by infrared thermal imaging sensors, depth values ​​are obtained using multi-view stereo matching or joint calibration with depth sensors; for data acquired by time-of-flight depth sensors, the depth values ​​output by the sensors are directly extracted. The annotation protocol for the skeleton sequence samples is as follows: two trained annotators independently annotate the action type of each skeleton sequence segment. When the annotation results of the two annotators are consistent, the annotation result is used as the final label; when the annotation results of the two annotators are inconsistent, a third senior annotator arbitrates, and the arbitration result is used as the final label. Furthermore, the self-built fall action dataset contains no fewer than 5000 skeleton sequence samples, of which no fewer than 1500 are fall action samples and no fewer than 3500 are daily action samples. Samples of each action category are divided into training, validation, and test sets according to a preset ratio.

[0064] Furthermore, the normalization process employs the same method in both the training and inference phases. During the training phase, for each skeleton sequence sample in the training dataset, normalization is performed as follows: using the hip center as the reference origin, the coordinates of all key points of the human skeleton are subtracted from the coordinates of the reference origin to obtain the translated skeleton coordinates; the distance from the shoulder center to the hip center is calculated as the reference scale, and the translated skeleton coordinates are divided by the reference scale to obtain the scale-normalized skeleton coordinates; the angle between the torso direction and the vertical direction is calculated, and the scale-normalized skeleton coordinates are rotated according to the angle to align the torso direction with the vertical direction, resulting in a normalized skeleton coordinate sequence. Furthermore, during the inference phase, the skeleton coordinate sequence corresponding to the current time window undergoes the same normalization process as in the training phase, i.e., the same method for determining the reference origin, calculating the reference scale, and performing the same rotation and alignment. Furthermore, the reference origin, reference scale, and rotation angle used in the normalization process are all calculated from the current skeleton coordinate sequence itself, without relying on global statistics in the training dataset, thereby ensuring that the normalization processing logic is consistent between the training and inference phases.

[0065] Furthermore, since falls occur infrequently in real-world scenarios, the number of fall action samples in the self-built fall action dataset is less than the number of daily action samples, resulting in class imbalance. To mitigate the impact of class imbalance on network training, at least one of the following methods is employed: First, different weight coefficients are set for different action categories in the cross-entropy loss function, with the weight coefficient for fall actions being greater than that for daily actions. These weight coefficients are determined based on the reciprocal or square root of the number of samples in each category. Second, an oversampling strategy is used for fall action samples during training, i.e., fall action samples are repeatedly selected according to a preset sampling ratio, ensuring that the ratio of fall action samples to daily action samples in each training batch remains within a preset range. Third, a data augmentation strategy is applied to the fall action samples, including at least one of random scaling on the time axis, random rotation in space, random perturbation of keypoint coordinates, and random occlusion of some keypoints. Through these methods, the spatiotemporal graph convolutional network can fully learn the features of fall actions during training, avoiding a decrease in the network's ability to recognize fall actions due to class imbalance.

[0066] Furthermore, the attention layer adopts a serial structure, that is, channel attention processing is performed first, followed by spatial attention processing. Specifically, the channel attention processing is implemented as follows: global average pooling and global max pooling are performed on the temporal feature map to obtain two channel description vectors with the number of feature channels as their dimension; the two channel description vectors are input into a shared multilayer perceptron, which contains two fully connected layers. The number of output channels of the first fully connected layer is one-quarter of the number of feature channels, and the number of output channels of the second fully connected layer is equal to the number of feature channels; the two output vectors of the multilayer perceptron are added element-wise and processed by an activation function to obtain a channel attention weight vector; the temporal feature map is multiplied by the channel attention weight vector channel-wise to obtain the channel-enhanced temporal feature map. Further, the spatial attention processing is implemented as follows: Global average pooling and global max pooling are performed on the channel-enhanced temporal feature map along the feature channel dimension to obtain two spatial description maps whose dimension is the number of time frames multiplied by the number of keypoints; the two spatial description maps are concatenated along the feature channel dimension to obtain a concatenated spatial description map; the concatenated spatial description map is input into a convolutional layer with a kernel size of 7x7, and after processing by an activation function, a spatial attention weight map is obtained; the channel-enhanced temporal feature map is multiplied element-wise with the spatial attention weight map to obtain an enhanced temporal feature map. Further, the channel attention processing and the spatial attention processing together constitute the attention layer, which is positioned after the graph convolutional layer and the temporal convolutional layer, and before the fully connected layer.

[0067] S4, determine whether the fall confidence level is greater than or equal to the first preset threshold.

[0068] In practice, the first preset threshold is a pre-set probability threshold value used to determine whether to enter the fall event confirmation process. The size of this threshold directly affects the system's sensitivity and false alarm rate: the lower the threshold is set, the more sensitive the system is to suspected fall actions, reducing the false alarm rate but increasing the false alarm rate; the higher the threshold is set, the more conservative the system's judgment of suspected fall actions, reducing the false alarm rate but increasing the false alarm rate. Therefore, the first preset threshold needs to be set in a balance between sensitivity and false alarm rate based on the specific requirements of the actual application scenario. In this embodiment, the first preset threshold can be set to 0.75, that is, only when the fall confidence level of the current time window is not lower than 0.75 is it considered that a suspected fall action has occurred, worthy of entering the subsequent confirmation process; if the fall confidence level is lower than 0.75, it is considered that the posture change within the current time window is insufficient to constitute a suspected fall, and the processing of the current time window is directly terminated.

[0069] S5, if the fall confidence level is greater than or equal to the first preset threshold, then the current moment is taken as the start moment of the stationary timing.

[0070] In practice, the current moment refers to the moment when the confidence level for detecting a fall reaches the first preset threshold, and this moment is recorded based on the system clock. This moment serves as the zero point for subsequent static state timing, marking the switch from the action recognition stage to the state verification stage. Furthermore, in the action recognition stage, a spatiotemporal graph convolutional network is used to analyze posture change patterns within a short time window; in the state verification stage, the focus shifts to whether the human body remains stationary after a suspected fall. Moreover, using the current moment as the starting point for static timing signifies the beginning of a continuous monitoring period during which the movement state of the monitored human body is evaluated in real time.

[0071] S6, starting from the start time of the static timing, continuously acquire the displacement rate of key points of the human skeleton in each frame of sensor data.

[0072] In specific implementation, after entering the static monitoring mode, the fall confidence inference is no longer based on the spatiotemporal graph convolutional network. Instead, the displacement rate of the human skeleton key points in subsequent frames from the start of the static timing is directly calculated. Displacement rate is a physical quantity that measures the intensity of human movement, representing the change in position of the human skeleton key points per unit time. For each newly received frame of sensor data, the coordinate information of the human skeleton key points in that frame is first extracted. Then, the coordinate change of each skeleton key point in that frame relative to the same skeleton key point in the previous frame is calculated, i.e., the Euclidean distance. This coordinate change is then divided by the inter-frame time interval to obtain the inter-frame displacement rate of each skeleton key point. Furthermore, the average or weighted average of the inter-frame displacement rates of all skeleton key points can be taken to obtain the comprehensive displacement rate of that frame; this is not specifically limited in this invention.

[0073] In some preferred embodiments, the step of continuously acquiring the displacement rate of human skeleton key points in each frame of sensor data from the start of the static timing includes: dividing all human skeleton key points into a trunk key point group and a limb key point group according to human body parts, wherein the trunk key point group includes human skeleton key points at the shoulder, hip, and spine, and the limb key point group includes human skeleton key points at the elbow, wrist, knee, and ankle; acquiring the coordinate change of the same human skeleton key point in consecutive frames from the start of the static timing as the displacement of the human skeleton key point; and calculating the displacement of each human skeleton key point in adjacent frames based on the displacement and the inter-frame time interval. The inter-frame displacement rate is calculated between frames. Based on the inter-frame displacement rates of all human skeleton keypoints within the torso keypoint group, the average displacement rate within the torso keypoint group is calculated to obtain the torso average displacement rate. Based on the inter-frame displacement rates of all human skeleton keypoints within the limb keypoint group, the average displacement rate within the limb keypoint group is calculated to obtain the limb average displacement rate. The torso average displacement rate and the limb average displacement rate are weighted and summed, wherein the weight of the torso average displacement rate is greater than the weight of the limb average displacement rate, to obtain the weighted average displacement rate of the current frame, and the weighted average displacement rate is used as the displacement rate.

[0074] In practice, displacement rate is a key indicator for determining whether a person is at rest during a fall. However, after a fall, the limbs may move slightly due to unconscious struggling, while the movement of the torso better reflects the overall displacement state of the body. If the displacement rates of all skeletal key points are averaged indiscriminately to calculate the overall displacement rate, slight movements of the limbs may cause the overall displacement rate to exceed the restlessness threshold, leading the system to misjudge that the person is not at rest, thus delaying or missing the confirmation of the fall event. Therefore, this embodiment divides the skeletal key points into a torso key point group and a limb key point group according to body parts, calculates the average displacement rate of the two groups and assigns different weights, so that the displacement rate of the torso dominates the fall determination, reducing the interference of slight limb movements on the determination result. The specific implementation method is as follows: First, all key points of the human skeleton are divided into two groups based on body parts: the trunk key point group and the limb key point group. The trunk key point group includes key points at the shoulders, hips, and spine, while the limb key point group includes key points at the elbows, wrists, knees, and ankles. The trunk key point group reflects the displacement and posture changes of the main body, namely the head, torso, and pelvic region. Key points in this group include, but are not limited to, key points at the left and right shoulders, left and right hips, and spine. The limb key point group reflects the displacement and posture changes of the limbs, namely the upper and lower limbs. Key points in this group include, but are not limited to, key points at the left and right elbows, left and right wrists, left and right knees, left and right ankles, and right ankles. This grouping method is based on ergonomic principles: in a fall scenario, the positional change of the torso is the core basis for determining whether the body as a whole has moved, while the limbs may exhibit localized movements even when the torso remains stationary. For example, after a fall, an elderly person may slightly move their arms or legs due to tension or discomfort, but these localized movements do not change the overall static state of the torso.

[0075] Furthermore, starting from the initial moment of the static timing, the coordinate changes of the same human skeleton key point in consecutive frames are obtained as the displacement of the human skeleton key point. For each human skeleton key point in each frame, the three-dimensional coordinates of the key point in the current frame are subtracted from the three-dimensional coordinates of the key point in the previous frame to obtain the coordinate change of the key point between the two frames. This coordinate change is a three-dimensional vector, whose three components correspond to the changes in the horizontal direction, the vertical direction, and the depth direction, respectively. The magnitude of this three-dimensional vector, i.e., the Euclidean distance, represents the magnitude of the spatial displacement of the key point between the two frames.

[0076] Furthermore, based on the displacement and inter-frame time interval, the inter-frame displacement rate of each human skeleton keypoint between adjacent frames is calculated. The inter-frame time interval is the time difference between two adjacent frames, determined by the sensor's sampling frame rate. For example, when the sensor sampling frame rate is 30 frames per second, the inter-frame time interval is approximately 33.3 milliseconds; when the sampling frame rate is 60 frames per second, the inter-frame time interval is approximately 16.7 milliseconds. Dividing the magnitude of the displacement of each keypoint by the inter-frame time interval yields the inter-frame displacement rate of that keypoint between adjacent frames. The physical meaning of the inter-frame displacement rate is the distance the keypoint moves per unit time, and its unit is consistent with the coordinate unit divided by the time unit.

[0077] Furthermore, based on the inter-frame displacement rates of all human skeletal keypoints within the torso keypoint group, the average displacement rate within the torso keypoint group is calculated to obtain the torso average displacement rate. The arithmetic mean of the inter-frame displacement rates of all human skeletal keypoints within the torso keypoint group is obtained by summing the values ​​and dividing by the number of keypoints in that group. This value reflects the intensity of the overall torso movement; when the human body is stationary, the average torso displacement rate approaches zero; when the human body moves as a whole, the average torso displacement rate increases significantly.

[0078] Furthermore, based on the inter-frame displacement rates of all human skeletal keypoints within the limb keypoint group, the average displacement rate within the limb keypoint group is calculated to obtain the average limb displacement rate. The arithmetic mean of the inter-frame displacement rates of all human skeletal keypoints within the limb keypoint group is then taken to obtain the average limb displacement rate. This value reflects the overall intensity of limb movement. In a static state after a fall, the average limb displacement rate may be slightly higher than the average trunk displacement rate due to subtle, unconscious movements.

[0079] Furthermore, the average displacement rate of the torso and the average displacement rate of the limbs are weighted and summed, with the weight of the average displacement rate of the torso being greater than that of the average displacement rate of the limbs, to obtain the weighted average displacement rate of the current frame, which is then used as the displacement rate. Since the torso keypoint group reflects the displacement of the main body of the human body, while the limb keypoint group reflects the displacement of the limbs, in a fall scenario, the movement of the torso more accurately reflects the overall displacement state of the human body. Therefore, the average displacement rate of the torso should dominate in fall detection. By assigning a larger weight to the average displacement rate of the torso, more attention is paid to the movement of the torso when calculating the overall displacement rate. Slight local movements of the limbs have a smaller impact on the weighted average displacement rate, reducing misjudgments of a stationary state due to slight local movements of the limbs. For example, the weight of the average displacement rate of the torso can be set to 0.7, and the weight of the average displacement rate of the limbs can be set to 0.3. Then, the weighted average displacement rate is equal to 0.7 multiplied by the average displacement rate of the torso plus 0.3 multiplied by the average displacement rate of the limbs. The specific values ​​of the weights can be adjusted according to the actual application scenario and human posture characteristics. For elderly people with strong mobility, the weights of the limbs can be appropriately increased to detect limb movements more sensitively; for elderly people with weak mobility, the weights of the limbs can be appropriately decreased to reduce misjudgments.

[0080] This embodiment divides all key points of the human skeleton into a trunk key point group and a limb key point group, and calculates the average displacement rate of each group, achieving differentiated assessment of the degree of movement of different body parts. Furthermore, by assigning a greater weight to the average displacement rate of the trunk than to the average displacement rate of the limbs, the calculated displacement rate focuses more on reflecting the movement state of the main body, reducing interference from slight struggling movements of the limbs after a fall that could affect the determination of a static state. This improves the accuracy of displacement rate as an indicator of static state and reduces the risk of misjudgment. Moreover, this grouping and weighting strategy is based on ergonomic physical principles and has clear technical significance in distinguishing between overall human displacement and local limb movements, enabling the system to more accurately determine whether a person after a fall is truly in a static state unable to get up, further improving the accuracy and reliability of fall detection.

[0081] S7, record the cumulative duration during which the displacement rate remains below the second preset threshold.

[0082] In practice, the second preset threshold is a pre-set displacement rate threshold used to determine whether the human body is stationary. When the overall displacement rate of a frame is lower than the second preset threshold, it indicates that the human body in that frame has not undergone significant movement and is in a relatively stationary state; when the overall displacement rate of a frame is higher than or equal to the second preset threshold, it indicates that the human body in that frame has significant movement. Furthermore, the unit of the second preset threshold is consistent with the unit of the displacement rate, and the specific value can be set according to factors such as sensor resolution and the distance from the human body to the sensor; for example, it can be set to 5 pixels / second. Furthermore, the initial value of the cumulative duration is zero. Starting from the start of the stationary timing, the displacement rate of each frame is judged. If the displacement rate of that frame is lower than the second preset threshold, the cumulative duration increases by one frame's time interval; if the displacement rate of that frame is not lower than the second preset threshold, the cumulative duration does not increase. In this way, the cumulative duration records the total duration for which the human body has been stationary since the start of the stationary timing.

[0083] S8. If the cumulative duration reaches the third preset threshold, then the fall event is confirmed.

[0084] In specific implementation, the third preset threshold is a pre-set duration threshold value used to determine whether the human body has remained still for a sufficiently long time to confirm a fall event. This threshold is set based on the following medical common sense: after a real fall, if an elderly person is unable to get up on their own, they will remain still for a long time, usually exceeding several seconds; while a normal, quick squatting motion to pick up an object will result in the elderly person quickly standing up and resuming movement. Therefore, by setting a sufficiently long stillness duration threshold, a real fall can be effectively distinguished from a normal, momentary movement. Further, in this embodiment, the third preset threshold can be set to 5 seconds, meaning that a fall event is only confirmed if the human body remains still for 5 seconds. It should be noted that the third preset threshold should be greater than or equal to the judgment period of the second preset threshold multiplied by the number of frames to ensure that the accumulated duration can reach this threshold.

[0085] Furthermore, if the cumulative duration reaches a third preset threshold, a fall event is confirmed.

[0086] Confirming a fall event means that, based on the combined results of the two-stage judgment, a genuine fall event has occurred at the currently monitored target. This confirmation result will serve as the trigger condition for generating subsequent early warning signals, and the confirmation status will be output to the early warning module. Furthermore, after confirming a fall event, relevant information about the current event can be recorded, including the time of the event and location identifier, providing a data foundation for subsequent early warnings and recording.

[0087] Furthermore, if the displacement rate is higher than or equal to the second preset threshold before the cumulative duration reaches the third preset threshold, then the cumulative duration is reset.

[0088] In specific implementation, this step describes a possible scenario during static monitoring: before the accumulated duration reaches the third preset threshold, the displacement rate of a certain frame recovers to a level higher than or equal to the second preset threshold. This indicates that the human body has resumed movement after a suspected fall, possibly by standing up, turning over to adjust posture, or engaging in other normal behaviors. At this point, it is determined that the current static state has been broken and no longer meets the characteristics of continuous stillness after a fall. Therefore, the accumulated duration is reset to its initial value of zero, and the current fall confirmation process is canceled. Furthermore, after the reset, the system returns to an idle monitoring state, waiting for the next event triggered when the fall confidence reaches the first preset threshold. Through this reset mechanism, the system can dynamically adapt to real-time changes in human behavior, avoiding misjudging non-continuous, brief periods of stillness as falls.

[0089] Furthermore, when the fall confidence level is less than the first preset threshold, it indicates that the match between the posture changes and the fall action pattern within the current time window is insufficient to classify it as a suspected fall. In this case, processing of the current time window is terminated directly, without any confirmation of the fall event or entering a static monitoring mode. The sensor data stream from the next time window is then processed to achieve continuous real-time monitoring.

[0090] Through the above steps, a complete fall detection closed loop is achieved, encompassing data acquisition, privacy enhancement processing, fall confidence inference, and static state verification. The first stage, short-window spatiotemporal graph convolutional network inference, is used to quickly identify suspected fall actions. The second stage, static state duration verification, is used to eliminate false alarms caused by instantaneous actions such as quickly squatting or bending over to pick up objects. Furthermore, by immediately discarding the original sensor data frames after skeleton extraction, the fall detection function is implemented while ensuring that the original images are not stored or transmitted, fundamentally protecting the privacy of the monitored object.

[0091] Furthermore, in this embodiment, parameters such as the frame rate, preset duration, first preset threshold, second preset threshold, and third preset threshold of the sensor data stream can be flexibly configured according to different application scenarios. For example, in scenarios involving the care of elderly people, since the movements of the elderly are generally slower, the first preset threshold can be appropriately lowered to improve sensitivity; in scenarios involving relatively active elderly people, the first preset threshold can be appropriately increased to reduce the false alarm rate. Similarly, the third preset threshold can also be adjusted according to the physical condition of the elderly person. For elderly people with weaker mobility, the third preset threshold can be appropriately shortened to respond more quickly to possible fall events. The specific values ​​of each threshold can be determined through experimental calibration and on-site debugging before system deployment, or can be dynamically adjusted through adaptive algorithms during system operation; this invention does not specifically limit this.

[0092] This embodiment extracts the coordinate information of key points of the human skeleton immediately after sensor data acquisition and discards the original sensor data frames, ensuring that the original sensor image data does not leave the device. This prevents the leakage of personal biometric information such as facial features and body characteristics from the data source, satisfying privacy protection requirements while achieving fall detection. Furthermore, by using a pre-trained spatiotemporal graph convolutional network to infer the skeleton coordinate sequence within a short time window, and utilizing the natural connection topology and temporal motion characteristics of the human skeleton, it can quickly and accurately identify fall movement patterns and output a fall confidence score reflecting the probability of a fall. Further, by using the moment when the fall confidence score reaches a first preset threshold as the start time for static timing, and continuously monitoring the cumulative duration when the displacement rate is lower than a second preset threshold from that moment, a fall event is confirmed only when the cumulative duration reaches a third preset threshold. This allows the system to effectively distinguish between instantaneous actions such as quickly crouching down to pick up an object and the continuous static state of being unable to get up after a real fall. Furthermore, when the displacement rate recovers to a level higher than or equal to the second preset threshold before the accumulated time reaches the third preset threshold, the accumulated time is reset, further reducing false alarms caused by brief periods of stillness and significantly decreasing the frequency of invalid manual verifications by caregivers due to false alarms. Furthermore, the entire method does not change the deployment of existing sensor hardware; it achieves a balance between privacy enhancement and low false alarm fall detection solely through algorithmic logic, demonstrating good practicality and deployability. Furthermore, the independent configuration and flexible adjustment capabilities of each preset threshold enable this embodiment to adapt to the specific needs of different monitoring scenarios and different monitored populations, exhibiting a wide range of applicability.

[0093] In some preferred embodiments, the method further includes: acquiring historical behavior time-series data of the target object within an initial time period; performing unsupervised clustering on the historical behavior time-series data to establish an individualized behavior baseline model of the target object; acquiring real-time behavior features of the target object and inputting the real-time behavior features into the individualized behavior baseline model to calculate the probability density value corresponding to the real-time behavior features; determining whether the probability density value is less than a fifth preset threshold; if the probability density value is less than the fifth preset threshold, generating a behavior deviation flag; in response to generating the behavior deviation flag, starting a deviation duration timer and recording the duration of the behavior deviation flag; determining whether the cumulative duration of the deviation duration timer is greater than or equal to a sixth preset threshold; if the cumulative duration of the deviation duration timer is greater than or equal to the sixth preset threshold, confirming a behavior deviation event; if the cumulative duration of the deviation duration timer is less than the sixth preset threshold, temporarily not confirming the behavior deviation event until the cumulative duration reaches the sixth preset threshold or the behavior deviation flag is released; in response to confirming the behavior deviation event, updating the individualized behavior baseline model according to the real-time behavior features at the current moment.

[0094] In practical implementation, within elderly care facilities, besides sudden accidents like falls, the slow deterioration of the elderly's health also requires close attention. When elderly individuals exhibit deviations in their behavioral patterns, such as reduced activity levels and altered sleep schedules, it often indicates a decline in health. For instance, a gradual decrease in activity may suggest a decline in bodily functions, while a significant increase in bed rest may indicate an early stage of illness. Furthermore, traditional fall detection systems only focus on instantaneous alarms for sudden events, lacking the ability to track and analyze long-term behavioral patterns in the elderly, and thus failing to provide preventative care reminders in the early stages of health deterioration. Therefore, this embodiment establishes an individualized behavioral baseline model specific to the target individual by performing unsupervised clustering of the target individual's historical behavioral time-series data within an initial period. It monitors the degree of matching between current behavioral characteristics and the baseline model in real time. When behavioral characteristics significantly deviate from the baseline, a behavioral deviation event is triggered, upgrading passive alarms to proactive care. The specific implementation method is as follows: The system acquires historical behavioral time-series data of the target individual during the initial period. The target individual is the monitored elderly person, each with unique daily routines, including wake-up time, toileting habits, and activity patterns. Furthermore, the initial period refers to a preset time interval after system startup, used to collect sufficient behavioral data to establish a reliable individualized baseline. The initial period needs to be long enough to cover the complete cycle of the target individual's daily behavior, while also being short enough to avoid including data from periods of significant change in the target individual's physical condition. In this embodiment, the initial period can be set to 7 days, meaning the system continuously collects behavioral data from the target individual for the first 7 days after startup. This 7-day data covers different work-rest patterns on weekdays and weekends, comprehensively reflecting the target individual's daily behavioral patterns. The historical behavioral time-series data includes records of various behavioral characteristics of the target individual during this initial period, which can characterize the target individual's daily life patterns from different dimensions.

[0095] Furthermore, unsupervised clustering is performed on the historical behavior time-series data to establish an individualized behavioral baseline model for the target object. Unsupervised clustering is a machine learning method that does not rely on labeled data. Its goal is to automatically classify data samples into several categories based on their inherent similarity, resulting in high similarity between samples of the same category and low similarity between samples of different categories. Further, in this embodiment, a Gaussian Mixture Model (GMM) is used to cluster the historical behavior time-series data. The GMM is a clustering method based on probability density estimation. The basic assumption of the GMM is that the data is generated by mixing several Gaussian distribution components, and each data point corresponds to an implicit category label indicating which Gaussian distribution component the data point originates from. Further, the parameters of the GMM include the mean vector, covariance matrix, and mixing weights of each Gaussian component. The mean vector reflects the central position of the data category, the covariance matrix reflects the distribution shape and direction of the data category, and the mixing weights reflect the proportion of the data category in the overall data. When establishing the baseline model, historical behavioral time-series data is used as training samples. The parameters of the Gaussian mixture model are iteratively estimated using the Expectation-Maximization (EM) algorithm. The EEM algorithm gradually optimizes the model parameters by alternately executing the expectation step and the maximization step, ultimately maximizing the likelihood probability of the model with respect to the training data. Furthermore, after fitting, this Gaussian mixture model serves as the individualized behavioral baseline model for the target object, representing the distribution of the target object's behavioral patterns under normal health conditions, i.e., the probability distribution of the target object's daily behavioral patterns in the feature space.

[0096] Further, the real-time behavioral characteristics of the target object are acquired and input into the individualized behavioral baseline model to calculate the probability density value corresponding to the real-time behavioral characteristics. Real-time behavioral characteristics are behavioral data of the target object collected at the current moment, possessing the same feature dimensions and format as historical behavioral time-series data. For example, if historical behavioral time-series data includes two features: the first time out of bed each day and the average range of bedridden activity per unit time period, then the real-time behavioral characteristics also include these two feature dimensions. Further, the real-time behavioral characteristics are used as a sample point and input into the established Gaussian mixture model to calculate the probability density value of the sample point under the model. The probability density value reflects the degree of matching between the current behavioral characteristics and the individualized behavioral baseline model of the target object. A higher probability density value indicates a higher probability of the current behavioral characteristics in the individualized behavioral baseline model, meaning the current behavior is more consistent with the normal behavioral pattern of the target object; a lower probability density value indicates a lower probability of the current behavioral characteristics in the individualized behavioral baseline model, meaning the current behavior deviates more from the normal behavioral pattern of the target object.

[0097] Further, it is determined whether the probability density value is less than a fifth preset threshold. If the probability density value is less than the fifth preset threshold, a behavior deviation flag is generated. The fifth preset threshold is a pre-set probability density threshold value used to determine whether the current behavior significantly deviates from the individualized baseline model. This threshold can be set to a small value, such as 0.01, indicating that when the probability of the current behavior feature in the individualized baseline model is less than 1%, the current behavior is considered to significantly deviate from the normal pattern. Further, when the probability density value is lower than this threshold value, it indicates that the probability of the current behavior feature in the individualized baseline model is extremely low, belonging to an abnormal behavior pattern rarely seen in the normal behavior pattern, and a behavior deviation flag is generated to identify this abnormal state. If the probability density value is greater than or equal to the fifth preset threshold, no behavior deviation flag is generated, i.e., the state remains normal.

[0098] Furthermore, in response to the generation of the behavior deviation flag, a deviation duration timer is started to record the duration of the behavior deviation flag. The deviation duration timer starts counting from the moment the behavior deviation flag is generated and is used to record the duration of the behavior deviation state. This design is further intended to avoid brief false triggers caused by instantaneous data fluctuations. Behavioral data may experience instantaneous fluctuations during acquisition due to various random factors, such as sensor noise and brief temporary behavioral changes of the target object. While these instantaneous fluctuations may cause the probability density value to temporarily fall below the fifth preset threshold, they do not necessarily indicate that the target object has actually experienced a sustained behavioral deviation.

[0099] Further, it is determined whether the cumulative duration of the deviation duration timer is greater than or equal to a sixth preset threshold. If the cumulative duration of the deviation duration timer is greater than or equal to the sixth preset threshold, a behavioral deviation event is confirmed. Further, the sixth preset threshold is a pre-set duration threshold value used to confirm whether the behavioral deviation is a persistent and genuine anomaly. This threshold can be set to 30 minutes, meaning that a genuine behavioral deviation event is only confirmed when the behavioral deviation state lasts for 30 minutes. This threshold avoids false triggers caused by short-term fluctuations while ensuring a timely response to genuine and persistent deviations.

[0100] Furthermore, if the cumulative duration of the deviation duration timer is less than the sixth preset threshold, the behavioral deviation event is temporarily not confirmed until the cumulative duration reaches the sixth preset threshold or the behavioral deviation flag is deactivated. If the behavioral deviation flag is automatically deactivated before the cumulative duration reaches the sixth preset threshold, it indicates that the deviation is temporary and recoverable; the current confirmation process is then canceled, and the deviation duration timer is reset to its initial value. This design filters out short-term fluctuations and only confirms persistent behavioral deviations.

[0101] Furthermore, in response to confirming the behavioral deviation event, the individualized behavioral baseline model is updated based on the real-time behavioral characteristics at the current moment. After confirming the behavioral deviation event, the real-time behavioral characteristics at the current moment are incorporated into the historical behavioral time-series data, and a Gaussian mixture model is refitted across all data, allowing the individualized behavioral baseline model to be gradually updated as the target subject's physical condition naturally changes. The significance of this update mechanism lies in the fact that the physical condition and daily behavioral patterns of the elderly evolve slowly with age and changes in health status. If the baseline model remains fixed, in the long run, it will deviate from the target subject's actual behavioral patterns, leading to detection failure. By updating the baseline model each time a behavioral deviation event is confirmed, the baseline model can adaptively adjust to the slow, natural changes in the target subject's behavioral patterns, maintaining the timeliness and accuracy of the baseline model.

[0102] This embodiment establishes a personalized behavioral baseline model by performing unsupervised clustering on the historical behavioral time-series data of the target object within the initial time period. This ensures that the criteria for judging behavioral deviations vary from person to person, with each elderly person having their own unique baseline model. This avoids misjudgments or omissions that might occur when using a uniform standard, reflecting respect for and adaptability to individual differences. Furthermore, by inputting real-time behavioral characteristics into the personalized behavioral baseline model to calculate probability density values, a quantitative assessment of the difference between the current behavioral pattern and the normal behavioral pattern is achieved. As a continuous numerical indicator, the probability density value can precisely reflect the degree of behavioral deviation. Furthermore, by comparing the probability density value with a fifth preset threshold, behavioral deviations can be detected objectively and accurately. By starting a deviation duration timer after the behavioral deviation flag is generated, a behavioral deviation event is only confirmed when the duration reaches a sixth preset threshold, avoiding false responses to instantaneous data fluctuations and improving the reliability of behavioral deviation detection. Furthermore, by updating the individualized behavioral baseline model based on the real-time behavioral characteristics at the current moment after confirming the behavioral deviation event, the baseline model can be adaptively updated to follow the slow and natural changes in the target object's behavioral patterns, maintaining the long-term effectiveness of the baseline model, avoiding monitoring failure caused by the solidification of the baseline model, and realizing adaptive maintenance of the behavioral baseline model.

[0103] In some preferred embodiments, acquiring the historical behavioral time-series data of the target object within an initial time period includes: continuously monitoring the target object's bedridden and out-of-bed states within the initial time period and generating a state timestamp log; extracting from the state timestamp log the target object's first time out of bed each day, the number of times it gets out of bed at night, and the duration of each toilet visit within the initial time period; acquiring the spatial coordinates of key points of the target object's skeletal structure in the bedridden state within each unit time period within the initial time period, and calculating the cumulative value of the positional change of each key point of the skeletal structure relative to the previous sampling time, using the cumulative value as the total bedridden activity within the unit time period; dividing the total bedridden activity within the unit time period by the duration of the unit time period to obtain the average bedridden activity amplitude within the unit time period; and using the first time out of bed each day, the number of times it gets out of bed at night, the duration of each toilet visit, and the average bedridden activity amplitude within the unit time period as the historical behavioral time-series data.

[0104] In practical implementation, within elderly care facilities, the daily behavioral patterns of the elderly can be quantified by monitoring their bedridden and ambulating states, as these are the most basic and regular behavioral state transitions in their daily lives. Furthermore, the first time an elderly person gets out of bed each day reflects their wake-up routine and is an important indicator for assessing their circadian rhythm; the number of times they get out of bed at night reflects their sleep quality and nocturia frequency, and is an important indicator for assessing their nighttime rest; the duration of each toilet visit reflects their toileting behavior pattern and is an important indicator for assessing their urinary system function and speed; and the range of motion while lying in bed within a unit of time reflects the intensity of their activity and the frequency of turning over during bedtime, and is an important indicator for assessing their sleep depth and comfort. Therefore, this embodiment continuously monitors the above behavioral indicators and quantifies them into specific numerical characteristics, providing a rich and accurate data foundation for establishing an individualized behavioral baseline model. The specific implementation method is as follows: During the initial time period, the bed-lying and bed-abandoning states of the target object are continuously monitored, generating a state timestamp log. Bed-lying state refers to the target object being in bed, while bed-abandoning state refers to the target object leaving the bed. Further, depth sensors deployed in the bed area can be used to identify whether the target object is in a bed-lying state, for example, by detecting the presence of human skeletal key points in the bed area. Further, when human skeletal key points are detected in the foreground human silhouette within the bed area and the positions of these key points conform to the characteristics of a bed-lying posture, the target object is determined to be in a bed-lying state; when no human skeletal key points are detected in the bed area, or the positions of the human skeletal key points do not conform to the characteristics of a bed-lying posture, the target object is determined to be in a bed-abandoning state. Further, the state timestamp log records the precise time point of each state change, for example, recording "September 1, 2026, 6:32:15 AM, switching from bed-lying state to bed-abandoning state" or "September 1, 2026, 10:15:30 PM, switching from bed-abandoning state to bed-lying state." Furthermore, the state timestamp log can be stored in the system's persistent storage device in tabular form, with each record containing the timestamp of the state transition and the state type after the transition.

[0105] Further, the daily first time out of bed, the number of times the target object gets out of bed at night, and the duration of each toilet visit are extracted from the status timestamp log during the initial period. The daily first time out of bed is the time when the target object first switches from a bed-resting state to a bed-above state each day, reflecting the target object's wake-up pattern. The first event in the status timestamp log that switches from a bed-resting state to a bed-above state each day is retrieved, and the timestamp of that event is extracted as the first time out of bed for that day. Further, the number of times the target object gets out of bed at night refers to the number of times the target object switches from a bed-resting state to a bed-above state during the nighttime period, reflecting the target object's nighttime activity frequency. The nighttime period can be determined based on actual sleep patterns, for example, set to 10 PM to 6 AM the next day. Further, the number of times the target object gets out of bed at night is obtained by counting the number of events in the status timestamp log that switch from a bed-resting state to a bed-above state within this nighttime period. The duration of each toilet visit refers to the duration the target object stays in the toilet-related area. The time difference between the time the target object enters and leaves the toilet area is calculated and used as the duration of that toilet visit. Furthermore, sensors can be deployed in the toilet area to detect whether a target is in the toilet area, or the spatial position of key points on the human skeleton can be analyzed to determine whether a target is in the toilet area.

[0106] Furthermore, within each unit time period of the initial time period, the spatial coordinates of key points of the target object's skeleton in a bedridden state are acquired, and the cumulative value of the positional change of each key point of the skeleton relative to the previous sampling time is calculated. This cumulative value is used as the total bedridden activity within the unit time period. Further, the unit time period can be set according to actual needs, for example, 1 hour. Within each unit time period, the spatial coordinates of key points of the target object's skeleton in a bedridden state are continuously acquired at a sensor sampling frame rate. For each pair of adjacent sampling times, the Euclidean distance between each key point of the skeleton and the two times is calculated, i.e., the positional change. Then, the positional changes of all key points are summed to obtain the total bedridden activity within that unit time period. The total bedridden activity reflects the total amplitude of the target object's body movement during bedridden periods. A larger total activity indicates more frequent and greater movement of the target object during bedridden periods; a smaller total activity indicates a more stable and less active state of the target object during bedridden periods.

[0107] Further, the total amount of bedridden activity within the unit time period is divided by the duration of the unit time period to obtain the average amplitude of bedridden activity within the unit time period. The average amplitude of bedridden activity is a normalization of the total amount of bedridden activity, and its physical meaning is the average intensity of bedridden movement per unit time. The larger this value, the greater the average amount of movement per minute of the target object during bed rest, i.e., the more unstable the bed rest; the smaller the value, the smaller the average amount of movement per minute of the target object during bed rest, i.e., the more stable the bed rest.

[0108] Furthermore, the first time the subject got out of bed each day, the number of times the subject got out of bed at night, the duration of each toilet visit, and the average range of motion in bed within a given time period are used as the historical behavioral time-series data. These data, when aggregated, constitute a multidimensional behavioral feature dataset of the target subject within the initial time period. Each record corresponds to a unit time period and contains values ​​for the aforementioned multiple feature dimensions. Multiple records are arranged chronologically to form complete historical behavioral time-series data. For example, for an initial time period of 7 days and a unit time period of 1 hour, the historical behavioral time-series data contains 168 records, each containing 5 feature dimensions: the average range of motion in bed for that hour, the first time the subject got out of bed that day, the number of times the subject got out of bed at night that day, and the duration of each toilet visit, etc.

[0109] This embodiment continuously monitors the target object's bedridden and ambient states during an initial time period and generates a state timestamp log, providing a precise timeline data foundation for subsequent feature extraction. The state timestamp log records the precise moment of each state transition, enabling subsequent feature extraction to be based on accurate time information. Furthermore, by extracting the first time out of bed each day, the number of times out of bed at night, and the duration of each toilet visit from the state timestamp log, these indicators with clear behavioral semantics are quantified into processable data features. These features characterize the target object's daily behavioral rhythms from a temporal perspective. Further, by obtaining the spatial coordinates of key points of the human skeleton in the bedridden state within a unit time period, calculating the cumulative value of positional changes as the total amount of bedridden activity, and dividing by the unit time period duration to obtain the average amplitude of bedridden activity, a quantitative assessment of activity intensity during bedridden periods is achieved. This feature characterizes the target object's bedridden behavioral characteristics from the perspective of movement intensity. Furthermore, the aforementioned features comprehensively characterize the daily behavior patterns of the target object from multiple dimensions, providing rich feature inputs for establishing an individualized behavior baseline model. This enables the behavior baseline to more accurately and comprehensively reflect the daily behavior patterns of the target object, improving the accuracy and reliability of behavior deviation detection.

[0110] In some preferred embodiments, the method further includes: monitoring the confirmation status of the fall event and the confirmation status of the behavioral deviation event; if the fall event has been confirmed, generating a first-level warning signal and triggering an emergency push process; in response to the triggering of the emergency push process, sending a drive signal to an audible and visual alarm terminal, and simultaneously pushing an emergency pop-up signal and an SMS signal to a pre-associated mobile terminal; after the first-level warning signal is generated, monitoring whether the confirmation status of the fall event remains continuously valid; if the fall event is not canceled within a preset seventh preset time period, repeatedly sending the drive signal, the emergency pop-up signal, and the SMS signal every seventh preset time period until the fall event is manually canceled; if the fall event... If the event is not confirmed but the behavior deviates from the event, a second-level warning signal is generated, with a lower priority than the first-level warning signal. In response to the generation of the second-level warning signal, a text reminder signal is pushed to the application message queue of the pre-associated mobile terminal. After the second-level warning signal is generated, it is determined whether a text reminder signal of the same behavior type has been generated within a preset eighth preset time period. If a text reminder signal of the same behavior type has been generated within the eighth preset time period, the push of the current text reminder signal is suppressed. If a text reminder signal of the same behavior type has not been generated within the eighth preset time period, the push of the current text reminder signal is allowed.

[0111] In practice, after fall and behavioral deviation events are confirmed, different levels of warning signals need to be generated based on the event type, and corresponding push control strategies need to be implemented. For confirmed fall events, the highest level of emergency warning needs to be triggered immediately, as falls can lead to serious physical injury, requiring immediate response and handling by nursing staff. Furthermore, the emergency warning needs to simultaneously notify nursing staff through audible and visual alarms and instant messaging to ensure that the warning information is perceived as soon as possible. Furthermore, behavioral deviation events, while less urgent than fall events, still require attention, as behavioral deviations may be an early sign of deteriorating health. Furthermore, for such events, gentle reminders need to be sent to nursing staff with a lower priority to avoid frequent non-urgent notifications interfering with their work. Furthermore, for non-urgent text reminders, a mechanism to prevent duplicate pushes needs to be set up to avoid repeated sending of the same type of reminder within a short period, causing notification fatigue. The specific implementation method is as follows: The system monitors the confirmation status of the fall event and the confirmation status of the behavioral deviation event. The confirmation status of the fall event includes two values: confirmed and unconfirmed. This status is generated and continuously updated by the aforementioned fall detection process. Similarly, the confirmation status of the behavioral deviation event also includes two values: confirmed and unconfirmed. This status is generated and continuously updated by the aforementioned behavioral deviation detection process. These two status signals are continuously received as input data for subsequent graded early warning determination.

[0112] Furthermore, if the fall event is confirmed, a Level 1 warning signal is generated, triggering an emergency push procedure. The Level 1 warning signal is the highest priority warning signal, indicating a fall event requiring immediate attention. This signal carries information such as the event type identifier, the time of the event, and the location of the target, enabling caregivers to quickly locate and respond.

[0113] Furthermore, in response to the triggering of the emergency push process, a drive signal is sent to the audible and visual alarm terminal, and an emergency pop-up signal and SMS signal are simultaneously pushed to the pre-associated mobile terminal. The audible and visual alarm terminal is a hardware alarm device deployed in locations such as nursing stations or corridors, typically including a high-power speaker and flashing indicator lights, capable of emitting high-intensity sound and flashing light to attract the attention of nursing staff. Furthermore, the drive signal is a control command used to trigger the operation of the audible and visual alarm terminal. This command is sent to the audible and visual alarm terminal via wired or wireless communication, and the terminal activates the audible and visual alarm upon receiving the drive signal. Furthermore, the pre-associated mobile terminal refers to the mobile phone or other mobile device of the on-duty nursing staff associated with the current target object. Furthermore, an association between the target object and the responsible nursing staff is established during the initialization phase; when a warning event occurs, the target mobile terminal is determined based on this association. Furthermore, an emergency pop-up signal refers to a high-priority notification window that is forcibly displayed on a mobile terminal. This pop-up has the highest notification priority and can be displayed in full-screen or banner mode even when the mobile terminal is locked or in silent mode, accompanied by vibration and ringtone alerts to ensure that the user notices it immediately. Additionally, an SMS signal refers to a text message sent to the mobile terminal's number via SMS service. SMS, as a communication channel independent of applications, serves as a backup notification channel when application notifications are not viewed in a timely manner, ensuring that notifications are reliably delivered.

[0114] Furthermore, after the first-level warning signal is generated, the system monitors whether the confirmation status of the fall event remains continuously valid. After the emergency warning signal is sent, the system continues to receive the fall event confirmation status signal to determine whether the event has been confirmed and handled by caregivers. This status signal is continuously read periodically at the frame rate of the sensor data.

[0115] Furthermore, if the fall event is not cancelled within a preset seventh preset time period, the drive signal, the emergency pop-up signal, and the SMS signal are repeatedly sent every seventh preset time period until the fall event is manually cancelled. Further, the seventh preset time period is a pre-set retransmission interval used to control the retransmission cycle of the emergency warning signal. This time period can be set to 60 seconds, meaning that after each emergency warning signal is sent, a 60-second wait is made. If the fall event status does not change from confirmed to unconfirmed within these 60 seconds, it indicates that the current warning signal may not have been noticed by the caregiver or has not received an effective response, and all emergency warning signals are automatically retransmitted. Further, the retransmission process continues, waiting for the seventh preset time period again after each transmission and checking the cancellation status of the fall event until the fall event is manually cancelled by the caregiver through the system interface. This continuous retransmission mechanism ensures the delivery of emergency warning signals and prevents delays in response due to a single push not being noticed in time.

[0116] Furthermore, if the fall event is not confirmed but the behavioral deviation event is confirmed, a second-level warning signal is generated. The second-level warning signal has a lower priority than the first-level warning signal. The second-level warning signal is a lower-priority warning signal used to indicate a behavioral deviation event that requires attention but not immediate emergency action. The lower priority of this signal means that its push notification method is primarily a gentle reminder, without resorting to forced interruption.

[0117] Furthermore, in response to the generation of the second-level warning signal, a text alert signal is pushed to the application message queue of the pre-associated mobile terminal. The application message queue is the message receiving queue of the nursing application running on the mobile terminal, and the text alert signal is presented to the caregiver in the form of an in-application notification. Moreover, this notification method does not have the characteristic of forced interruption or forced reminder; caregivers can view it at their convenience without disrupting their normal work rhythm.

[0118] Furthermore, after the second-level warning signal is generated, it is determined whether a text reminder signal of the same behavior type has been generated within a preset eighth preset time period. Further, the eighth preset time period is a pre-set anti-repetition suppression window duration, used to prevent non-urgent reminders of the same category from being repeatedly pushed within a short period. Further, this time period can be set to 7200 seconds, i.e., 2 hours. Further, the same behavior type refers to behavioral deviation events of the same category, such as all belonging to bed-staying-overtime deviation, all belonging to prolonged-out-of-bed deviation, or all belonging to toilet-use-overtime deviation, etc., determined by checking the behavior type identifier carried in the previously generated second-level warning signal to determine whether they belong to the same type.

[0119] Furthermore, if a text alert signal of the same behavior type has already been generated within the eighth preset time period, the push of the current text alert signal is suppressed. Furthermore, suppressing the push means that the currently generated text alert signal will not be sent to the application message queue of the mobile terminal, thereby avoiding repeated pushes of the same type of alert and preventing notification fatigue caused to caregivers by non-urgent notifications.

[0120] Furthermore, if no text alert signal of the same behavior type has been generated within the eighth preset time period, then the current text alert signal is allowed to be pushed. Furthermore, allowing the push means that the currently generated text alert signal is normally sent to the application message queue of the mobile terminal, and the caregiver can learn about the behavior deviation event by checking the application notification.

[0121] Furthermore, if neither the fall event nor the behavioral deviation event is confirmed, no warning signal is generated. When neither event is confirmed, the target is in a safe and normal monitoring state, and the system remains silent, not sending any notifications to caregivers.

[0122] This embodiment monitors the confirmation status of fall events and behavioral deviation events, generating warning signals of different priorities based on different event types. This implements a tiered warning mechanism, ensuring that emergency events receive a high-priority, immediate response, while non-emergency events do not cause unnecessary disruption to nursing staff, reflecting the rational allocation and effective utilization of nursing resources. Furthermore, for confirmed fall events, by sending drive signals to the audible and visual alarm terminal and simultaneously pushing emergency pop-up signals and SMS signals to mobile terminals, parallel multi-channel notifications from hardware devices and mobile terminals are achieved, ensuring that emergency warning information is delivered to nursing staff with the highest priority and in the most reliable manner. Furthermore, by repeatedly sending all emergency warning signals every seventh preset time interval if the fall event is not canceled within a preset seventh preset time interval, until the fall event is manually canceled, the continuity and delivery of emergency warnings are guaranteed, preventing delays in response due to a single push not being noticed in time, and ensuring timely response to critical events. Furthermore, for confirmed behavioral deviation events, by suppressing the push of text reminder signals of the same behavioral type within a preset eighth time period, notification fatigue caused by repeated push of similar non-urgent reminders in a short period of time is avoided, enabling nursing staff to pay more attention to and process different types of reminder information more effectively, thereby improving the efficiency and quality of nursing work.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0124] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0125] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0126] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a privacy-enhancing fall detection method based on machine vision.

[0127] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0128] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a privacy-enhancing fall detection method based on machine vision.

[0129] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0130] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a privacy-enhancing fall detection method based on machine vision provided in any of the above method embodiments.

[0131] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0132] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0133] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of a machine vision-based privacy-enhancing fall detection method provided in any of the above method embodiments.

[0134] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0136] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0137] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0139] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0140] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A privacy-enhancing fall detection method based on machine vision, characterized in that, include: In response to the received sensor data stream, sensor data within a preset time window is obtained from the sensor data stream, and the preset time window is used as the current time window; Extract the coordinate information of key points of the human skeleton from each frame of sensor data within the current time window, generate the skeleton coordinate sequence corresponding to the current time window, and discard the corresponding original sensor data frames; Based on the skeleton coordinate sequence, inference is performed through a pre-trained spatiotemporal graph convolutional network to obtain the fall confidence of the current time window; Determine whether the fall confidence level is greater than or equal to a first preset threshold; If the fall confidence level is greater than or equal to the first preset threshold, then the current moment is taken as the start moment of the stationary timing. Starting from the moment of the static timing, the displacement rate of key points of the human skeleton in each frame of sensor data is continuously acquired; Record the cumulative duration for which the displacement rate remains below the second preset threshold; If the cumulative duration reaches the third preset threshold, a fall event is confirmed.

2. The privacy-enhancing fall detection method based on machine vision according to claim 1, characterized in that, The step of extracting coordinate information of key points of the human skeleton from each frame of sensor data within the current time window and generating a skeleton coordinate sequence corresponding to the current time window includes: Perform human contour detection on each frame of sensor data within the current time window to obtain the human contour region; Obtain the sensor type corresponding to the sensor data stream; Select a background segmentation strategy that matches the sensor type; Based on the selected background segmentation strategy, background segmentation is performed on the human body contour region to obtain the foreground human body contour. Based on the foreground human body contour, extract the two-dimensional coordinates and depth values ​​of a preset number of human skeleton key points in the image plane, and use them as the initial coordinate data of each human skeleton key point. The initial coordinate data of each key point of the human skeleton are scored with confidence to obtain the confidence score value of each key point of the human skeleton. The human skeleton key points whose confidence scores are lower than the fourth preset threshold are marked as low-confidence human skeleton key points, and the low-confidence human skeleton key points are removed from all the current human skeleton key points. For the low-confidence human skeleton key points that are removed, interpolation is performed based on the coordinate data of the adjacent human skeleton key points of the low-confidence human skeleton key points in the current frame and multiple frames before and after the current frame to generate alternative coordinate data for the low-confidence human skeleton key points, and the alternative coordinate data is added to the human skeleton key points. Based on the coordinate data of all key points of the human skeleton after the supplementation, single-frame skeleton coordinate data corresponding to each frame of sensor data within the current time window is formed. The skeleton coordinate data of multiple consecutive frames within the current time window are arranged according to the acquisition time sequence to generate the skeleton coordinate sequence corresponding to the current time window.

3. The privacy-enhancing fall detection method based on machine vision according to claim 2, characterized in that, The step of performing background segmentation on the human body contour region based on the selected background segmentation strategy to obtain the foreground human body contour includes: If the sensor type is infrared thermal imaging, then obtain the thermal radiation intensity value of each pixel in the sensor data; Based on the distribution of thermal radiation intensity within the human body contour area, calculate the statistical characteristics of thermal radiation intensity within the human body contour area, including the mean thermal radiation intensity and the standard deviation of thermal radiation intensity. Based on the statistical characteristics of the thermal radiation intensity, a dynamic segmentation threshold matching the sensor type is determined. The dynamic segmentation threshold is the weighted sum of the mean thermal radiation intensity and the standard deviation of the thermal radiation intensity. Pixels with thermal radiation intensity values ​​lower than the dynamic segmentation threshold within the human body contour area are identified as background pixels and removed to obtain the foreground human body contour. If the sensor type is time-of-flight depth type, then obtain the depth value of each pixel in the sensor data; Histogram statistics are performed on all depth values ​​within the human body contour area, and the depth layer corresponding to the depth value with the highest frequency in the histogram is determined as the background depth layer. Pixels whose depth values ​​within the human body contour area belong to the background depth layer are identified as background pixels and removed to obtain the foreground human body contour.

4. The privacy-enhancing fall detection method based on machine vision according to claim 1, characterized in that, The step of inferring the fall confidence score for the current time window based on the skeleton coordinate sequence using a pre-trained spatiotemporal graph convolutional network includes: Normalization processing is performed on the coordinates of each human skeleton key point in the skeleton coordinate sequence to align the coordinates of each human skeleton key point to a unified standard pose space, thus obtaining a normalized skeleton coordinate sequence. Obtain the predefined natural connectivity topology of the human skeleton from the pre-trained spatiotemporal graph convolutional network, and construct a spatial adjacency matrix based on the natural connectivity topology of the human skeleton. The normalized skeleton coordinate sequence is input into the pre-trained spatiotemporal graph convolutional network, which includes a graph convolutional layer, a temporal convolutional layer, an attention layer, and a fully connected layer. The attention layer is set after the graph convolutional layer and the temporal convolutional layer, and before the fully connected layer. The spatial dependency relationships between key points of each human skeleton are extracted by the graph convolutional layer based on the spatial adjacency matrix to obtain a spatial feature map; The spatial feature map is convolved along the time dimension by the temporal convolutional layer to extract the temporal motion features between frames within the current time window, thus obtaining the temporal feature map. The attention layer weights the temporal feature map along the channel dimension and the spatial dimension respectively to obtain the enhanced temporal feature map. The enhanced temporal feature map is input into the fully connected layer for mapping, and the fall confidence of the current time window is output.

5. The privacy-enhancing fall detection method based on machine vision according to claim 1, characterized in that, The step of continuously acquiring the displacement rate of key points of the human skeleton in each frame of sensor data starting from the static timing start time includes: All key points of the human skeleton are divided into the trunk key point group and the limb key point group according to the human body parts. The trunk key point group includes the human skeleton key points at the shoulders, hips and spine, and the limb key point group includes the human skeleton key points at the elbows, wrists, knees and ankles. Starting from the moment of the static timing, the coordinate change of the same human skeleton key point in consecutive frames is obtained as the displacement of the human skeleton key point. Based on the displacement and the inter-frame time interval, calculate the inter-frame displacement rate of each human skeleton key point between adjacent frames. Based on the inter-frame displacement rate of all human skeleton key points in the trunk key point group, the average displacement rate within the trunk key point group is calculated to obtain the average displacement rate of the trunk. Based on the inter-frame displacement rate of all human skeleton key points in the limb key point group, the average displacement rate within the limb key point group is calculated to obtain the average displacement rate of the limbs. The average displacement rate of the torso and the average displacement rate of the limbs are weighted and summed, wherein the weight of the average displacement rate of the torso is greater than the weight of the average displacement rate of the limbs, to obtain the weighted average displacement rate of the current frame, and the weighted average displacement rate is used as the displacement rate.

6. The privacy-enhancing fall detection method based on machine vision according to claim 1, characterized in that, The method further includes: Obtain historical behavior time-series data of the target object within the initial time period; Unsupervised clustering is performed on the historical behavior time series data to establish an individualized behavior baseline model for the target object; The real-time behavioral features of the target object are obtained, and the real-time behavioral features are input into the individualized behavioral baseline model to calculate the probability density value corresponding to the real-time behavioral features. Determine whether the probability density value is less than a fifth preset threshold; If the probability density value is less than the fifth preset threshold, a behavior deviation flag is generated; In response to the generation of the behavior deviation flag, a deviation duration timer is started to record the duration of the behavior deviation flag; Determine whether the cumulative duration of the deviation duration timer is greater than or equal to a sixth preset threshold; If the cumulative duration of the deviation duration timer is greater than or equal to the sixth preset threshold, then a behavior deviation event is confirmed. If the cumulative duration of the deviation duration timer is less than the sixth preset threshold, the behavior deviation event will not be confirmed until the cumulative duration reaches the sixth preset threshold or the behavior deviation flag is removed. In response to the confirmation of the behavioral deviation event, the individualized behavioral baseline model is updated based on the real-time behavioral characteristics at the current moment.

7. The privacy-enhancing fall detection method based on machine vision according to claim 6, characterized in that, The acquisition of historical behavior time-series data of the target object within the initial time period includes: During the initial time period, the bed rest status and bed alighting status of the target object are continuously monitored, and a status timestamp log is generated. Extract the target object's first time out of bed each day, number of times it gets out of bed at night, and duration of each toilet visit from the status timestamp log during the initial period. Within each unit time period of the initial time period, the spatial coordinates of each key point of the human skeleton of the target object in the bedridden state are obtained, and the cumulative value of the position change of each key point of the human skeleton relative to the previous sampling time is calculated. The cumulative value is used as the total amount of bedridden activity within the unit time period. Divide the total amount of bed rest activities within the unit time period by the duration of the unit time period to obtain the average range of bed rest activities within the unit time period. The first time of getting out of bed each day, the number of times getting out of bed at night, the duration of each toilet visit, and the average range of bed activity within a unit time period are used as the historical behavior time series data.

8. The privacy-enhancing fall detection method based on machine vision according to claim 6, characterized in that, The method further includes: Monitor the confirmation status of the fall event and the confirmation status of the behavioral deviation event; If the fall event is confirmed, a Level 1 warning signal is generated, and an emergency push process is triggered. In response to the emergency push process being triggered, a drive signal is sent to the audible and visual alarm terminal, and an emergency pop-up signal and an SMS signal are simultaneously pushed to the pre-associated mobile terminal. After the first-level warning signal is generated, monitor whether the confirmation status of the fall event remains valid. If the fall event is not cancelled within the preset seventh preset time period, the drive signal, the emergency pop-up signal and the SMS signal will be sent repeatedly every seventh preset time period until the fall event is manually cancelled. If the fall event is not confirmed but the behavioral deviation event is confirmed, a second-level warning signal is generated, and the priority of the second-level warning signal is lower than that of the first-level warning signal. In response to the generation of the second-level warning signal, a text alert signal is pushed to the application message queue of the pre-associated mobile terminal; After the second-level warning signal is generated, it is determined whether the same type of text reminder signal has been generated within a preset eighth preset time period; If the same type of text reminder signal has been generated within the eighth preset time period, then the push of the current text reminder signal will be suppressed. If no text alert signal of the same behavior type has been generated within the eighth preset time period, then the current text alert signal can be pushed.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images