Human body skeleton action recognition falling detection method and device based on comparative learning
The human skeleton motion recognition method based on comparative learning solves the problem of motion recognition in occluded and multi-person scenes, improves the accuracy of fall detection and feature extraction efficiency, and reduces annotation costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF RADIO METROLOGY & MEASUREMENT
- Filing Date
- 2025-12-08
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies lack accuracy in recognizing human skeletal movements in occluded scenes, suffer from severe feature interference in multi-person scenes, have insufficient discriminative power for movement features, and have high annotation costs.
A human skeleton motion recognition method based on contrastive learning is adopted. By acquiring video data, occlusion adaptation and multi-human separation processing are performed. Motion features are extracted using a three-dimensional convolutional neural network, and model parameters are optimized by a contrastive loss function to predict fall motion.
It improves the accuracy of fall detection in occluded scenes, reduces interference in multi-person scenes, lowers annotation costs, and improves the efficiency and accuracy of action feature extraction.
Smart Images
Figure CN121963295A_ABST
Abstract
Description
A Fall Detection Method and Device Based on Comparative Learning for Human Skeletal Movement Recognition Technical Field
[0001] This invention relates to the field of image processing, and in particular to a method and apparatus for fall detection based on contrastive learning for human skeletal motion recognition. Background Technology
[0002] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.
[0003] In the human motion recognition technology system, motion recognition based on skeletal keypoints has become one of the mainstream research branches due to its unique advantages. Currently, the acquisition of skeletal keypoints mainly relies on pose estimation algorithms. Through two-dimensional top-down or bottom-up pose estimation models, the coordinates and confidence information of human joints are extracted from video frames, providing basic data for subsequent motion recognition. Although existing technologies provide a foundation for skeletal motion recognition, many intractable bottlenecks remain in practical applications. These include insufficient feature integrity in occluded scenes, severe feature interference in multi-person scenes, insufficient discriminative power of motion features, weak scene generalization ability, and high annotation costs. A solution that can specifically address these problems is urgently needed. Summary of the Invention
[0004] One objective of this application is to provide a fall detection method based on contrastive learning for human skeletal motion recognition, which improves the accuracy of fall detection in occluded scenes. Another objective of this application is to provide a fall detection device based on contrastive learning for human skeletal motion recognition.
[0005] To achieve the above objectives, this application discloses a fall detection method based on contrastive learning for human skeletal motion recognition. The method includes: acquiring video data containing human motion; extracting a human skeletal keypoint dataset from each frame of the video data; performing occlusion adaptation processing on the human skeletal keypoint dataset to obtain a completed skeletal keypoint dataset; performing multi-human separation processing on the completed skeletal keypoint dataset to obtain a target human skeletal sequence dataset; converting the target human skeletal sequence dataset into a 3D heatmap volume; inputting the 3D heatmap volume into a 3D convolutional neural network for training; optimizing the model parameters of the 3D convolutional neural network using a contrastive loss function to obtain motion feature representations; inputting the motion feature representations into a classification head; mapping the probability distribution of motion categories through the classification head; and obtaining a prediction result of the fall motion based on the probability distribution.
[0006] Optionally, the human skeleton key point dataset includes the coordinate information and confidence information of each skeleton key point.
[0007] Optionally, the occlusion adaptation process includes identifying high-frequency occlusion areas of skeletal key points in a home scene, randomly masking the skeletal key points within the high-frequency occlusion areas, and performing feature completion on the missing key points resulting from the masking.
[0008] Optionally, the multi-human separation process includes: assigning a unique identifier to each human body in the video data and tracking its corresponding skeletal sequence, adding a feature mask to the skeletal sequence of each human body based on the unique identifier, and filtering the skeletal sequence of the background human body in the completed skeletal keypoint dataset.
[0009] Optionally, extracting the human skeleton keypoint dataset from each frame of the video data includes: using a human detector to perform human target detection on each frame of the video data to determine the human body region in each frame; using a two-dimensional top-down pose estimation model to extract skeletal keypoints from the human body region to obtain the coordinates and confidence scores of each skeletal keypoint; and organizing the coordinates and confidence scores of all skeletal keypoints in each frame into the human skeleton keypoint dataset.
[0010] Optionally, identifying high-frequency occlusion regions of skeletal key points in a home setting includes: collecting a preset number of home scene human motion video samples, the home scene human motion video samples covering occlusion scenes corresponding to common occlusion objects in the home environment; annotating each frame of each home scene human motion video sample, recording the occlusion location and corresponding occlusion frequency of each skeletal key point; calculating the occlusion frequency ratio of each skeletal key point, the occlusion frequency ratio being the ratio of the total occlusion frequency of the skeletal key point to the total number of frames of the home scene human motion video samples, and determining the area where the occlusion frequency ratio of the skeletal key points exceeds a preset ratio as the high-frequency occlusion region.
[0011] Optionally, feature completion for missing keypoints generated after masking includes: determining the adjacent joints of the missing keypoint, wherein the adjacent joints are skeletal keypoints that have a direct skeletal connection with the missing keypoint; extracting the coordinates of the adjacent joints in adjacent frames of the frame containing the missing keypoint in the video data, and calculating the motion trend of the adjacent joints in the time dimension; inferring the coordinate change pattern of the missing keypoint based on the motion trend, and calculating the completed coordinates of the missing keypoint using an interpolation algorithm; assigning the completed coordinates to the missing keypoint, and integrating the coordinates and confidence scores of all skeletal keypoints to obtain the completed skeletal keypoint dataset.
[0012] Optionally, assigning a unique identifier to each human body in the video data and tracking its corresponding skeletal sequence includes: using a target tracking algorithm to extract the appearance features and motion information of the human body in each frame of the video data, wherein the appearance features are constructed based on the image features of the human body region, and the motion information includes the displacement state of the human body; calculating the correlation degree of human bodies in adjacent frames based on the appearance features and motion information, and assigning the same unique identifier to human bodies whose correlation degree meets the preset correlation conditions; and organizing the skeletal key points of each frame corresponding to the same unique identifier into the skeletal sequence of that human body in chronological order, thereby realizing continuous tracking of the skeletal sequences of each human body.
[0013] Optionally, filtering the background human skeleton sequence in the completed skeletal keypoint dataset includes: constructing a fall action skeletal sequence template, the fall action skeletal sequence template containing the coordinate change features of each skeletal keypoint during the fall; calculating the similarity between each masked human skeleton sequence and the fall action skeletal sequence template; retaining the masked human skeleton sequences whose similarity meets the preset similarity conditions, and filtering the masked human skeleton sequences whose similarity does not meet the preset similarity conditions, to obtain the target human skeleton sequence dataset.
[0014] This application also provides a fall detection device for human skeletal motion recognition based on contrastive learning. The device includes: a data acquisition module: acquiring video data containing human motion, extracting a human skeletal keypoint dataset from the video data, performing occlusion adaptation processing on the human skeletal keypoint dataset to obtain a completed skeletal keypoint dataset, and performing multi-human separation processing on the completed skeletal keypoint dataset to obtain a target human skeletal sequence dataset; a training module: converting the target human skeletal sequence dataset into a three-dimensional heatmap volume, inputting the three-dimensional heatmap volume into a three-dimensional convolutional neural network for training, optimizing the model parameters of the three-dimensional convolutional neural network using a contrastive loss function to obtain motion feature representations; and a prediction module: inputting the motion feature representations into a classification head, mapping the probability distribution of motion categories through the classification head, and obtaining a prediction result of the fall motion based on the probability distribution.
[0015] The beneficial effects of this application are as follows: This application discloses a fall detection method for human skeleton action recognition based on contrastive learning. By performing occlusion adaptation processing on the human skeleton key point dataset, it first identifies high-frequency occlusion areas and then performs feature completion on missing key points, which can fill the feature gaps caused by occlusion and improve the accuracy of fall detection in occluded scenarios. At the same time, through multi-human separation processing, a unique identifier is assigned to each human body and the skeleton sequence is tracked. Feature masks are added to filter background human bodies, which can filter out the skeleton sequence of the target human body from the features of multiple human bodies and eliminate the interference of irrelevant human bodies. Finally, the target human skeleton sequence dataset is converted into a three-dimensional heat map volume. This volume integrates the spatial position information of the skeleton key points and the temporal dimension of action change information. Then, a three-dimensional convolutional neural network is used to extract features, which improves the efficiency and accuracy of extracting spatiotemporal features of actions. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the accompanying drawings: Figure 1 shows a flowchart of a specific embodiment of the fall detection method for human skeletal motion recognition based on contrastive learning according to an embodiment of this application; Figure 2 shows a flowchart of steps S110-S130 of the specific embodiment of the fall detection method for human skeletal motion recognition based on contrastive learning according to an embodiment of this application; Figure 3 shows a flowchart of steps S140-S160 of the specific embodiment of the fall detection method for human skeletal motion recognition based on contrastive learning according to an embodiment of this application; Figure 4 shows a flowchart of steps S210-S240 of the specific embodiment of the fall detection method for human skeletal motion recognition based on contrastive learning according to an embodiment of this application; Figure 5 shows a flowchart of steps S310-S330 of the specific embodiment of the fall detection method for human skeletal motion recognition based on contrastive learning according to an embodiment of this application; Figure 6 shows a flowchart of steps S340-S360 of the specific embodiment of the fall detection method for human skeletal motion recognition based on contrastive learning according to an embodiment of this application; Figure 7 shows a structural schematic diagram of a specific module of the fall detection device for human skeletal motion recognition based on contrastive learning according to an embodiment of this application. Detailed Implementation
[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.
[0018] It should be understood that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "include" and "have," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or programmable logic device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or programmable logic devices.
[0019] References to embodiments in this application mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0020] To address at least one of the problems existing in the prior art, according to one aspect of this application, as shown in FIG1, this embodiment discloses a fall detection method for human skeleton motion recognition based on contrastive learning. The method includes: S100: acquiring video data containing human motion, extracting human skeleton key point datasets from each frame of the video data, performing occlusion adaptation processing on the human skeleton key point datasets to obtain a completed skeleton key point dataset, and performing multi-human separation processing on the completed skeleton key point datasets to obtain a target human skeleton sequence dataset.
[0021] Specifically, the falling action is a continuous temporal process. Video data can completely record the temporal changes of the action, while single-frame images cannot capture this dynamic information. Therefore, video data is chosen as the input. After acquiring the video data, a human skeleton keypoint dataset is extracted from each frame. Each frame corresponds to an instantaneous state in the action process. Extracting details frame by frame ensures that no action details are missed, thus completely constructing the temporal features of the action. Secondly, the extracted human skeleton keypoint dataset undergoes occlusion adaptation processing. This occlusion adaptation processing addresses the problem of missing keypoints caused by common occlusions such as tables, chairs, and sofas in home scenes. This processing yields a complete skeleton keypoint dataset, avoiding the impact of incomplete features on subsequent recognition. Subsequently, the complete skeleton keypoint dataset undergoes multi-human separation processing. This processing is to solve the problem of interference between different human skeleton features in scenarios with many people gathering, such as community activity centers, ultimately obtaining the target human skeleton sequence dataset.
[0022] It should be noted that there are various ways to acquire video data containing human movements. It can be done in real time by using surveillance cameras deployed in living rooms, bedrooms, community activity centers, etc., or by retrieving stored video files from a pre-built human movement video database. This application is not limited to these methods.
[0023] S200: Convert the target human skeleton sequence dataset into a three-dimensional heatmap volume, input the three-dimensional heatmap volume into a three-dimensional convolutional neural network for training, and optimize the model parameters of the three-dimensional convolutional neural network using a contrastive loss function to obtain action feature representation.
[0024] Specifically, after obtaining the target human skeletal sequence dataset, it is converted into a 3D heatmap volume. The 3D heatmap volume can simultaneously contain spatial location information of skeletal key points and temporal motion change information. This spatiotemporal fusion feature is more suitable for processing by a 3D convolutional neural network. The 3D heatmap volume is then input into the 3D convolutional neural network for training, and the model parameters are optimized using a contrastive loss function. The principle of the contrastive loss function is to narrow the feature distance between positive samples of normal human motion and data-augmented motion, and widen the feature distance between negative samples, such as a falling motion and a normal standing motion, enabling the model to learn more discriminative motion features, thereby obtaining motion feature representations.
[0025] S300: Input the action feature representation into the classification head, map the probability distribution of the action category through the classification head, and obtain the prediction result of the fall action based on the probability distribution.
[0026] Specifically, the action feature representation is input into the classification head, which maps the high-dimensional action feature representation into a probability distribution of different action categories. Each element in the probability distribution corresponds to the predicted probability of a certain action category. Based on this probability distribution, the action category with the highest probability is selected to obtain the prediction result of the fall action. It should be noted that in the entire method flow, the output of the previous step is used as the input of the next step, forming a complete data flow loop.
[0027] In an optional implementation, the human skeletal key point dataset includes the coordinate information and confidence information of each skeletal key point.
[0028] Specifically, the human skeletal keypoint dataset is the foundational data for all subsequent feature processing and model training. The coordinate and confidence information of each skeletal keypoint serves different functions, each with a clearly defined source and purpose. The coordinate information of the skeletal keypoints characterizes the spatial position of each keypoint within a video frame. This information is typically obtained by analyzing the human body region in the video frame using a pose estimation model. Specifically, the pose estimation model first determines the pixel positions of keypoints such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles in the image based on the image pixel coordinate system of the video frame. Then, it converts these pixel positions into corresponding coordinate values, thus forming the coordinate information. The coordinate information provides spatial location information for subsequent construction of 3D heatmap volumes and analysis of movement trends. For example, by comparing the coordinate changes of the same keypoint in different frames, the direction and distance of movement of that keypoint can be determined, thereby inferring the overall movement state. The confidence information of the skeletal keypoints is a parameter generated simultaneously by the pose estimation model when outputting the keypoint coordinates. It reflects the reliability of the keypoint detection result. The confidence level information is generated by the pose estimation model comprehensively evaluating factors such as the matching degree of image features around the keypoint and the rationality of the topological relationship with adjacent keypoints when detecting each keypoint. If the matching degree is high and the topological relationship is reasonable, a higher confidence level is output; otherwise, a lower confidence level is output. When the confidence level of a keypoint is lower than a certain level, it can be preliminarily determined that the keypoint may have occlusion or detection error, thereby triggering the feature completion process to correct or complete the coordinates of the keypoint. Incorporating coordinate information and confidence level information into the human skeleton keypoint dataset can avoid misjudgments caused by relying solely on coordinate information, such as including erroneous coordinates with extremely low confidence levels. This provides data support for subsequent occlusion adaptation in this application, which is not limited to this.
[0029] In an optional implementation, the occlusion adaptation process includes identifying high-frequency occlusion areas of skeletal key points in a home scene, randomly masking the skeletal key points within the high-frequency occlusion areas, and performing feature completion on the missing key points resulting from the masking.
[0030] Specifically, occlusion adaptation processing is a process specifically designed to address the issue of occlusion of skeletal key points in home scenarios. It achieves adaptation to the occlusion problem through three consecutive operations: identifying high-frequency occlusion areas, random masking, and feature completion. The first step is to identify high-frequency occlusion areas for skeletal key points in home scenarios. These areas refer to those in the home environment where skeletal key points have a high probability of being missed due to overlap with common occlusion objects such as tables, chairs, sofas, cabinets, and beds. This identification process requires a large number of actual video samples from home scenarios. Through statistical analysis of these samples, areas with high occlusion probabilities are selected. In a living room scenario, when a person sits down or stands up, the knee and ankle key points are easily obscured by the sofa; in a kitchen scenario, the wrist key point is easily obscured by cabinets. These areas containing easily obscured key points are the high-frequency occlusion areas that need to be identified. The purpose of identifying high-frequency occlusion areas is to provide a clear target range for the subsequent random masking operation, avoiding unnecessary data loss caused by indiscriminate masking and making the masking operation more consistent with the occlusion situation in real home scenarios. Next, key points in the skeleton within high-frequency occlusion areas are randomly masked. This step simulates occlusion scenarios that may occur in real-world home environments, allowing subsequent model training to encounter similar incomplete features in advance, thereby improving the robustness of the trained model to occluded scenarios. Random masking involves randomly selecting some key points within the identified high-frequency occlusion areas according to preset rules and marking their coordinates or confidence information as missing. The randomness of masking is not entirely random; the proportion and scope of masking are controlled to simulate occlusion without rendering the entire dataset completely invalid. Finally, feature completion is performed on the missing key points resulting from masking. Since random masking or real-world occlusion can lead to the loss of some key point information, directly inputting an incomplete dataset into subsequent processing may affect the accuracy of action recognition. Therefore, feature completion is needed to fill in these missing information. The principle of feature completion is to utilize the topological structure of the human skeleton and the temporal continuity of actions, inferring the reasonable coordinates of the missing key points based on the motion information of adjacent joints. This step can repair the incomplete skeletal keypoint dataset into a complete skeletal keypoint dataset, providing a complete feature foundation for subsequent multi-person separation processing and avoiding misjudgments in multi-person separation due to feature gaps.
[0031] In an optional implementation, the multi-human separation process includes assigning a unique identifier to each human body in the video data and tracking its corresponding skeletal sequence, adding a feature mask to the skeletal sequence of each human body based on the unique identifier, and filtering the skeletal sequence of the background human body in the completed skeletal keypoint dataset.
[0032] Specifically, multi-person separation processing addresses the interference caused by the overlapping of skeletal features of different individuals in crowded public areas, making it difficult for models to accurately identify a falling person. It achieves this by assigning a unique identifier to each person and tracking them, adding a feature mask, and filtering background individuals. First, a unique identifier is assigned to each person in the video data, and their corresponding skeletal sequence is tracked. In multi-person scenarios, if the identities of different individuals cannot be distinguished, subsequent processing will mix up the skeletal features of all individuals. Therefore, it is necessary to assign a unique identifier to each person. This unique identifier is typically generated using a target tracking algorithm. This algorithm extracts appearance features such as grayscale histograms and contour features of the human body from each frame of the video, as well as motion information such as displacement direction and velocity. Then, by comparing the similarity of appearance features and the matching degree of motion information between adjacent frames, the correlation degree of the individuals is calculated. Individuals whose correlation degree meets the criteria are assigned the same unique identifier, thus enabling the same person to be continuously identified in consecutive video frames. Simultaneously, the skeletal keypoints corresponding to the same unique identifier in each frame are organized into a skeletal sequence for that human body, enabling continuous tracking of each human body's skeletal sequence. An independent identity profile is created for each human body to distinguish the skeletal features of different individuals. Next, feature masks are added to the skeletal sequences of each human body based on their unique identifiers. A feature mask is a marker used to distinguish the skeletal features of different human bodies; essentially, it assigns a unique feature marker to the skeletal sequences corresponding to different unique identifiers. The addition of feature masks is implemented by constructing a mask matrix matching the feature dimensions of each skeletal sequence corresponding to a unique identifier. In the matrix, the feature positions corresponding to that sequence are marked as valid, while the feature positions corresponding to other sequences are marked as invalid. Then, the mask matrix is fused with the skeletal sequence features. This isolates the skeletal features of different human bodies, preventing the features of different human bodies from overlapping and interfering with each other, allowing the model to analyze the skeletal sequence of each human body independently. Finally, the skeletal sequences of background human bodies are filtered from the completed skeletal keypoint dataset. These background human bodies are those whose action features differ significantly from the falling action. If they are not filtered, a large number of irrelevant features will be input into the model, affecting the model's accuracy in recognizing the falling action. The filtering principle involves constructing a template of skeletal sequences depicting a fall. Each masked human skeletal sequence is compared to this template, and the similarity is calculated. Sequences that meet the similarity criteria are retained, while those that do not are filtered out, ultimately yielding a dataset of target human skeletal sequences. This process eliminates interference from irrelevant individuals, focusing on target individuals who may be performing a fall, thus providing input data for subsequent model training and prediction.
[0033] In an optional implementation, as shown in Figure 2, extracting the human skeleton key point dataset from each frame of the video data includes: S110: using a human detector to perform human target detection on each frame of the video data to determine the human body region in each frame.
[0034] Specifically, the first step is to use a human detector to detect human targets in each frame of the video data, identifying the human body regions within each frame. A human detector is an algorithmic model capable of recognizing and locating human bodies in an image. Its working principle involves analyzing features such as texture, color, and contour in the image to identify regions that match human characteristics and outputting the bounding box coordinates of these regions, which represent the location and size of the human body. Human target detection is necessary because video frames typically contain a large amount of background information such as walls, furniture, and the ground. If keypoint extraction is performed directly on the entire frame, this background information can interfere with the detection results, leading to false positives or false negatives. After identifying the human body regions through human detection, subsequent keypoint extraction only needs to be performed on these regions, significantly improving the accuracy and efficiency of keypoint detection.
[0035] It should be noted that the choice of human detector can be determined according to the actual needs of the scenario. Commonly used object detection models such as Faster-RCNN (Faster Region-based Convolutional Neural Network) and YOLO (You Only Look Once) are all applicable, and this application is not limited to them.
[0036] S120: A two-dimensional top-down pose estimation model is used to extract skeletal key points from the human body region, and the coordinates and confidence scores of each skeletal key point are obtained.
[0037] Next, a two-dimensional top-down pose estimation model is used to extract skeletal keypoints from the human body region, obtaining the coordinates and confidence score of each keypoint. The two-dimensional top-down pose estimation model first locates the entire human body region and then detects individual skeletal keypoints within that region. The difference between this and the bottom-up pose estimation model, which detects keypoints first and then assembles them into a human body, lies in the processing order. The top-down approach is more suitable for keypoint detection in single-person or small-person scenarios, effectively avoiding confusion between multiple human keypoints. When processing the human body region, this model detects the position of each skeletal keypoint one by one based on the topological structure of the human skeleton, such as the connection between the head and neck, or the connection between the neck and shoulders, and outputs the corresponding coordinates. Simultaneously, based on the reliability of feature matching during the detection process, a confidence score for each keypoint is generated. The purpose of this step is to extract the coordinates and confidence scores of skeletal keypoints from the human body region, providing raw data for subsequent dataset processing.
[0038] S130: Compile the coordinates and confidence scores of all skeletal keypoints in each frame into the human skeletal keypoint dataset.
[0039] Finally, the coordinates and confidence scores of all skeletal keypoints in each frame are compiled into a human skeletal keypoint dataset. This compilation process typically involves storing the coordinates and confidence scores of all keypoints in each frame sequentially according to a predefined skeletal keypoint sequence, forming a structured dataset. The advantage of this structured storage is that it allows for rapid location of specific keypoints during subsequent occlusion adaptation and multi-human separation processing, facilitating targeted operations.
[0040] In an optional implementation, as shown in Figure 3, identifying high-frequency occlusion areas of skeletal key points in a home scene includes: S140: collecting a preset number of home scene human motion video samples, wherein the home scene human motion video samples cover occlusion scenes corresponding to common occlusion objects in the home environment.
[0041] Specifically, a predetermined number of human motion video samples in home scenarios will be collected, and the samples must cover occlusion scenarios corresponding to common obstructions in the home environment. There are various types of obstructions in the home environment, such as sofas and coffee tables in the living room, beds and wardrobes in the bedroom, and cabinets and dining tables in the kitchen. The occlusion scenarios and key points obscured by different obstructions are all different. Therefore, the collected video samples need to cover these different home scenarios and also include various human movements in these scenarios, such as standing, sitting, walking, bending over, and standing up, to enrich the sample diversity. Samples can be collected either through real-time recording using cameras deployed in different apartment layouts or by selecting qualified samples from publicly available human motion video databases.
[0042] S150: Label each frame of each of the aforementioned home scene human motion video samples, and record the location of each skeletal key point that is occluded and the corresponding occlusion frequency.
[0043] Specifically, each frame of the human motion video sample for each home scene is labeled, recording the occlusion locations and corresponding occlusion frequencies of each skeletal keypoint. The core purpose of the labeling is to obtain information on whether each skeletal keypoint in each frame is occluded. The labeling content mainly includes: the sequence number of the occluded skeletal keypoint in the frame, and whether the keypoint is occluded in the frame. If occluded, it is recorded as 1 occlusion; if not occluded, it is recorded as 0 occlusions. The labeling process can combine manual and automatic labeling. First, an automatic labeling tool, such as an occlusion detection tool based on image segmentation, is used to initially label the occlusion information. Then, the labeling results are manually calibrated to correct errors in the automatic labeling, such as misclassifying unoccluded areas as occluded areas or omitting occluded keypoints, making the labeling information more accurate. The purpose of this step is to provide raw occlusion data for subsequent statistical analysis, and the accuracy of the labeling results affects the accuracy of high-frequency occlusion region identification.
[0044] S160: Calculate the occlusion frequency ratio of each skeletal key point. The occlusion frequency ratio is the ratio of the total occlusion frequency of the skeletal key point to the total number of frames of the home scene human motion video sample. The area where the skeletal key point has an occlusion frequency ratio exceeding a preset ratio is determined as the high-frequency occlusion area.
[0045] Specifically, the occlusion frequency percentage of each skeletal keypoint is statistically analyzed, and areas containing keypoints with a percentage exceeding a preset threshold are identified as high-frequency occlusion regions. The occlusion frequency percentage is calculated by first counting the total number of times a keypoint is marked as occluded across all frames in all acquired samples, and then dividing this total frequency by the total number of frames in all samples to obtain the occlusion frequency percentage for that keypoint. A higher occlusion frequency percentage indicates a greater probability that the keypoint is occluded in a home environment. The preset threshold is set based on the actual occlusion situation in home environments and application requirements, with the aim of identifying keypoint areas that are indeed prone to occlusion.
[0046] In an optional implementation, as shown in Figure 4, feature completion for missing key points generated after masking includes: S210: determining the adjacent joints of the missing key points, wherein the adjacent joints are skeletal key points that have a direct skeletal connection with the missing key points.
[0047] Specifically, the adjacent joints of the missing keypoint are identified, and these adjacent joints are skeletal keypoints that have a direct skeletal connection with the missing keypoint. The human skeleton has a fixed topological structure, and each skeletal keypoint has a direct skeletal connection with other specific keypoints, such as the wrist keypoint directly connecting to the elbow keypoint, the knee keypoint directly connecting to the hip keypoint, and the ankle keypoint directly connecting to the other keypoint. This connection is the basis for identifying adjacent joints. The purpose of identifying adjacent joints is to use the motion information of adjacent joints to infer the location of the missing keypoint. Because adjacent joints are connected to the missing keypoint through bones, their motion trends are strongly correlated, and the motion state of adjacent joints can indirectly reflect the motion state of the missing keypoint. For example, if the missing keypoint is the wrist, its adjacent joint is the elbow keypoint. By analyzing the motion direction and velocity of the elbow keypoint, the motion trend of the wrist keypoint can be roughly inferred.
[0048] S220: Extract the coordinates of the adjacent joints in the adjacent frames of the frame containing the missing key point in the video data, and calculate the motion trend of the adjacent joints in the time dimension.
[0049] Specifically, the coordinates of adjacent joints in the frames adjacent to the frame containing the missing keypoint in the video data are extracted, and the motion trend of the adjacent joints in the time dimension is calculated. Adjacent frames typically refer to the frame before and after the frame containing the missing keypoint. These two frames are chosen because the action has temporal continuity, and the coordinate changes of adjacent joints in adjacent frames can reflect the joint's motion state over a short period. The motion trend is calculated by comparing the coordinate differences of adjacent joints in adjacent frames. For example, calculating the displacement of adjacent joints in the x-axis and y-axis directions between the previous and current frames, or the displacement of adjacent joints in the next and current frames. These displacements are used to analyze the motion direction and speed of adjacent joints, thus forming the motion trend of adjacent joints. The purpose of this step is to provide a reliable motion basis for inferring the coordinates of the missing keypoint, ensuring that the completed coordinates conform to the temporal continuity of the action.
[0050] S230: Based on the aforementioned movement trend, infer the coordinate change pattern of the missing key points, and use an interpolation algorithm to calculate the complete coordinates of the missing key points.
[0051] Specifically, based on motion trends, the coordinate change patterns of missing keypoints are inferred, and interpolation algorithms are used to calculate the completed coordinates of the missing keypoints. Since adjacent joints and missing keypoints have direct skeletal connections, their motion trends are correlated. Therefore, the coordinate change patterns of the missing keypoints in the current frame can be inferred from the coordinate change patterns of adjacent joints. Interpolation algorithms are mathematical methods for inferring unknown data points from known data points. In feature completion, the historical coordinates of missing keypoints in adjacent frames and the motion trends of adjacent joints are typically used to calculate the completed coordinates of the missing keypoints in the current frame. For example, if the coordinates of the missing keypoints in the previous frame are known, and the motion trend of adjacent joints in the current frame is a rightward displacement of 5 pixels, it can be inferred that the missing keypoints should also be displaced a similar distance to the right. The specific completed coordinates are then calculated using the interpolation algorithm. This step calculates the coordinates of the missing keypoints and fills in the feature gaps.
[0052] S240: Assign the completed coordinates to the missing keypoints, integrate the coordinates and confidence scores of all skeletal keypoints, and obtain the completed skeletal keypoint dataset.
[0053] Specifically, the missing keypoints are assigned their completed coordinates, and the coordinates and confidence scores of all skeletal keypoints are integrated to obtain a completed skeletal keypoint dataset. The integration process involves arranging the coordinates of the completed keypoints, along with the coordinates and confidence scores of the remaining keypoints, according to their original sequence and structure. Simultaneously, for each completed keypoint, a corresponding confidence score is generated based on the reliability of the completion; the more comprehensive the evidence for completion, the higher the confidence score. It should be noted that the integrated completed skeletal keypoint dataset can avoid subsequent misjudgments in multi-person separation and model training caused by missing keypoints.
[0054] In an optional implementation, as shown in Figure 5, assigning a unique identifier to each human body in the video data and tracking its corresponding skeletal sequence includes: S310: using a target tracking algorithm to extract the appearance features and motion information of each human body in each frame of the video data. The appearance features are constructed based on the image features of the human body region, and the motion information includes the displacement state of the human body.
[0055] Specifically, a target tracking algorithm is used to extract the appearance features and motion information of the human body in each frame of the video data. Appearance features are visual attributes used to distinguish different human bodies. They are constructed based on image features of the human body region, such as grayscale histograms, HOG (Histogram of Oriented Gradients) features, or CNN (Convolutional Neural Network) features. These features can effectively distinguish the external differences between different human bodies. Motion information reflects the positional changes of the human body in consecutive frames, including the displacement state of the human body, such as the direction and velocity of displacement in adjacent frames. Motion information is obtained by calculating the coordinate changes of the bounding box of the human body region in adjacent frames. The purpose of extracting appearance features and motion information is to provide two core dimensions for subsequently judging the correlation between human bodies in adjacent frames, avoiding association errors caused by relying on only a single feature.
[0056] S320: Calculate the correlation degree of human bodies in adjacent frames based on the appearance features and motion information, and assign the same unique identifier to human bodies whose correlation degree meets the preset correlation conditions.
[0057] Specifically, the correlation between human bodies in adjacent frames is calculated based on appearance features and motion information. Human bodies whose correlation meets preset conditions are assigned the same unique identifier. The calculation of correlation is the core of target tracking. Its principle is to comprehensively evaluate the similarity of appearance features and the matching degree of motion information between a human body in the current frame and all human bodies in the previous frame. A higher similarity in appearance features indicates that the visual attributes of the two bodies are closer, suggesting they may be the same human body; a higher matching degree of motion information indicates that the displacement trends of the two bodies are more consistent, also suggesting they may be the same human body. The final correlation degree is obtained by fusing appearance feature similarity and motion information matching degree according to preset weights. The preset correlation conditions are set according to the accuracy requirements of actual application scenarios. For example, when the correlation degree is higher than a certain threshold, the human body in the current frame is determined to be the same object as a human body in the previous frame, and the same unique identifier is assigned to it. If the correlation degree is lower than the threshold, it is determined to be a newly appearing human body, and a new unique identifier is assigned. The purpose of this step is to establish a unique identifier for each human body, enabling accurate identification of the same human body in consecutive frames and avoiding identity confusion.
[0058] S330: Organizes the skeletal key points of each frame corresponding to the same unique identifier into a skeletal sequence of the human body in chronological order, so as to realize continuous tracking of the skeletal sequence of each human body.
[0059] Specifically, the skeletal keypoints corresponding to the same unique identifier in each frame are organized into a skeletal sequence for the human body, enabling continuous tracking of each human skeletal sequence. The organization process involves arranging skeletal keypoints with the same unique identifier, including coordinates and confidence levels, from different frames in chronological order of the video frames to form a continuous skeletal sequence. This sequence can completely reflect the human body's movement changes over a period of time, such as a complete sequence of movements from standing to bending over and then falling. It should be noted that by continuously tracking the human skeletal sequence, continuous motion feature data can be provided for subsequent feature masking and background human body filtering, avoiding misjudgments such as misidentifying a bending posture in a single frame as a fall, which can occur due to analyzing only single-frame features.
[0060] In an optional implementation, as shown in Figure 6, filtering the skeleton sequence of the background human body in the completed skeleton key point dataset includes: S340: constructing a fall action skeleton sequence template, wherein the fall action skeleton sequence template contains the coordinate change features of each skeleton key point during the fall.
[0061] Specifically, a skeletal sequence template for falling motions is constructed, which includes the coordinate change characteristics of key skeletal points during the fall. Falling motions have typical temporal characteristics, such as a gradual descent of the body's center of gravity before the fall, significant changes in the extension angles of the limbs during the fall, and the body remaining relatively still after the fall. These characteristics are reflected in the coordinate changes of key skeletal points. The template construction process typically involves: collecting a large number of standard falling motion video samples of people of different ages and body types in different scenarios; extracting the skeletal key point sequences from these samples; analyzing the coordinate change patterns of each key point in each sample before, during, and after the fall; statistically analyzing and refining these patterns to remove individual differences and scene interference, obtaining universally applicable coordinate change characteristics of key points in falling motions, and finally forming the falling motion skeletal sequence template. This template serves as a standard reference for determining whether a skeletal sequence to be detected belongs to a falling motion.
[0062] S350: Calculate the similarity between each masked human skeleton sequence and the fall motion skeleton sequence template.
[0063] Specifically, the similarity between each masked human skeleton sequence and the template template of a fall motion skeleton sequence is calculated. The similarity calculation is based on a consistency assessment of motion features, which compares the coordinate change trends of corresponding key points in the target sequence and the overall motion logic between the template sequence and the target sequence. For example, the Euclidean distance between corresponding key points in the two sequences at the same time step can be calculated; a smaller distance indicates closer coordinates. Alternatively, the overall motion trends of the two sequences can be analyzed, such as the similarity of the curves showing changes in the human body's center of gravity and the angles of the limb joints; the more consistent the trends, the more similar the motions. The similarity between the target sequence and the template is calculated using these methods. A higher similarity value indicates that the human motion corresponding to that sequence is closer to the fall motion. This step provides a quantitative basis for distinguishing between the target human body and the background human body, avoiding screening errors caused by subjective judgment alone.
[0064] S360: Retain masked human skeleton sequences whose similarity meets the preset similarity conditions, and filter masked human skeleton sequences whose similarity does not meet the preset similarity conditions to obtain the target human skeleton sequence dataset.
[0065] Specifically, masked human skeleton sequences that meet the preset similarity criteria are retained, while masked human skeleton sequences that do not meet the preset similarity criteria are filtered out to obtain the target human skeleton sequence dataset. The preset similarity criteria are set based on the accuracy and false negative rate requirements for fall detection in actual application scenarios. During the filtering process, sequences that meet the similarity criteria are determined to indicate that the corresponding human body may be in a fall and are retained; sequences that do not meet the similarity criteria, such as sequences of normal walking, standing, or sitting, are determined to be background human sequences and are filtered out.
[0066] As shown in Figure 7, another aspect of this application discloses a human skeleton motion recognition and fall detection device based on contrastive learning. The device includes: a data acquisition module 11: acquiring video data containing human motion, extracting a human skeleton key point dataset from the video data, performing occlusion adaptation processing on the human skeleton key point dataset to obtain a completed skeleton key point dataset, and performing multi-human separation processing on the completed skeleton key point dataset to obtain a target human skeleton sequence dataset.
[0067] Training module 12: Converts the target human skeleton sequence dataset into a three-dimensional heatmap volume, inputs the three-dimensional heatmap volume into a three-dimensional convolutional neural network for training, optimizes the model parameters of the three-dimensional convolutional neural network using a contrastive loss function, and obtains action feature representations.
[0068] Prediction module 13: Inputs the action feature representation into the classification head, maps the probability distribution of action categories through the classification head, and obtains the prediction result of the fall action based on the probability distribution.
[0069] Since the principle by which this device solves the problem is similar to the methods described above, the implementation of this device can be found in the implementation of the methods, and will not be repeated here.
[0070] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0071] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A fall detection method based on contrastive learning for human skeletal motion recognition, characterized in that, The method includes: acquiring video data containing human movements; extracting a human skeletal keypoint dataset from each frame of the video data; performing occlusion adaptation processing on the human skeletal keypoint dataset to obtain a completed skeletal keypoint dataset; performing multi-human separation processing on the completed skeletal keypoint dataset to obtain a target human skeletal sequence dataset; converting the target human skeletal sequence dataset into a 3D heatmap volume; inputting the 3D heatmap volume into a 3D convolutional neural network for training; optimizing the model parameters of the 3D convolutional neural network using a contrastive loss function to obtain a motion feature representation; inputting the motion feature representation into a classification head; mapping the probability distribution of motion categories through the classification head; and obtaining a prediction result of a fall based on the probability distribution.
2. The method according to claim 1, characterized in that, The human skeletal key point dataset includes the coordinate information and confidence information of each skeletal key point.
3. The method according to claim 1, characterized in that, The occlusion adaptation process includes identifying high-frequency occlusion areas of skeletal key points in a home scene, randomly masking the skeletal key points within the high-frequency occlusion areas, and performing feature completion on the missing key points resulting from the masking.
4. The method according to claim 1, characterized in that, The multi-human separation process includes: assigning a unique identifier to each human body in the video data and tracking its corresponding skeletal sequence; adding a feature mask to the skeletal sequence of each human body based on the unique identifier; and filtering the skeletal sequence of the background human body in the completed skeletal keypoint dataset.
5. The method according to claim 1, characterized in that, Extracting the human skeletal keypoint dataset from each frame of the video data includes: using a human detector to detect human targets in each frame of the video data and determining the human body region in each frame; using a two-dimensional top-down pose estimation model to extract skeletal keypoints from the human body region and obtain the coordinates and confidence scores of each skeletal keypoint; and organizing the coordinates and confidence scores of all skeletal keypoints in each frame into the human skeletal keypoint dataset.
6. The method according to claim 3, characterized in that, Identifying high-frequency occlusion regions of skeletal key points in home scenes includes: collecting a preset number of home scene human motion video samples, which cover occlusion scenes corresponding to common occlusion objects in the home environment; annotating each frame of each home scene human motion video sample, recording the occlusion location and corresponding occlusion frequency of each skeletal key point; calculating the occlusion frequency ratio of each skeletal key point, where the occlusion frequency ratio is the ratio of the total occlusion frequency of the skeletal key point to the total number of frames of the home scene human motion video samples; and determining the areas where the occlusion frequency ratio of skeletal key points exceeds a preset ratio as the high-frequency occlusion regions.
7. The method according to claim 3, characterized in that, Feature completion for missing keypoints resulting from masking includes: identifying adjacent joints of the missing keypoint, wherein the adjacent joints are skeletal keypoints that have a direct skeletal connection with the missing keypoint; extracting the coordinates of the adjacent joints in adjacent frames of the frame containing the missing keypoint in the video data, and calculating the motion trend of the adjacent joints in the time dimension; inferring the coordinate change pattern of the missing keypoint based on the motion trend, and calculating the completed coordinates of the missing keypoint using an interpolation algorithm; assigning the completed coordinates to the missing keypoint, and integrating the coordinates and confidence scores of all skeletal keypoints to obtain the completed skeletal keypoint dataset.
8. The method according to claim 4, characterized in that, Assigning a unique identifier to each human body in the video data and tracking its corresponding skeletal sequence includes: using a target tracking algorithm to extract the appearance features and motion information of the human body in each frame of the video data, wherein the appearance features are constructed based on the image features of the human body region, and the motion information includes the displacement state of the human body; calculating the correlation degree of human bodies in adjacent frames based on the appearance features and motion information, and assigning the same unique identifier to human bodies whose correlation degree meets the preset correlation conditions; and organizing the skeletal key points of each frame corresponding to the same unique identifier into the skeletal sequence of that human body in chronological order, thereby realizing the continuous tracking of the skeletal sequence of each human body.
9. The method according to claim 4, characterized in that, Filtering the background human skeleton sequence in the completed skeletal key point dataset includes: constructing a fall action skeletal sequence template, which contains the coordinate change features of each skeletal key point during the fall; calculating the similarity between each masked human skeleton sequence and the fall action skeletal sequence template; retaining the masked human skeleton sequences whose similarity meets the preset similarity conditions, and filtering the masked human skeleton sequences whose similarity does not meet the preset similarity conditions, to obtain the target human skeleton sequence dataset.
10. A fall detection device for human skeletal motion recognition based on contrastive learning, characterized in that, The device includes: an acquisition module for acquiring video data containing human movements, extracting a human skeletal keypoint dataset from the video data, performing occlusion adaptation processing on the human skeletal keypoint dataset to obtain a completed skeletal keypoint dataset, and performing multi-human separation processing on the completed skeletal keypoint dataset to obtain a target human skeletal sequence dataset; a training module for converting the target human skeletal sequence dataset into a 3D heatmap volume, inputting the 3D heatmap volume into a 3D convolutional neural network for training, optimizing the model parameters of the 3D convolutional neural network using a contrastive loss function to obtain a motion feature representation; and a prediction module for inputting the motion feature representation into a classification head, mapping the probability distribution of motion categories through the classification head, and obtaining a prediction result of the fall motion based on the probability distribution.