A double-head action recognition prediction method and system based on skeleton sequence and spatiotemporal velocity feature fusion

Abstract

The application discloses a double-head action recognition and prediction method and system based on skeleton sequence and space-time speed feature fusion, relates to the technical field of computer vision, and comprises the following steps: acquiring video data; extracting the spatial position information of each human body skeleton key point on continuous frames from the video data to construct a human body skeleton key point sequence; calculating the motion change information of adjacent frames based on the human body skeleton key point sequence, and fusing the spatial position information of the skeleton key point to construct space-time-dynamics fusion features; and taking the space-time-dynamics fusion features as input, identifying the current action category and predicting the future action category based on a time series modeling network model. The application supports the identification and early warning of multiple abnormal behavior types, and can adapt to different application scenarios.

Description

Technical Field

[0001] This invention relates to a dual-head action recognition and prediction method and system based on the fusion of skeleton sequence and spatiotemporal velocity features, belonging to the field of computer vision technology. Background Technology

[0002] With the accelerating aging of the population, the need for safety monitoring of elderly people living alone and patients with chronic diseases in their home environments is becoming increasingly prominent. How to achieve continuous perception of their behavioral status and timely response to abnormal events has become a critical technical problem that urgently needs to be solved in the fields of smart healthcare and home-based elderly care. Existing intelligent monitoring systems mostly rely on traditional video surveillance technology, which achieves behavior recognition and event detection by analyzing video images, but there are still many shortcomings in practical applications.

[0003] First, in terms of early warning capabilities, most existing technologies are based on post-event identification mechanisms, that is, detection and alarm are only carried out after abnormal events such as falls or sudden illnesses have occurred. They lack the ability to dynamically model and predict risks in the stage before abnormal behavior occurs, making it difficult to provide an effective time window for external intervention in a timely manner.

[0004] Secondly, in terms of privacy protection, traditional data processing methods based on RGB images usually contain sensitive information such as facial features, clothing information, and living environment. They are difficult to promote and apply in privacy-sensitive scenarios such as bedrooms and bathrooms, and pose a significant risk of data security and privacy leakage.

[0005] Furthermore, in terms of model generalization ability, existing methods generally adopt a dataset construction method based on frame-level random partitioning, which is prone to introducing the problem of time series information leakage. This results in the model achieving high performance during the training phase, but being unstable in real application scenarios. At the same time, since the proportion of normal behavior samples is much higher than that of abnormal behavior samples in actual monitoring scenarios, the category distribution exhibits obvious long-tail characteristics, which further exacerbates the model's inadequacy in identifying abnormal behavior.

[0006] Finally, in terms of system performance and functional expansion, some methods rely on video feature extraction models with high computational complexity, which are difficult to deploy on resource-constrained edge devices; and existing systems mostly focus on single abnormal behaviors (such as fall detection), which is insufficient to support the identification and early warning of multiple types of medical abnormal behaviors, making it difficult to meet the needs of practical applications.

[0007] Therefore, there is an urgent need to propose an intelligent behavior monitoring method and system that, while ensuring privacy and security, possesses the ability to provide early warning of abnormal behavior, has good generalization performance and low computational complexity, and supports the extension of multiple types of abnormal behavior, so as to improve the safety and reliability of home medical monitoring. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a dual-head action recognition and prediction method and system based on the fusion of skeleton sequence and spatiotemporal velocity features, which supports the recognition and early warning of various abnormal behavior types and can be adapted to different application scenarios.

[0009] To achieve the above objectives, the present invention is implemented using the following technical solution: On one hand, this invention provides a dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features, comprising: Acquire video data; The spatial location information of each human skeleton key point in consecutive frames is extracted from video data to form a human skeleton key point sequence. Motion change information between adjacent frames is calculated based on the sequence of key points of human skeleton, and then fused with the spatial location information of the key points of skeleton to construct spatiotemporal-dynamic fusion features. Using spatiotemporal-dynamic fusion features as input, the current action category is identified based on a temporal modeling network model, and the future action category is predicted.

[0010] Furthermore, the temporal modeling network model adopts a bidirectional two-layer long short-term memory network model, which includes a backbone network and an attention mechanism module connected in sequence. The output of the attention mechanism module is connected to the classification head and the prediction head. The backbone network is used to extract features from the spatiotemporal-dynamic fusion features to obtain a temporal feature sequence, the expression of which is: ; in, Represents a time-series characteristic sequence. Indicates the first Temporal features corresponding to frames Indicates the number of frames in the video data; The attention mechanism module is used to perform weighted and normalized processing on each temporal feature in the temporal feature sequence, and then sum them up to obtain the global spatiotemporal context vector. The classification head is used to output the current action category based on the global spatiotemporal context vector; The prediction head is used to predict future action categories based on the global spatiotemporal context vector.

[0011] Furthermore, the attention mechanism module is used to perform weighted and normalized processing on each temporal feature in the temporal feature sequence, and to obtain a global spatiotemporal context vector by weighted summation, including: The temporal attention mechanism is used to weight each temporal feature in the temporal feature sequence, resulting in an attention score, which is expressed as follows: ; in, Indicates the first Attention score for the corresponding frame , Indicates learnable parameters, This represents the matrix transpose operation. Indicates the first Temporal features corresponding to frames Indicates the number of frames in the video data; The attention scores are normalized and then weighted and summed to obtain the global spatiotemporal context vector, which is expressed as follows: ; ; in, Indicates the first The normalized attention score corresponding to the frame, Represents an exponential function. Indicates the first Temporal features corresponding to frames Represents the global spatiotemporal context vector. Indicates the first Attention score for each frame; The data processing expression for the classification head is: ; in, This indicates the output of the classification header. This represents the behavior recognition mapping function. Represents the global spatiotemporal context vector. This represents the activation function. , , , All are learnable parameters; The data processing expression for the prediction head is: ; in, This indicates the output of the prediction head. This represents a mapping function for predicting the risk of abnormal behavior. , , , All are learnable parameters; The , , , and , , , They are independent of each other.

[0012] Furthermore, it also includes pre-training the time-series modeling network model, the pre-training method comprising: Acquire video data; The spatial location information of each human skeleton key point in consecutive frames is extracted from the video data to form a human skeleton key point sequence, and each frame in the human skeleton key point sequence is labeled with an action category label. An asymmetric sliding window was constructed to sample the sequence of key points of the human skeleton to obtain multiple window data. For each window of data, the corresponding judgment label is obtained based on the action category label of each frame in the window data, and the motion change information of adjacent frames is calculated based on the human skeleton key point sequence. The spatiotemporal-dynamic fusion feature is then fused with the spatial position information of the human skeleton key points to construct the corresponding spatiotemporal-dynamic fusion feature, which serves as the training sample. The training samples are used as input to train the temporal modeling network model. During the training process, the loss function is calculated, and the model parameters are adjusted according to the loss function until a fixed number of training rounds are reached to obtain the pre-trained temporal modeling network model.

[0013] Furthermore, the determination label includes a current action category identification label and a future action category prediction label; The method for obtaining the current action category identification tag includes: An indicator function is used to calculate whether a frame is abnormal based on the action category label of each frame in the window data. The expression is as follows: ; in, Indicates the first An indicator function to determine if a frame is abnormal. Indicates the first Frame action category labels, Represents a set of abnormal action categories; The number of all abnormal frames in the window data is counted based on the indicator function indicating whether each frame in the window data is abnormal. The expression is as follows: ; in, Indicates the number of abnormal frames. Indicates the number of frames contained in the window data; The current action category identification label of the window data is obtained by counting all abnormal frames within the window data, and its expression is: ; in, This indicates the current action category identification label. This indicates the threshold for the number of abnormal frames. The mode operator is used to represent the number of elements in a given number. This means selecting the action category that appears most frequently in the set of abnormal action categories as the current action category. This indicates that it is normal; The method for obtaining the future action category prediction label includes: Using the current window as a baseline, take the subsequent frames as the future observation interval; If the current action category identification label of the current window is normal, and an abnormal action category appears within the future observation interval, then the future action category prediction label of the current window is set to the abnormal action category; otherwise, the current action category identification label of the current window is the future action category prediction label, and its expression is: ; in, Indicates a label predicting future action categories. This represents the set of abnormal action categories within the future observation period. This indicates the last element operation. This indicates that the current action category is identified as normal. This indicates the number of frames included in the future observation interval.

[0014] Furthermore, the calculation expression for the loss function is as follows: ; ; ; ; ; ; ; ; ; in, Represents the loss function. This represents an improved multi-class focus loss function based on the focus loss function. This indicates the current action category identification label. This indicates the current action category output by the model. This represents the model hyperparameters for current action category recognition. The focus parameter represents the focus loss function. Indicates a label predicting future action categories. This indicates the future action category output by the model. The hyperparameters of the model representing the prediction of future action categories; Indicates the first The current action category of each training sample is identified as the action category. The probability, Indicates the action category in the current action category recognition. Normalized weights, Indicates the action category in the current action category recognition. The weight, Indicates the total number of action categories. Indicates action category The total number, Represents a positive infinitesimal. Represents the smoothing factor. Dictionary of action categories A set of key-value pairs; Indicates the first The future action category prediction for each training sample is the action category. The probability, Indicates the action category in future action category prediction Normalized weights, Indicates the action category in future action category prediction The weight.

[0015] Furthermore, the expression of the action category dictionary is {1: normal, 2: fall warning signs, 3: fall, 4: back pain, 5: blowing nose, 6: headache, 7: nausea / vomiting, 8: neck pain, 9: sneezing / coughing, 10: unsteady standing}.

[0016] Furthermore, the method for constructing the sequence of key points of the human skeleton includes: Extract the coordinates of key points of the human skeleton in each frame of the video data to obtain the position representation of key points of the human skeleton in each frame. The key point representations of the human skeleton in each frame are normalized to map them uniformly to a preset range, resulting in the sequence of key points of the human skeleton for each frame. The processing expression is as follows: ; ; in, Indicates the current frame's... Key points of the human skeleton Axis coordinates Indicates the height of the current frame. Indicates the width of the current frame. Indicates the current frame's... Key points of the human skeleton Axis coordinates Indicates the current frame's... Normalization of key points of the human skeleton Axis coordinates Indicates the current frame's... Normalization of key points of the human skeleton Axis coordinates Indicates the first A sequence of key points on the human skeleton in a frame. This indicates a normalization operation. Indicates the current frame's... Normalized coordinates of key points in an individual's skeletal structure. This represents the total number of key points on the human skeleton in the current frame.

[0017] Furthermore, the calculation of motion change information at adjacent time steps based on the sequence of key points in the human skeleton, and the fusion of this information with the spatial location information of the key points to construct a spatiotemporal-dynamic fusion feature, includes: By performing corresponding difference operations on the normalized coordinates of human skeletal keypoints in adjacent frames, motion change features are obtained, the expression of which is: ， ； ; in, Indicates the characteristics of motion changes. Indicates the first A sequence of key points on the human skeleton in a frame. Indicates the first A sequence of key points on the human skeleton in a frame. This indicates that the sequence of key points in the human skeleton has been completed. , Indicates the number of frames in the video data; Flattening the sequence of human skeletal key points corresponding to each frame into a vector form yields the spatial location features, the expression of which is: ; in, Indicates spatial location characteristics, Indicates the first The sequence of key points of the human skeleton in a frame; By concatenating motion change features and spatial location features, a spatiotemporal-dynamic fusion feature is obtained, the expression of which is: ; in, This indicates the spatiotemporal-dynamic fusion characteristics.

[0018] On the other hand, the present invention also provides a dual-head action recognition and prediction system based on the fusion of skeleton sequence and spatiotemporal velocity features, for implementing the dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features as described in any of the above claims, comprising: The video data acquisition module is configured to acquire video data. The human skeleton key point sequence extraction module is configured to extract the spatial location information of each human skeleton key point in consecutive frames from video data to form a human skeleton key point sequence. The spatiotemporal-dynamic fusion feature construction module is configured to calculate motion change information of adjacent frames based on the sequence of human skeleton key points, and fuse it with the spatial position information of the skeleton key points to construct spatiotemporal-dynamic fusion features. The action recognition and prediction module is configured to take spatiotemporal-dynamic fusion features as input, identify the current action category based on a temporal modeling network model, and predict the future action category.

[0019] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This invention constructs a feature representation that integrates spatiotemporal information and motion change information of key points of the human skeleton, and combines temporal modeling and a forward-looking label mapping mechanism. This enables the model to not only identify the current behavioral state, but also assess the risk of abnormal behavior in the future time range, thereby achieving early warning of abnormal behavior and improving the system's response time to emergencies. This invention effectively reduces the impact of time series information leakage by using a time series-based sample construction method, combined with physically isolated data partitioning and differentiated sampling strategies, and alleviates the problem of class imbalance to a certain extent, thereby improving the model's generalization ability and stability under different subjects and complex application scenarios. This invention adopts a feature representation method based on key points of the human skeleton, which only models the abstract human structure information, avoiding direct processing of sensitive information such as face and environment, which is conducive to improving the privacy and security of the system; at the same time, the feature representation and model structure adopted have low computational complexity, making it easy to deploy on resource-constrained devices. This invention, through a modular system architecture design, enables the system to support the identification and early warning of multiple abnormal behavior types within a unified framework. By expanding the definition of behavior categories, it can be adapted to different application scenarios, thereby enhancing the system's versatility and scalability. Attached Figure Description

[0020] Figure 1This is a flowchart illustrating a dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features in one embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the weight allocation of key frames in the fall process by the temporal attention mechanism of a dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features in one embodiment of the present invention. Figure 3 This is a schematic diagram of the temporal modeling network model of a dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features in one embodiment of the present invention. Figure 4 This is a schematic diagram of the changes in total training loss and verification accuracy during the pre-training process of a dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features in one embodiment of the present invention. The left side is a schematic diagram of the changes in total training loss, and the right side is a schematic diagram of the changes in verification accuracy. Figure 5 This is a schematic diagram of the confusion matrix of the classification head of the dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features in one embodiment of the present invention, on 10 types of abnormal medical behaviors. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention. Example 1

[0022] like Figure 1 As shown, this embodiment of the invention provides a dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features, specifically including the following steps: A temporal modeling network model is constructed. In this embodiment, the temporal modeling network model adopts a bidirectional two-layer long short-term memory network model (Bi-LSTM), and an attention mechanism is introduced. The specific structure is as follows: Figure 3 As shown.

[0023] The temporal modeling network model consists of a backbone network and an attention mechanism module connected sequentially. The backbone network is used to extract features from the spatiotemporal-dynamic fusion features to obtain a temporal feature sequence. The expression for the temporal feature sequence is: ; in, Represents a time-series characteristic sequence. Indicates the first Temporal features corresponding to frames This indicates the number of frames in the video data.

[0024] like Figure 2 As shown, the attention mechanism module is used to calculate the contribution weight of each time step in the time series to the final output, in order to enhance the feature representation ability of key behavioral stages. Specifically: The temporal attention mechanism is used to weight each temporal feature in the temporal feature sequence, resulting in an attention score, which is expressed as follows: ; in, Indicates the first Attention score for the corresponding frame , Indicates learnable parameters, This represents the matrix transpose operation.

[0025] The attention scores are normalized and then weighted and summed to obtain the global spatiotemporal context vector, which is expressed as follows: ; ; in, Indicates the first The normalized attention score corresponding to the frame, Represents an exponential function. This represents the global spatiotemporal context vector, used to extract the temporal dependencies of actions. Indicates the first Attention score for each frame.

[0026] The global spatiotemporal context vector is input to both the classification head and the prediction head. The classification head is used to output the current action category based on the global spatiotemporal context vector, and its data processing expression is as follows: ; in, This indicates the output of the classification header. This represents the behavior recognition mapping function. Represents the global spatiotemporal context vector. This represents the activation function. , , , All of these are learnable parameters.

[0027] The prediction head is used to predict future action categories based on the global spatiotemporal context vector, in order to predict the risk of abnormal behavior within a future time range. Its data processing expression is as follows: ; in, This indicates the output of the classification header. This represents the behavior recognition mapping function. Represents the global spatiotemporal context vector. This represents the activation function. , , , All are learnable parameters It should be noted that, , , , and , , , These are two sets of independent and learnable parameters.

[0028] The constructed temporal modeling network model is pre-trained. The pre-training method includes the following steps: Obtain video data.

[0029] The spatial location information of key points of the human skeleton in consecutive frames is extracted from video data to form a sequence of key points of the human skeleton, which serves as a feature representation reflecting the human motion state. In this embodiment, the extraction of human skeleton keypoint coordinates is achieved using an existing open-source human pose estimation model. This yields the position representation of the human skeleton keypoints in each frame, which is then normalized to map to a fixed range, thus eliminating the influence of data from different resolutions. The processing can be represented as follows: ; ; in, Indicates the current frame's... Key points of the human skeleton Axis coordinates Indicates the height of the current frame. Indicates the width of the current frame. Indicates the current frame's... Key points of the human skeleton Axis coordinates Indicates the current frame's... Normalization of key points of the human skeleton Axis coordinates Indicates the current frame's... Normalization of key points of the human skeleton Axis coordinates Indicates the first A sequence of key points on the human skeleton in a frame. This indicates a normalization operation. Indicates the current frame's... Normalized coordinates of key points in an individual's skeletal structure. This represents the total number of key points on the human skeleton in the current frame.

[0030] Each frame in the human skeleton keypoint sequence is labeled with an action category tag.

[0031] Construct a forward-looking label generation mechanism based on a time window, where the time window length is . L An asymmetric sliding window was constructed to sample the sequence of key points of the human skeleton to obtain multiple window data.

[0032] For each window of data, a corresponding decision label is obtained based on the action category label of each frame within the window. The decision label includes the current action category identification label and the future action category prediction label.

[0033] The methods for obtaining the current action category identification label include: An indicator function is used to calculate whether a frame is abnormal based on the action category label of each frame in the window data. The expression is as follows: ; in, Indicates the first An indicator function to determine if a frame is abnormal. i = 1, 2, …, L， Indicates the first Frame action category labels, This represents a set of abnormal action categories.

[0034] The number of all abnormal frames in the window data is counted based on the indicator function indicating whether each frame in the window data is abnormal. The expression is as follows: ; in, Indicates the number of abnormal frames. Indicates the number of frames contained in the window data.

[0035] The current action category identification label of the window data is obtained by counting all abnormal frames within the window data, and its expression is: ; in, This indicates the current action category identification label. This indicates the threshold for the number of abnormal frames. The mode operator is used to represent the number of elements in a given number. This means selecting the action category that appears most frequently in the set of abnormal action categories as the current action category. This indicates that everything is normal.

[0036] Methods for obtaining future action category prediction labels include: Using the current window as a baseline, the subsequent frames are taken as the future observation interval.

[0037] The risk is recalibrated for the current window based on the behavior status within the future observation period. If the current action category identification label for the current window is normal, and an abnormal action category appears within the future observation period, then the predicted future action category label for the current window is set to that abnormal action category. Otherwise, the current action category identification label for the current window is the predicted future action category label, and its expression is: ; in, Indicates a label predicting future action categories. This represents the set of abnormal action categories within the future observation period. This indicates the last element operation. This indicates that the current action category is identified as normal. This indicates the number of frames included in the future observation interval.

[0038] Motion change information between adjacent frames is calculated based on the sequence of key points on the human skeleton, and then fused with the spatial location information of the key points on the human skeleton to construct the corresponding spatiotemporal-dynamic fusion feature. Specifically: The normalized coordinates of human skeletal keypoints in adjacent frames are subjected to corresponding difference operations to obtain motion change features characterizing human kinematics. The expression is as follows: ， ； ; in, Indicates the characteristics of motion changes. Indicates the first A sequence of key points on the human skeleton in a frame. Indicates the first A sequence of key points on the human skeleton in a frame. This indicates that the sequence of key points in the human skeleton has been completed. To ensure consistency over time, This indicates the number of frames in the video data. In this embodiment, the number of frames in the video data is 30.

[0039] Flattening the sequence of human skeletal key points corresponding to each frame into a vector form yields the spatial location features, the expression of which is: ; in, Indicates spatial location characteristics, Indicates the first The sequence of key points of the human skeleton in a frame; By concatenating motion change features and spatial location features along the channel dimension, a spatiotemporal-dynamic fusion feature is obtained, the expression of which is: ; in, This represents the spatiotemporal-dynamic fusion features, which are used as training samples.

[0040] The training samples are used as input to train the temporal modeling network model. During the training process, the loss function is calculated, and the model parameters are adjusted according to the loss function until a fixed number of training rounds are reached to obtain the pre-trained temporal modeling network model.

[0041] In this embodiment, the expression for the loss function is: ; ; ; ; ; ; ; ; ; in, Represents the loss function. This represents an improved multi-class focus loss function based on the focus loss function. This indicates the current action category identification label. This indicates the current action category output by the model. This represents the model hyperparameters for current action category recognition. The focus parameter represents the focus loss function. Indicates a label predicting future action categories. This indicates the future action category output by the model. This represents the model hyperparameters for predicting future action categories.

[0042] Indicates the first The current action category of each training sample is identified as the action category. The probability, Indicates the action category in the current action category recognition. Normalized weights, Indicates the action category in the current action category recognition. The weight, This represents the total number of action categories and also serves as a scaling factor, ensuring that the weighted average loss is on the same order of magnitude as the defined total loss, thereby ensuring the effectiveness of the preset learning rate. Indicates action category The total number, It represents a positive infinitesimal, preventing division by zero due to a certain action category being zero. This represents a smoothing factor used to mitigate imbalances in action categories. Dictionary of action categories The key-value set, the expression of the action category dictionary is {1: normal, 2: fall warning signs, 3: fall, 4: back pain, 5: blowing nose, 6: headache, 7: nausea / vomiting, 8: neck pain, 9: sneezing / coughing, 10: unsteady standing}.

[0043] Indicates the first The future action category prediction for each training sample is the action category. The probability, Indicates the action category in future action category prediction Normalized weights, Indicates the action category in future action category prediction The weight.

[0044] In some embodiments, a validation set is also included. However, it should be noted that during the training phase, a dense sliding sampling strategy with a step size smaller than the window length is adopted to implicitly expand the transitional state samples from the precursor to the occurrence of abnormal behavior. During the validation and application phases, an independent sampling strategy with a step size not less than the window length is adopted to physically isolate adjacent sample blocks and prevent the leakage of time sequence information caused by high-frequency sampling.

[0045] In this embodiment, the curves showing the changes in total training loss and validation accuracy during the pre-training process are as follows: Figure 4 As shown, with the increase of the number of training rounds, the total training loss of the model continues to decrease and converges rapidly, while the accuracy of the validation set gradually increases and stabilizes at a high level. This indicates that the multi-task joint training strategy, asymmetric sliding window sampling mechanism, and improved weighted focus loss function proposed in this invention can effectively alleviate the problems of class imbalance and temporal information leakage, enabling the model to maintain good convergence and stability during training and possess strong generalization ability and prediction reliability.

[0046] During the application phase, video data is acquired; The spatial location information of each human skeleton key point in consecutive frames is extracted from the video data to form a human skeleton key point sequence.

[0047] Motion change information between adjacent frames is calculated based on the sequence of key points on the human skeleton, and then fused with the spatial location information of the key points to construct a spatiotemporal-dynamic fusion feature. The specific steps of the spatiotemporal-dynamic fusion feature are the same as those described above, and will not be repeated here.

[0048] Using spatiotemporal-dynamic fusion features as input, a pre-trained temporal modeling network model is used to identify the current action category and predict the future action category.

[0049] The confusion matrix of the classification head in this embodiment for 10 types of abnormal medical behaviors is as follows: Figure 5 As shown, this invention achieves high accuracy in identifying key categories such as normal behavior, falls, and gait instability. The accuracy rate for the normal behavior category reaches 98%, while the accuracy rates for high-risk abnormal behaviors like falls and gait instability are 88% and 87%, respectively, indicating that the model has a good ability to identify high-priority safety-related behaviors. Simultaneously, the model also demonstrates a certain degree of discrimination against transitional or non-rigid movements such as fall precursors, back pain, and nose blowing. The overall identification results show high consistency with the real labels, validating the spatiotemporal-dynamic fusion feature and temporal modeling method proposed in this invention. This method can effectively capture the differences in action patterns of different medical behaviors and exhibits good robustness and practicality in multi-category medical behavior identification tasks. Example 2

[0050] This embodiment also provides a dual-head action recognition and prediction system based on the fusion of skeleton sequence and spatiotemporal velocity features, used to implement the dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features as described in Embodiment 1, including: The video data acquisition module is configured to acquire video data.

[0051] The human skeleton key point sequence extraction module is configured to extract the spatial location information of each human skeleton key point in consecutive frames from video data to form a human skeleton key point sequence.

[0052] The spatiotemporal-dynamic fusion feature construction module is configured to calculate motion change information of adjacent frames based on the sequence of human skeletal key points, and fuse it with the spatial position information of the skeletal key points to construct spatiotemporal-dynamic fusion features.

[0053] The action recognition and prediction module is configured to take spatiotemporal-dynamic fusion features as input, identify the current action category based on a temporal modeling network model, and predict the future action category.

[0054] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features, characterized in that, include: Acquire video data; The spatial location information of each human skeleton key point in consecutive frames is extracted from video data to form a human skeleton key point sequence. Motion change information between adjacent frames is calculated based on the sequence of key points of human skeleton, and then fused with the spatial location information of the key points of skeleton to construct spatiotemporal-dynamic fusion features. Using spatiotemporal-dynamic fusion features as input, the current action category is identified based on a temporal modeling network model, and the future action category is predicted.

2. The dual-head action recognition and prediction method based on skeleton sequence and spatiotemporal velocity feature fusion according to claim 1, characterized in that, The temporal modeling network model adopts a bidirectional two-layer long short-term memory network model, which includes a backbone network and an attention mechanism module connected in sequence. The output of the attention mechanism module is connected to the classification head and the prediction head. The backbone network is used to extract features from the spatiotemporal-dynamic fusion features to obtain a temporal feature sequence, the expression of which is: ； in, Represents a time-series characteristic sequence. Indicates the first Temporal features corresponding to frames Indicates the number of frames in the video data; The attention mechanism module is used to perform weighted and normalized processing on each temporal feature in the temporal feature sequence, and then sum them up to obtain the global spatiotemporal context vector. The classification head is used to output the current action category based on the global spatiotemporal context vector; The prediction head is used to predict future action categories based on the global spatiotemporal context vector.

3. The dual-head action recognition and prediction method based on skeleton sequence and spatiotemporal velocity feature fusion according to claim 2, characterized in that, The attention mechanism module is used to perform weighted and normalized processing on each temporal feature in the temporal feature sequence, and then sum the weighted values ​​to obtain a global spatiotemporal context vector, including: The temporal attention mechanism is used to weight each temporal feature in the temporal feature sequence, resulting in an attention score, which is expressed as follows: ； in, Indicates the first Attention score for the corresponding frame , Indicates learnable parameters, This represents the matrix transpose operation. Indicates the first Temporal features corresponding to frames Indicates the number of frames in the video data; The attention scores are normalized and then weighted and summed to obtain the global spatiotemporal context vector, which is expressed as follows: ； ； in, Indicates the first The normalized attention score corresponding to the frame, Represents an exponential function. Represents the global spatiotemporal context vector. Indicates the first Attention score for each frame; The data processing expression for the classification head is: ； in, This indicates the output of the classification header. This represents the behavior recognition mapping function. Represents the global spatiotemporal context vector. This represents the activation function. , , , All are learnable parameters; The data processing expression for the prediction head is: ； in, This indicates the output of the prediction head. This represents a mapping function for predicting the risk of abnormal behavior. , , , All are learnable parameters; The , , , and , , , They are independent of each other.

4. The dual-head action recognition and prediction method based on skeleton sequence and spatiotemporal velocity feature fusion according to claim 1, characterized in that, It also includes pre-training the time-series modeling network model, the pre-training method including: Acquire video data; The spatial location information of each human skeleton key point in consecutive frames is extracted from the video data to form a human skeleton key point sequence, and each frame in the human skeleton key point sequence is labeled with an action category label. An asymmetric sliding window was constructed to sample the sequence of key points of the human skeleton to obtain multiple window data. For each window of data, the corresponding judgment label is obtained based on the action category label of each frame in the window data, and the motion change information of adjacent frames is calculated based on the human skeleton key point sequence. The spatiotemporal-dynamic fusion feature is then fused with the spatial position information of the human skeleton key points to construct the corresponding spatiotemporal-dynamic fusion feature, which serves as the training sample. The training samples are used as input to train the temporal modeling network model. During the training process, the loss function is calculated, and the model parameters are adjusted according to the loss function until a fixed number of training rounds are reached to obtain the pre-trained temporal modeling network model.

5. The dual-head action recognition and prediction method based on skeleton sequence and spatiotemporal velocity feature fusion according to claim 4, characterized in that, The determination label includes a current action category identification label and a future action category prediction label; The method for obtaining the current action category identification tag includes: An indicator function is used to calculate whether a frame is abnormal based on the action category label of each frame in the window data. The expression is as follows: ； in, Indicates the first An indicator function to determine if a frame is abnormal. Indicates the first Frame action category labels, Represents a set of abnormal action categories; The number of all abnormal frames in the window data is counted based on the indicator function indicating whether each frame in the window data is abnormal. The expression is as follows: ； in, Indicates the number of abnormal frames. This indicates the number of frames contained in the window data; The current action category identification label of the window data is obtained based on the number of all abnormal frames within the window data. The expression is as follows: ； in, This indicates the current action category identification label. This indicates the threshold for the number of abnormal frames. The mode operator is used to represent the number of elements in a given number. This means selecting the action category that appears most frequently in the set of abnormal action categories as the current action category. This indicates that it is normal; The method for obtaining the future action category prediction label includes: Using the current window as a baseline, take the subsequent frames as the future observation interval; If the current action category identification label of the current window is normal, and an abnormal action category appears within the future observation interval, then the future action category prediction label of the current window is set to the abnormal action category; otherwise, the current action category identification label of the current window is the future action category prediction label, and its expression is: ； in, Indicates a label for predicting future action categories. This represents the set of abnormal action categories within the future observation period. This indicates the last element operation. This indicates that the current action category is identified as normal. This indicates the number of frames included in the future observation interval.

6. The dual-head action recognition and prediction method based on skeleton sequence and spatiotemporal velocity feature fusion according to claim 4, characterized in that, The expression for calculating the loss function is as follows: ； ； ； ； ； ； ； ； ； in, Represents the loss function. This represents an improved multi-class focus loss function based on the focus loss function. This indicates the current action category identification label. This indicates the current action category output by the model. This represents the model hyperparameters for current action category recognition. The focus parameter represents the focus loss function. Indicates a label for predicting future action categories. This indicates the future action category output by the model. The hyperparameters of the model representing the prediction of future action categories; Indicates the first The current action category of each training sample is identified as the action category. The probability, Indicates the action category in the current action category recognition. Normalized weights, Indicates the action category in the current action category recognition. The weight, Indicates the total number of action categories. Indicates action category The total number, Represents a positive infinitesimal. Represents the smoothing factor. Dictionary of action categories A set of key-value pairs; Indicates the first The future action category prediction for each training sample is the action category. The probability, Indicates the action category in future action category prediction Normalized weights, Indicates the action category in future action category prediction The weight.

7. The dual-head action recognition and prediction method based on skeleton sequence and spatiotemporal velocity feature fusion according to claim 6, characterized in that, The expression for the action category dictionary is {1: normal, 2: fall warning signs, 3: fall, 4: back pain, 5: blowing nose, 6: headache, 7: nausea / vomiting, 8: neck pain, 9: sneezing / coughing, 10: unsteady standing}.

8. The dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features according to claim 1 or 4, characterized in that, The method for constructing the sequence of key points of the human skeleton includes: Extract the coordinates of key points of the human skeleton in each frame of the video data to obtain the position representation of key points of the human skeleton in each frame. The key point representations of the human skeleton in each frame are normalized to map them uniformly to a preset range, resulting in the sequence of key points of the human skeleton for each frame. The processing expression is as follows: ； ； in, Indicates the current frame's... Key points of the human skeleton Axis coordinates Indicates the height of the current frame. Indicates the width of the current frame. Indicates the current frame's... Key points of the human skeleton Axis coordinates Indicates the current frame's... Normalization of key points of the human skeleton Axis coordinates Indicates the current frame's... Normalization of key points of the human skeleton Axis coordinates Indicates the first A sequence of key points on the human skeleton in a frame. This indicates a normalization operation. Indicates the current frame's... Normalized coordinates of key points in an individual's skeletal structure. This represents the total number of key points on the human skeleton in the current frame.

9. The dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features according to claim 1 or 4, characterized in that, The process involves calculating motion change information at adjacent time steps based on the sequence of key points on the human skeleton, and fusing this information with the spatial location information of the key points to construct a spatiotemporal-dynamic fusion feature, including: By performing corresponding difference operations on the normalized coordinates of human skeletal keypoints in adjacent frames, motion change features are obtained, the expression of which is: ； ； in, Indicates the characteristics of motion changes. Indicates the first A sequence of key points on the human skeleton in a frame. Indicates the first A sequence of key points on the human skeleton in a frame. This indicates that the sequence of key points in the human skeleton has been completed. , Indicates the number of frames in the video data; Flattening the sequence of human skeletal key points corresponding to each frame into a vector form yields the spatial location features, the expression of which is: ； in, Indicates spatial location characteristics, Indicates the first The sequence of key points of the human skeleton in a frame; By concatenating motion change features and spatial location features, a spatiotemporal-dynamic fusion feature is obtained, the expression of which is: ； in, This indicates the spatiotemporal-dynamic fusion characteristics.

10. A dual-head action recognition and prediction system based on the fusion of skeleton sequence and spatiotemporal velocity features, characterized in that, The method for implementing the dual-head action recognition and prediction method based on the fusion of skeleton sequence and spatiotemporal velocity features as described in any one of claims 1 to 9 includes: The video data acquisition module is configured to acquire video data. The human skeleton key point sequence extraction module is configured to extract the spatial location information of each human skeleton key point in consecutive frames from video data to form a human skeleton key point sequence. The spatiotemporal-dynamic fusion feature construction module is configured to calculate motion change information of adjacent frames based on the sequence of human skeleton key points, and fuse it with the spatial position information of the skeleton key points to construct spatiotemporal-dynamic fusion features. The action recognition and prediction module is configured to take spatiotemporal-dynamic fusion features as input, identify the current action category based on a temporal modeling network model, and predict the future action category.

Images