Laboratory abnormal action recognition method based on skeleton point space-time sequence

Abstract

The invention discloses a laboratory abnormal action recognition method based on a skeleton point space-time sequence, belongs to the field of human body action recognition, and aims to solve the problems that a traditional human body action recognition method is difficult to adapt to complex backgrounds, shielding conditions and personnel dressing characteristics in a laboratory, and the recognition precision and robustness are remarkably insufficient. The method comprises the following steps: collecting typical abnormal action video data of laboratory personnel under the condition of wearing protective clothing, labeling the action data, extracting human body key skeleton points by using an MMpose model, and inputting the key skeleton point data and motion characteristics into an improved LSTM model for preliminary classification; meanwhile, the spatial position information of the key skeleton points is fused for joint decision, and finally an abnormal action category judgment result is output. According to the method, the abnormal action occurring when the person wears the protective clothing in the laboratory scene can be quickly identified, and the method has the characteristics of high identification accuracy and strong real-time performance.

Description

Technical Field

[0001] This invention belongs to the field of human motion recognition, specifically relating to a laboratory abnormal motion recognition method based on spatiotemporal sequences of skeletal points. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, human motion recognition, as a key technology in fields such as intelligent monitoring and human-computer interaction, has received widespread attention. Especially in high-risk, highly enclosed environments such as laboratories, the recognition of abnormal human movements is of great significance for safety protection, emergency response, and behavioral regulation.

[0003] During experiments, operators typically wear protective suits, breathing masks, and other specialized equipment, which restricts their freedom of movement and makes abnormal states harder to detect. Therefore, real-time detection and intelligent recognition of key human movements are crucial technologies for improving the level of intelligent laboratory management and ensuring personnel safety. Actions such as touching the head or waist, squatting, bending over, and falling are particularly important safety indicators: for example, touching the head or waist may indicate discomfort, leaks, or pressure from protective clothing or breathing equipment; frequent or abnormally large squatting or bending may indicate physical exhaustion or equipment malfunction; and falling is a typical emergency action requiring rapid response to prevent accidents. Therefore, accurate detection of these actions by the system helps to identify risks early and provide timely emergency response.

[0004] However, traditional human motion recognition methods struggle to adapt to complex backgrounds, occlusions, and clothing characteristics in laboratory settings, exhibiting significant shortcomings in recognition accuracy and robustness. Furthermore, the recognition results output by a single deep learning model often lack interpretability and have poor fault tolerance, making reliable application in high-risk scenarios difficult. Summary of the Invention

[0005] In order to address the problem that traditional human motion recognition methods are difficult to adapt to the complex backgrounds, occlusions, and clothing characteristics of personnel in the laboratory, and have significant deficiencies in recognition accuracy, this invention provides a laboratory abnormal motion recognition method based on spatiotemporal sequences of skeletal points.

[0006] A laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points, which is implemented through the following steps;

[0007] The first step is to train the improved LSTM model on a sample set:

[0008] S1. Constructing the sample set:

[0009] Collect video data of laboratory personnel's movements while wearing protective clothing, including six types of movements: normal state, touching head, touching waist, bending over, squatting, and falling to the ground;

[0010] S2. Preprocess the sample set:

[0011] Each sample's motion video is divided into multiple original images according to the video frame rate (FPS), with each frame corresponding to one original image.

[0012] Based on the original image data, the MMPose model is used to extract the coordinate information of key human skeletal points, and the velocity and acceleration are calculated based on the key point coordinate sequence.

[0013] Each frame of the image is annotated with actions to obtain corresponding action label information, wherein the action label information is the action category corresponding to that frame;

[0014] S3. Construct a training set based on the preprocessed sample set and train the improved LSTM model:

[0015] The coordinates, velocity, and acceleration sequences of key human skeletal points corresponding to each frame of the original image are used as input data in the training set. The velocity of the first frame is set to 0, and the acceleration of the first and second frames is set to 0. The action category label corresponding to the input data is used as the ground truth. An improvement was made to the original LSTM model by adding an attention mechanism and optimizing the loss function. The LSTM model is then trained to determine the model's weight parameters and complete the training.

[0016] Secondly, the improved LSTM model after training was used to identify abnormal actions in the laboratory.

[0017] L1. Acquire real-time monitoring images and perform personnel detection:

[0018] The system acquires images during real-time monitoring, processes each frame using the YOLOv8 model, identifies laboratory personnel in the images, and outputs the bounding box information of the personnel.

[0019] L2. Pose estimation based on personnel detection results:

[0020] The output personnel bounding box information is used as input to the MMPose model to perform skeletal key point detection and extract the coordinate information of key skeletal points of the target human body for subsequent action recognition and judgment.

[0021] L3. Extracting kinematic features:

[0022] Based on the coordinate information of key human skeletal points output by the MMPose model, the velocity and acceleration of the key points in the time series are calculated.

[0023] L4. Preliminary recognition of human movements based on an improved LSTM model:

[0024] The coordinates, velocity, and acceleration features of key human skeletal points are input into a pre-trained improved LSTM model to make a preliminary judgment on human actions corresponding to multiple consecutive frames of images and output the corresponding action category label.

[0025] L5. Fusion judgment based on the positional relationships of key human skeletal points:

[0026] The action category labels output by the improved LSTM model are fused with the action category labels determined based on the positional relationship of key human skeletal points. The action recognition results are then reconfirmed, and human action category labels are output. At the same time, optimization strategies are adopted to enhance the real-time performance of the overall detection.

[0027] Furthermore, in step S2, the motion video of each sample is divided into multiple frames of original images according to the video frame rate (FPS), where FPS = 30 f / s, and f / s is frames per second.

[0028] Furthermore, in step S2, action annotation is performed on each frame of the image. Specifically, the COCO17 key point annotation format is selected as the inference format for pose estimation, namely, nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0029] Furthermore, in step S2, the MMPose model is used to extract the coordinate information of key human skeletal points. The MMPose model used is an improved MMPose model. The specific improvements include replacing the original HRNet backbone network in the MMPose model with the EfficientNet backbone network to improve the computational efficiency of skeletal key point detection and the model inference speed, while ensuring the accuracy of key point localization.

[0030] Furthermore, the specific method for calculating velocity and acceleration based on the key point coordinate sequence in step S2 is as follows:

[0031] The MMPose model predicts the 2D coordinates of 17 skeletal points of the human body and performs normalization. Then, the skeletal point data of the current frame is stored in the historical sequence, ensuring that a maximum of three frames of historical data are retained for subsequent calculations. When a frame is lost, the system automatically uses the keypoint data from the previous frame to complete the calculation. After obtaining three frames of historical keypoints, the center point position is first calculated based on the coordinates of the target bounding box. The calculation formula is as follows:

[0032] (1)

[0033] (2)

[0034] Among them, X min Y min X max Y max Given the coordinates of the top-left and bottom-right corners of the target bounding box, and based on historical center point data from three consecutive frames, the target's velocity and acceleration are further calculated using the following formula:

[0035] (3)

[0036] (4)

[0037] Where f is the video frame rate, and X... t and X t-1 These are the center point coordinate vectors of the current frame and the previous frame, respectively;

[0038] Furthermore, the improved method for the LSTM model in step S3 is as follows:

[0039] An additive attention mechanism was added to the LSTM model. Specifically, this involved: first, encoding the coordinates, velocities, and acceleration sequences of the skeletal points using an LSTM layer to obtain the hidden state output at each time step; then, using the additive attention mechanism to calculate the importance weights of the hidden states at each time step, with the score e at time step t being calculated. t The calculation formula is:

[0040] (5)

[0041] in, Here, W1 represents the first weight matrix, W2 represents the second weight matrix, b1 represents the bias vector, b2 represents the scalar bias, tanh is the activation function, and h is the learnable quantity. t For the output of the LSTM model at time step t, e t It is a scalar used to explain the importance of the hidden state at time step t;

[0042] After calculating the attention weights, the weights are normalized using the softmax function. The formula for calculating the attention weights at time step t is as follows:

[0043] (6)

[0044] Where T represents the total number of time steps in the sequence, i.e., all e t Quantity;

[0045] Next, the normalized attention weights are applied to the LSTM output to obtain a weighted summation context vector, calculated as follows:

[0046] (7)

[0047] Finally, the context vector is input into the fully connected layer, and the corresponding action category prediction result is output.

[0048] Building upon the above, to further improve the model's convergence speed and generalization ability, a batch normalization layer and a dropout mechanism are introduced after the fully connected layer. Batch normalization is applied to the hidden state of the last time step of the LSTM output to normalize the hidden vector. Given a mini-batch of input features... x is the output of the LSTM hidden layer. The coordinates, velocity, and acceleration are input into the LSTM and processed to obtain a feature representation. It is a high-dimensional feature vector that integrates coordinate, velocity, and acceleration information, rather than the original coordinates or kinematic quantities themselves. First, the mean and variance need to be calculated using the following formulas:

[0049] (8)

[0050] (9)

[0051] in, The mean, Let Variance be the variance.

[0052] Next, normalization is performed using the mean obtained from formula (8) and the variance obtained from formula (9). The calculation formula is as follows:

[0053] (10)

[0054] in, It is a very small constant used to avoid division by zero errors;

[0055] Finally, a linear transformation is performed, including scaling and translation, calculated using the following formula:

[0056] (11)

[0057] in, and These are learnable parameters. Indicates scalar scaling. This represents the translation vector, used to restore the network's expressive power;

[0058] Through this process, the data is standardized to a distribution with a mean of 0 and a variance of 1;

[0059] This model adds a Dropout layer after batch normalization and before the fully connected layer. The calculation formula is as follows:

[0060] (12)

[0061] Where x is the batch-normalized feature vector, and p=0.5 is the discard probability;

[0062] A multi-task loss combination was designed, consisting of three parts: cross-entropy loss (CE_Loss), mean squared error loss (MSE_Loss), and temporal smoothness loss (Tempt_Loss). The cross-entropy loss measures the model's accuracy in action classification, and its calculation formula is as follows:

[0063] (13)

[0064] Where N is the number of samples and C is the number of categories. For the one-hot encoding of sample i in class k, Predict the probability that sample i belongs to class k for the model;

[0065] The mean squared error loss is used to compare the numerical error between the model's predicted key points and the actual key points. The calculation formula is as follows:

[0066] (14)

[0067] Where N is the number of samples, For the j-th predicted value of the i-th sample, This represents the j-th true value of the i-th sample.

[0068] The temporal smoothness loss is used to constrain the model's prediction output over time frames, ensuring that the prediction results change continuously and smoothly over time, avoiding sudden changes or jumps in the results. The calculation formula is as follows:

[0069] (15)

[0070] Where T is the number of time frames. Let be the class probability vector output by the model in frame t;

[0071] After calculating the above three losses, we continue to calculate the overall loss function, which is calculated using the following formula:

[0072] (16)

[0073] in, =1, =0.5, =0.1;

[0074] Furthermore, the action category labels predicted by the LSTM model are compared with the labels obtained based on the rules governing the positional relationships of skeletal keypoints. When the results are consistent, the category label is directly adopted as the final recognition result; when the results are inconsistent, the result based on the rule-based judgment is given priority to correct the model output, thereby achieving a secondary confirmation of the action recognition result and effectively improving the accuracy and confidence of the recognition.

[0075] Furthermore, the rules based on the positional relationships of skeletal keypoints include:

[0076] (1) Head-touching judgment: When the LSTM model initially predicts the action category as "normal", the distance between the hand bone point and the nose bone point is further calculated. If the distance is less than Z1 pixels, the action is judged as "head-touching".

[0077] (2) Waist palpation judgment: Calculate the waist position. Based on the average coordinates of the key points on the left and right hips, determine the center horizontal coordinate and initial vertical coordinate of the waist; then offset upwards by Z2 pixels to obtain the actual vertical coordinate of the waist. The calculation formula is:

[0078] (17)

[0079] (18)

[0080] Among them, X L Y L X R X R , representing the coordinates of the left hip and right hip respectively;

[0081] After obtaining the waist coordinates, the following three rules are used for further judgment: the distance between the hand and the waist is less than Z3 pixels; the angle formed by the skeletal points of the shoulder, elbow, and hand is less than 110 degrees; and the difference in vertical coordinates between the hand and the waist is less than Z4 pixels. If all three conditions are met, the action is judged as "touching the waist".

[0082] (3) Fall detection: A person is considered to be in a fall state when at least three of the following conditions are met: the angle between the line connecting the shoulder to the hip and the vertical direction is greater than 70 degrees; the ratio of the height difference between the head and ankle key points to the height difference between the hip and ankle key points is less than 0.5; the aspect ratio of the detection box is less than 0.6; the angle of the line connecting at least one leg from the knee joint to the ankle joint is less than 30 degrees or greater than 150 degrees, and there are fewer than 9 visible skeletal key points; the movement speed of the head, hip, or knee is >1.0 and the acceleration is >1.2.

[0083] (4) Squatting judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the hip, knee and ankle. If it is less than 100 degrees, the action is judged as "squatting".

[0084] (5) Bending judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the shoulder, hip and knee. If it is less than 100 degrees, the action is judged as "bending".

[0085] Furthermore, when the action category label initially output by the improved LSTM model is "normal", this label is directly used as the final recognition result; when the action category label initially output by the improved LSTM model is "abnormal", the action category is further judged based on the positional relationship of key human skeletal points. If the two judgment results are inconsistent, the judgment result based on the positional relationship of key human skeletal points is used as the final recognition result. When an action does not belong to any of the conditions of the rule judgment, the result predicted by LSTM is used.

[0086] The beneficial effects of this application compared to the prior art are:

[0087] This invention proposes a method for identifying abnormal actions in laboratories based on spatiotemporal sequences of skeletal points. This method enables intelligent monitoring and risk identification of the actions of laboratory personnel in a closed laboratory environment. By fully considering the impact of protective clothing on actions and incorporating multi-source spatiotemporal features for modeling, this method enhances the model's ability to identify abnormal actions in complex environments, providing support for safety protection and emergency response, thereby improving the level of intelligent laboratory safety management.

[0088] This invention collects video data on the behavior of personnel in a laboratory setting while wearing protective clothing, including six types of actions: normal, touching the head, touching the waist, bending over, squatting, and falling. The video is then segmented into multiple frames of raw images, and each frame is labeled with an action category. Subsequently, key skeletal point information of the human body is extracted from the images using an MMPose model to obtain the human motion posture features under protective clothing conditions. Furthermore, velocity and acceleration features are calculated based on the extracted skeletal point data. The extracted skeletal key point coordinates, velocity, and acceleration features are then input into an improved LSTM model for preliminary classification. Simultaneously, the spatial location information of the key skeletal points is fused for joint decision-making, ultimately achieving real-time identification of abnormal behaviors of laboratory personnel. Attached Figure Description

[0089] Figure 1This is a flowchart of the abnormal action recognition process of the present invention;

[0090] Figure 2 This is the annotation format for COCO's 17 key points;

[0091] Figure 3 This is the result of the laboratory abnormal action recognition based on the spatiotemporal sequence of skeletal points in this invention;

[0092] Figure 4 The confusion matrix for training results of the improved LSTM model;

[0093] Figure 5 This is the accuracy confusion matrix of a laboratory abnormal action recognition method based on spatiotemporal sequences of skeletal points.

[0094] Figure 3 (1) indicates a normal action, (2) indicates a head-touching action, (3) indicates a waist-touching action, (4) indicates a bending-over action, (5) indicates a squatting action, and (6) indicates a falling action. Detailed Implementation

[0095] Specific implementation method one: Combining Figures 1 to 5 This embodiment describes a method for identifying abnormal actions in a laboratory setting based on spatiotemporal sequences of skeletal points. This method is implemented through the following steps:

[0096] The first step is to train the improved LSTM model on a sample set:

[0097] S1. Constructing the sample set:

[0098] Collect video data of laboratory personnel's movements while wearing protective clothing, including six types of movements: normal state, touching head, touching waist, bending over, squatting, and falling to the ground;

[0099] S2. Preprocess the sample set:

[0100] Each sample's motion video is divided into multiple original images according to the video frame rate (FPS), with each frame corresponding to one original image.

[0101] Based on the original image data, the MMPose model is used to extract the coordinate information of key human skeletal points, and the velocity and acceleration are calculated based on the key point coordinate sequence.

[0102] Each frame of the image is annotated with actions to obtain corresponding action label information, wherein the action label information is the action category corresponding to that frame;

[0103] S3. Construct a training set based on the preprocessed sample set and train the improved LSTM model:

[0104] The coordinates, velocity, and acceleration sequences of key human skeletal points corresponding to each frame of the original image are used as input data in the training set. The velocity of the first frame is set to 0, and the acceleration of the first and second frames is set to 0. The action category label corresponding to the input data is used as the ground truth. An improvement was made to the original LSTM model by adding an attention mechanism and optimizing the loss function. The LSTM model is then trained to determine the model's weight parameters and complete the training.

[0105] Secondly, the improved LSTM model after training was used to identify abnormal actions in the laboratory.

[0106] L1. Acquire real-time monitoring images and perform personnel detection:

[0107] The system acquires images during real-time monitoring, processes each frame using the YOLOv8 model, identifies laboratory personnel in the images, and outputs the bounding box information of the personnel.

[0108] L2. Pose estimation based on personnel detection results:

[0109] The output personnel bounding box information is used as input to the MMPose model to perform skeletal key point detection and extract the coordinate information of key skeletal points of the target human body for subsequent action recognition and judgment.

[0110] L3. Extracting kinematic features:

[0111] Based on the coordinate information of key human skeletal points output by the MMPose model, the velocity and acceleration of the key points in the time series are calculated.

[0112] L4. Preliminary recognition of human movements based on an improved LSTM model:

[0113] The coordinates, velocity, and acceleration features of key human skeletal points are input into a pre-trained improved LSTM model to make a preliminary judgment on human actions corresponding to multiple consecutive frames of images and output the corresponding action category label.

[0114] L5. Fusion judgment based on the positional relationships of key human skeletal points:

[0115] The action category labels output by the improved LSTM model are fused with action category labels determined based on the positional relationship of key human skeletal points to perform secondary verification of the action recognition results and output human action category labels with higher confidence. At the same time, optimization strategies are adopted to enhance the real-time performance of the overall detection.

[0116] Specific Implementation Method Two: Combining Figures 1 to 5This embodiment further defines step S2 in specific embodiment one. In step S2, the motion video of each sample is divided into multiple original images according to the video frame rate (FPS), where FPS = 30 f / s, and f / s is frames per second. Other components and connection methods are the same as in specific embodiment one.

[0117] Specific implementation method three: Combining Figures 1 to 5 This embodiment further defines step S2 in Specific Embodiment 1. In step S2, action annotation is performed on each frame of the image. Specifically, the COCO17 keypoint annotation format is used as the inference format for pose estimation, namely, nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. Other components and connections are the same as in Specific Embodiment 1.

[0118] Specific implementation method four: Combination Figures 1 to 5 This embodiment further defines step S2 in Specific Embodiment 1. In step S2, the MMPose model is used to extract the coordinate information of key skeletal points in the human body. The MMPose model used is an improved version, specifically replacing the original HRNet backbone network in the MMPose model with an EfficientNet backbone network to improve the computational efficiency of skeletal keypoint detection and the model inference speed, while ensuring the accuracy of keypoint localization. Other methods and steps are the same as in Specific Embodiment 1.

[0119] Specific Implementation Method Five: Combining Figures 1 to 5 This embodiment further defines step S2 in specific embodiment one. The specific method for calculating velocity and acceleration based on the key point coordinate sequence in step S2 is as follows:

[0120] The MMPose model predicts the 2D coordinates of 17 skeletal points of the human body and performs normalization. Then, the skeletal point data of the current frame is stored in the historical sequence, ensuring that a maximum of three frames of historical data are retained for subsequent calculations. When a frame is lost, the system automatically uses the keypoint data from the previous frame to complete the calculation. After obtaining three frames of historical keypoints, the center point position is first calculated based on the coordinates of the target bounding box. The calculation formula is as follows:

[0121] (1)

[0122] (2)

[0123] Among them, X min Y min X maxY max Given the coordinates of the top-left and bottom-right corners of the human target's bounding box, and based on historical center point data from three consecutive frames, the target's velocity and acceleration are further calculated using the following formula:

[0124] (3)

[0125] (4)

[0126] Where f is the video frame rate, and X... t and X t-1 These are the coordinate vectors of the center points of the current frame and the previous frame, respectively. The other steps are the same as in Specific Implementation Method Four.

[0127] Specific Implementation Method Six: Combination Figures 1 to 5 This embodiment further defines step S3 in specific embodiment one. The improved method of the LSTM model in step S3 is as follows:

[0128] An additive attention mechanism was added to the LSTM model. Specifically, this involved: first, encoding the coordinates, velocities, and acceleration sequences of the skeletal points using an LSTM layer to obtain the hidden state output at each time step; then, using the additive attention mechanism to calculate the importance weights of the hidden states at each time step, with the score e at time step t being calculated. t The calculation formula is:

[0129] (5)

[0130] in, Here, W1 represents the first weight matrix, W2 represents the second weight matrix, b1 represents the bias vector, b2 represents the scalar bias, tanh is the activation function, and h is the learnable quantity. t The output of the LSTM model at time step t.

[0131] e t It is a scalar used to explain the importance of the hidden state at time step t;

[0132] After calculating the attention weights, the weights are normalized using the softmax function. The formula for calculating the attention weights at time step t is as follows:

[0133] (6)

[0134] Where T represents the total number of time steps in the sequence, i.e., all e t Quantity;

[0135] Next, the normalized attention weights are applied to the LSTM output to obtain a weighted summation context vector, calculated as follows:

[0136] (7)

[0137] Finally, the context vector is input into the fully connected layer, and the corresponding action category prediction result is output.

[0138] Building upon the above, to further improve the model's convergence speed and generalization ability, a batch normalization layer and a dropout mechanism are introduced after the fully connected layer. Batch normalization is applied to the hidden state of the last time step of the LSTM output to normalize the hidden vector. Given a mini-batch of input features... x is the output of the LSTM hidden layer. The coordinates, velocity, and acceleration are input into the LSTM and processed to obtain a feature representation. It is a high-dimensional feature vector that integrates coordinate, velocity, and acceleration information, rather than the original coordinates or kinematic quantities themselves. First, the mean and variance need to be calculated using the following formulas:

[0139] (8)

[0140] (9)

[0141] in, The mean, Let Variance be the variance.

[0142] Next, normalization is performed using the mean obtained from formula (8) and the variance obtained from formula (9). The calculation formula is as follows:

[0143] (10)

[0144] in, It is a very small constant used to avoid division by zero errors;

[0145] Finally, a linear transformation is performed, including scaling and translation, calculated using the following formula:

[0146] (11)

[0147] in, and These are learnable parameters. Indicates scalar scaling. This represents the translation vector, used to restore the network's expressive power;

[0148] Through this process, the network input is standardized to a distribution with a mean of 0 and a variance of 1;

[0149] This model adds a Dropout layer after batch normalization and before the fully connected layer. The calculation formula is as follows:

[0150] (12)

[0151] Where x is the batch-normalized feature vector, and p=0.5 is the discard probability;

[0152] A multi-task loss combination was designed, consisting of three parts: cross-entropy loss (CE_Loss), mean squared error loss (MSE_Loss), and temporal smoothness loss (Tempt_Loss). The cross-entropy loss measures the model's accuracy in action classification, and its calculation formula is as follows:

[0153] (13)

[0154] Where N is the number of samples and C is the number of categories. For the one-hot encoding of sample i in class k, Predict the probability that sample i belongs to class k for the model;

[0155] The mean squared error loss is used to compare the numerical error between the model's predicted key points and the actual key points. The calculation formula is as follows:

[0156] (14)

[0157] Where N is the number of samples, For the j-th predicted value of the i-th sample, This represents the j-th true value of the i-th sample.

[0158] The temporal smoothness loss is used to constrain the model's prediction output over time frames, ensuring that the prediction results change continuously and smoothly over time, avoiding sudden changes or jumps in the results. The calculation formula is as follows:

[0159] (15)

[0160] Where T is the number of time frames. Let be the class probability vector output by the model in frame t;

[0161] After calculating the above three losses, we continue to calculate the overall loss function, which is calculated using the following formula:

[0162] (16)

[0163] in, =1, =0.5, =0.1.

[0164] The other methods and steps are the same as in Specific Implementation Method 5.

[0165] Specific implementation method seven: Combination Figures 1 to 5 This implementation differs from Implementation Method Six in that, in step L5, the action category label predicted by the LSTM model is compared with the label obtained based on the rule-based determination of skeletal keypoint position relationships. When the results are consistent, the category label is directly adopted as the final recognition result; when the results are inconsistent, the result based on the rule-based determination is given priority to correct the model output, achieving a secondary confirmation of the action recognition result, thereby effectively improving the accuracy and confidence of the recognition. Other methods and steps are the same as in Implementation Method Six.

[0166] Specific implementation method eight: Combination Figures 1 to 5 This embodiment differs from specific embodiment seven in that the rules based on the positional relationships of skeletal key points in step L5 include:

[0167] (1) Head-touching judgment: When the LSTM model initially predicts the action category as "normal", the distance between the hand bone point and the nose bone point is further calculated. If the distance is less than 25 pixels, the action is judged as "touching the head".

[0168] (2) Waist palpation judgment: Calculate the waist position. Based on the average coordinates of the key points on the left and right hips, determine the center horizontal coordinate and initial vertical coordinate of the waist; then offset upwards by 15 pixels to obtain the actual vertical coordinate of the waist. The calculation formula is:

[0169] (17)

[0170] (18)

[0171] Among them, X L Y L X R X R , representing the coordinates of the left hip and right hip respectively;

[0172] After obtaining the waist coordinates, the following three rules are used for further judgment: the distance between the hand and the waist is less than 25 pixels; the angle formed by the skeletal points of the shoulder, elbow, and hand is less than 110 degrees; and the difference in vertical coordinates between the hand and the waist is less than 30 pixels. If all three conditions are met, the action is judged as "touching the waist".

[0173] (3) Fall detection: A person is considered to be in a fall state when at least three of the following conditions are met: the angle between the line connecting the shoulder to the hip and the vertical direction is greater than 70 degrees; the height difference between the head and ankle key points is less than 0.5 relative to the height difference between the hip and ankle key points; the aspect ratio of the detection box is less than 0.6; the angle between the line connecting the knee and ankle of at least one leg is less than 30 degrees or greater than 150 degrees, and there are fewer than 9 visible skeletal key points; the movement speed of the head, hip, or knee is >1.0 and the acceleration is >1.2.

[0174] (4) Squatting judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the hip, knee and ankle. If it is less than 100 degrees, the action is judged as "squatting".

[0175] (5) Bending judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the shoulder, hip and knee. If it is less than 100 degrees, the action is judged as "bending". Other methods and steps are the same as in Specific Implementation Method 7.

[0176] Specific Implementation Method Nine: Combining Figures 1 to 5 This embodiment differs from Specific Embodiment Eight in that the specific principle of fusing the action category label output by the improved LSTM model with the action category label determined based on the positional relationship of key human skeletal points in step L5 is as follows: When the action category label initially output by the improved LSTM model is "normal," this label is directly used as the final recognition result; when the action category label initially output by the improved LSTM model is "abnormal," further action category determination is performed based on the positional relationship of key human skeletal points. If the two determination results are inconsistent, the determination result based on the positional relationship of key human skeletal points is used as the final recognition result. Other methods and steps are the same as in Specific Embodiment Eight.

[0177] As described in Specific Implementation Methods One through Nine, in practical applications, this application can acquire RGB video data using a Hikvision infrared network camera and construct a three-layer LSTM network under a configured PyTorch framework for feature extraction and temporal modeling. Each layer contains 128 hidden units, and the action recognition model used can be implemented using existing technologies. First, the input features are normalized using the sigmoid activation function. The model training uses the cross-entropy loss function, and a batch normalization layer and a dropout layer are added after the LSTM output, with the dropout ratio set to 0.5. The optimizer is Adam, with an initial learning rate of 0.0005, and a StepLR learning rate scheduling strategy is adopted, reducing the learning rate by a decay factor of 0.5 every 10 training epochs.

[0178] This application visualizes the loss function values ​​of the training and testing parts in the LSTM model training results. After 20 rounds of training, both the training and testing parts begin to converge and gradually stabilize at around 0.89. While this result has limited accuracy as a preliminary judgment, its accuracy can be further improved when combined with judgments based on keypoint location information.

[0179] To improve the interpretability of abnormal action recognition results, the motion trajectory of skeletal points is graphically displayed.

[0180] During the preprocessing of the sample set, firstly, the video is segmented into multiple original images according to the video frame rate (FPS), where FPS = 30 f / s, and f / s represents frames per second; secondly, based on the original image data, the coordinate information of key points of the human skeleton is extracted using a pose estimation model, and the velocity and acceleration of each joint are calculated based on the key point coordinate sequence; thirdly, action annotation is performed on each frame to obtain corresponding action label information, where the action label information is the action category corresponding to that frame. There are six action categories in the annotated video data: normal, touching the head, touching the waist, bending over, squatting, and falling.

[0181] Each frame of the original image contains a single person instance. The COCO 17 keypoint annotation format is used as the inference format for pose estimation: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. (See details below.) Figure 2 .

[0182] (a) Implementation of Euclidean distance between skeletal points in each frame of the original image: The two-dimensional coordinates of 17 skeletal points for each person are obtained through the MMPose model and normalized to obtain the normalized two-dimensional coordinate values ​​of each skeletal point. Using the normalized two-dimensional coordinates of 17 skeletal points for each person, the Euclidean distance between the skeletal points is calculated. ; , .

[0183] (II) Implementation of the angles between skeletal points in each frame of the original image: The two-dimensional coordinates of 17 skeletal points for each person are obtained through the MMPose model and then normalized. Let the two-dimensional coordinate value of the target key point be A. Its adjacent key point is B. The key point at the remote end is Vectors can be constructed:

[0184] (18)

[0185] (19)

[0186] Further calculate the angle θ between the skeletal points;

[0187] (20)

[0188] (III) Implementation of 2D vector velocity and acceleration of each skeletal point in the original image of each frame: The 2D coordinates of 17 skeletal points of the human body are obtained through the MMPose model and normalized. Then, the skeletal point data of the current frame is stored in the historical sequence, ensuring that a maximum of three frames of historical data are retained for subsequent calculations. When a frame is lost, the system automatically uses the keypoint data from the previous frame to complete the calculation. After obtaining three frames of historical keypoints, the center point position is first calculated based on the coordinates of the target bounding box. The calculation formula is:

[0189]

[0190]

[0191] Among them, X min Y min X max Y max Given the coordinates of the top-left and bottom-right corners of the human target's bounding box, and based on historical center point data from three consecutive frames, the target's velocity and acceleration are further calculated using the following formula:

[0192]

[0193]

[0194] Where f is the video frame rate, and X t and X t-1These are the coordinate vectors of the center point of the current frame and the previous frame, respectively.

[0195] Example 1:

[0196] The following describes the process of using this application to perform motion recognition on personnel wearing protective clothing in a laboratory setting:

[0197] Hikvision infrared network cameras were deployed in a laboratory environment and connected to the software platform of this system to collect real-time data from laboratory personnel wearing protective clothing. Typical human movements were categorized and numbered as follows: normal (number 0), touching head (number 1), touching waist (number 2), bending over (number 3), squatting (number 4), and falling (number 5).

[0198] Based on real-time data acquisition, a database of abnormal actions in a laboratory setting was constructed. This database includes: complete video recordings of actions performed while wearing protective suits, extracted image files of six typical actions (including .jpg image data and corresponding .xml annotation files), and corresponding action feature data and tag information.

[0199] After data collection and labeling, the dataset is divided into training and validation sets. The system uses an LSTM model to train the model, optimizes the loss function, and iteratively updates the network parameters until the model converges on the validation set.

[0200] After model training is complete, the model is deployed to this system platform to infer and judge the behavior of laboratory personnel in real-time image streams. First, an LSTM model is used to initially determine the action category. If the model outputs an action category of "normal," the system will further refine the judgment based on rules:

[0201] (1) Head-touching judgment: Calculate the distance between the hand bone point and the nose bone point. If the distance is less than 25, the action is judged as head-touching.

[0202] (2) Waist-touching judgment: First, calculate the waist position. Determine the center coordinate of the waist by averaging the left and right hips, then subtract 15 to get the actual waist height. The body is judged to be in a "waist-touching" state when the following three conditions are met: After obtaining the waist coordinates, further judge according to the following three rules: if the distance between the hand and the waist is less than 25 pixels; the angle formed by the skeletal points of the shoulder, elbow and hand is less than 110 degrees; the longitudinal coordinate difference between the hand and the waist is less than 30 pixels. If the three conditions are met, the action is judged as "waist-touching".

[0203] (3) Fall detection: A person is considered to be in a fall state when at least three of the following conditions are met: the angle between the line connecting the shoulder to the hip and the vertical direction is greater than 70 degrees; the height difference between the head and ankle key points is less than 0.5 relative to the height difference between the hip and ankle key points; the aspect ratio of the detection box is less than 0.6; the angle between the line connecting at least one leg from the knee joint to the ankle joint is less than 30 degrees or greater than 150 degrees, and there are fewer than 9 visible skeletal key points; the movement speed of the head, hip, or knee is greater than 1.0 and the acceleration is greater than 1.2.

[0204] (4) Squatting judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the hip, knee and ankle. If it is less than 100 degrees, the action is judged as "squatting".

[0205] (5) Bending over judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the shoulder, hip and knee. If it is less than 100 degrees, the action is judged as "bending over".

[0206] When the inference results of the LSTM model conflict with the rule-based judgment results, the result based on the positional relationship of key skeletal points in the human body shall prevail to ensure the accuracy and reliability of the recognition. When the system recognizes abnormal actions (such as touching the head, touching the waist, bending over, squatting, or falling), it will synchronize the recognition results and corresponding images to the front-end interface in real time and trigger the alarm module to prompt relevant personnel to respond in a timely manner.

[0207] The present invention has been disclosed above with preferred embodiments, but it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed structure and technical content to create equivalent embodiments without departing from the scope of the present invention. However, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for identifying abnormal actions in a laboratory based on spatiotemporal sequences of skeletal points, characterized in that: This method is implemented through the following steps: L1. Acquire images during real-time monitoring, process each frame of image using a network model, identify laboratory personnel in the images, and output the bounding box information of the personnel. L2. The output personnel bounding box information is used as the input of the MMPose model to perform skeletal key point detection and extract the coordinate information of key skeletal points of the target human body for subsequent action recognition and judgment. L3. Based on the coordinate information of key human skeletal points output by the MMPose model, calculate the velocity and acceleration of the key points in the time series. L4. Input the coordinates, velocity and acceleration features of key human skeletal points into the trained improved LSTM model to make a preliminary judgment on human actions corresponding to multiple consecutive frames of images and output the corresponding action category label. L5. The action category labels output by the LSTM model are fused with the action category labels determined based on the positional relationship of key human skeletal points. The action recognition results are then confirmed a second time, and human action category labels are output. At the same time, optimization strategies are adopted to enhance the real-time performance of the overall detection.

2. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 1, characterized in that: The LSTM model is an improved LSTM model, and the improvement method is as follows: An additive attention mechanism was added to the original LSTM model. Specifically, this involves: first, encoding the coordinates, velocities, and acceleration sequences of the skeletal points using LSTM layers to obtain the hidden state output at each time step; then, using the additive attention mechanism to calculate the importance weights of the hidden states at each time step, with the score e at time step t being calculated. t The calculation formula is: in, Here, W1 represents the first weight matrix, W2 represents the second weight matrix, b1 represents the bias vector, b2 represents the scalar bias, tanh is the activation function, and h is the learnable quantity. t For the output of the LSTM model at time step t, e t It is a scalar used to explain the importance of the hidden state at time step t; After calculating the attention weights, the weights are normalized using the softmax function. The formula for calculating the attention weights at time step t is as follows: Where T represents the total number of time steps in the sequence, i.e., all e t Quantity; Next, the normalized attention weights are applied to the LSTM output to obtain a weighted summation context vector, calculated as follows: Finally, the context vector is input into the fully connected layer, and the corresponding action category prediction result is output.

3. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 2, characterized in that: Add a batch normalization layer and a random deactivation layer after the LSTM output and before the fully connected layer; Batch normalization is applied to the hidden state of the last time step of the LSTM output to normalize the hidden vector, given the input features. First, the mean needs to be calculated. and variance Next, using the obtained mean and the variance obtained Normalization is performed, and the calculation formula is as follows: in, It is a very small constant used to avoid division by zero errors; Finally, a linear transformation is performed, including scaling and translation, calculated using the following formula: in, and These are learnable parameters. Indicates scalar scaling. y represents the translation vector used to recover the network's expressive power. i This refers to the final output of batch normalization; The formula for calculating the random deactivation layer is: Where x is the batch-normalized feature vector and p is the discard probability.

4. A laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 2 or 3, characterized in that: The improved LSTM model is pre-trained, and the training process is as follows: S1. Collect video data of laboratory personnel's movements while wearing protective clothing, including six types of movements: normal state, touching head, touching waist, bending over, squatting and falling to the ground; S2. Divide the motion video of each sample into multiple original images according to the video frame rate (FPS), with each frame corresponding to one original image. Based on the original image data, the MMPose model is used to extract the coordinate information of key human skeletal points, and the velocity and acceleration are calculated based on the key point coordinate sequence. Each frame of the image is annotated with actions to obtain corresponding action label information, wherein the action label information is the action category corresponding to that frame; S3. Take the coordinates, velocity, and acceleration sequences of the key human skeleton points corresponding to each frame of the original image as input data in the training set. The velocity of the first frame is set to 0, and the acceleration of the first and second frames is set to 0. Take the action category label corresponding to the input data as the ground value, train the LSTM model, determine the model's weight parameters, and complete the training.

5. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 4, characterized in that: The loss function used when training the improved LSTM model is as follows: Where CE_Loss is the cross-entropy loss; MSE_Loss is the mean squared error loss; Tempt_Loss is the time smoothness loss; α, β and γ are all weighting coefficients; The cross-entropy loss is used to measure the accuracy of the model in action classification, and its calculation formula is as follows: Where N is the number of samples and C is the number of categories. For the one-hot encoding of sample i in class k, Predict the probability that sample i belongs to class k for the model; The mean squared error loss is used to compare the numerical error between the model's predicted key points and the actual key points. The calculation formula is as follows: Where N is the number of samples, For the j-th predicted value of the i-th sample, This represents the j-th true value of the i-th sample. The temporal smoothness loss is used to constrain the model's prediction output over time frames, ensuring that the prediction results change continuously and smoothly over time, avoiding sudden changes or jumps in the results. The calculation formula is as follows: Where T is the number of time frames. Let be the class probability vector output by the model in frame t.

6. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 4, characterized in that: In step S2, action annotation is performed on each frame of the image. Specifically, the COCO17 keypoint annotation format is selected as the inference format for pose estimation.

7. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 4, characterized in that: In step S2, the MMPose model is used to extract the coordinate information of key human skeletal points. The MMPose model used is an improved MMPose model. The specific improvement includes replacing the original HRNet backbone network in the MMPose model with the EfficientNet backbone network.

8. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 4, characterized in that: The specific method for calculating velocity and acceleration based on the key point coordinate sequence in step S2 is as follows: The MMPose model was used to predict the 2D coordinates of 17 human skeletal points, which were then normalized. The skeletal point data of the current frame was then stored in the historical sequence, ensuring that a maximum of three frames of historical data were retained for subsequent calculations. After obtaining the three historical keypoints, the center point position was first calculated based on the coordinates of the target bounding box. The calculation formula is as follows: Among them, X min Y min X max Y max Given the coordinates of the top-left and bottom-right corners of the human target's bounding box, and based on historical center point data from three consecutive frames, the target's velocity and acceleration are further calculated using the following formula: Where f is the video frame rate. and These are the coordinate vectors of the center point of the current frame and the previous frame, respectively.

9. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 1, characterized in that: The rules based on the positional relationships of skeletal keypoints in step L5 include: (1) Head-touching judgment: When the LSTM model initially predicts the action category as "normal", the distance between the hand bone point and the nose bone point is further calculated. If the distance is less than Z1 pixels, the action is judged as "head-touching". (2) Waist palpation judgment: Calculate the waist position. Based on the average coordinates of the key points on the left and right hips, determine the center horizontal coordinate and initial vertical coordinate of the waist. Then, shift upwards by Z2 pixels to obtain the actual vertical coordinate of the waist. The calculation formula is as follows: Among them, X L Y L X R X R , representing the coordinates of the left hip and right hip respectively; After obtaining the waist coordinates, the following three rules are used to make a judgment: if the distance between the hand and the waist is less than Z3 pixels; the angle formed by the skeletal points of the shoulder, elbow and hand is less than 110 degrees; and the difference in the vertical coordinates between the hand and the waist is less than Z4 pixels, the action is judged as "touching the waist". (3) Fall detection: A person is considered to be in a fall state when at least three of the following conditions are met: the angle between the line connecting the shoulder to the hip and the vertical direction is greater than 70 degrees; the height difference between the head and ankle key points is less than 0.5 relative to the height difference between the hip and ankle key points; the aspect ratio of the detection box is less than 0.6; the angle between the line connecting the knee and ankle of at least one leg is less than 30 degrees or greater than 150 degrees, and there are fewer than 9 visible skeletal key points; the movement speed of the head, hip, or knee is >1.0 and the acceleration is >1.

2. (4) Squatting judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the hip, knee and ankle. If it is less than 100 degrees, the action is judged as "squatting". (5) Bending judgment: Calculate the vertical distance between the nose and the ankle and compare it with the height difference between the hip and the ankle. If the ratio of the height difference between the nose and the ankle to the height difference between the hip and the ankle is less than 0.6, then further calculate the angle between the shoulder, hip and knee. If it is less than 100 degrees, the action is judged as "bending".

10. The laboratory abnormal action recognition method based on spatiotemporal sequence of skeletal points as described in claim 9, characterized in that: The specific principle of fusing the action category label output by the improved LSTM model with the action category label determined based on the positional relationship of key human skeletal points in step L5 is as follows: when the action category label initially output by the improved LSTM model is "normal", this label is directly used as the final recognition result; when the action category label initially output by the improved LSTM model is "abnormal", the action category is further determined based on the positional relationship of key human skeletal points. If the two determination results are inconsistent, the determination result based on the positional relationship of key human skeletal points is used as the final recognition result. When an action does not belong to any of the conditions of the rule determination, the result predicted by LSTM is used.

Images