An ICU patient body movement video recognition method based on human body skeleton key point detection

Abstract

This invention relates to a method for recognizing body movements in ICU patient videos based on human skeletal keypoint detection, overcoming the shortcomings of existing technologies in detecting human body movements in videos. The invention includes the following steps: dataset acquisition and preprocessing; construction of a human skeletal keypoint recognition model; training of the human skeletal keypoint recognition model; acquisition of the video stream data to be recognized; human keypoint detection and keypoint coordinate information storage; human atomic motion recognition; and body movement quantization classification based on CPOT scoring. This invention introduces a CA attention module into the backbone network, enhancing the model's ability to recognize keypoint location information and improving the accuracy of keypoint detection. Simultaneously, by introducing depthwise separable convolution, the number of model parameters is effectively reduced, detection speed is improved, and the problems of poor convergence and keypoint loss in the original algorithm on a self-made ICU patient dataset are solved.

Description

Technical Field

[0001] This invention relates to the field of video image recognition technology, specifically a method for recognizing the body movements of ICU patients based on the detection of key points in the human skeleton. Background Technology

[0002] In the ICU, monitoring patient movement is crucial. According to the Critical Care Pain Observation Scale (CPOT) published in the Chinese Journal of Critical Care Medicine, movement is categorized into three types: no movement, protective movement, and agitated movement. No movement indicates immobility but not necessarily the absence of pain; protective movement involves slow or cautious movements aimed at seeking attention; and agitated movement is characterized by violent limb swinging. ICU doctors and nurses currently primarily assess ICU patients' movement status by observing head movements, shoulder shrugging, and attempts to break free from restraints, allowing for timely medical intervention.

[0003] Traditional methods of motion monitoring in the ICU have significant drawbacks. Manual monitoring is labor-intensive, error-prone, and may miss high-risk movements within a short period. In many local hospitals, manual monitoring in ICUs is limited by manpower and human fatigue. Statistics show that approximately 25% of ICU patients experience undetected body movements, including vigorous movements caused by pain or discomfort. While ICU patients may be in critical condition, even in relatively stable situations, undetected body movements can lead to self-harm or dislodgement of medical equipment. Accurate motion monitoring is particularly crucial for patients experiencing discomfort, as ICU patients may feel unwell due to illness, medication, or interventions, but they may struggle to articulate their needs. Therefore, timely detection of their body movements is critical for assessing the degree of pain, discomfort, or potential self-harm and for taking appropriate action.

[0004] Existing automated motion detection technologies, such as wearable sensors, bed exit alarms, and video surveillance systems, while offering some assistance, still suffer from false alarms and missed alarms, as well as the risk of hardware failure, posing challenges to accuracy and reliability.

[0005] The YOLO algorithm, based on deep learning, is a promising technology. YOLO is a real-time object detection system that is faster and more accurate than traditional methods, and can identify the human body and its skeletal key points in a single frame. However, existing YOLO algorithms have not been sufficiently studied and improved for monitoring postural activity in ICU patients.

[0006] Therefore, automatic motion detection technology based on key points of the human skeleton is a technical problem that urgently needs to be solved. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies in detecting human body movements in videos, and to provide a method for recognizing body movements in ICU patients based on the detection of key points in the human skeleton to solve the above problems.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A method for recognizing body movement in ICU patients based on human skeletal key point detection includes the following steps:

[0010] Dataset acquisition and preprocessing: Acquire ICU patient video datasets, convert them to JPG format, and perform preprocessing;

[0011] Construction of a human skeleton key point recognition model: A human skeleton key point recognition model was built based on YOLOv7-Pose;

[0012] Training of the human skeleton key point recognition model: The human skeleton key point recognition model was trained using a preprocessed ICU patient video dataset;

[0013] Acquisition of video stream data to be identified: Acquire video data of ICU patients to be identified and perform preprocessing;

[0014] Human body key point detection and key point coordinate information storage: The preprocessed ICU patient video data is input into the trained human skeleton key point recognition model for detection, and the detected key point coordinate information is stored in the database to obtain a key point information database.

[0015] Human atomic motion recognition: The 17 skeletal key points of the human body are divided into head, upper limbs and lower limbs. Atomic motion recognition is performed based on the changes in the position information of key points at different times in the key point information database.

[0016] Quantitative classification of body movement based on CPOT score: The CPOT score table is used to give the amplitude score of movement at the amplitude threshold, and the body movement of the patient is classified by summing all scores using the quantitative body movement score formula.

[0017] The acquisition and preprocessing of the dataset includes the following steps:

[0018] Acquire video data of ICU patients and use FFmpeg technology to extract frames from them, saving the extracted frames in JPG format;

[0019] Perform data augmentation to increase the amount of data and enhance the model's generalization ability;

[0020] First, the OpenCV image processing library was used to flip the images horizontally, vertically, and horizontally and vertically respectively. Then, a Python script was used to add Gaussian noise to all the photos in the dataset and perform shifting, scaling, and random cropping.

[0021] The augmented images were divided into training and validation sets in an 8:2 ratio, and the datasets were labeled.

[0022] In this context, annotation refers to using the Labelme tool to annotate the key points of the human body and human skeleton in the experimental dataset. The format of the annotated dataset is then converted to YOLO format using a Python script. The training set is used to train the YOLOv7-Pose network model and extract features. The validation set is used to adjust the hyperparameters of the YOLOv7-Pose network model.

[0023] The construction of the human skeleton key point recognition model includes the following steps:

[0024] The human skeleton key point recognition model is based on YOLOv7-Pose and consists of four parts: Input, Backbone, Neck layer and Head layer.

[0025] The input layer is configured to use Mosaic data augmentation, adaptive initial anchor box calculation, and image scaling and cropping to preprocess the image before outputting it to the backbone network.

[0026] The backbone network is set up to extract features from the processed input image. The extracted feature maps are then processed by the Neck layer to perform multi-scale feature fusion to obtain four different sizes of feature maps. Finally, the fused features are sent to the Head layer.

[0027] The Head layer outputs the results after detection. The Head layer uses CIoU Loss to calculate localization and confidence loss for bounding box supervision. The optimized OKS Loss is used to calculate the loss of human skeleton keypoints and supervise the skeleton keypoints. The concept of IOU loss is extended from bounding boxes to keypoints. In keypoints, OKS Loss is treated as IOU, and different error penalties are set for different keypoints.

[0028] A coordinate attention mechanism (CA) is introduced into the backbone network, and CA is added after the 3rd, 4th, 5th and 6th convolutions respectively.

[0029] Lightweight improvements to the ELAN modules in the backbone network are made by adopting the depthwise separable convolution lightweighting method proposed in MobileNet to replace the ordinary convolutions in the ELAN modules of the backbone network. Specifically, the four 3×3 convolutions used by the five ELAN modules in the backbone network are replaced with depthwise separable convolutions.

[0030] The training of the human skeletal key point recognition model includes the following steps:

[0031] Set the training parameters for the YOLOv7-Pose network model, including the number of iterations, batch size, and initial learning rate. The number of iterations (epochs) is 300, the batch size is 32, and the initial learning rate (lr0) is set to 0.001. Cosine annealing is used to reduce the learning rate.

[0032] Set the data augmentation parameters for the YOLOv7-Pose network model. The data augmentation parameters include hue (hsv_h), saturation (hsv_s), brightness (hsv_v), rotation angle (degrees), vertical flip probability (flipud), and horizontal flip probability (fliplr). The values ​​are: hsv_h = 0.015, hsv_s = 0.7, hsv_v = 0.4, degrees = 0, flipud = 0, and fliplr = 0.5.

[0033] Input the preprocessed ICU patient dataset into the human skeletal key point recognition model:

[0034] The input module uses Mosaic data augmentation to randomly scale, crop, and arrange the images.

[0035] The training images output from the Input module are fed into the improved Backbone module. The Backbone module, which incorporates an attention mechanism and a lightweight ELAN module, extracts the feature information of the training images and outputs the feature information of the training images.

[0036] The feature information at different scales extracted by the backbone module is fed into the Neck layer to achieve the fusion of shallow graphic features and deep semantic features;

[0037] The feature maps of different scales output from the Neck layer are fed into the Head layer of the detection head, which has four detection heads of different scales.

[0038] Finally, there are two decoupled detection heads for predicting detection boxes and key points;

[0039] During the prediction process, the IOU loss is extended from the predicted bounding box to the skeletal keypoints. The object keypoint similarity is used to evaluate the skeletal keypoints. After non-maximum suppression (NMS), the predicted results of the human skeletal keypoints are finally output.

[0040] The detection of key human body points and the storage of key point coordinate information include the following steps:

[0041] Obtain the video stream data to be identified;

[0042] This paper describes a method for detecting key points in the human skeleton based on the YOLOv7-Pose algorithm model. For the input video stream data, it performs feature extraction, feature fusion, and key point detection.

[0043] If no human target is detected, human keypoint detection will not be performed, and None will be output.

[0044] If a human target is detected, the key points and bounding box of the human body will be detected directly.

[0045] After detecting key points of the human skeleton, the system outputs the human target bounding box data and the two-dimensional coordinate information of the key points of the human skeleton, and stores the key point coordinate information of different frames into a database to form a key point coordinate information database.

[0046] The human atomic motion recognition includes the following steps:

[0047] The key points of the human skeleton are divided into three categories: head, upper limbs, and lower limbs, as follows:

[0048] The key points of the head correspond to the nose, left eye, right eye, left ear, and right ear, totaling 5 key points;

[0049] The key points of the upper limbs correspond to the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip, totaling 8 key points;

[0050] The key points of the lower limbs correspond to the left knee, right knee, left ankle, and right ankle, totaling 4 key points;

[0051] Human atomic movements are defined, with a total of 9 types: small-amplitude head movements, upper limb movements, and lower limb movements; medium-amplitude head movements, upper limb movements, and lower limb movements; and large-amplitude head movements, upper limb movements, and lower limb movements.

[0052] Obtain keypoint coordinate information from the database, select a stable frame as the reference frame, and take the Euclidean distance between the keypoint coordinates of the reference frame and the origin of the bounding box as the reference distance. Calculate the relative change between the Euclidean distance of the keypoint coordinates between the current frame and the reference frame and the reference distance as the motion amplitude threshold d. i To perform atomic action recognition, where the threshold d iThe calculation formula is as follows, where f i For the current frame, f r For the reference frame, d(f) r () is the reference distance.

[0053]

[0054] Atomic action recognition includes: no atomic action corresponding to a threshold of less than 0.5%, small-amplitude atomic action corresponding to a threshold of 0.5% to 2.5%, medium-amplitude atomic action corresponding to a threshold of 2.5% to 5%, and large-amplitude atomic action corresponding to a threshold of more than 5%.

[0055] After the atomic action is identified, the next step is to classify the body movement based on the CPOT score.

[0056] The motion recognition and classification based on CPOT scores includes the following steps:

[0057] Based on the CPOT rating tool, bodily movement is defined into four categories: no bodily movement, protective bodily movement, bodily discomfort, and restlessness.

[0058] The identified atomic actions are quantitatively scored: no atomic action is scored as 0 points, small-amplitude atomic action is scored as 1 point, medium-amplitude atomic action is scored as 2 points, and large-amplitude atomic action is scored as 3 points.

[0059] The scores of the above atomic movements are summed up using the body movement quantification calculation formula to obtain a total score, and the patient's body movement status is determined.

[0060] If the recognition result is less than the set threshold, the system will not output a body movement alarm signal.

[0061] If the body movement exceeds the set threshold, the system will output a body movement alarm signal and the judgment criteria. The alarm signal includes protective body movement, body movement discomfort, and restlessness. The judgment criteria include small amplitude, medium amplitude, and large amplitude atomic movement types.

[0062] The judgment criteria are based on large-amplitude head movements and medium-amplitude upper limb movements. The formula for quantifying body movement is as follows: S total It is the total score of physical activity; S h S u S l These are different threshold values ​​d for the head, upper limbs, and lower limbs, respectively. i Physical activity score; d i The corresponding thresholds are d1, d2, d3, and d4, respectively.

[0063] S total =S h (d i )+Su (d i )+S l (d i ).

[0064] Beneficial effects

[0065] This invention presents a method for recognizing body movement in ICU patients based on human skeletal keypoint detection. Compared with existing technologies, this method significantly improves the detection accuracy and speed of the YOLOv7-Pose model through an improvement. First, this invention introduces a CA attention module into the backbone network, enhancing the model's ability to recognize keypoint location information and improving keypoint detection accuracy. Simultaneously, by introducing depthwise separable convolution, the number of model parameters is effectively reduced, improving detection speed and solving the problems of poor convergence and keypoint loss in the original algorithm on a self-made ICU patient dataset.

[0066] This invention provides a detailed classification of body movement behavior in ICU patients, including small-amplitude, medium-amplitude, and large-amplitude movements, specifically involving head, upper limb, and lower limb movements. This classification method allows for more precise capture of body movements in different parts of the patient's body, providing crucial information for further diagnosis and treatment. Furthermore, by summing quantitative scores of patient body movements, body movements are categorized into no movement, protective movement, movement-related discomfort, and restlessness, further refining the criteria for body movement identification and classification.

[0067] Experimental results show that the improved model achieved an average accuracy (AP) of 69.1% for keypoint detection, which is 1.5 percentage points higher than the original model. At the same time, the detection speed is 1.3 times faster than the original model, which can meet the requirements of ICU patient movement recognition and classification, and greatly improve the efficiency and accuracy of clinical monitoring. Attached Figure Description

[0068] Figure 1 This is a sequence diagram of the method of the present invention;

[0069] Figure 2 This is a logic flowchart of the present invention;

[0070] Figure 3 This is a schematic diagram of the ICU dataset involved in this invention;

[0071] Figure 4 This is a diagram of the YOLOv7pose network structure involved in this invention;

[0072] Figure 5 This is a network diagram of the CA attention mechanism involved in this invention;

[0073] Figure 6 This is a structural diagram of the improved ELAN module involved in this invention;

[0074] Figure 7 This is a diagram showing the results of key point recognition of the human skeleton involved in this invention.

[0075] Figure 8 This is a diagram showing the body movement recognition results involved in this invention. Detailed Implementation

[0076] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0077] like Figure 1 and Figure 2 As shown, the present invention provides a method for recognizing body movement in ICU patients based on the detection of key points in the human skeleton, comprising the following steps:

[0078] The first step is to acquire and preprocess the dataset: acquire the video dataset of ICU patients, convert it to JPG format, and perform preprocessing.

[0079] (1) Acquire video data of ICU patients and perform frame extraction processing using FFmpeg technology. Save the extracted frames as JPG format. Part of the dataset is shown below. Figure 3 As shown.

[0080] (2) Perform data augmentation to increase the amount of data and enhance the generalization ability of the model;

[0081] First, the OpenCV image processing library was used to flip the images horizontally, vertically, and horizontally and vertically respectively. Then, a Python script was used to add Gaussian noise to all the photos in the dataset and perform shifting, scaling, and random cropping.

[0082] (3) Divide the augmented images into training and validation sets in an 8:2 ratio and label the datasets;

[0083] In this context, annotation refers to using the Labelme tool to annotate the key points of the human body and human skeleton in the experimental dataset. The format of the annotated dataset is then converted to YOLO format using a Python script. The training set is used to directly participate in the training of the YOLOv7-Pose network model for feature extraction. The validation set is used to adjust the hyperparameters of the YOLOv7-Pose network model.

[0084] The second step is to build a human skeleton key point recognition model: a human skeleton key point recognition model is built based on YOLOv7-Pose.

[0085] To detect human skeletal keypoints from video stream data, this invention uses the YOLOv7-Pose model to construct a human skeletal keypoint recognition model. YOLOv7 is a powerful and flexible object detection and segmentation model, offering higher performance and greater flexibility compared to previous YOLO versions. The YOLOv7-Pose model is a deep convolutional network based on YOLOv7, possessing high accuracy and adding human pose estimation functionality while performing object detection. Its network structure mainly consists of three parts: Backbone, which uses an improved backbone network architecture, combining the advantages of ResNet residual modules and CSPNet cross-stage local networks, enhancing the efficiency and expressive power of feature extraction. It can more effectively capture multi-scale information while reducing computational cost and parameter count, improving the overall performance of the model. Neck, which employs multi-scale fusion technology to fuse and process features from different scales from the Backbone, enhancing the model's robustness and detection capabilities. The Head consists of an object detection head and a pose estimation head. The detection head contains a series of convolutional and deconvolutional layers to generate detection results, ultimately achieving the task of human skeleton keypoint detection and classification.

[0086] The specific steps are as follows:

[0087] (1) As Figure 4 As shown, a human skeleton key point recognition model is set up based on YOLOv7-Pose, which consists of four parts: Input, Backbone, Neck layer and Head layer.

[0088] (2) The input end is set to use Mosaic data augmentation, adaptive initial anchor box calculation, and image scaling and cropping to preprocess the image and output it to the backbone network.

[0089] (3) The backbone network is set to extract features from the processed input image. The extracted feature maps are processed by the Neck layer to perform multi-scale feature fusion to obtain four different sizes of feature maps. Finally, the fused features are sent to the Head layer.

[0090] (4) Set the output of the Head layer after detection. The Head layer uses CIoU Loss to calculate the localization and confidence loss for bounding box supervision. The optimized OKS Loss is used to calculate the loss of human skeleton key points and supervise the skeleton key points. The concept of IOU loss is extended from the bounding box to the key points. In the key points, OKS Loss is used as IOU, and different error penalties are set for different key points.

[0091] (5) Figure 5 As shown, a coordinate attention mechanism (CA) is introduced into the backbone network, and CA is added after the 3rd, 4th, 5th and 6th convolutions respectively.

[0092] Introducing the coordinate attention mechanism (CA) into the backbone network improves image feature extraction capabilities and adaptability to complex environments by considering not only channel information but also orientation-related positional information. This alleviates the problem of difficulty in recognizing internal texture features of images, thereby improving the efficiency and accuracy of task processing.

[0093] (6) Figure 6 As shown, lightweight improvements are made to the ELAN modules in the backbone network. The depthwise separable convolution lightweight method proposed in MobileNet is adopted to replace the ordinary convolutions of the ELAN modules in the backbone network. Specifically, the four 3×3 convolutions used by the five ELAN modules in the backbone network are replaced with depthwise separable convolutions.

[0094] The ELAN module in the backbone network is lightweighted and improved. Addressing the issue of excessive parameters in the human skeletal keypoint recognition model, and mitigating the impact of redundant feature information on feature fusion during feature extraction, this approach aims to improve the network model's inaccuracy in locating skeletal keypoints and further enhance the YOLOv7-Pose model's performance in the ICU to meet detection speed requirements. The depthwise separable convolution lightweight method proposed in MobileNet replaces some ordinary convolutions in the ELAN module of the backbone. This effectively reduces the number of network parameters and computational cost without compromising accuracy, better meeting the detection speed requirements.

[0095] The third step is to train the human skeletal key point recognition model: the human skeletal key point recognition model is trained using the preprocessed ICU patient video dataset.

[0096] To detect human skeletal keypoints from video stream data, this invention uses the YOLOv7-Pose model to train a human skeletal keypoint dataset. This model leverages the efficient architecture of YOLOv7 to extract human skeletal keypoints through a convolutional neural network, thereby achieving keypoint recognition. Training the YOLOv7-Pose model faces certain technical challenges. First, it requires a large, accurately labeled dataset of human skeletal keypoints to ensure its generalization ability and accuracy. Second, the model needs improvement and optimization to adapt to the computational capabilities and requirements of different devices and scenarios. Furthermore, improving the model's robustness and accuracy under complex backgrounds and multi-pose interference is also a significant challenge.

[0097] (1) Set the training parameters of the YOLOv7-Pose network model. The training parameters include the number of iterations, batch size, and initial learning rate. The number of iterations is 300 epochs, the batch size is 32, and the initial learning rate lr0 of the model is set to 0.001. The cosine annealing algorithm is used to reduce the learning rate.

[0098] (2) Set the data augmentation parameters of the YOLOv7-Pose network model. The data augmentation parameters include hue hsv_h, saturation hsv_s, brightness hsv_v, rotation angle degrees, vertical flip probability flipud, and horizontal flip probability fliplr. Among them, hsv_h is 0.015, hsv_s is 0.7, hsv_v is 0.4, degrees is 0, flipud is 0, and fliplr is 0.5.

[0099] Input the preprocessed ICU patient dataset into the skeletal keypoint recognition model:

[0100] The input module uses Mosaic data augmentation to randomly scale, crop, and arrange the images.

[0101] The training images output from the Input module are fed into the improved Backbone module. The Backbone module, which incorporates an attention mechanism and a lightweight ELAN module, extracts the feature information of the training images and outputs the feature information of the training images.

[0102] The feature information at different scales extracted by the backbone module is fed into the Neck layer to achieve the fusion of shallow graphic features and deep semantic features;

[0103] The feature maps of different scales output from the Neck layer are fed into the Head layer of the detection head, which has four detection heads of different scales.

[0104] Finally, there are two decoupled detection heads for predicting detection boxes and key points;

[0105] During the prediction process, the IOU loss is extended from the predicted bounding box to the skeletal keypoints. The object keypoint similarity is used to evaluate the skeletal keypoints. After non-maximum suppression (NMS), the predicted results of the human skeletal keypoints are finally output.

[0106] The fourth step is to acquire the video stream data to be identified: acquire the video data of the ICU patients to be identified and perform preprocessing.

[0107] The fifth step is to detect key human body points and input key point coordinate information into the database: The preprocessed ICU patient video data is input into the trained human skeleton key point recognition model for detection, and the detected key point coordinate information is input into the database to obtain a key point information database.

[0108] (1) Obtain the video stream data to be identified.

[0109] (2) Human skeleton key point detection is performed based on the YOLOv7-Pose algorithm model. For the input video stream data, feature extraction, feature fusion and head key point detection are performed.

[0110] If no human target is detected, human keypoint detection will not be performed, and None will be output.

[0111] If a human target is detected, the model will directly detect the human key points and the human target bounding box.

[0112] (3) After detecting the key points of the human skeleton, output the human target box data information and the two-dimensional coordinate information of the key points of the human skeleton, and put the key point coordinate information of different frames into the database to form a key point coordinate information database.

[0113] Step 6, Human Atomic Motion Recognition: Divide the 17 skeletal key points of the human body into head, upper limbs and lower limbs, and perform atomic motion recognition based on the changes in the key point position information at different times in the key point information database.

[0114] (1) The key points of the human skeleton are divided into three categories: head, upper limbs and lower limbs, as follows:

[0115] The key points of the head correspond to the nose, left eye, right eye, left ear, and right ear, totaling 5 key points;

[0116] The key points of the upper limbs correspond to the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip, totaling 8 key points;

[0117] The key points of the lower limbs correspond to the left knee, right knee, left ankle, and right ankle, for a total of 4 key points.

[0118] (2) Define human atomic movements. A total of 9 types of human atomic movements are defined, namely small-amplitude head movements, upper limb movements and lower limb movements, medium-amplitude head movements, upper limb movements and lower limb movements, and large-amplitude head movements, upper limb movements and lower limb movements.

[0119] (3) Obtain key point coordinate information from the database, take a stable frame as the reference frame, and take the Euclidean distance between the key point coordinates of the reference frame and the origin coordinates of the bounding box as the reference distance. Calculate the relative change between the Euclidean distance of the key point coordinates between the current frame and the reference frame and the reference distance as the motion amplitude threshold d. iTo perform atomic action recognition, where the threshold d i The calculation formula is as follows, where f i For the current frame, f r For the reference frame, d(f) r () is the reference distance;

[0120]

[0121] (64) Atomic action recognition includes: no atomic action corresponding to a threshold of less than 0.5%, small-amplitude atomic action corresponding to a threshold of 0.5% to 2.5%, medium-amplitude atomic action corresponding to a threshold of 2.5% to 5%, and large-amplitude atomic action corresponding to a threshold of more than 5%.

[0122] (5) After the atomic action is identified, the next step is to classify the body movement based on the CPOT score.

[0123] Step 7, Quantitative classification of body movement based on CPOT score: Use the corresponding CPOT score table to give the amplitude of movement score under a certain amplitude threshold, and use the quantitative body movement score formula to sum all scores to classify the patient's body movement.

[0124] (1) Based on the CPOT rating tool, body movement is defined as four categories: no body movement, protective body movement, body movement discomfort, and restlessness.

[0125] (2) The identified atomic actions are quantitatively scored. No atomic action is scored as 0 points, small atomic action is scored as 1 point, medium atomic action is scored as 2 points, and large atomic action is scored as 3 points.

[0126] (3) The scores of the above atomic movements are accumulated by the body movement quantification calculation formula to obtain the total score, and the patient's body movement status is judged;

[0127] If the recognition result is less than the set threshold, the system will not output a body movement alarm signal.

[0128] If the body movement exceeds the set threshold, the system will output a body movement alarm signal and the judgment criteria. The alarm signal includes protective body movement, body movement discomfort, and restlessness. The judgment criteria include small amplitude, medium amplitude, and large amplitude atomic movement types.

[0129] The judgment criteria are based on large-amplitude head movements and medium-amplitude upper limb movements. The formula for quantifying body movement is as follows: S total It is the total score of physical activity; S h S u S l These are different threshold values ​​d for the head, upper limbs, and lower limbs, respectively. i Physical activity score; d iThe corresponding thresholds are d1, d2, d3, and d4, respectively.

[0130] S total =S h (d i )+S u (d i )+S l (d i ).

[0131] like Figure 7 As shown, YOLOv7-Pose, based on two improvements—the CA coordinate attention mechanism and the lightweight ELAN module—can accurately identify key points of the human skeleton in video stream data images.

[0132] like Figure 8 As shown, the present invention can store the accurate coordinate information of human body key points into a skeletal key point information database after obtaining accurate human body key point coordinate information, and then perform frame-by-frame processing on the coordinate information of different frames in the key point information database to determine the patient's body movement state.

[0133] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for recognizing body movement in ICU patients based on human skeletal key point detection, characterized in that, Includes the following steps: 11) Data Acquisition and Preprocessing: Acquire ICU patient video datasets, convert them to JPG format, and perform preprocessing; 12) Construction of human skeleton key point recognition model: A human skeleton key point recognition model was constructed based on YOLOv7-Pose; The construction of the human skeleton key point recognition model includes the following steps: 121) Based on YOLOv7-Pose, a human skeleton key point recognition model is set up, which consists of four parts: Input, Backbone, Neck layer and Head layer. 122) The input terminal is configured to use Mosaic data augmentation, adaptive initial anchor box calculation, and image scaling and cropping to preprocess the image, and the output is sent to the backbone network. 123) The backbone network is set up to extract features from the processed input image. The extracted feature maps are processed by the Neck layer to perform multi-scale feature fusion to obtain four different sizes of feature maps. Finally, the fused features are sent to the Head layer. 124) The output of the Head layer after detection is set. The Head layer uses CIoU Loss to calculate the localization and confidence loss for bounding box supervision. The optimized OKS Loss is used to calculate the loss of human skeleton key points and supervise the skeleton key points. The concept of IOU loss is extended from bounding box to key points. In key points, OKS Loss is used as IOU, and different error penalties are set for different key points. 125) A coordinate attention mechanism (CA) is introduced into the backbone network, and CA is added after the 3rd, 4th, 5th and 6th convolutions respectively; 126) Lightweight improvement of ELAN modules in the backbone network: The depth-separable convolution lightweight method proposed in MobileNet is adopted to replace the ordinary convolutions of ELAN modules in the backbone network. That is, the four 3×3 convolutions used by the five ELAN modules in the backbone network are replaced with depth-separable convolutions. 13) Training of the human skeleton key point recognition model: The human skeleton key point recognition model was trained using the preprocessed ICU patient video dataset; 14) Acquisition of video stream data to be identified: Acquire video data of ICU patients to be identified and perform preprocessing; 15) Human body key point detection and key point coordinate information storage: The preprocessed ICU patient video data is input into the trained human skeleton key point recognition model for detection, and the detected key point coordinate information is stored in the database to obtain a key point information database. 16) Human Atomic Motion Recognition: The 17 skeletal key points of the human body are divided into head, upper limbs and lower limbs. Atomic motion recognition is performed based on the changes in the key point position information at different times in the key point information database. The human atomic motion recognition includes the following steps: 161) The key points of the human skeleton are divided into three categories: head, upper limbs, and lower limbs, as follows: The key points of the head correspond to the nose, left eye, right eye, left ear, and right ear, totaling 5 key points; The key points of the upper limbs correspond to the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip, totaling 8 key points; The key points of the lower limbs correspond to the left knee, right knee, left ankle, and right ankle, totaling 4 key points; 162) Define human atomic movements, a total of 9 types of human atomic movements are defined, namely small-amplitude head movements, upper limb movements and lower limb movements, medium-amplitude head movements, upper limb movements and lower limb movements, and large-amplitude head movements, upper limb movements and lower limb movements; 163) Obtain keypoint coordinate information from the database, take a stable frame as the reference frame, and take the Euclidean distance between the keypoint coordinates of the reference frame and the origin coordinates of the bounding box as the reference distance. Calculate the relative change between the Euclidean distance of the keypoint coordinates between the current frame and the reference frame and the reference distance as the motion amplitude threshold d. i To perform atomic action recognition, where the threshold d i The calculation formula is as follows, where f i For the current frame, f r For the reference frame, d(f) r () is the reference distance. ； 164) Atomic action recognition includes: no atomic action corresponds to a threshold of less than 0.5%, small-amplitude atomic action corresponds to a threshold of 0.5% to 2.5%, medium-amplitude atomic action corresponds to a threshold of 2.5% to 5%, and large-amplitude atomic action corresponds to a threshold of more than 5%. 165) After the atomic action is identified, the next step is to classify the body movement based on the CPOT score; 17) Quantitative classification of body movement based on CPOT score: The CPOT score table is used to give the amplitude score of movement under the amplitude threshold, and the body movement of the patient is classified by summing all scores using the quantitative body movement score formula.

2. The method for ICU patient motion video recognition based on human skeletal key point detection according to claim 1, characterized in that, The acquisition and preprocessing of the dataset includes the following steps: 21) Acquire video data of ICU patients and use FFmpeg technology to extract frames from them, saving the extracted frames as jpg format; 22) Perform data augmentation to increase the amount of data and enhance the model's generalization ability; First, the OpenCV image processing library was used to flip the images horizontally, vertically, and horizontally and vertically respectively. Then, a Python script was used to add Gaussian noise to all the photos in the dataset and perform shifting, scaling, and random cropping. 23) Divide the augmented images into training and validation sets in an 8:2 ratio, and label the datasets accordingly; In this context, annotation refers to using the Labelme tool to annotate the key points of the human body and human skeleton in the experimental dataset. The format of the annotated dataset is then converted to YOLO format using a Python script. The training set is used to train the YOLOv7-Pose network model and extract features. The validation set is used to adjust the hyperparameters of the YOLOv7-Pose network model.

3. The method for ICU patient motion video recognition based on human skeletal key point detection according to claim 1, characterized in that, The training of the human skeletal key point recognition model includes the following steps: 31) Set the training parameters of the YOLOv7-Pose network model. The training parameters include the number of iterations, batch size, and initial learning rate. The number of iterations is 300 epochs, the batch size is 32, and the initial learning rate lr0 is set to 0.

001. The cosine annealing algorithm is used to reduce the learning rate. 32) Set the data augmentation parameters for the YOLOv7-Pose network model. The data augmentation parameters include hue (hsv_h), saturation (hsv_s), brightness (hsv_v), rotation angle (degrees), vertical flip probability (flipud), and horizontal flip probability (fliplr). Among them, hsv_h is 0.015, hsv_s is 0.7, hsv_v is 0.4, degrees is 0, flipud is 0, and fliplr is 0.

5. Input the preprocessed ICU patient dataset into the human skeletal key point recognition model: The input module uses Mosaic data augmentation to randomly scale, crop, and arrange the images. The training images output from the Input module are fed into the improved Backbone module. The Backbone module, which incorporates an attention mechanism and a lightweight ELAN module, extracts the feature information of the training images and outputs the feature information of the training images. The feature information at different scales extracted by the backbone module is fed into the Neck layer to achieve the fusion of shallow graphic features and deep semantic features; The feature maps of different scales output from the Neck layer are fed into the Head layer of the detection head, which has four detection heads of different scales. Finally, there are two decoupled detection heads for predicting detection boxes and key points; During the prediction process, the IOU loss is extended from the predicted bounding box to the skeletal keypoints. The object keypoint similarity is used to evaluate the skeletal keypoints. After non-maximum suppression (NMS), the predicted results of the human skeletal keypoints are finally output.

4. The method for recognizing ICU patient body movement video based on human skeletal key point detection according to claim 1, characterized in that, The detection of key human body points and the storage of key point coordinate information include the following steps: 41) Obtain the video stream data to be identified; 42) Human skeleton keypoint detection based on YOLOv7-Pose algorithm model. For input video stream data, feature extraction, feature fusion, and head keypoint detection are performed. If no human target is detected, human keypoint detection will not be performed, and None will be output. If a human target is detected, the key points and bounding box of the human body will be detected directly. 43) After detecting key points of the human skeleton, output the human target bounding box data and the two-dimensional coordinate information of the human skeleton key points, and input the key point coordinate information of different frames into the database to form a key point coordinate information database.

5. The method for ICU patient motion video recognition based on human skeletal key point detection according to claim 1, characterized in that, The motion recognition and classification based on CPOT scores includes the following steps: 51) Based on the CPOT rating tool, bodily movement is defined as follows: bodily movement is divided into four categories: no bodily movement, protective bodily movement, bodily discomfort, and restlessness. 52) Perform body motion quantification scoring on the identified atomic actions. No atomic action body motion quantification score is 0 points, small-amplitude atomic action body motion quantification score is 1 point, medium-amplitude atomic action body motion quantification score is 2 points, and large-amplitude atomic action body motion quantification score is 3 points. 53) The scores of the above atomic movements are summed up using the body movement quantification calculation formula to obtain the total score, and the patient's body movement status is determined; If the recognition result is less than the set threshold, the system will not output a body movement alarm signal. If the body movement exceeds the set threshold, the system will output a body movement alarm signal and the judgment criteria. The alarm signal includes protective body movement, body movement discomfort, and restlessness. The judgment criteria include small amplitude, medium amplitude, and large amplitude atomic movement types. The judgment criteria are based on large-amplitude head movements and medium-amplitude upper limb movements. The formula for quantifying body movement is as follows: S total It is the total score of physical activity; S h S u S l These are different threshold values ​​d for the head, upper limbs, and lower limbs, respectively. i Physical activity score; d i The corresponding thresholds are d1, d2, d3, and d4, respectively. 。

Images