A method for monitoring patient's bed posture

Abstract

The present invention discloses a method for monitoring a patient's posture in bed, comprising the following steps: S1) collecting image information of human posture behavior through a depth camera; S2) extracting human skeleton information from the collected image information to obtain human action skeleton diagram information; S3) identifying the patient's posture in bed according to the human action skeleton diagram. The step S3 uses a classification feature algorithm to classify relevant image postures, and compares the classified posture type results with the national standard bed posture. The method for monitoring a patient's posture in bed provided by the present invention can accurately identify whether the patient's posture is correct, so that the behavior of postoperative or seriously ill patients resting in bed can be automatically standardized and corrected.

Description

Technical Field

[0001] The invention relates to a method for recognizing human body posture, and in particular to a method for monitoring the posture of a patient in bed. Background Art

[0002] The number of patients who are bedridden for a long time and cannot take care of themselves due to various factors is increasing day by day. According to relevant statistics, there are many patients in China who need rehabilitation nursing equipment. For paralyzed patients, they spend most of their time in bed, and they need to turn over frequently to prevent the bed from dragging. At present, there are nursing beds that can turn over, but they are either too simple in function, require the assistance of medical staff and are difficult to meet the requirements of turning over, or are expensive and difficult to popularize. In addition, patients with limited mobility who are bedridden for a long time will inevitably have certain psychological disorders, which are specifically manifested as sleep disorders, decreased mental activity, excessive thinking, depression, pessimism, self-blame, loss of appetite, physical decline and delusion, etc., which have a great impact on the development and recovery of their illness.

[0003] The field of human posture recognition and analysis mainly analyzes and understands individual posture behaviors from videos or images, and then determines the correctness and effectiveness of individual postures. Human behavior recognition technology is widely used in many fields, such as security monitoring systems, human-computer interaction systems, etc., and its application in the medical field is not common, but this does not mean that posture behavior technology is not applicable in medical care. On the contrary, with the rapid development of information digitization today, human behavior recognition technology has broad development prospects in the medical field.

[0004] At present, the improvement of nursing methods in the medical field is mainly reflected in the improvement of medical equipment and voice control. There are few methods for identifying, monitoring and analyzing the patient's behavior posture during nursing. These improvements are mainly reflected in increasing the patient's experience and avoiding secondary damage caused during the use process. The patient's behavior data cannot be effectively used after use, resulting in a certain amount of data waste. Moreover, the current improvements cannot be accurately evaluated and it is difficult to improve the existing systems and functions and provide an effective theoretical basis. The patient's bed posture data and behavior data are easily affected by irrelevant factors, such as clothing, lighting, gauze bandages, etc. If a large amount of image data is not pre-processed for noise reduction, it is difficult to accurately identify whether the patient's posture is correct or not, and to distinguish the standard patient's sitting, lying, leaning and other behaviors.

[0005] Patients and their families in the wards always hear doctors and nurses repeatedly emphasize that patients with impaired consciousness or paralysis should be turned over regularly and placed in a proper position. However, since their families are not aware of the importance of this, a considerable number of patients still develop complications such as pressure sores or aspiration pneumonia. These sequelae not only make the patients lose their ability to take care of themselves, but also increase the burden on their families and society. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for monitoring a patient's posture in bed, which can accurately identify whether the patient's posture is correct or not, thereby automatically and standardizedly correcting the behavior of postoperative or seriously ill patients resting in bed.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is to provide a method for monitoring a patient's bed posture, comprising the following steps: S1) collecting image information of human posture behavior through a depth camera; S2) extracting human skeleton information from the collected image information to obtain human motion skeleton map information; S3) identifying the patient's bed posture according to the human motion skeleton map.

[0008] Compared with the prior art, the present invention has the following beneficial effects: the patient bed posture monitoring method provided by the present invention can accurately identify whether the patient's posture is correct or not, thereby automatically and standardizedly correcting the behavior of postoperative or seriously ill patients resting in bed. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of the monitoring process of a patient's bed posture according to the present invention;

[0010] Figure 2 This is a schematic diagram of the vectorized standard labels file format after processing by the present invention. DETAILED DESCRIPTION

[0011] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0012] The method for monitoring the patient's bed posture provided by the present invention includes image information of human body posture behavior and key point depth information of a human skeleton diagram. The extracted image is extracted into an original database through a posture recognition algorithm for behavior comparison fitting, and the extracted data is fitted into a human skeleton diagram cloud. Feature extraction is performed on the image data of the skeleton diagram, and then a 3D coordinate unit structure diagram is constructed. A classification feature algorithm is used to classify related image postures, and the matching rate of the classified posture type results and the posture relative position coordinates of the original national standard unit are calculated, the three-dimensional angle of skeletal joint offset is calculated, and the precise numerical value of the joint space vector is calculated.

[0013] See also Figure 1The method for monitoring the patient's bed posture provided by the present invention has the following specific processing steps:

[0014] 1. Data monitoring

[0015] Data collection uses a depth camera to collect video and image data from multiple angles. Since the patient's head is at the head of the bed, and due to the ward obstacles and indoor lighting, the depth camera and surveillance camera are placed above the ward ceiling, in front, on the left, and on the right to obtain depth information in four directions. Ensure that the camera can capture the patient's multi-directional image information, and there are no obstacles blocking the line of sight within the range.

[0016] 2. Data processing

[0017] In the process of identifying and analyzing the patient's bed posture, we can focus on the human skeleton information map. The Alphapose optimization model highlights the patient's internal human information such as the skeleton while in bed and is not affected by the indoor environment, such as the patient's clothing, ward decorations, medical equipment, lighting and other information, thereby reducing interference in recognition and improving recognition accuracy.

[0018] The depth camera can be used to collect video information of human behavior and depth information of key points of the human skeleton. The collected video data is used to extract human skeleton information. Human skeleton information can be extracted through Alphapose, which uses a bottom-up method to detect human skeletal joints. First, the confidence of human body part detection is predicted through the network, and the position of key bone points of the human body is detected. After obtaining the position of the key bone points, the affinity between the parts is associated through the network prediction. Finally, the confidence and affinity are analyzed through the greedy algorithm, and the key points are connected to obtain the human action skeleton information.

[0019] The acquired images are extracted into a patient bed skeleton using the Alphapose posture recognition algorithm. The skeleton consists of 25 joint points and 24 bone edges, which is a natural structural representation of human joints and bones. The raw skeleton data in each frame is always provided as a vector sequence, each vector representing the two-dimensional or three-dimensional coordinates of the corresponding human joint.

[0020] Taking the center of the bed as the origin O coordinate, in order to measure the accuracy of the posture standard, the three-dimensional vector offset and offset angle of the human skeleton are calculated. Assuming that the patient's standard posture offset is (x 1 , x 2 , x 3 ), the actual patient skeleton offset is (y 1 ,y 2 ,y 3), then the actual error offset vector is (x 1 -y 1 , x 2 -y 2 , x 3 -y 3 )

[0021] 3. Posture recognition

[0022] Alphapose human target detection uses the yolov5 algorithm for network training modules. Data acquisition submodule, data standardization submodule, model training submodule. The main process is to obtain the image files of the standard examples through the data interface, and the data can be updated and shared in real time. For the network training module of the acquired network, it includes the preparation and preprocessing of the data set, training for the processed data labels, etc.

[0023] 1) Data acquisition submodule

[0024] It is mainly used for training the image and video data of the posture recognition model. Specifically, the main sources are obtained from three aspects. The first is the sample image files of the national standards stored in the standard library. The demonstration posture image data can be automatically classified. The second is mainly the real-time storage data through test comparison when the hospital detects the patient's behavior in real time.

[0025] 2) Data standardization submodule

[0026] The data standardization submodule includes vectorizing the national standard sample image files stored in the standard library, training the network using standard txt annotation files, converting the original image dataset files used for training into txt file data format, and the processed vectorized standard labels files are as follows: Figure 2 As shown: In the labels file, each object has one line, and each line is in the format of class, x_center, y_center, width, and height. The box coordinates must be in the standardized xywh format (from 0 to 1). Since the target detection target in this invention only considers targets such as patient bed posture, there is only one line of labels file corresponding to each picture, and the category is set to 0. The coordinate points of the upper left and lower right of the patient image are extracted from the picture name, and after a series of operations, the labels file in txt format is output.

[0027] The calculation formula is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] The coordinates of the center point of the normalized detection patient image rectangular frame are expressed as (x_center, y_center), and the width of the detection image rectangular frame is width, and the height is height. w and h represent the width and length of the normalized detection patient image rectangular frame, respectively.

[0033] 3) Model training submodule

[0034] Randomly import pictures from the national standard patient posture standard data set as the data set of the present invention, and use 70% of the data set as the training set and 30% as the verification set. For a large training set, the training data is similar to the pre-training data, so the pre-trained weight parameters can be used to initialize the network, and then start training. In order to obtain a better model, the present invention studied and trained 300 epochs and saved the optimal model. Select the Adam optimizer, the learning rate decreases as the number of training times increases, and the network parameters are continuously adjusted. The main hyperparameter changes are mainly weight decay, learning rate, momentum, etc. Among them, the learning rate is an indispensable hyperparameter. Adjust the learning rate according to the training performance. It should be noted that if its value is too low, the function convergence speed will decrease, and if its value is too high, it will make it difficult for the objective function to obtain the global optimal value. In order to prevent the occurrence of overfitting of the model, weight decay is included, and the normative term is included in the cost function. Learning rate decay can prevent the objective function from converging at the local optimal value while maintaining the convergence speed.

[0035] 4) Posture recognition submodule

[0036] The extracted skeleton graph is feature extracted using a graph convolutional neural network. Given a coordinate sequence of body joints in 2D or 3D form, a naturally connected spatiotemporal graph with joints as nodes and human body structure and time as edges is constructed. Therefore, the input of the model is the joint coordinate vector on the graph node. This can be seen as an image-based analog CNN, where the input is formed by a pixel vector located on a 2D image grid. Multi-layer spatiotemporal graph convolution operations are performed on the input data to generate higher-level feature maps on the graph. It will then be classified into the corresponding action category by a standard SoftMax classifier.

[0037] The characteristics of the human skeleton include spatial relative characteristics and motion characteristics. Spatial characteristics are extracted from joints and bones. Joints are represented as 3D coordinates, and bones are represented as the difference between two joint coordinates. Motion characteristics are generally represented as motion information, which includes displacement, joint direction, motion speed, acceleration, etc.

[0038] For features including spatial relative features, which include the calculation of the relative vector size of the joint angle and the calculation of the error size, it is assumed that the spatial vector size L of the specification file (referring to the training file, that is, the file in the national standard library) M =(x m ,y m , z m ), where M is the definition name of each joint, the label is M∈{1, 2, 3....m}, and the joint space vector detected in real time is L N =(x n ,y n , z n ), where N is the definition name of each joint detected in real time, the label is N∈{1, 2, 3....n}, and the patient joint space vector detected in real time is the error p calculation formula is The larger the value of p, the lower the image model matching rate, indicating that the patient's posture is more different from the standard posture, and the posture feedback needs to be corrected in time; otherwise, the matching degree is higher and no correction is needed. If the threshold value of p is greater than 0.5, it means that the patient's posture difference is too large and an alarm is required. Otherwise, the data can be automatically stored in the data storage submodule to be called as training data for model training, thereby forming a closed-loop detection storage system.

[0039] The motion feature consists of computing the motion information of a joint as the coordinate difference along the time dimension. Similarly, the deformation of a bone is represented as the difference between the vectors of the same bone in consecutive frames. Formally, the motion of a joint v in time t is calculated as The deformed joints of the patient's skeleton are defined similarly as: The acceleration of the patient's joint l is defined as follows: where v t1 Refers to the instantaneous velocity of the patient at time t1 captured by the camera, where v t2 It refers to the instantaneous speed of the patient at time t2 captured by the camera; the average movement trend direction of the patient's joints is The patient's movement trend direction is the direction of the AB vector, and the starting point coordinate of the patient's initial movement is A=(x 1 ,y 1 ,z 1 ), the final joint coordinate position is B = (x 2 ,y 2 ,z 2 ).

[0040] Finally, the human skeleton information and bone information obtained in the first few stages can be fed to the posture recognition module for recognition and analysis, mainly for the recognition and analysis of the standardized recovery rest posture behavior of postoperative or seriously ill patients when they are resting in bed, such as (supine position, lateral position).

[0041] The present invention realizes closed-loop management of the system and secondary utilization of data. The Alphapose optimization model adopted by the present invention is compared with the traditional openpose model. This type of algorithm first detects the human body and then obtains the key points and skeleton. Therefore, its accuracy and AP value (AP measures the quality of the learned model in each category) are significantly higher than openpose.

[0042] The target detection algorithm of the present invention: YOLOv5 is implemented in PyTorch. It benefits from the mature PyTorch ecosystem, supports simpler and easier deployment. Compared with YOLOv4, YOLOv5 has the following advantages:

[0043] (1) Faster. In the YOLOv5 Colab notebook, running TeslaP100, we can see that the inference time for each image is only 0.007 seconds, which means 140 frames per second (FPS), which is more than twice the speed of YOLOv4.

[0044] (2) Higher accuracy. In Roboflow's test on the blood cell counting and detection (BCCD) dataset, it achieved an average precision (mAP) of about 0.895 after only 100 epochs of training. It is true that EfficientDet and YOLOv4 have comparable performance, but it is very rare to have such a comprehensive performance improvement without any loss in accuracy.

[0045] (3) Smaller size. The weight file of YOLOv5 is 27 megabytes. The weight file of YOLOv4 (using Darknet architecture) is 244 megabytes. YOLOv5 is nearly 90% smaller than YOLOv4! This means that YOLOv5 can be more easily deployed on embedded devices.

[0046] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the definition of the claims.

Claims

1. A method for monitoring a patient's bed posture, It is characterized in that The steps include: S1) collecting image information of human posture behavior through a depth camera; S2) extracting human skeleton information from the collected image information to obtain human action skeleton graph information; S3) identifying the patient's bed posture according to the human body motion skeleton diagram; The depth camera in step S1 is connected to a plurality of cameras, and the plurality of cameras are placed above the ceiling of the ward, in front, on the left and on the right, respectively acquiring depth information in four directions; The step S3 uses a classification feature algorithm to classify the postures of the relevant images, and compares the classified posture type results with the national standard bed posture, specifically including: S31) obtaining the national standard sample image files stored in the standard library and performing vectorization processing to generate a patient posture standard data set; S32) randomly importing pictures from the patient posture standard dataset as the target dataset of the neural network model, using 70% of the target dataset as a training set and 30% as a validation set; S33) extracting features from the obtained human action skeleton graph using a graph convolutional neural network, constructing a naturally connected spatiotemporal graph with joints as nodes and human body structure and time as edges given a coordinate sequence of body joints in 2D or 3D form; using a multi-layer spatiotemporal graph convolution operation on the input data to generate a higher-level feature graph, and then classifying it into corresponding action categories through a standard SoftMax classifier; The characteristics of the human skeleton include motion characteristics, and step S33 includes: The joint motion information is calculated as the coordinate difference along the time dimension, and the bone deformation is set as the vector difference of the same bone in consecutive frames; then the average joint motion trend direction of the patient is The patient's movement trend direction is the direction of the AB vector, and the patient's starting point coordinates are A = (x 1 ,y 1 ,z 1 ), the final joint coordinate position is B = (x 2 ,y 2 ,z 2 ).

2. The method for monitoring the patient's bed posture as claimed in claim 1, It is characterized in that The step S2 detects the human skeleton joints in a bottom-up manner through the Alphapose model, specifically comprising: First, the confidence of human body part detection is predicted through the network, and the position of the key bone points of the human body is detected; after obtaining the position of the key bone points, the affinity between the related parts is predicted through the network; finally, the confidence and affinity are analyzed through the greedy algorithm, and the key points are connected to obtain the patient's human body motion skeleton diagram.

3. The method for monitoring the patient's bed posture as claimed in claim 2, It is characterized in that The human motion skeleton graph consists of 25 joint points and 24 bone edges. The original skeleton data in each frame is always provided as a vector sequence, and each vector represents the two-dimensional or three-dimensional coordinates of the corresponding human joint; the step S2 also includes calculating the three-dimensional vector offset and offset angle of the human skeleton with the center position of the bed as the origin O coordinate.

4. The method for monitoring the patient's bed posture as claimed in claim 1, It is characterized in that In the vectorized standard labels file processed in step S31, each object is a line, and each line includes category, x_center, y_center, width and height format, and (x_center, y_center) is the coordinate of the center point of the normalized patient image rectangle.

5. The method for monitoring the patient's bed posture as claimed in claim 1, It is characterized in that The features of the human skeleton include spatial relative features extracted from joints and bones, and step S33 includes: Calculate the relative vector size and error size of the joint angle, assuming that the space vector size L of the specification file M =(x m ,y m , z m ), where M is the definition name of each joint, and the label is M∈{1, 2, 3....m}; Real-time detection of joint space vector L N =(x n ,y n , z n ), where N is the definition name of each joint detected in real time, and the label is N∈{1, 2, 3....n}; Real-time detection of the patient's joint space vector error p, The larger the value of p, the lower the image model matching rate, which means that the patient's posture is more different from the standard posture; if the value of p is greater than the set threshold, it is determined that the patient's posture difference is too large, and real-time feedback is given to correct the posture and an alarm is performed; otherwise, the data is automatically stored in the data storage submodule and called as training data for the neural network model, thereby forming a closed-loop detection storage system.

Images