A warning method for self-extubation behavior of ICU patients based on RGB video monitoring

Abstract

The present invention discloses a method for early warning of ICU patient self-extubation behavior based on RGB video monitoring, which belongs to the field of computer vision behavior recognition. It includes constructing an ICU patient self-extubation behavior dataset, obtaining a patient self-extubation early warning classification model based on a neural network and based on optical flow behavior features in an ICU scenario based on the dataset, obtaining a video image texture feature classification result by the patient self-extubation early warning classification model in an ICU scenario based on a neural network, and obtaining a patient self-extubation optical flow behavior feature classification result by the patient self-extubation early warning classification model in an ICU scenario based on optical flow behavior features; the above two classification results are fused and used as input to build a dual-stream ICU scenario patient self-extubation early warning model, thereby realizing ICU patient self-extubation early warning work based on RGB monitoring video. The present invention solves the problem of incomplete feature extraction from a single-stream structure, and has the advantages of high real-time performance, high accuracy, full automation and contactlessness.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision behavior recognition, and in particular relates to an ICU patient self-extubation behavior early warning method based on RGB video monitoring. Background Art

[0002] Unplanned extubation usually refers to the premature removal of the patient's endotracheal tube, which is mainly divided into two categories: patient self-extubation and accidental extubation. Patient self-extubation is the intentional extubation behavior caused by the patient's discomfort with the tube, while accidental extubation is caused by the patient's inappropriate movement or unintentional action. ICU patients' self-extubation behavior often brings great risks to patients and has an adverse effect on their postoperative recovery. At the same time, frequent intubation and extubation supervision will also add a lot of burden to medical staff. Based on the above reasons, it is very important to actively understand the physical condition of ICU patients and communicate with patients to avoid patients' self-extubation behavior.

[0003] However, patients in the intensive care unit (ICU) often cannot verbalize due to factors such as intubation or pain, resulting in varying degrees of communication difficulties. This can often trigger negative emotions in patients, frustration in healthcare professionals, and increased incidence of adverse events, thus impairing patient recovery. Therefore, several communication support strategies have emerged to facilitate communication between nurses and patients, including unassisted communication strategies, primary communication strategies, and novel communication strategies. Unassisted communication strategies refer to spontaneous communication methods between nurses and patients, such as facial expressions, gestures, and eye contact. Primary communication strategies involve the use of simple, practical, affordable, readily available, and portable non-electronic devices such as communication boards, homemade charts, and writing boards. Novice communication strategies involve the use of high-tech electronic devices to assist patient communication, such as eye-tracking devices and touch-screen software with voice. Although these methods have eased communication difficulties between doctors and patients, they still require further improvement to prevent unplanned extubation.

[0004] Although numerous studies have focused on analyzing unplanned extubation in intensive care units (ICUs), most have focused on qualitative and quantitative studies of the outcomes of extubation. Aydogan et al. evaluated the risk of unplanned extubation in adult ICU patients and found that patients experiencing sedation, analgesia, or respiratory insufficiency were at risk for unintended extubation. Chang et al. analyzed risk factors for unplanned extubation and found that most patients who experienced unplanned extubation were physically restrained. Fabregat et al. used machine learning models to investigate the predictive effects of programmed extubation, and Hsieh et al. also used machine learning to investigate the relationship between unplanned extubation and mortality. These studies explored numerous factors influencing unplanned extubation and analyzed their consequences, but none provided early warning of unplanned extubation. Chiang et al. developed a wearable sensor that detects the distance between a patient's hand and the intubation site. This sensor has the advantage of effectively warning patients of self-extubation, but the disadvantage is that the wearable sensor is bulky and unsuitable for postoperative patients.

[0005] Therefore, the ability to achieve low-cost, high-performance, and fully automated early warning of patients' self-extubation behavior in intensive care units has become a new demand for patients and medical staff. Summary of the Invention

[0006] The object of the present invention is to provide an ICU patient self-extubation behavior warning method based on RGB video monitoring to address the problems or improvement needs of the above-mentioned prior art. The method obtains a video image texture feature classification result by constructing a patient self-extubation warning classification model in an ICU scenario based on a neural network, and obtains a patient self-extubation behavior feature classification result by constructing a patient self-extubation warning classification model in an ICU scenario based on light flow behavior features. The above two classification results are fused and used as input to build a dual-stream ICU scenario patient self-extubation warning model, thereby realizing ICU patient self-extubation warning work based on RGB monitoring video.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for early warning of self-extubation behavior of ICU patients based on RGB video monitoring, comprising the following steps:

[0009] S1. Obtain video sequence data of ICU patients' self-extubation behavior captured by an RGB depth camera. The video data includes three stages of ICU patients' self-extubation behavior: tendency stage, intention stage, and extubation stage.

[0010] S2: The ICU surveillance video collected in S1 is cropped at equal intervals and labeled according to the recommendations of professional medical staff to form a dataset of ICU patients' self-extubation behavior. This dataset includes four types of human behavior: patients in a non-moving state, patients with no extubation tendency, patients with extubation tendency, and patients with medical staff around.

[0011] S3: Build a C3D convolutional neural network model based on the PyTorch deep learning framework and train it using the public behavior recognition dataset UCF101 to obtain a pre-trained model based on this behavior recognition dataset. Based on the transfer learning method, the pre-trained model is trained using the ICU patient self-extubation behavior dataset constructed in S2 to obtain a neural network-based patient self-extubation warning classification model in the ICU scenario. This model is used to analyze the texture features of video images and output the analysis results.

[0012] The authors extracted bed regions of interest from a dataset of ICU patient self-extubation behaviors, tracked patient motion behaviors based on the LK sparse optical flow algorithm, and extracted three types of lightweight patient motion feature operators from the tracked patient motion behaviors, including optical flow corner point number features, trajectory distance features, and wavelet change features. These three types of lightweight patient motion feature operators were used as optical flow behavioral feature information and input into a support vector machine for classification training. This resulted in a patient self-extubation warning classification model for ICU scenarios based on optical flow behavioral features. This model was used to classify the optical flow behavioral features of patient self-extubation and output the classification results.

[0013] S4. Use the maximum pooling method to fuse the video image texture analysis results and the light flow behavior feature classification results, and use the fused features as the output to build a dual-stream ICU scenario patient self-extubation warning model;

[0014] S5. Real-time acquisition of RGB video sequence data. The ICU patient self-extubation warning classification model built using S5D is used to process and test the video sequence at equal time intervals. The classification results of ICU patient human behavior movements at each moment are obtained, and a real-time warning is issued for dangerous self-extubation behaviors based on the classification results.

[0015] Furthermore, in S3, the YOLO v3 target detection algorithm is used to detect and extract the bed region of interest.

[0016] Furthermore, the process of extracting three types of lightweight patient motion feature operators in S3 is as follows:

[0017] (1) Extract the number of optical flow corner points used to represent the time dimension of the action:

[0018] (1.1) Let M represent the number of corner points per frame obtained based on the LK sparse optical flow algorithm, Represents the position coordinate information of the i-th corner point in the t-th frame, and the status information of the i-th corner point in the current frame is Determined by the following formula:

[0019]

[0020] Where d represents the decision threshold; when the coordinate transformation of the same corner point in the current adjacent frame represented by the above formula is greater than d, then The state information of is set to 0, that is, the current corner point is removed, and the number of corner points per frame feature N(t) can be obtained;

[0021]

[0022] (1.2) Calculate the number of corner points N(T) of the last frame T of the video according to (1.1) and use it as the optical flow corner point number feature;

[0023] (2) Extract trajectory distance features used to represent the action space dimension:

[0024] (2.1) Using the optical flow corner point number feature N(T) obtained in (1.2) as a benchmark, calculate the corner point trajectory length s(t) between adjacent frames:

[0025]

[0026] (2.2) Calculate the average value of all s(t) in the current video sequence and use it as the trajectory distance feature S of the current video sequence:

[0027]

[0028] (3) Extract wavelet transform features used to represent the spatial and temporal dimensions of the action:

[0029] (3.1) Perform wavelet transform on the optical flow corner point number feature N(t) according to the following formula to obtain the spatial / frequency components WT(α,τ) of the number of optical flow corner points at different times:

[0030]

[0031] Among them, α represents the scale factor to control the expansion and contraction of the wavelet function φ, and τ represents the translation factor to control the translation amount of the wavelet function φ;

[0032] (3.2) Add WT(α,τ) obtained in (3.1) in the time dimension to obtain the spatiotemporal feature W(α) representing the action:

[0033]

[0034] After adopting the above technical solution, the present invention has the following advantages:

[0035] 1. This invention uses transfer learning to solve the problem that small sample data sets are difficult to train huge models. It also uses the LK sparse optical flow algorithm to track the patient's movement behavior in the area of ​​interest of the bed, and selects lightweight feature operators from the tracked patient movement behavior, which solves the problem of long inference time of traditional algorithms and ensures the real-time warning of ICU patients' self-extubation behavior. The fusion of the two results improves the warning performance and solves the problem of incomplete feature extraction from the single-stream structure.

[0036] 2. The dual-flow ICU patient self-extubation early warning model constructed by the present invention realizes the ICU patient self-extubation early warning work based on RGB monitoring video, with the advantages of high real-time performance, high accuracy, full automation and contactlessness. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the early warning of ICU patients' self-extubation behavior based on the dual-flow structure;

[0038] Figure 2 This is a schematic diagram of the early warning model for ICU patients' self-extubation behavior based on transfer learning methods;

[0039] Figure 3 Schematic diagram of the early warning model for ICU patients' self-extubation behavior based on the LK optical flow sparse algorithm; DETAILED DESCRIPTION

[0040] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0041] like Figure 1 As shown, a method for early warning of self-extubation behavior of ICU patients based on RGB video monitoring includes the following steps:

[0042] S1. Obtain video data of ICU patients' self-extubation behavior captured by an RGB depth camera. The data collected here mainly consists of RGB video sequences, covering the entire process of ICU patients' extubation from a few minutes before to the end of extubation. The process can be mainly divided into three stages: tendency stage, intention stage, and extubation stage.

[0043] The tendency phase refers to the patient's physical discomfort and slight limb movements 2-3 minutes before extubation;

[0044] The intention phase refers to the patient's cannula-touching behavior, which begins approximately 10 seconds before the cannula removal action;

[0045] The extubation phase refers to the patient's self-extubation behavior.

[0046] S2. Perform equal-interval cropping on the original, unprocessed ICU surveillance video. First, find the end time of ICU patient extubation based on the video content, and crop the video before this time at equal intervals. The interval length is determined by the advance amount of the required warning time. In this embodiment, the equal interval length is set to 5 seconds. The content of different video segments varies, and the limb behaviors of ICU patients also vary. Considering the suggestions of professional medical staff and the relevant standards for dataset construction, the video segments are classified and labeled based on the video content to construct an ICU patient extubation behavior dataset. Based on the video content and the relevant suggestions of medical staff, the dataset is mainly divided into the following four categories of human behavior: patients in no motion state, patients with no extubation tendency, patients with extubation tendency, and patients with medical staff around.

[0047] Patient non-movement state: This means the patient is in a static state with no limb movement, including sleeping and lying flat;

[0048] Patients’ non-extubation-inclined movements: These movements refer to minor movements and non-extubation behaviors of the patient, such as eating and touching the skin.

[0049] Patients have a tendency to extubate: This refers to patients having abnormal behaviors and a tendency to extubate themselves;

[0050] Patients are surrounded by medical staff: This means that there are doctors and nurses around the patients to diagnose and care for them.

[0051] S3. Obtain a warning classification model for patient self-extubation in an ICU scenario based on a neural network and light flow behavioral features.

[0052] The early warning acquisition process of the patient self-extubation early warning classification model in the ICU scenario based on the neural network is as follows:

[0053] (a1) A C3D convolutional neural network model is constructed based on the PyTorch deep learning framework. The C3D convolutional neural network consists of 5 three-dimensional convolutional neural network layers, 5 maximum pooling layers, 3 linear layers, and a relu activation function. The last three layers of the three-dimensional convolutional neural network can be divided into two categories, a and b, and the dropout is set to 0.5. W represents the weight matrix of the convolutional layer, X represents the input in matrix form, and b represents the bias of the convolutional layer. Represents the relu activation function, and Y represents the output in vector form;

[0054]

[0055] The loss function adopts the cross entropy loss function, which is defined as Loss;

[0056]

[0057] The constructed C3D convolutional neural network model is trained using the public human behavior recognition classification dataset UCF101 to obtain a pre-trained model based on the behavior recognition dataset.

[0058] (a2) Subsequently, based on the transfer learning method, the previously trained C3D pre-trained model was fine-tuned using the constructed ICU patient self-extubation behavior dataset. The fine-tuning process is as follows:

[0059] The weights of the pre-trained model trained using the public human behavior recognition and classification dataset UCF101 were used as initialization conditions. The model was retrained using the ICU patient self-extubation behavior dataset constructed by S2 to obtain an ICU patient self-extubation warning classification model based on a neural network. The output of this model is the result of texture feature analysis of the video image. For details, please refer to Figure 2 .

[0060] like Figure 3 As shown, the warning acquisition process of the patient self-extubation warning classification model in the ICU scenario based on light flow behavior characteristics is as follows:

[0061] (b1) The YOLO v3 target detection algorithm is used to detect and extract the patient bed area of ​​interest. It is trained on the ICU patient self-extubation behavior dataset to complete the positioning of the patient bed area in the ICU surveillance video, thereby eliminating the noisy ICU environment. The bed area detection model predicts the coordinate position information of each box area, where t x , t y , t w , t h Represents the center coordinates, width and height of the box respectively. (c x ,c y ) represents the offset of the box from the upper left corner of the image. w and p h Represent the prior width and height of the box respectively. The entire prediction process is as follows:

[0062] b x =σ(t x )+c x

[0063] b y =σ(t y )+c y

[0064]

[0065]

[0066] Among them (bx ,b y ) represents the true center coordinate of the box, b w and b h Represents the actual width and height of the box respectively.

[0067] (b2) Track the patient's motion behavior within the bed's region of interest based on the LK sparse optical flow algorithm, and extract three types of lightweight patient motion feature operators from the tracked patient motion behavior. The three types of lightweight patient motion feature operators are the optical flow corner point number feature, trajectory distance feature, and wavelet change feature. The process of extracting the three types of lightweight patient motion feature operators is as follows:

[0068] (1) Extract the number of optical flow corner points:

[0069] (1.1) Let M represent the number of corner points per frame obtained based on the LK sparse optical flow algorithm, Represents the position coordinate information of the i-th corner point in the t-th frame, and the status information of the i-th corner point in the current frame is Determined by the following formula:

[0070]

[0071] Where d represents the decision threshold; when the coordinate transformation of the same corner point in the current adjacent frame represented by the above formula is greater than d, then The state information of is set to 0, that is, the current corner point is removed. This formula is used to delete the corner points of abnormal tracking. Then the number of corner points per frame feature N(t) can be obtained. N(t) is used to represent the action feature in the time dimension:

[0072]

[0073] (1.2) In practical applications, there may be corner points that fail to be tracked during the tracking process. To facilitate the calculation of subsequent trajectory distance features, this embodiment selects the number of corner points N(T) in the last frame T of the video as the optical flow corner point number feature.

[0074] (2) Extract trajectory distance features:

[0075] (2.1) Using the optical flow corner point number feature N(T) obtained in (1) as a benchmark, calculate the corner point trajectory length s(t) between adjacent frames:

[0076]

[0077] (2.2) Considering that the number of corner points in each frame is different, it is necessary to take the average value of all s(t) in the current video sequence and use it as the trajectory distance feature S of the current video sequence. This feature is used to represent the action feature in the spatial dimension.

[0078]

[0079] (3) Extracting wavelet transform features:

[0080] By performing a wavelet transform on the optical flow corner point number feature N(t), we can not only obtain the frequency domain component information of the time domain signal, but also know the time when each component appears as a hidden feature of the patient's self-extubation behavior. The wavelet transform of N(t) is shown as follows:

[0081]

[0082] Among them, α represents the scale factor to control the expansion and contraction of the wavelet function φ, and τ represents the translation factor to control the translation amount of the wavelet function φ;

[0083] WT(α,τ) represents the spatial / frequency components of the number of optical flow corner points at different moments, and summing them up in the time dimension to obtain W(α) to characterize the spatiotemporal characteristics of the action.

[0084]

[0085] Finally, the three types of features are concatenated to obtain the space-time feature f = [N(T), S, W T ] T The data is sent to the support vector machine for classification and early warning.

[0086] S4. Use the maximum pooling method to fuse the video image texture analysis results and the light flow behavior feature classification results, and use the fused features as input to build a dual-stream ICU scenario patient self-extubation warning model.

[0087] Y=max(Y network ,Y optical-flow )

[0088] Y represents the final warning classification result, Y network Represents the warning classification probability vector output based on neural network, Y optical-flow Represents the warning classification probability vector output based on the optical flow method.

[0089] S5. Real-time monitoring of ICU patient behavior and actions, issuing real-time warnings for dangerous self-extubation behaviors. Using an RGB depth camera, the system captures RGB video surveillance footage in real time. The video segments are fed into the Shuangliu ICU Self-Extubation Warning Classification Model at fixed intervals (5 seconds) to classify and identify ICU patient behaviors. If the identified action is "patient's tendency to extubate," the ICU Self-Extubation Warning System issues a real-time alarm signal, prompting medical staff to pay attention to the patient.

[0090] In summary, this invention uses an RGB depth camera to capture real-time body movements of patients in the intensive care unit (ICU), easily reducing implementation costs. Furthermore, it can provide early warning of self-extubation without requiring patients to wear wearable sensors. This low-cost, high-performance approach offers a new solution for early warning of unplanned extubation in ICU patients.

Claims

1. A method for early warning of self-extubation behavior of ICU patients based on RGB video monitoring, characterized in that: The following steps are involved: S1. Obtain video sequence data of ICU patients' self-extubation behavior captured by an RGB depth camera. The video data includes three stages of ICU patients' self-extubation behavior: tendency stage, intention stage, and extubation stage. S2: The ICU surveillance video collected in S1 is cropped at equal intervals and labeled according to the recommendations of professional medical staff to form a dataset of ICU patients' self-extubation behavior. This dataset includes four types of human behavior: patients in a non-moving state, patients with no extubation tendency, patients with extubation tendency, and patients with medical staff around. S3: Build a C3D convolutional neural network model based on the PyTorch deep learning framework and train it using the public behavior recognition dataset UCF101 to obtain a pre-trained model based on this behavior recognition dataset. Based on the transfer learning method, the pre-trained model is trained using the ICU patient self-extubation behavior dataset constructed in S2 to obtain a neural network-based patient self-extubation warning classification model in the ICU scenario, which is used to analyze the texture features of video images and output the analysis results. The researchers extracted bed regions of interest from a dataset of ICU patient self-extubation behaviors, tracked the patient's motion behavior based on the LK sparse optical flow algorithm, and extracted three types of lightweight patient motion feature operators from the tracked patient motion behavior. These three types of lightweight patient motion feature operators included the optical flow corner point number feature, trajectory distance feature, and wavelet change feature. The three types of lightweight patient motion feature operators mentioned above are used as optical flow behavior feature information to input into the support vector machine for classification training. This results in a patient self-extubation warning classification model based on optical flow behavior features in the ICU scenario. This model is used to classify the patient self-extubation optical flow behavior features and output the classification results. S4. Use the maximum pooling method to fuse the video image texture analysis results and the light flow behavior feature classification results, and use the fused features as the output to build a dual-stream ICU scenario patient self-extubation warning model; S5. Real-time acquisition of RGB video sequence data. The ICU patient self-extubation warning classification model built using S5D is used to process and test the video sequence at equal time intervals. The classification results of ICU patient human behavior movements at each moment are obtained, and a real-time warning is issued for dangerous self-extubation behaviors based on the classification results.

2. The method for early warning of self-extubation behavior of ICU patients based on RGB video monitoring according to claim 1, characterized in that: The S3 uses the YOLO v3 target detection algorithm to detect and extract the bed area of ​​interest.

3. The method for early warning of ICU patient self-extubation behavior based on RGB video monitoring according to claim 1 or 2, characterized in that: The S3 extracts three types of lightweight patient motion feature operators through the following steps: (1) Extract the number of optical flow corner points used to represent the time dimension of the action: (1.1) Let M represent the number of corner points per frame obtained based on the LK sparse optical flow algorithm, Represents the position coordinate information of the i-th corner point in the t-th frame, and the status information of the i-th corner point in the current frame is Determined by the following formula: Where d represents the decision threshold; when the coordinate transformation of the same corner point in the current adjacent frame represented by the above formula is greater than d, then The state information of is set to 0, that is, the current corner point is removed, and the number of corner points per frame feature N(t) can be obtained; (1.2) Calculate the number of corner points N(T) of the last frame T of the video according to (1.1) and use it as the optical flow corner point number feature; (2) Extract trajectory distance features used to represent the action space dimension: (2.1) Using the optical flow corner point number feature N(T) obtained in (1.2) as a benchmark, calculate the corner point trajectory length s(t) between adjacent frames: (2.2) Calculate the average value of all s(t) in the current video sequence and use it as the trajectory distance feature S of the current video sequence: (3) Extract wavelet transform features used to represent the spatial and temporal dimensions of the action: (3.1) Perform wavelet transform on the optical flow corner point number feature N(t) according to the following formula to obtain the spatial / frequency components WT(α,τ) of the number of optical flow corner points at different times: Among them, α represents the scale factor to control the expansion and contraction of the wavelet function φ, and τ represents the translation factor to control the translation amount of the wavelet function φ; (3.2) Add WT(α,τ) obtained in (3.1) in the time dimension to obtain the spatiotemporal feature W(α) representing the action:

Images