ICU patient behavior and safety monitoring system and method based on instant detection

Abstract

The application discloses an ICU patient behavior and safety monitoring system and method based on instant detection. The system comprises a multi-modal visual acquisition module, a posture evaluation and behavior recognition module, a facial expression detection and pain evaluation module, a pipeline position monitoring module, a risk evaluation and alarm module, a motion and environment perception module. By adopting the motion and environment perception module to drive the POCT to move in front of the patient bed to be monitored, the current state of the patient is collected and preprocessed, the feature of the image data of the patient is extracted based on the preprocessed data, and the analysis is carried out, the risk level of the patient is evaluated based on the analysis result, and the alarm is carried out. The application sets the motion and environment perception module to control the mobile POCT robot to actively patrol the bed, and the patient condition is autonomously recognized and analyzed, and the analysis result and alarm information are pushed to the nurse station, so that timely early warning can be carried out when the patient state is abnormal.

Description

Technical Field

[0001] This invention relates to the field of medical monitoring technology, specifically to an ICU patient behavior and safety monitoring system and method based on real-time detection. Background Technology

[0002] ICU patients are critically ill and often present with altered consciousness, agitation, and delirium. Currently, ICU monitoring primarily relies on vital sign monitoring equipment (ECG, blood oxygen saturation, blood pressure, etc.), but monitoring of patient behavior is severely lacking. Clinical challenges include: 1. Lack of monitoring for accidental extubation: Incidents of patients removing life-saving tubes such as endotracheal tubes, central venous catheters, gastric tubes, and urinary catheters by themselves occur frequently. Each extubation may endanger life, but currently, it relies entirely on nurses' regular rounds to detect such cases, and there is no real-time warning.

[0003] 2. Risk of falling out of bed: Agitated patients may climb over the bed rails and fall out of bed. Improper use of restraints may also cause secondary injuries. There is a lack of automatic recognition and alarm mechanisms.

[0004] 3. Lack of recognition of pain and distress expressions: ICU patients are often unable to express themselves verbally due to intubation, and their pain, suffocation and other distress states can only be judged by medical staff based on their experience, which is highly subjective and has a delayed response.

[0005] 4. Heavy workload of manpower patrol: ICU night shift nurses need to monitor multiple critically ill patients at the same time, making it difficult to provide continuous attention to each patient.

[0006] Existing fixed surveillance cameras only provide passive recording and do not have intelligent analysis capabilities. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides an ICU patient behavior and safety monitoring system based on real-time detection, comprising a multimodal visual acquisition module, a posture assessment and behavior recognition module, a facial expression detection and pain assessment module, a tube position monitoring module, a risk assessment and alarm module, and a motion and environmental perception module. The multimodal vision acquisition module is used to acquire the patient's current state information and preprocess the acquired data; wherein, the patient's current state information specifically includes: the patient's current three-dimensional pose image data, the patient's facial expression, and the patient's current body surface temperature; The posture assessment and behavior recognition module is used to analyze and judge the patient's skeletal key points based on preprocessed data and to perform behavioral state analysis. The facial expression detection and pain assessment module is used to extract and analyze the features of the patient's facial condition based on the preprocessed data, and generate a pain score in combination with the clinical pain assessment scale. The tube position monitoring module is used to monitor the patient's current intubation position in real time based on the patient's current three-dimensional posture image data. The risk assessment and alarm module is used to assess the safety risk level and issue alarms based on the analysis results of the posture assessment and behavior recognition module, facial expression detection and pain assessment module, and pipeline position monitoring module. The motion and environment perception module is used to control the POCT to move to the bedside of the patient to be monitored, and to perceive and identify the environmental conditions around the current trajectory of the POCT using lidar.

[0008] Furthermore, the multimodal vision acquisition module specifically acquires the patient's current status information by using an RGB-D depth camera and an infrared thermal imaging camera.

[0009] Furthermore, the posture assessment and behavior recognition module specifically employs a lightweight MoveNet model to identify and extract key skeletal points of the patient; The patient's behavioral states in the behavioral state analysis specifically include: lying flat, lying on their side, sitting up, climbing over the bed rails, and tube removal gestures.

[0010] Furthermore, the security risk levels in the risk assessment and alarm module specifically include: Red: A red alert is issued when the patient's current behavior is detected as a gesture to remove the tube or climbing over the bed rails, or when the patient's current pain score is ≥5. Yellow: When a patient is currently intubated and their behavior changes from lying flat, lying on their side, or sitting up to tending to climb over the bed rails, or when a patient's current pain score is ≥3 and <5, a yellow warning is issued. Green: When the patient's current condition is detected as stable, it is marked as green and no alarm is triggered.

[0011] Furthermore, the motion and environment perception module specifically uses lidar to perceive and identify the environment around the POCT's current trajectory.

[0012] A method for monitoring ICU patient behavior and safety based on real-time detection includes the following steps: S1. The POCT is driven to move to the bedside of the patient to be monitored by using a motion and environment perception module, and the surrounding environment of the movement trajectory is perceived and identified by LiDAR. S2. Collect and preprocess the patient's current condition; S3. Based on the preprocessed data, extract features from the patient's image data and perform analysis: S31. Extract the patient's skeletal key points and perform behavioral state analysis based on the location coordinates of the patient's skeletal key points; S32. Extract key points of the patient's facial expressions and score pain based on these key points; S33. Obtain the patient's current intubation position based on the patient's current three-dimensional pose image data, and make a displacement judgment based on historical intubation position data; S4. Based on the analysis results, assess the risk level of the patient's condition and issue an alert.

[0013] Furthermore, step S31 specifically includes the following sub-steps: S31-1. Trim the patient's current 3D pose image data to 192×192 and normalize it; S31-2. Use Lightning in MoveNet to identify and extract key points of the human skeleton in image data; S31-3. Multiply the extracted human skeleton key point coordinates by the width and height of the original image to restore them; S31-4. Calculate the relative distance between each key point, set the judgment threshold, and analyze and identify the patient's current behavioral state based on the judgment threshold; The key points of the human skeleton specifically include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, neck, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; The patient's current behavioral state specifically includes: lying flat, lying on their side, sitting up, climbing over the bed rails, and extubation gestures; among them, when the distance between the key point of the wrist and the endotracheal tube area is less than a threshold and the palm is facing the face, it is judged as an extubation gesture.

[0014] Furthermore, the coordinates of the key points on the neck are taken as the midpoint between the coordinates of the left shoulder and the right shoulder.

[0015] Furthermore, in step S32, key facial expression points of the patient are extracted using a two-stream network architecture, which specifically includes: Spatial flow: used to extract facial motion units, including eyebrow lowering, eyelid tightening, and nasal wing lifting; Time-stream: Used to analyze changes in a patient's facial expressions over a period of time.

[0016] Furthermore, in step S32, CPOT is specifically used for pain scoring.

[0017] This invention provides a system and method for monitoring the behavior and safety of ICU patients based on real-time detection, which has the following beneficial effects: This invention controls a mobile POCT robot to actively patrol the bed by setting up motion and environmental perception modules, and autonomously identifies and analyzes the patient's condition, and pushes the analysis results and alarm information to the nurse station, which can provide timely warnings when the patient's condition is abnormal. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the system structure provided by the present invention; Figure 2 The method flowchart provided by the present invention. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] The following detailed description of the implementation method of the present invention is in conjunction with the accompanying drawings. The description is only a partial embodiment and not all embodiments. For clarity, representations and descriptions unrelated to the present invention are omitted in the drawings and description.

[0022] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the following detailed description of the technical solution is provided. Obviously, the described embodiments are only a portion of the embodiments of this invention, not all of them, and should not be construed as limiting the scope of implementation of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this invention.

[0023] like Figure 1 As shown, the present invention provides an ICU patient behavior and safety monitoring system based on real-time detection, including a multimodal visual acquisition module, a posture assessment and behavior recognition module, a facial expression detection and pain assessment module, a tube position monitoring module, a risk assessment and alarm module, and a motion and environmental perception module.

[0024] The multimodal vision acquisition module is used to collect the patient's current state information and preprocess the collected data. Specifically, the patient's current state information includes: the patient's current 3D pose image data, the patient's facial expression, and the patient's current body surface temperature. This information is acquired using an RGB-D depth camera and an infrared thermal imaging camera.

[0025] The posture assessment and behavior recognition module is used to analyze and judge the patient's skeletal key points based on preprocessed data and to perform behavioral state analysis. Specifically, a lightweight MoveNet model is used to identify and extract the patient's skeletal key points; the patient's behavioral states in the behavioral state analysis include: lying flat, lying on their side, sitting up, climbing over the bed rails, and extubation gestures.

[0026] The facial expression detection and pain assessment module is used to extract and analyze facial features based on preprocessed data, and generate a pain score by combining it with a clinical pain assessment scale.

[0027] The tube position monitoring module is used to monitor the patient's current intubation position in real time.

[0028] The risk assessment and alarm module is used to assess the safety risk level and issue alarms based on the analysis results of the posture assessment and behavior recognition module, facial expression detection and pain assessment module, and pipeline position monitoring module.

[0029] The specific safety risk levels include: Red: A red alert is issued when the patient's current behavior is detected as a gesture to remove the tube or climbing over the bed rails, or when the patient's current pain score is ≥5. Yellow: When a patient is currently intubated and their behavior changes from lying flat, lying on their side, or sitting up to tending to climb over the bed rails, or when a patient's current pain score is ≥3 and <5, a yellow warning is issued. Green: When the patient's current condition is detected as stable, it is marked as green and no alarm is triggered.

[0030] The motion and environment perception module is used to control the movement of the POCT and to perceive and identify the environment around the POCT's current trajectory. Specifically, it uses LiDAR to perceive and identify the environment around the POCT's current trajectory.

[0031] Methods for monitoring ICU patient behavior and safety based on real-time detection, such as Figure 2 As shown, it includes the following steps: S1. The POCT is driven to move to the bedside of the patient to be monitored by using a motion and environment perception module, and the surrounding environment of the movement trajectory is perceived and identified by LiDAR.

[0032] S2. Collect and preprocess the patient's current condition.

[0033] S3. Based on the preprocessed data, extract features from the patient's image data and perform analysis: S31. Extract the patient's skeletal key points and perform behavioral state analysis based on the location coordinates of these key points: S31-1. Trim the patient's current 3D pose image data to 192×192 and normalize it; S31-2. Use Lightning in MoveNet to identify and extract key points of the human skeleton in image data; S31-3. Multiply the extracted human skeleton key point coordinates by the width and height of the original image to restore them; S31-4. Calculate the relative distances between each key point, set a judgment threshold, and analyze and identify the patient's current behavioral state based on the judgment threshold; wherein, the key points of the human skeleton specifically include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, neck, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The coordinates of the neck key point are taken as the midpoint between the coordinates of the left shoulder and the right shoulder.

[0034] S32. Extract key points of the patient's facial expressions and score pain based on these key points; specifically, extract key points of the patient's facial expressions using a dual-stream network architecture, which includes: spatial stream: used to extract facial action units, including lowering of eyebrows, tightening of eyelids, and raising of the nostrils; temporal stream: used to analyze changes in the patient's facial expressions over a period of time.

[0035] S33. Obtain the patient's current intubation position based on the patient's current three-dimensional posture image data, and determine displacement based on historical intubation position data. Specifically, use CPOT to score facial expression pain, and combine the patient's current intubation position with the CPOT / BPS scale for comprehensive scoring and judgment.

[0036] The CPOT (Critical Care Pain Observation Scale) score is a professional scale specifically designed for assessing pain in critically ill patients who are unable to communicate verbally. It uses objective behavioral indicators instead of subjective pain descriptions, scoring patients across four dimensions: facial expressions, body movement, muscle tension, and patient-ventilator synchrony. This helps healthcare professionals accurately determine if a patient is experiencing pain. Both the CPOT and BPS (Behavioral Pain Scale) are standardized pain assessment tools for ICU patients unable to communicate verbally. The CPOT has four dimensions (0-8 points), while the BPS has three dimensions (3-12 points). The BPS covers three items: facial expressions, upper limb movement, and ventilation compliance (or vocalization), each scored from 1-4 points, for a total score of 3-12 points. It is specifically designed for mechanically ventilated or sedated comatose patients, using vocalization as a substitute for ventilation indicators when not intubated.

[0037] S4. Based on the analysis results, assess the risk level of the patient's condition and issue an alert.

[0038] When the distance between the wrist key point and the endotracheal tube area is less than a threshold and the palm is facing the face, it is judged as an extubation gesture. When an extubation gesture is detected for N consecutive frames, an extubation warning is immediately triggered.

[0039] All video data is processed in real time within the ICU patient behavior and safety monitoring system, and only structured alarm information ("Bed 3, extubation warning, risk red") is transmitted to the nurses' station. The original video is not stored or transmitted externally, meeting ICU privacy compliance requirements.

[0040] This invention controls a mobile POCT robot to actively patrol the bed by setting up motion and environmental perception modules, and autonomously identifies and analyzes the patient's condition, and pushes the analysis results and alarm information to the nurse station, which can provide timely warnings when the patient's condition is abnormal.

[0041] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. An ICU patient behavior and safety monitoring system based on real-time detection, characterized in that, It includes a multimodal visual acquisition module, a posture assessment and behavior recognition module, a facial expression detection and pain assessment module, a pipeline position monitoring module, a risk assessment and alarm module, and a motion and environmental perception module; The multimodal vision acquisition module is used to acquire the patient's current state information and preprocess the acquired data; wherein, the patient's current state information specifically includes: the patient's current three-dimensional pose image data, the patient's facial expression, and the patient's current body surface temperature; The posture assessment and behavior recognition module is used to analyze and judge the patient's skeletal key points based on preprocessed data and to perform behavioral state analysis. The facial expression detection and pain assessment module is used to extract and analyze the features of the patient's facial condition based on the preprocessed data, and generate a pain score in combination with the clinical pain assessment scale. The tube position monitoring module is used to monitor the patient's current intubation position in real time based on the patient's current three-dimensional posture image data. The risk assessment and alarm module is used to assess the safety risk level and issue alarms based on the analysis results of the posture assessment and behavior recognition module, facial expression detection and pain assessment module, and pipeline position monitoring module. The motion and environment perception module is used to control the POCT to move to the bedside of the patient to be monitored, and to perceive and identify the environmental conditions around the current trajectory of the POCT using lidar.

2. The ICU patient behavior and safety monitoring system based on real-time detection according to claim 1, characterized in that, The multimodal vision acquisition module specifically uses an RGB-D depth camera and an infrared thermal imaging camera to acquire the patient's current status information.

3. The ICU patient behavior and safety monitoring system based on real-time detection according to claim 1, characterized in that, The posture assessment and behavior recognition module specifically uses a lightweight MoveNet model to identify and extract key skeletal points of the patient. The patient's behavioral states in the behavioral state analysis specifically include: lying flat, lying on their side, sitting up, climbing over the bed rails, and tube removal gestures.

4. The ICU patient behavior and safety monitoring system based on real-time detection according to claim 3, characterized in that, The security risk levels in the risk assessment and alarm module specifically include: Red: A red alert is issued when the patient's current behavior is detected as a gesture to remove the tube or climbing over the bed rails, or when the patient's current pain score is ≥5. Yellow: When a patient is currently intubated and their behavior changes from lying flat, lying on their side, or sitting up to tending to climb over the bed rails, or when a patient's current pain score is ≥3 and <5, a yellow warning is issued. Green: When the patient's current condition is detected as stable, it is marked as green and no alarm is triggered.

5. The ICU patient behavior and safety monitoring system based on real-time detection according to claim 1, characterized in that, Specifically, the motion and environment perception module uses lidar to perceive and identify the environment around the POCT's current trajectory.

6. A method for monitoring the behavior and safety of ICU patients based on real-time detection, based on the real-time detection-based ICU patient behavior and safety monitoring system according to any one of claims 1 to 5, characterized in that, Includes the following steps: S1. The POCT is driven to move to the bedside of the patient to be monitored by using a motion and environment perception module, and the surrounding environment of the movement trajectory is perceived and identified by LiDAR. S2. Collect and preprocess the patient's current condition; S3. Based on the preprocessed data, extract features from the patient's image data and perform analysis: S31. Extract the patient's skeletal key points and perform behavioral state analysis based on the location coordinates of the patient's skeletal key points; S32. Extract key points of the patient's facial expressions and score pain based on these key points; S33. Obtain the patient's current intubation position based on the patient's current three-dimensional pose image data, and make a displacement judgment based on historical intubation position data; S4. Based on the analysis results, assess the risk level of the patient's condition and issue an alert.

7. The method for monitoring ICU patient behavior and safety based on real-time detection according to claim 6, characterized in that, Step S31 specifically includes the following sub-steps: S31-1. Trim the patient's current 3D pose image data to 192×192 and normalize it; S31-2. Use Lightning in MoveNet to identify and extract key points of the human skeleton in image data; S31-3. Multiply the extracted human skeleton key point coordinates by the width and height of the original image to restore them; S31-4. Calculate the relative distance between each key point, set the judgment threshold, and analyze and identify the patient's current behavioral state based on the judgment threshold; The key points of the human skeleton specifically include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, neck, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; The patient's current behavioral state specifically includes: lying flat, lying on their side, sitting up, climbing over the bed rails, and extubation gestures; among them, when the distance between the key point of the wrist and the endotracheal tube area is less than a threshold and the palm is facing the face, it is judged as an extubation gesture.

8. The method for monitoring ICU patient behavior and safety based on real-time detection according to claim 7, characterized in that, The coordinates of the key points on the neck are taken as the midpoint between the coordinates of the left shoulder and the right shoulder.

9. The method for monitoring ICU patient behavior and safety based on real-time detection according to claim 6, characterized in that, In step S32, key facial expression points of the patient are extracted using a dual-stream network architecture, which specifically includes: Spatial flow: used to extract facial motion units, including eyebrow lowering, eyelid tightening, and nasal wing lifting; Time-stream: Used to analyze changes in a patient's facial expressions over a period of time.

10. The method for monitoring ICU patient behavior and safety based on real-time detection according to claim 9, characterized in that, In step S32, CPOT is specifically used to score pain.

Images