A patient abnormal behavior recognition method and system for ICU intensive care
By employing multimodal vision-driven facial expression perception and posture recognition methods, combined with pain scoring, the problem of accurately identifying pain states and abnormal behaviors in critically ill ICU patients was solved, enabling more efficient clinical monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- AFFILIATED HOSPITAL OF ZUNYI UNIV
- Filing Date
- 2026-04-24
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies struggle to accurately identify pain states and abnormal behaviors in critically ill ICU patients, leading to misjudgments or overlooking of key symptoms and impacting clinical monitoring outcomes.
We employ a multimodal vision-driven expression perception method, combined with an abnormal movement recognition method based on posture and facial information, and a state assessment method based on pain scores and abnormal movement classification. By acquiring images through RGB, near-infrared, and depth cameras, we construct micro-expression features and movement tension features to understand the patient's state from multiple perspectives.
It improves the accuracy and timeliness of abnormal behavior identification, provides timely and reliable early warning information, and enhances the ability to provide early warning of patient suffering and potential risks.
Smart Images
Figure CN122435677A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of smart healthcare technology, specifically referring to a method and system for identifying abnormal patient behaviors in ICU critical care. Background Technology
[0002] This device, designed for identifying abnormal patient behavior in ICU critical care, utilizes image perception and artificial intelligence technologies to continuously monitor and analyze the behavioral status of critically ill ICU patients during their hospitalization. It enables automatic identification and risk warning of abnormal behavior, aiming to improve the accuracy and timeliness of abnormal behavior identification, assist medical staff in carrying out clinical interventions efficiently, and comprehensively enhance patient safety and the quality of nursing services in the ICU environment.
[0003] However, in the existing process of identifying abnormal patient behavior, there are technical problems. Critically ill patients are often sedated, intubated, or have impaired consciousness, making it difficult for them to actively express their discomfort. Nursing staff cannot accurately assess the patient's pain level through facial expressions, easily leading to missed diagnoses of key symptoms. Furthermore, the subtle and complex movements of critically ill patients make it difficult to fully reflect their true condition solely through body posture or facial expressions, resulting in misjudgments or oversights of abnormal movements, failing to meet the clinical need for real-time and precise monitoring. Finally, relying solely on external movements makes it difficult to accurately determine the patient's true condition. Actions such as "turning over" or "shaking" are not always high-risk behaviors; some critically ill patients with relatively good physical function may be making natural postural adjustments due to discomfort, which can easily lead to false alarms or inappropriate interventions, affecting the effectiveness of clinical monitoring. Summary of the Invention
[0004] To address the above-mentioned issues and overcome the shortcomings of existing technologies, this invention provides a method and system for identifying abnormal patient behaviors in ICU critical care. Addressing the technical problem that existing methods for identifying abnormal patient behaviors often result in critically ill patients being sedated, intubated, or experiencing impaired consciousness, making it difficult for them to actively express their discomfort, and hindering nurses from accurately assessing their pain state through facial expressions, thus easily leading to missed diagnoses of key symptoms, this solution creatively employs a multimodal vision-driven expression perception method. By integrating multimodal image information, it stably identifies subtle dynamic changes in key facial areas under different lighting conditions and quantifies the expression of discomfort in patients experiencing pain, thereby assisting the system in inferring the patient's potential pain state and effectively improving the accuracy of abnormal behavior identification. Furthermore, it addresses the issue that existing methods for identifying abnormal patient behaviors often involve subtle and complex movements in critically ill patients, making it difficult to fully reflect the patient's true state solely through body posture or facial expressions, leading to misjudgments or oversights of abnormal movements. To address the technical challenges of meeting the clinical need for real-time and precise monitoring, this solution creatively employs an abnormal movement recognition method that combines posture and facial information. This method comprehensively perceives patient movements and facial expressions, effectively improving the detection accuracy and response time of abnormal movements, and enhancing the early warning capability for patient pain, discomfort, and potential risks. Furthermore, to address the technical problem in existing patient abnormal behavior recognition processes where relying solely on external movements is insufficient to accurately determine the patient's true condition—movements such as "turning over" or "shaking" are not always high-risk behaviors, and some critically ill patients with relatively good physical function may be making natural postural adjustments due to discomfort—leading to false alarms or inappropriate interventions that affect clinical monitoring effectiveness, this solution creatively adopts a state assessment method that combines pain scores and abnormal movement classification. This enables a multi-faceted understanding of abnormal patient behavior, allowing for the assessment of the patient's pain level while observing abnormal movements, thereby providing medical staff with more timely and reliable early warning information.
[0005] The technical solution adopted by this invention is as follows: This invention provides a method for identifying abnormal patient behaviors in ICU critical care, the method comprising the following steps:
[0006] Step S1: Patient image acquisition;
[0007] Step S2: Patient facial expression perception;
[0008] Step S3: Abnormal action identification;
[0009] Step S4: Patient status assessment.
[0010] As a further improvement to this solution, in step S1, the patient image acquisition specifically involves deploying RGB cameras, near-infrared cameras, and depth cameras in the ICU intensive care unit to monitor the ICU intensive care unit in real time, and using a frame synchronization mechanism to unify timestamps and collaboratively acquire three-modal patient image streams.
[0011] The trimodal patient image stream includes an RGB patient image stream, a near-infrared patient image stream, and a depth patient image stream.
[0012] As a further improvement to this solution, in step S2, the patient expression perception is used to construct the micro-expression features of ICU critically ill patients. Specifically, based on the three-modal patient image stream, a multimodal vision-driven expression perception method is used to perceive patient expressions and obtain five-dimensional micro-expression tension features, including the following steps:
[0013] Step S21: Facial region recognition, specifically by constructing a deep perception segmentation network model to extract facial region classification probability maps from each frame of the trimodal patient image stream. Then, by weighting and fusing the trimodal facial region classification probability maps by pixels, the comprehensive facial region probability is calculated, and five types of key facial regions are extracted.
[0014] Step S22: Micro-expression tension recognition, used to quantify pain-related micro-expression dynamic features. Specifically, based on five key facial regions, the image gradient of each key facial region in two consecutive frames is differentially calculated to evaluate the change amplitude of local texture structure, obtain micro-expression tension scores for the five key facial regions, and then stitch them together to obtain five-dimensional micro-expression tension features.
[0015] As a further improvement to this solution, in step S3, the abnormal action recognition is used to identify abnormal actions of ICU critically ill patients. Specifically, based on the three-modal patient image stream and five-dimensional micro-expression tension features, an abnormal action recognition method combining posture and facial information is used to perform abnormal action recognition and obtain abnormal action recognition reference data, including the following steps:
[0016] Step S31: Key point pose extraction, specifically, by constructing a lightweight pose estimation network model, processing each frame of the RGB patient image stream in the three-modal patient image stream, detecting and locating the two-dimensional coordinates of 18 key points on the patient's whole body in real time, and obtaining a set of key point coordinates;
[0017] The patient's 18 key points include 5 facial key points, 6 upper limb key points, 1 trunk key point, and 6 lower limb key points;
[0018] Step S32: Motion amplitude tension recognition, used to quantify the changes in motion amplitude at key points of the patient's body. Specifically, based on the set of key point coordinates, the Euclidean distance between the coordinates of each key point in two consecutive frames of the RGB patient image stream is calculated to obtain motion amplitude tension scores for 18 key points, which are then stitched together to obtain whole-body motion tension features.
[0019] Step S33: Facial movement synchronization scoring, specifically, extracting the movement amplitude tension scores of 5 facial key points from the whole body movement tension features, and splicing them to form facial key point movement tension features. Then, linear mapping is performed on the five-dimensional micro-expression tension features and the facial key point movement tension features respectively. Then, the cosine similarity measure is used to quantify the consistency between facial emotion expression and actual facial movement to obtain the expression movement synchronization scoring index.
[0020] Step S34: Multimodal fusion feature construction, specifically by concatenating five-dimensional micro-expression tension features, full-body movement tension features, and facial expression and movement synchronization scoring indicators to construct multimodal fusion features;
[0021] Step S35: Abnormal action classification, specifically, through graph structure construction, spatiotemporal joint modeling and classification output, abnormal actions are classified to obtain the patient's action category;
[0022] The graph structure is constructed by taking key points as nodes, using the motion amplitude tension score of each key point as node features, and establishing the connection relationship between nodes through spatial connection edge construction and temporal connection edge construction to construct a patient's body structure graph.
[0023] The construction of the spatial connection edge specifically involves establishing spatial connection edges in the same frame of the RGB patient image stream according to the skeleton topology between each key point.
[0024] The construction of the temporal connection edge specifically involves connecting the nodes of the same key point in the previous frame and the current frame in two consecutive frames of the RGB patient image stream to establish a temporal connection edge.
[0025] The spatiotemporal joint modeling specifically involves constructing a temporal graph convolutional network model to perform spatiotemporal joint modeling of the patient's body structure map, thereby generating temporal features of the patient's posture.
[0026] The classification output is specifically obtained by concatenating the patient's posture temporal features and multimodal fusion features to obtain high-level semantic features, and then classifying the high-level semantic features through a gated recurrent unit to obtain the patient's action category.
[0027] Step S36: Abnormal action recognition reference data generation, specifically, through the key point pose extraction, the action amplitude tension recognition, the facial action synchronous scoring, the multimodal fusion feature construction, and the abnormal action classification, abnormal action recognition reference data is obtained. The abnormal action recognition reference data includes multimodal fusion features and patient action categories.
[0028] As a further improvement to this scheme, in step S4, the patient status assessment specifically involves using a status assessment method that combines pain scores and abnormal movement classification based on abnormal movement identification reference data to assess the patient's status and obtain a patient status report, including the following steps:
[0029] Step S41: Pain score prediction, which maps the multimodal fusion features to a CPOT total pain score that meets clinical nursing standards. Specifically, the multimodal fusion features are input into four parallel sub-scoring channels. Each sub-scoring channel corresponds to the four dimensions of facial expression, body activity, ventilation dependence, and muscle tension in the CPOT pain scoring system. In each sub-scoring channel, score prediction is performed through linear mapping and ReLU activation. Then, the score prediction results of each sub-scoring channel are added together to obtain the patient's total pain score.
[0030] Step S42: Status report generation. Specifically, a patient status report is generated based on the patient's action category and total pain score. When the patient's action is abnormal and the total pain score is higher than 5, an alert message is automatically generated for medical staff to refer to.
[0031] The present invention provides a patient abnormal behavior recognition system for ICU critical care, comprising: a patient image acquisition module, a patient expression perception module, an abnormal action recognition module, and a patient status assessment module;
[0032] The patient image acquisition module is used for patient image acquisition, acquiring a trimodal patient image stream from the ICU intensive care unit, and sending the trimodal patient image stream to the patient expression perception module and the abnormal action recognition module;
[0033] The patient expression perception module is used for patient expression perception. Through patient expression perception, it obtains five-dimensional micro-expression tension features and sends the five-dimensional micro-expression tension features to the abnormal action recognition module.
[0034] The abnormal action recognition module is used for abnormal action recognition, obtaining abnormal action recognition reference data through abnormal action recognition, and sending the abnormal action recognition reference data to the patient status assessment module.
[0035] The patient status assessment module is used to assess the patient's status and generate a patient status report.
[0036] The beneficial effects achieved by the present invention using the above solution are as follows:
[0037] (1) In the existing process of identifying abnormal patient behavior, there is a technical problem that critically ill patients are often accompanied by sedation, intubation or impaired consciousness, making it difficult for them to actively express their discomfort. Nursing staff cannot accurately judge the patient's pain status by observing facial expressions, and key symptoms are easily missed. This solution creatively adopts a multimodal vision-driven expression perception method to perceive the patient's expression, integrates multimodal image information, stably identifies subtle dynamic changes in key areas of the patient's face under different lighting conditions, and quantifies the expression of discomfort in the patient's pain situation, thereby assisting the system in reasoning about the patient's potential pain status and effectively improving the accuracy of abnormal behavior identification.
[0038] (2) In the existing process of identifying abnormal behavior in patients, the movements of critically ill patients are subtle and complex. Relying solely on body posture or facial expressions is insufficient to fully reflect the patient's true condition, which leads to the easy misjudgment or neglect of abnormal movements and makes it difficult to meet the clinical needs for real-time and accurate monitoring. This solution creatively adopts an abnormal movement identification method that combines posture and facial information to identify abnormal movements, fully perceive the patient's movements and facial expressions, effectively improve the detection accuracy and response time of abnormal movements, and enhance the early warning capability for patients' pain, discomfort and potential risks.
[0039] (3) In the existing process of identifying abnormal patient behavior, it is difficult to accurately judge the patient's true state based solely on external actions. Actions such as "turning over" or "shaking" are not always high-risk behaviors. Some critically ill patients with relatively good physical function may be making natural adjustments to their body position due to discomfort. This can easily lead to false alarms or inappropriate interventions, affecting the effectiveness of clinical monitoring. This solution creatively adopts a state assessment method that combines pain scores and abnormal action classification to assess the patient's state. This enables a multi-faceted understanding of the patient's abnormal behavior and allows for the assessment of the patient's pain level while the action is abnormal, thereby providing medical staff with more timely and reliable early warning information. Attached Figure Description
[0040] Figure 1 This is a flowchart illustrating a method for identifying abnormal patient behaviors in ICU critical care provided by the present invention.
[0041] Figure 2 This is a schematic diagram of a patient abnormal behavior recognition system for ICU critical care provided by the present invention;
[0042] Figure 3 This is a flowchart illustrating step S3.
[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation
[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0045] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0046] Example 1, see Figure 1 The present invention provides a method for identifying abnormal patient behaviors in ICU critical care, the method comprising the following steps:
[0047] Step S1: Patient image acquisition;
[0048] Step S2: Patient facial expression perception;
[0049] Step S3: Abnormal action identification;
[0050] Step S4: Patient status assessment.
[0051] Example 2, see Figure 1 This embodiment is based on the above embodiment. In step S1, the patient image acquisition specifically involves deploying RGB cameras, near-infrared cameras, and depth cameras in the ICU intensive care unit to monitor the ICU intensive care unit in real time, and using a frame synchronization mechanism to unify timestamps and collaboratively acquire three-modal patient image streams.
[0052] The trimodal patient image stream includes an RGB patient image stream, a near-infrared patient image stream, and a depth patient image stream.
[0053] Example 3, see Figure 1This embodiment is based on the above embodiment. In step S2, the patient expression perception is used to construct the micro-expression features of ICU critically ill patients. Specifically, based on the three-modal patient image stream, a multi-modal vision-driven expression perception method is used to perceive patient expressions and obtain five-dimensional micro-expression tension features, including the following steps:
[0054] Step S21: Facial region recognition, specifically, involves constructing a deep perception segmentation network model to extract facial region classification probability maps from each frame of the trimodal patient image stream. Then, the trimodal facial region classification probability maps are weighted and fused pixel-wise to calculate the comprehensive facial region probability, extracting five categories of key facial regions. The calculation formula is as follows:
[0055] ;
[0056] ;
[0057] In the formula, R face (i) represents the region category corresponding to the i-th pixel, where i is the pixel index, and argmax is the region category. c (·) represents the region category that maximizes the overall probability of the facial region, P. total (i,c) represents the combined facial region probability that the i-th pixel belongs to the c-th region category, where c is the region category index. It is the RGB modal weight, P RGB (i,c) is the probability that the i-th pixel in the RGB modality belongs to the c-th region category. It is the near-infrared mode weight, P NIR (i,c) is the probability that the i-th pixel in the near-infrared mode belongs to the c-th region category. It is the depth modality weight, P ToF (i,c) is the probability that the i-th pixel in the depth modality belongs to the c-th region category;
[0058] The five key facial areas include the glabella area, eyelid area, nasal wing area, corner of mouth area, and jaw area;
[0059] Preferably, the deep perception segmentation network model is the BiFusion-Seg model;
[0060] Step S22: Micro-expression tension recognition, used to quantify pain-related micro-expression dynamic features. Specifically, based on five key facial regions, the image gradient of each key facial region in two consecutive frames is differentially calculated to evaluate the magnitude of changes in local texture structure, obtaining micro-expression tension scores for the five key facial regions. These scores are then concatenated to obtain five-dimensional micro-expression tension features. The calculation formula is as follows:
[0061] ;
[0062] ;
[0063] In the formula, T c (t) represents the micro-expression tension score for the c-th region category at time step t, U is the pixel set for the c-th region category, and ||·|| is the L2 norm operator used to measure Euclidean distance. It is the image gradient of the c-th region category at time step t, at the i-th pixel. F is the image gradient of the c-th region category at the i-th pixel at time step t-1. expr (t) represents the five-dimensional micro-expression tension features, concat(·) is the concatenation operation, and T eyebro (t) is the micro-expression tension score in the glabella area, T eyelid (t) is the micro-expression tension score in the eyelid region, T nostril (t) is the micro-expression tension score of the nasal alar region, T mouth (t) is the micro-expression tension score of the corner of the mouth area, T jaw (t) is the micro-expression tension score of the mandibular region.
[0064] By performing the above operations, this solution addresses the technical problem in existing patient abnormal behavior identification processes: critically ill patients are often sedated, intubated, or have impaired consciousness, making it difficult for them to actively express their discomfort. Nursing staff cannot accurately assess the patient's pain state through facial expressions, easily leading to missed diagnoses of key symptoms. This solution creatively employs a multimodal vision-driven expression perception method to perceive patient expressions. It integrates multimodal image information to stably identify subtle dynamic changes in key areas of the patient's face under different lighting conditions and quantifies the expression of discomfort in patients experiencing pain. This assists the system in inferring the patient's potential pain state, effectively improving the accuracy of abnormal behavior identification.
[0065] Example 4, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S3, the abnormal action recognition is used to identify abnormal actions of ICU critically ill patients. Specifically, it uses an abnormal action recognition method that combines posture and facial information based on the three-modal patient image stream and five-dimensional micro-expression tension features to perform abnormal action recognition and obtain abnormal action recognition reference data. This includes the following steps:
[0066] Step S31: Key point pose extraction, specifically, by constructing a lightweight pose estimation network model, processing each frame of the RGB patient image stream in the three-modal patient image stream, detecting and locating the two-dimensional coordinates of 18 key points on the patient's whole body in real time, and obtaining a set of key point coordinates;
[0067] The patient's 18 key points include 5 facial key points, 6 upper limb key points, 1 trunk key point, and 6 lower limb key points;
[0068] The five key facial points include the tip of the nose, the left eye, the right eye, the left corner of the mouth, and the right corner of the mouth;
[0069] The six key points of the upper limbs include the left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist;
[0070] The one key point of the torso includes the center of the neck;
[0071] The six key points of the lower limbs include the left hip, right hip, left knee, right knee, left ankle, and right ankle;
[0072] Preferably, the lightweight pose estimation network model is the YOLOPose model;
[0073] Step S32: Motion amplitude tension recognition, used to quantify the changes in motion amplitude at various key points of the patient's body. Specifically, based on the set of key point coordinates, the Euclidean distance between the coordinates of each key point in two consecutive frames of the RGB patient image stream is calculated to obtain motion amplitude tension scores for 18 key points. These scores are then stitched together to obtain the whole-body motion tension features. The calculation formula is as follows:
[0074] ;
[0075] ;
[0076] In the formula, A v (t) is the Euclidean distance of the v-th keypoint at time step t, used to represent the motion amplitude tension score, where v is the keypoint index, and q is the distance between the v-th keypoint and the v-th keypoint index. v (t) represents the two-dimensional coordinates of the v-th keypoint at time step t, and q v (t-1) is the two-dimensional coordinate of the v-th keypoint at time step t-1, F motion (t) represents the overall body tension characteristic, A1(t) is the Euclidean distance of the first keypoint at time step t, and A2(t) is the Euclidean distance of the second keypoint at time step t. 18 (t) is the Euclidean distance of the 18th keypoint at time step t;
[0077] Step S33: Facial movement synchronization scoring. Specifically, this involves extracting the movement amplitude tension scores of five facial key points from the whole-body movement tension features, and concatenating them to form facial key point movement tension features. Then, linear mapping is performed on the five-dimensional micro-expression tension features and the facial key point movement tension features, respectively. Finally, cosine similarity measurement is used to quantify the consistency between facial emotion expression and actual facial movement, resulting in an expression-movement synchronization scoring index. The calculation formula is as follows:
[0078] ;
[0079] ;
[0080] In the formula, It is the five-dimensional micro-expression tension feature after linear mapping, W e It is the linear mapping weight of micro-expressions, b e It is the bias term of the micro-expression linear mapping. It is the facial key point motion tension feature after linear mapping, W m It is a linear mapping weight of facial key point movements. It is the facial key point movement tension characteristics, b m It is the linear mapping bias term for facial keypoint movements, and Sync(t) is the facial expression and movement synchronization scoring index. cos (·) is the cosine similarity measure function;
[0081] Step S34: Multimodal fusion feature construction, specifically, involves concatenating five-dimensional micro-expression tension features, full-body motion tension features, and facial expression / motion synchronization scoring indicators to construct multimodal fusion features. The calculation formula is as follows:
[0082] ;
[0083] In the formula, Z(t) is the multimodal fusion feature;
[0084] Step S35: Abnormal action classification, specifically, through graph structure construction, spatiotemporal joint modeling and classification output, abnormal actions are classified to obtain the patient's action category;
[0085] The graph structure is constructed by taking key points as nodes, using the motion amplitude tension score of each key point as node features, and establishing the connection relationship between nodes through spatial connection edge construction and temporal connection edge construction to construct a patient's body structure graph.
[0086] The construction of the spatial connection edges specifically involves establishing spatial connection edges in the same frame of the RGB patient image stream according to the skeleton topology between each key point. Preferably, in the facial region, the tip of the nose is connected to the left and right eyes, and the tip of the nose is connected to the left and right corners of the mouth. In the upper limb region, the left shoulder is connected to the left elbow, the left elbow to the left wrist, the right shoulder to the right elbow, the right elbow to the right wrist, and the center of the neck to the left and right shoulders. In the lower limb region, the left hip is connected to the left knee, the left knee to the left ankle, the right hip to the right knee, and the right knee to the right ankle.
[0087] The construction of the temporal connection edge specifically involves connecting the nodes of the same key point in the previous frame and the current frame in two consecutive frames of the RGB patient image stream to establish a temporal connection edge.
[0088] The spatiotemporal joint modeling specifically involves constructing a temporal graph convolutional network model to perform spatiotemporal joint modeling of the patient's body structure map, thereby generating temporal features of the patient's posture.
[0089] The classification output is specifically obtained by concatenating the patient's posture temporal features and multimodal fusion features to obtain high-level semantic features, and then classifying the high-level semantic features through a gated recurrent unit to obtain the patient's action category.
[0090] The patient's movement categories include normal, struggling, tube removal, turning over, shaking, and falling;
[0091] Step S36: Abnormal action recognition reference data generation, specifically, through the key point pose extraction, the action amplitude tension recognition, the facial action synchronous scoring, the multimodal fusion feature construction, and the abnormal action classification, abnormal action recognition reference data is obtained. The abnormal action recognition reference data includes multimodal fusion features and patient action categories.
[0092] By performing the above operations, this solution addresses the technical problem in the existing process of identifying abnormal patient behavior. The movements of critically ill patients are subtle and complex, and relying solely on body posture or facial expressions is insufficient to fully reflect the patient's true condition, leading to misjudgments or oversights of abnormal movements and failing to meet the clinical need for real-time, precise monitoring. This solution creatively adopts an abnormal movement recognition method that combines posture and facial information. This comprehensively perceives patient movements and facial expressions, effectively improving the detection accuracy and response time of abnormal movements, and enhancing the early warning capability for patient pain, discomfort, and potential risks.
[0093] Example 5, see Figure 1This embodiment is based on the above embodiment. In step S4, the patient status assessment specifically involves using a status assessment method that combines pain scores and abnormal movement classification based on abnormal movement identification reference data to assess the patient status and obtain a patient status report. This includes the following steps:
[0094] Step S41: Pain score prediction, used to map multimodal fusion features to a CPOT total pain score that conforms to clinical nursing standards. Specifically, the multimodal fusion features are input into four parallel sub-scoring channels. Each sub-scoring channel corresponds to one of the four dimensions of the CPOT pain scoring system: facial expression, body activity, ventilation dependence, and muscle tension. Score prediction is performed in each sub-scoring channel using linear mapping and ReLU activation. The prediction results of each sub-scoring channel are then summed to obtain the patient's total pain score. The calculation formula is as follows:
[0095] ;
[0096] ;
[0097] In the formula, S CPOT (t) is the patient's overall pain score, k is the sub-score channel index, and s k (t) is the predicted rating result for the k-th sub-rating channel, ReLU(·) is the ReLU activation function, and W k It is the linear mapping weight of the k-th sub-rating channel, b k It is the linear mapping bias term of the k-th sub-scoring channel;
[0098] Step S42: Status report generation. Specifically, a patient status report is generated based on the patient's action category and total pain score. When the patient's action is abnormal and the total pain score is higher than 5, an alert message is automatically generated for medical staff to refer to.
[0099] The patient's abnormal movements included struggling, pulling out tubes, turning over, shaking, and falling.
[0100] By performing the above operations, this solution addresses the technical problem in the existing process of identifying abnormal patient behavior. It is difficult to accurately determine the patient's true condition based solely on external actions. Actions such as "turning over" or "shaking" are not always high-risk behaviors; some critically ill patients with relatively good physical function may be making natural postural adjustments due to discomfort. This can easily lead to false alarms or inappropriate interventions, affecting the effectiveness of clinical monitoring. This solution creatively adopts a state assessment method that combines pain scores and abnormal action classification to assess the patient's state. This enables a multi-faceted understanding of abnormal patient behavior and allows for the assessment of the patient's level of pain while observing abnormal actions, thus providing medical staff with more timely and reliable early warning information.
[0101] Example 6, see Figure 2 Based on the above embodiments, this embodiment provides a patient abnormal behavior recognition system for ICU critical care, comprising: a patient image acquisition module, a patient expression perception module, an abnormal action recognition module, and a patient status assessment module;
[0102] The patient image acquisition module is used for patient image acquisition, acquiring a trimodal patient image stream from the ICU intensive care unit, and sending the trimodal patient image stream to the patient expression perception module and the abnormal action recognition module;
[0103] The patient expression perception module is used for patient expression perception. Through patient expression perception, it obtains five-dimensional micro-expression tension features and sends the five-dimensional micro-expression tension features to the abnormal action recognition module.
[0104] The abnormal action recognition module is used for abnormal action recognition, obtaining abnormal action recognition reference data through abnormal action recognition, and sending the abnormal action recognition reference data to the patient status assessment module.
[0105] The patient status assessment module is used to assess the patient's status and generate a patient status report.
[0106] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0107] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.
[0108] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.
Claims
1. A method for identifying abnormal patient behaviors in ICU critical care, characterized in that: The method includes the following steps: Step S1: Patient image acquisition, obtaining a three-modal patient image stream; Step S2: Patient expression perception, specifically based on the three-modal patient image stream, using a multimodal vision-driven expression perception method to perceive patient expressions and obtain five-dimensional micro-expression tension features, including the following steps: Step S21 facial region recognition and Step S22 micro-expression tension recognition; Step S3: Abnormal action recognition, specifically based on the three-modal patient image stream and five-dimensional micro-expression tension features, adopts an abnormal action recognition method that combines posture and facial information to perform abnormal action recognition and obtain abnormal action recognition reference data, including the following steps: Step S31 Key point posture extraction, Step S32 Action amplitude tension recognition, Step S33 Simultaneous facial action scoring, Step S34 Multimodal fusion feature construction, Step S35 Abnormal action classification, and Step S36 Abnormal action recognition reference data generation; In step S35, the abnormal action classification specifically involves classifying abnormal actions through graph structure construction, spatiotemporal joint modeling, and classification output to obtain the patient's action category; Step S4: Patient status assessment. Specifically, based on the abnormal movement identification reference data, a status assessment method combining pain score and abnormal movement classification is used to assess the patient status and obtain a patient status report. This includes the following steps: Step S41 Pain score prediction and Step S42 Status report generation.
2. The method for identifying abnormal patient behavior in ICU critical care according to claim 1, characterized in that: In step S35, the graph structure construction specifically involves using key points as nodes, the motion amplitude tension score of each key point as node features, and establishing connection relationships between nodes through spatial connection edge construction and temporal connection edge construction to construct a patient's body structure graph. The construction of the spatial connection edge specifically involves establishing spatial connection edges in the same frame of the RGB patient image stream according to the skeleton topology between each key point. The construction of the temporal connection edge specifically involves connecting the nodes of the same key point in the previous frame and the current frame in two consecutive frames of the RGB patient image stream to establish a temporal connection edge. The spatiotemporal joint modeling specifically involves constructing a temporal graph convolutional network model to perform spatiotemporal joint modeling of the patient's body structure map, thereby generating temporal features of the patient's posture. The classification output is specifically obtained by concatenating the patient's posture temporal features and multimodal fusion features to obtain high-level semantic features, and then classifying the high-level semantic features through a gated recurrent unit to obtain the patient's action category.
3. The method for identifying abnormal patient behavior in ICU critical care according to claim 2, characterized in that: In step S21, the facial region recognition specifically involves constructing a deep perception segmentation network model to extract facial region classification probability maps from each frame of the trimodal patient image stream. Then, by weighting and fusing the trimodal facial region classification probability maps by pixels, a comprehensive facial region probability is calculated, and five types of key facial regions are extracted. In step S22, the micro-expression tension recognition is used to quantify pain-related micro-expression dynamic features. Specifically, based on five types of key facial regions, the image gradient of each key facial region in two consecutive frames is differentially calculated to evaluate the change amplitude of local texture structure, obtain micro-expression tension scores for the five types of key facial regions, and then stitch them together to obtain five-dimensional micro-expression tension features.
4. The method for identifying abnormal patient behavior in ICU critical care according to claim 3, characterized in that: In step S31, the key point pose extraction specifically involves constructing a lightweight pose estimation network model to process each frame of the RGB patient image stream in the trimodal patient image stream, and to detect and locate the two-dimensional coordinates of 18 key points on the patient's whole body in real time, thereby obtaining a set of key point coordinates. The patient's 18 key points include 5 facial key points, 6 upper limb key points, 1 trunk key point, and 6 lower limb key points; In step S32, the motion amplitude tension recognition is used to quantify the changes in motion amplitude at various key points of the patient's body. Specifically, based on the set of key point coordinates, the Euclidean distance between the coordinates of each key point in two consecutive frames of the RGB patient image stream is calculated to obtain motion amplitude tension scores for 18 key points, which are then stitched together to obtain whole-body motion tension features.
5. The method for identifying abnormal patient behavior in ICU critical care according to claim 4, characterized in that: In step S33, the facial movement synchronization scoring specifically involves extracting the movement amplitude tension scores of five facial key points from the whole-body movement tension features, splicing them together to form facial key point movement tension features, then performing linear mapping on the five-dimensional micro-expression tension features and facial key point movement tension features respectively, and then using cosine similarity measurement to quantify the consistency between facial emotion expression and actual facial movement to obtain the expression movement synchronization scoring index. In step S34, the construction of the multimodal fusion feature specifically involves splicing together the five-dimensional micro-expression tension feature, the whole-body movement tension feature, and the facial expression and movement synchronization scoring index to construct the multimodal fusion feature.
6. A method for identifying abnormal patient behavior in ICU critical care according to claim 5, characterized in that: In step S36, the abnormal action recognition reference data is generated specifically by extracting the pose of key points, recognizing the amplitude and tension of movements, scoring facial movements synchronously, constructing multimodal fusion features, and classifying abnormal actions. The abnormal action recognition reference data includes multimodal fusion features and patient movement categories.
7. A method for identifying abnormal patient behavior in ICU critical care according to claim 6, characterized in that: In step S41, the pain score prediction is used to map the multimodal fusion features into a CPOT pain total score that meets clinical nursing standards. Specifically, the multimodal fusion features are input into four parallel sub-scoring channels. Each sub-scoring channel corresponds to the four dimensions of facial expression, body activity, ventilation dependence, and muscle tension in the CPOT pain scoring system. In each sub-scoring channel, score prediction is performed through linear mapping and ReLU activation. Then, the score prediction results of each sub-scoring channel are added together to obtain the patient's total pain score. In step S42, the status report is generated specifically by generating a patient status report based on the patient's action category and total pain score. When the patient's action is abnormal and the total pain score is higher than 5, a reminder message is automatically generated for medical staff to refer to.
8. A method for identifying abnormal patient behavior in ICU critical care according to claim 7, characterized in that: In step S1, the patient image acquisition specifically involves deploying RGB cameras, near-infrared cameras, and depth cameras in the ICU intensive care unit to monitor the ICU intensive care unit in real time, and using a frame synchronization mechanism to unify timestamps and collaboratively acquire a three-modal patient image stream. The trimodal patient image stream includes an RGB patient image stream, a near-infrared patient image stream, and a depth patient image stream.
9. A patient abnormal behavior recognition system for ICU critical care, used to implement the patient abnormal behavior recognition method for ICU critical care as described in any one of claims 1-8, characterized in that: It includes a patient image acquisition module, a patient expression perception module, an abnormal action recognition module, and a patient status assessment module; The patient image acquisition module is used for patient image acquisition, acquiring a trimodal patient image stream from the ICU intensive care unit, and sending the trimodal patient image stream to the patient expression perception module and the abnormal action recognition module; The patient expression perception module is used for patient expression perception. Through patient expression perception, it obtains five-dimensional micro-expression tension features and sends the five-dimensional micro-expression tension features to the abnormal action recognition module. The abnormal action recognition module is used for abnormal action recognition, obtaining abnormal action recognition reference data through abnormal action recognition, and sending the abnormal action recognition reference data to the patient status assessment module. The patient status assessment module is used to assess the patient's status and generate a patient status report.