Multi-mode pain assessment method and pain monitoring system for ICU (Intensive Care Unit) patient

Abstract

The invention belongs to the field of medical treatment, and particularly relates to a multi-mode pain assessment method and a pain monitoring system for an ICU patient. The method comprises the following steps: constructing a pain recognition model for generating a pain state of a patient according to multi-modal data; the pain recognition model comprises a feature extraction module, a feature fusion module, a personalized module and a pain evaluation module; basic information, face videos and physiological monitoring data of ICU patients meeting requirements are obtained to form multi-modal data, and label information is added according to CPOT pain assessment results; setting a loss function according to task types to obtain a large number of sample data sets; the training set, the test set and the verification set are divided into a training set, a test set and a verification set for training, testing and verifying the pain recognition module; the validated model is retained and used for pain assessment of the ICU patient. According to the invention, the defects of hysteresis and insufficient reliability of a patient pain assessment scheme adopted in the existing ICU are overcome.

Description

Technical Field

[0001] This invention belongs to the medical field, specifically relating to a multimodal pain assessment method for ICU patients, its corresponding computer program product, and a pain monitoring system for ICU wards. Background Technology

[0002] Pain is a major symptom in intensive care unit (ICU) patients, manifesting as a complex sensory and emotional experience caused by actual or potential tissue damage, inflammation, or psychological stress. Most critically ill patients experience moderate to severe pain. Sources of pain include underlying diseases, inflammatory responses, endotracheal intubation, prolonged immobilization, and related treatments and procedures (such as repositioning and suctioning). Pain responses increase oxygen demand, endogenous catecholamine release, and excessive metabolism, thus delaying patient recovery and negatively impacting the condition of critically ill patients (e.g., cardiac instability, impaired respiratory function, and immunosuppression). Untreated conditions can reduce treatment effectiveness, cause unnecessary suffering, and negatively affect the quality of life of patients after ICU discharge. Therefore, pain is one of the important symptoms that requires attention in critically ill patients. Continuous and objective monitoring of pain is a prerequisite for precise adjustment of analgesics and sedatives in clinical practice; it also helps in preventative analgesia to avoid severe surgical pain and adverse events.

[0003] Pain assessment methods in clinical settings primarily include self-assessment tools and behavioral observation scales. Critically ill patients, due to their altered clinical condition, level of consciousness, mechanical ventilation, or sedation, often cannot accurately report their pain. Commonly used pain assessment tools include the Behavioral Pain Scale (BPS) and the Critical Care Pain Observation Tool (CCPOT). Similar to emotional expression, facial expressions are an important indicator of pain assessment. Many researchers have made significant efforts in developing comprehensive, automated pain assessment based on facial features. However, as... Figure 1 As shown, most ICU patients require invasive endotracheal intubation or non-invasive mask ventilation, resulting in severe facial obstruction. This affects the reliability of facial-based pain assessment protocols. In recent years, advancements in computer vision and machine learning technologies have also driven the development of automated pain recognition.

[0004] Pain is multidimensional and subjective, and rating scales alone cannot objectively and comprehensively reflect it. Although widely used, they are particularly inconvenient for critically ill patients with impaired communication abilities. Furthermore, these methods cannot achieve continuous dynamic monitoring, easily leading to delays in adjusting sedation and analgesia medications. Therefore, developing a multidimensional, automated, and continuous pain assessment system tailored to the characteristics of critically ill patients in the ICU is extremely important for the sedation and analgesia treatment and management of ICU patients. However, current technologies still lack corresponding solutions. Summary of the Invention

[0005] To address the shortcomings of existing pain assessment schemes, such as lag and insufficient reliability, and the lack of online pain assessment methods suitable for ICU patients, this invention provides a multimodal pain assessment method for ICU patients, along with a corresponding computer program product and a pain monitoring system for ICU wards.

[0006] This invention is achieved using the following technical solution: A multimodal pain assessment method for ICU patients includes the following steps: Construct a pain recognition model to generate a patient's pain state based on input multimodal data. The pain recognition model includes a feature extraction module, a feature fusion module, a personalization module, and a pain assessment module.

[0007] The feature extraction module includes physiological and facial branches. The physiological branch uses MLP and activation functions to map physiological monitoring data into physiological features of a specified dimension. The facial branch performs frame segmentation, face recognition, and keypoint detection on facial videos to segment local images containing pain semantic information, and then extracts facial features from the local features using facial attention. The fusion module concatenates physiological and facial features, and then generates fused features based on the sequence features of the concatenated vectors using a temporal attention module. The personalization module processes the patient's basic information through a text extraction network and generates corresponding moderating factors; then, the moderating factors are multiplied by the fused features to obtain the pain features. The pain assessment module consists of fully connected layers, ReLU activation layers, Dropout layers, and an output layer, and is used to generate the patient's pain state based on the pain features.

[0008] Basic information of eligible ICU patients, along with facial videos and physiological monitoring data during hospitalization, were collected to form multimodal data. CPOT pain assessments were performed on ICU patients by medical staff, and the assessment results served as multimodal labeling information. A large amount of labeled multimodal data was acquired as sample data to form a sample dataset.

[0009] The sample dataset was divided into training, testing, and validation sets. The loss function was set according to the task type of the pain assessment module, and the pain recognition module was trained, tested, and validated. The model parameters that met the performance requirements after validation were retained and used for pain assessment of ICU patients.

[0010] As a further improvement of the present invention, the patient's basic information in the multimodal data includes: medical record number, name, gender, age, height, weight, disease type, past medical history, laboratory and diagnostic information.

[0011] As a further improvement of the present invention, the physiological monitoring data includes any one or more of the following: heart rate, blood pressure, respiratory rate, blood oxygen, and electrocardiogram.

[0012] When physiological monitoring data includes multiple types, the physiological branch in the feature extraction module performs feature extraction on each type of physiological monitoring signal to generate multiple corresponding physiological features.

[0013] As a further improvement of this invention, during the sample data collection phase, patients older than 18 years who were transferred to the ICU after surgery and were able to complain of pain were defined as eligible ICU patients. Pain status assessments were performed on ICU patients at any two manipulation stimuli, and at 30 min, 1 h, 8 h, and 12 h after transfer to the ICU in a resting state, to generate corresponding time-period label information.

[0014] Among the collected multimodal data, facial video and physiological monitoring data were collected simultaneously; basic information was obtained through the hospital's HIS system; and physiological monitoring data was collected in real time through various physiological monitoring devices worn by patients.

[0015] As a further improvement of the present invention, the process of generating facial features based on facial video by the face branch includes: OpenCV was used to convert facial videos into frame-by-frame images. The Mediapipe face detection model was employed to perform face recognition and key point detection on each frame. Key facial activity units (MAUs) containing pain semantic information and unobstructed features were identified, and corresponding local images were segmented. These key MAUs included lowering eyebrows, tightening eye sockets, contracting levator muscles, and closing eyes. Each segmented local image was then fed into a facial feature extraction unit composed of facial attention and convolutional networks to extract region features. The facial representation was obtained by vertically concatenating the region features of all associated local images within the same frame.

[0016] As a further improvement of the present invention, in the facial feature extraction unit, the input local image is processed by a facial attention module consisting of a ResNet-50 feature extraction layer and a fully connected layer to obtain the regional features of each local image.

[0017] As a further improvement of the present invention, in the feature fusion module, the time-synchronized physiological features and facial features are spliced ​​together in chronological order. The spliced ​​features arranged in time are input into the time-series attention module for processing in order to capture the rich relationships between features at different frame times.

[0018] The temporal attention module comprises multiple attention layers connected in sequence and an MLP classification head. Each attention layer includes a multi-head self-attention module, a multi-attention random deactivation module, a pre-summing and normalization layer, a multilayer perceptron, and a post-summing and normalization layer, all connected in sequence. Furthermore, the input of the multi-head self-attention module is residually concatenated with the output of the multi-attention random deactivation module and used as the input to the pre-summing and normalization layer. The output of the pre-summing and normalization layer is residually concatenated with the output of the multilayer perceptron and used as the input to the post-summing and normalization layer.

[0019] As a further improvement of the present invention, the personalized module of the pain recognition model also includes a voice assistance unit; the voice assistance unit is used to classify the input patient's voice data; and generate a weight coefficient based on the classification result to weight the fusion features.

[0020] The speech classification results of the speech-assisted unit include three categories: normal speech, sighing, and screaming. When the patient speaks normally, the fusion vector is multiplied by a weight coefficient of 1; when the patient sighs, the fusion vector is multiplied by a weight coefficient of 1.2; and when the patient screams, the fusion vector is multiplied by a weight coefficient of 1.5.

[0021] As a further improvement of the present invention, during the training phase of the pain recognition model, when the pain assessment module is used to perform classification tasks and achieve pain grading, cross-entropy loss is used as the loss function to minimize the relative entropy between the observed and estimated values.

[0022] When the pain assessment module is used to perform regression tasks and generate pain quantification values, the mean squared error loss is used as the loss function to minimize the mean of the sum of squares of the errors between corresponding points in the predicted data and the original data.

[0023] The present invention also includes a computer program product comprising a computer program that, when executed by a processor, creates a pain recognition model trained in the aforementioned multimodal pain assessment method for ICU patients, thereby enabling the identification of the pain status of ICU patients based on input multimodal data.

[0024] The present invention also includes a pain monitoring system for an ICU ward, comprising: an audio-visual component, a data acquisition component, a pain recognition component, and a decision support component.

[0025] The audio and video components include a camera and may also include a microphone. The camera is used to capture video stream data including the patient's face; the microphone is used to capture the patient's voice data.

[0026] The data acquisition component is used to acquire video stream data and possibly audio data collected by the audio-visual component; and to acquire the patient's basic information and physiological monitoring data. The synchronously acquired video stream data, physiological monitoring data, and audio data, together with the basic information registered by the patient upon admission, constitute the patient's multimodal data.

[0027] The pain recognition component includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a pain recognition model trained in the aforementioned multimodal pain assessment method for ICU patients, thereby enabling the recognition of the pain status of ICU patients based on the input multimodal data.

[0028] The decision support component comprises an analysis unit, a decision-making unit, and a feedback unit. The analysis unit integrates the patient's pain status, liver and kidney function indicators, and pharmacokinetic parameters to predict the plasma concentration-time curves of various sedative-analgesics during metabolism in the current patient, and generates individualized medication and dosage adjustment recommendations for sedative-analgesics. The decision-making unit dynamically adjusts the current patient's dosing regimen based on a reinforcement learning framework to balance analgesic efficacy with adverse reaction risks. The feedback unit continuously collects the patient's physiological indicators after medication via bedside monitoring devices and analyzes the drug efficacy decay trend using a sliding time window to trigger dose reassessment, thereby achieving closed-loop optimization of sedative-analgesic drug decision-making.

[0029] The technical solution provided by this invention has the following beneficial effects: This invention designs a novel pain recognition model tailored to the characteristics and data acquisition conditions of ICU patients. This network model deeply integrates multimodal data, overcoming the limitations of single-modal assessment. It combines dynamic changes in facial expressions with autonomic nervous system responses (such as variability in various electrical signals, blood pressure, respiration, and heart rate) to establish a multimodal pain recognition model, thereby improving the specificity and sensitivity of pain recognition. To address group and individual differences among patients, this invention further introduces a moderating factor generated based on basic patient admission information and a weighting coefficient generated by combining user voice classification. These two factors optimize the features generated by the network model, enhancing its generalization ability.

[0030] Multimodal data contains numerous dimensions. Based on the collection, linking, and annotation of clinical data, this invention develops and upgrades modeling techniques to simultaneously process large-scale image, physiological signal, or text data, establishing a large database for multimodal pain assessment of critically ill patients. This database enables refined, fine-grained data analysis and application. Using the trained pain recognition model, this invention can also monitor patients' pain status in real time and develop individualized sedation and analgesia plans, achieving a closed-loop management system of real-time dynamic pain assessment at the bedside, sedation and analgesia drug intervention, and efficacy feedback, thus solving the lag problem of traditional intermittent assessment. Attached Figure Description

[0031] Figure 1 This is a photograph of an ICU patient under mechanical ventilation, as described in the background section.

[0032] Figure 2 This is a flowchart of the multimodal pain assessment method for ICU patients provided in Embodiment 1 of the present invention.

[0033] Figure 3 This is a network architecture diagram of the pain recognition model constructed in Embodiment 1 of the present invention.

[0034] Figure 4 This is a structural diagram of the temporal attention module used in the pain recognition model of Embodiment 1 of the present invention.

[0035] Figure 5 This is a flowchart illustrating the operation of the decision support component in Embodiment 3 of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0037] Example 1 Given that most existing pain assessment schemes based on facial recognition and physiological information are trained and tested on publicly available datasets, with subjects being recruited adults whose faces are unobstructed, there are significant differences compared to ICU patients in terms of age, physiological state, and treatment methods. Therefore, network models trained on publicly available data cannot be directly applied in clinical settings. To address this issue, this embodiment provides a multimodal pain assessment method specifically for ICU patients. This method designs a novel pain recognition model architecture tailored to the physiological characteristics of ICU patients and their medical monitoring conditions, thereby improving the specificity and sensitivity of pain recognition. This network model can achieve more accurate online monitoring of the patient's pain state based on multimodal data such as real-time facial video, physiological monitoring signals, and basic information from admission registration. Furthermore, it enables refined, fine-grained data analysis and application to optimize the medication regimens for sedatives and analgesics in ICU patients, reducing pain levels, minimizing adverse drug events, improving dosage adjustment accuracy, and ensuring optimal patient recovery.

[0038] In this embodiment, the pain recognition model extracts various physiological monitoring signals from the patient in real time and inputs them into a feature mapping network to improve its representation dimensionality. Facial video stream data is acquired, segmented into frames, and then subjected to feature point detection, image segmentation, convolution, and facial attention processing to extract facial features. Next, all features are fused and input into the Transformer temporal attention module to learn the relationships between sequences; and a moderating factor is designed based on the patient's basic information to overcome patient variability. Finally, the pain assessment module monitors the patient's pain state online based on the extracted feature information and outputs corresponding prediction results.

[0039] Specifically, such as Figure 2 As shown, the multimodal pain assessment method for ICU patients provided in this embodiment includes the following steps: I. Constructing a pain recognition model In the technical solution provided in this embodiment, the pain recognition model is a network model used to generate the patient's pain state based on the input multimodal data. Among the multimodal data input to the pain recognition model, facial video refers to real-time video stream data containing the patient's face. Physiological monitoring data refers to any one or more of the following: heart rate, blood pressure, respiratory rate, blood oxygen saturation, electrocardiogram, etc. These data have been proven in previous studies to be related to the patient's pain state and contain key information for assessing the patient's pain level. The raw signals of the above-mentioned physiological monitoring data can be collected using existing life monitoring equipment in the ICU ward and extracted and analyzed using tools such as Pandas; no additional hardware for signal acquisition is required. The patient's basic information in the multimodal data may include: medical record number, name, gender, age, height, weight, disease type, past medical history, laboratory and diagnostic information, etc. This information is usually registered proactively by the patient upon admission; in this embodiment, the patient's basic information can be collected directly through the hospital's HIS (Hospital Information System).

[0040] In detail, in this embodiment, the network architecture of the pain recognition model is as follows: Figure 3 As shown, the model mainly includes a feature extraction module, a feature fusion module, a personalization module, and a pain assessment module. The feature extraction module includes a physiological branch and a facial branch. The physiological branch uses MLP and activation functions to map physiological monitoring data into physiological features of a specified dimension; the facial branch performs frame segmentation, face recognition, and keypoint detection on facial videos to segment local images containing pain semantic information, and then uses facial attention and convolutional networks to extract facial features from the local features. The feature fusion module concatenates physiological and facial features, and then uses a temporal attention module to generate fused features based on the sequence features of the concatenated vectors. The personalization module processes the patient's basic information through a text extraction network and generates corresponding moderating factors; then, the moderating factors are multiplied by the fused features to obtain the pain features. The pain assessment module generates the patient's pain state based on the pain features.

[0041] To make the principles and performance advantages of the pain recognition model in this embodiment more apparent, the four functional modules in the model are described in detail below: 1.1 Feature Extraction Module The feature extraction module is used to extract feature information related to the patient's pain level from the collected multimodal data. This embodiment overcomes the limitations of single-modal assessment by comprehensively extracting dynamic changes in the patient's facial expressions from facial videos and autonomic nervous system responses (such as variability in various electrical signals, blood pressure, respiration, and heart rate) from physiological monitoring data. Furthermore, it utilizes the extracted multidimensional information to jointly model the user's pain level, thereby obtaining richer semantic information and improving the specificity and sensitivity of pain recognition.

[0042] In this embodiment, pain-related feature information mainly comes from facial videos and physiological monitoring data in multimodal data. Therefore, the feature extraction module of the designed pain recognition model contains two feature extraction branches. One branch is used to extract physiological features representing the user's pain level from various physiological monitoring signals; this embodiment refers to this branch as the physiological branch. The other branch is used to identify the patient's face from the facial video, locate and segment key regions of the patient's face, and then extract the problem features and other deeper information contained therein to obtain facial features representing the user's pain level. This embodiment refers to this branch as the facial branch.

[0043] In the physiological branch, raw physiological monitoring data is transmitted by various monitors at the second-by-second level. In practical applications, technicians can use Pandas tools to extract signals such as heart rate, blood pressure, and respiratory rate, and then perform Kalman filtering on them. At the time axis level, to accurately assess changes in the patient's pain state at a given moment, the sampling frequency of physiological parameters should correspond to the frame rate of the facial video; this allows the feature extraction module to output temporally synchronized physiological and facial features. Regarding the dimensionality of feature information, considering the rich semantic information contained in each frame of the facial video, the extracted facial features have a high feature dimension; while physiological monitoring data only contains feature information of different types of signals, resulting in a relatively low feature dimension. Therefore, this embodiment uses a multilayer perceptron (MLP) and activation functions to perform multilayer linear calculations and nonlinear activations on the input physiological monitoring signals to map the physiological detection signals from low dimension to high dimension; and matches them with facial features of a specified dimension to facilitate subsequent feature fusion and enhancement.

[0044] It should be noted that in practical applications, physiological monitoring data in multimodal data typically includes multiple signals. In the physiological branch of the feature extraction module in this embodiment, the same strategy is used to independently extract features from the various synchronously acquired physiological monitoring signals, thereby generating multiple different types of physiological features synchronized across time axes, such as heart rate features, blood pressure features, and respiratory rate features.

[0045] In the facial representation branch, this embodiment first uses OpenCV to process the input video stream data into frames, extracting frame-by-frame images, and then performs face recognition and key region localization on each frame. Addressing the common facial occlusion issues in ICU patients caused by intubation and mechanical ventilation, this embodiment selects key regions from the patient's face that are not obstructed and carry rich pain-related semantic information. Accurate facial landmark recognition is crucial for identifying the corresponding facial activity unit regions, ensuring that the remaining facial landmarks are captured regardless of whether the face is frontal, side-facing, or occluded. In practical application, this embodiment uses the Mediapipe face detection model to perform face recognition and landmark detection on each extracted frame. This face detection model can estimate 468 3D facial landmarks in real time, and then segment the corresponding 24 square regions of the face based on the positions of the 3D facial landmarks and the positions of six designated facial activity units.

[0046] Based on the local images identified, located, and segmented from each frame of the video stream data, this embodiment extracts regional features through a facial feature extraction unit composed of facial attention. This yields the regional features of all local images associated with the same frame. In the facial feature extraction unit used in this embodiment, each input local image is processed by a facial attention module consisting of a ResNet-50 feature extraction layer and a fully connected layer to obtain the regional features of each local image. The facial feature extraction unit in this embodiment automatically learns the importance weights of regions for pain expression through a lightweight network, thereby enhancing the model's sensitivity to key facial expression changes. The weighted regions are input into a pre-trained convolutional neural network for deep visual feature extraction, and feature aggregation is performed on the video frame sequence using temporal pooling or a recurrent neural network, ultimately obtaining a visual feature vector that reflects the dynamic changes in facial expressions.

[0047] 1.2 Feature Fusion Module The aforementioned feature extraction module can extract features from the input physiological monitoring data and facial videos to obtain corresponding physiological and facial features. It is important to emphasize that, to ensure the network model can learn the semantic information representing pain levels implicit in the patient's facial expressions and various physiological parameters at the same time using multimodal data, this embodiment needs to ensure that the physiological and facial features output by the feature extraction module remain synchronized on the timeline. That is, for each frame of image from which facial features are extracted, there is a corresponding physiological feature at that time. The feature extraction module is used to generate a series of temporally arranged physiological and facial feature sequences based on the input multimodal data. When the input physiological monitoring data includes multiple signals, the physiological feature sequence can be further divided into multiple feature subsequences.

[0048] Based on this, the extraction and fusion module can concatenate each set of physiological and facial features output by the feature extraction module in chronological order to obtain the corresponding concatenated vector. Then, fused features are generated based on the sequence features of the concatenated vector. Attention is an important component of human cognitive function. When faced with massive amounts of information, humans can focus on some information while ignoring others. In the pain recognition model of this embodiment, when processing the large number of sequence features of the concatenated vector, the engineers drew on the attention mechanism of the human brain and introduced the temporal attention module in Transformer to explore the rich relationships between the sequence features corresponding to the concatenated vector at different frame times. Then, they selected some key information contained therein for input processing to improve the efficiency of the network model.

[0049] Specifically, such as Figure 4 As shown, the temporal attention module used in the network model of this embodiment includes M attention layers connected in sequence and an MLP classification head. M is preferably 3. Each attention layer includes a multi-head self-attention module (MSA), a multi-attention random deactivation module (MAD), an add-and-normalize layer (Add&Norm), a multilayer perceptron (MLP), and a post-add-and-normalize layer (Add&Norm) connected in sequence. The outputs of the aforementioned layers serve as the outputs of subsequent layers. Furthermore, to avoid gradient vanishing and gradient exploding and to promote rapid model convergence, the input of the multi-head self-attention module is residually concatenated with the output of the multi-attention random deactivation module and used as the input of the add-and-normalize layer; the output of the add-and-normalize layer is residually concatenated with the output of the multilayer perceptron and used as the input of the post-add-and-normalize layer; this ensures the effective preservation of information during multi-layer transmission.

[0050] In this embodiment, the concatenated vectors are sorted according to the time axis and used as input features. x The inputs are then fed sequentially into the temporal attention module. The multi-head self-attention module within the attention layer captures semantic relationships of different dimensions in parallel using multiple attention heads. Each attention head independently calculates its weights, and the results are then merged. Specifically, the MSA module first processes the inputs... x The linear transformation yields a query vector Q, a keyword vector K, and a value vector V. Attention weights are then calculated using the QKV formula, and finally, all values ​​are weighted and summed to calculate the sequence feature vector. The resulting sequence feature vector is then randomly deactivated using MAD (Multimodal Analysis) to avoid overfitting. After normalization, it is input into an MLP (Multimodal Processing Level) where a mapping layer transforms it to a specified dimension, and after normalization, outputs it as the desired fusion feature. This fusion feature is the feature data generated by the pain recognition model in this embodiment based on multimodal data, capable of characterizing the patient's pain level.

[0051] 1.3 Personalization Module In this embodiment, considering the significant differences in age, body type, and condition among patients in the ICU, the pain recognition model, which only uses fused features extracted from facial and physiological monitoring signals to predict pain status, may suffer from insufficient generalization. Therefore, this embodiment designs a special personalization module in the network model. This module can combine the basic information of different users to generate an adaptive adjustment factor, adjusting the model output to obtain more accurate prediction results and improve the reliability and universality of the network model.

[0052] Specifically, the personalization module in this embodiment includes a text extraction network that can extract basic information of ICU patients registered upon admission from the HIS. Then, combining the extracted differences in gender, age, height, weight, disease type, past medical history, laboratory tests, and diagnostic information among different patients, the network dynamically adjusts the parameters of neurons in each layer during the backpropagation of the prediction results, and finally outputs a modulating factor. α The fused features output by the feature fusion module are combined with the adjustment factor. α By multiplying these values, we can obtain the personalized pain characteristics of the current patient, which can be used to accurately predict the pain status of different users.

[0053] In practical applications, a patient's pain state is not only related to basic information such as gender, age, body type, and condition, exhibiting group differences, but also to individual differences in subjective feelings and tolerance. These individual differences in pain perception can be expressed through the patient's voice data. For example, most patients with vocal expression abilities may groan or sigh to alleviate moderate to severe pain; in severe cases, some patients may even scream. To address this, in a further optimized solution of this embodiment, a voice assistance unit can be added to the personalization module. This voice assistance unit is used to classify the input patient's voice data; the voice classification results of the voice assistance unit include three categories: normal speech, sighing, and screaming. Then, a weighting coefficient is generated for the current patient based on the classification results. β Weighting coefficient β Used to weight the fused features.

[0054] Specifically, in this embodiment, the weighting coefficient β The values ​​are 1, 1.2, and 1.5. When the patient can speak normally, it indicates mild pain that is tolerable; the fusion vector is multiplied by a weight coefficient of 1. When the patient sighs, it indicates high pain, and the fusion vector is multiplied by a weight coefficient of 1.2. When the patient screams, it indicates extremely severe pain that is unbearable, and the fusion vector is multiplied by a weight coefficient of 1.5.

[0055] 1.4 Pain Assessment Module In the pain recognition model designed in this embodiment, the input multimodal data undergoes feature extraction, feature fusion, and personalization processing to obtain pain features containing rich semantic information. This embodiment uses a pain assessment module to generate the patient's current pain state based on these pain features.

[0056] The pain assessment module in this embodiment consists of a fully connected layer, a ReLU activation layer, a Dropout layer, and an output layer. The fully connected layer maps the generated high-dimensional pain vector to the final prediction result. The ReLU activation function introduces non-linearity into the network, improving the model's expressive power. The added Dropout layer provides regularization to some extent, adapting to the limited training samples in the ICU medical scenario of this invention and preventing overfitting of the trained model.

[0057] In practical applications, the pain assessment module in this embodiment can be adjusted to either a pain regression task or a pain classification task according to different needs. Pain regression refers to the pain assessment module generating a quantitative value representing the degree of pain based on pain characteristics. Pain classification, on the other hand, refers to the pain assessment module generating a pain level representing the degree of pain based on pain characteristics.

[0058] II. Constructing a multimodal database of ICU patients Considering that existing public databases are incompatible with the multimodal data required in this embodiment in terms of data type, completeness, and other aspects, they are unsuitable for training the pain recognition model in this embodiment. Therefore, this embodiment requires the use of a self-built database to complete model training. The database construction process of this embodiment includes: Basic information of eligible ICU patients, along with facial videos and physiological monitoring data during hospitalization, were collected to form multimodal data. CPOT pain assessments were performed on ICU patients by medical staff, and the assessment results served as multimodal labeling information. A large amount of labeled multimodal data was acquired as sample data to form a sample dataset.

[0059] Specifically, during the sample data collection phase, patients older than 18 years who were transferred to the ICU after surgery and were able to complain of pain were considered eligible ICU patients. Patients meeting any of the following criteria were excluded during this database construction phase: a) those unable or unwilling to complain of pain; b) patients with quadriplegia, muscle dysfunction, deep sedation (RASS score ≤ -3), or those using neuromuscular blocking agents; c) patients with facial trauma, burns, facial dressings, facial paralysis, or other conditions significantly affecting facial imaging data acquisition; d) patients with a history of Alzheimer's disease, mental illness, or diagnosed with delirium; e) patients lacking physiological monitoring data.

[0060] The clinical data collection aims to gather all pain-related information from patients in the ICU, including basic patient information such as medical record number, name, gender, age, height, weight, disease type, past medical history, laboratory and diagnostic information, facial video data, and physiological monitoring data such as heart rate, blood pressure, respiratory rate, blood oxygen saturation, and electrocardiogram.

[0061] A complete multimodal dataset needs to ensure the real-time nature and consistency of different information. Physiological monitoring data for ICU patients can be provided by ventilators and monitors. Basic patient information and past medical history are provided by the hospital's HIS (Hospital Information System). Patient facial video data is acquired by external cameras. The dataset itself can be equipped with information acquisition components and data processing modules, simultaneously recording data from different sources. For example, using a hospital intranet machine, a camera with a standby microphone can simultaneously acquire voice data and facial video stream data. To ensure data security, patient information can be fully accessed from the HIS system. Ventilators and monitors send the patient's current physiological information to the central workstation every second, which is then transmitted to the acquisition computer via the hospital's wireless intranet. The acquisition computer builds a graphical interface for acquisition and finally stores the data named according to the patient's medical record number and the acquisition time.

[0062] Postoperative patients transferred to the ICU are categorized into intubated and non-intubated patients. In clinical application, pain rating scales can be used to collect patient status data under conditions of deep sedation, daily pain, suctioning, and back percussion. The collection order is dynamically adjusted according to the actual situation. For example, daily pain may occur during suctioning or back percussion. Suctioning and back percussion are performed according to normal ICU care procedures, without additional procedures on the patient. If the patient is transferred without endotracheal intubation, only the T8-T10 process is collected. CPOT pain assessment is jointly scored by a nurse and a physician, with a difference threshold set (typically 2). If the difference is within the threshold, the average of the two scores is used as the actual score; if it exceeds the threshold, the data is discarded. For patients without endotracheal intubation or after extubation, VAS self-report data is also collected as another scoring indicator.

[0063] Based on the collected multimodal data, this embodiment can assess the pain status of ICU patients at any two operational stimuli (such as endotracheal intubation and suctioning) and at 30 min, 1 h, 8 h, and 12 h after being transferred to the ICU in a resting state, so as to generate label information for the corresponding time period.

[0064] III. Training and Application of Pain Recognition Models After acquiring a large amount of sample data and constructing the aforementioned multimodal database, this embodiment divides the sample dataset into a training set, a test set, and a validation set. The loss function is set according to the task type of the pain assessment module, and the pain recognition module is trained, tested, and validated. The model parameters that meet the performance requirements after validation are retained and used for pain assessment of ICU patients.

[0065] Specifically, during the training phase of the pain recognition model, when the pain assessment module is used to perform classification tasks and achieve pain grading, the following cross-entropy loss is used as the loss function to minimize the relative entropy between the observations and the estimates.

[0066] When the pain assessment module is used to perform regression tasks and generate pain quantification values, the mean squared error loss is used as the loss function to minimize the mean of the sum of squares of the errors between corresponding points in the predicted data and the original data.

[0067] Example 2 The multimodal pain assessment method for ICU patients provided in Example 1 is essentially a data processing method. To better apply this method, this example further provides a computer program product, a storage medium, and a pain assessment component.

[0068] The computer program product provided in this embodiment includes a computer program. When the computer program is executed by the processor, it creates a pain recognition model trained in the multimodal pain assessment method for ICU patients as in Embodiment 1, thereby enabling the recognition of the pain status of ICU patients based on the input multimodal data.

[0069] The storage medium provided in this embodiment stores a computer program. When the computer program is executed by the processor, it creates a pain recognition model trained in the multimodal pain assessment method for ICU patients as in Embodiment 1, thereby enabling the recognition of the pain status of ICU patients based on the input multimodal data.

[0070] The pain assessment component provided in this embodiment includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a pain recognition model trained in the multimodal pain assessment method for ICU patients as described in Embodiment 1, thereby enabling the identification of the pain state of ICU patients based on the input multimodal data.

[0071] The pain assessment component provided in this embodiment is essentially a computer device. In practical applications, this computer device can be embedded and deployed in designated medical equipment to directly monitor the patient's pain status online based on the collected multimodal data. Alternatively, it can be a standalone computer device, such as a laptop, tablet, desktop computer, or a large-scale computer device capable of executing computer programs, such as a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers). The multimodal data output from the front end is then processed at the back end to obtain the monitoring results of the patient's pain status.

[0072] The computer device in this embodiment includes, but is not limited to, a memory and a processor that can be interconnected via a system bus. In this embodiment, the memory (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0073] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device.

[0074] Example 3 Based on the solution in Embodiment 2, this embodiment further provides a pain monitoring system for an ICU ward, which includes: an audio and video component, a data acquisition component, a pain recognition component, and a decision support component.

[0075] The audio-visual component includes at least one camera, which can be a camera with a microphone and synchronous audio-visual acquisition capabilities. The camera is used to acquire video stream data containing the patient's face; the microphone is used to acquire the patient's voice data. In practical applications of this embodiment, the cameras can be installed above each bed in the ICU ward to achieve higher quality data acquisition. To obtain omnidirectional facial video, multiple cameras can also be installed around the beds.

[0076] The data acquisition component is used by users to collect and organize multimodal data; specifically, it includes acquiring video stream data and possibly audio data collected by the audio-visual component; and acquiring basic patient information and physiological monitoring data. The synchronously acquired video stream data, physiological monitoring data, and audio data, together with the basic information registered by the patient upon admission, constitute the patient's multimodal data.

[0077] The pain recognition component, as described in Embodiment 2, includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a pain recognition model trained in the multimodal pain assessment method for ICU patients as described in Embodiment 1, thereby enabling the recognition of the pain state of ICU patients based on the input multimodal data. Wherein, when the audio-visual component includes multiple cameras, the facial branch in the feature extraction module should be upgraded to a network architecture with multi-source video stream data processing capabilities.

[0078] Decision support components include analysis units, decision-making units, and feedback units. For example... Figure 5 As shown, the analysis unit integrates the patient's pain status, liver and kidney function indicators, and pharmacokinetic parameters to predict the blood concentration-time curves of various sedative-analgesics (such as fentanyl, dexmedetomidine, and propofol) during metabolism in the current patient, and generates individualized medication and dosage adjustment suggestions for sedative-analgesics. The decision-making unit dynamically adjusts the current patient's dosing regimen based on a reinforcement learning framework to balance analgesic efficacy with adverse reaction risks. The feedback unit continuously collects the patient's physiological indicators after medication through bedside monitoring equipment and analyzes the drug efficacy decay trend using a sliding time window to trigger dose reassessment, thereby achieving closed-loop optimization of sedative-analgesic drug decision-making.

[0079] In practical applications, this scheme can be used to set up short-term closed loops when adjusting medication, i.e., updating the dosage recommendation every 30 minutes. Alternatively, it can be set up as a long-term loop, i.e., continuously optimizing the global model parameters by aggregating data from multiple centers to improve population adaptability.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of multi-modal pain assessment of an ICU patient, characterized in that, It comprises: constructing a pain recognition model for generating a pain state of a patient according to inputted multi-modal data; the pain recognition model comprises a feature extraction module, a feature fusion module, a personalization module and a pain assessment module; the feature extraction module comprises a physiological branch and a facial branch; the physiological branch is used for mapping physiological monitoring data into physiological features of a specified dimension by adopting an MLP and an activation function; the facial branch is used for frame segmentation, face recognition and key point detection on a facial video to segment out a local image containing pain semantic information, and then extracting facial features from the local features by facial attention; the feature fusion module is used for splicing the physiological features and the facial features, and then generating fusion features according to the sequence characteristics of the spliced vectors by a time sequence attention module; the personalization module processes the basic information of the patient by a text extraction network and generates corresponding adjustment factors; then the adjustment factors are multiplied with the fusion features to obtain pain features; the pain assessment module is composed of a full connection layer, a ReLU activation layer, a Dropout layer and an output layer, and is used for generating a pain state of the patient according to the pain features; obtaining the basic information of the ICU patient meeting the requirements and the facial video and physiological monitoring data during hospitalization to constitute multi-modal data; performing CPOT pain assessment on the ICU patient by medical staff, and taking the assessment result as the label information of the multi-modal data; obtaining a large amount of multi-modal data with labels as sample data to constitute a sample data set; dividing the sample data set into a training set, a test set and a validation set, setting a loss function according to the task type of the pain assessment module, and training, testing and validating the pain recognition model; retaining the model parameters meeting the performance requirements after validation, and using them for pain assessment of the ICU patient.

2. The method of multi-modal pain assessment of an ICU patient as claimed in claim 1, wherein, The basic information of the patient in the multi-modal data includes: medical record number, name, gender, age, height, weight, disease, medical history, laboratory and diagnosis information; and / or, the physiological monitoring data includes any one or more of heart rate, blood pressure, respiratory rate, blood oxygen and electrocardiogram; when the physiological monitoring data includes multiple types, the physiological branch extracts features from each type of physiological monitoring signal to generate multiple corresponding physiological features.

3. The method of multi-modal pain assessment of an ICU patient as claimed in claim 1, wherein: In the sample data acquisition stage, patients aged more than 18 years old, transferred to the ICU after surgery and able to complain of pain are taken as ICU patients meeting the requirements; the pain state of the ICU patient is assessed at any 2 times of operation stimulation and in a resting state 30 min, 1 h, 8 h and 12 h after being transferred to the ICU to generate label information corresponding to the time period; in the collected multi-modal data, the facial video and the physiological monitoring data are collected synchronously; the basic information is obtained through the HIS system in the hospital; the physiological monitoring data is collected in real time by various physiological monitoring devices worn by the patient.

4. The method of multi-modal pain assessment of an ICU patient as claimed in claim 1, wherein, The process of generating facial features from the facial video by the facial branch comprises: The face video is converted into frame-by-frame images by using OpenCV; the frame-by-frame images are subjected to face recognition and key point detection by using a Mediapipe face detection model; a key face activity unit without occlusion and containing pain semantic information is recognized, and a local image of the corresponding region is segmented; the key face activity unit includes lowering eyebrows, tightening eye sockets, muscle contraction and closing eyes; the segmented local images are respectively input into a face feature extraction unit composed of a face attention and convolution network to extract regional features; the regional features of all local images associated with the same frame image are vertically connected to obtain a face representation.

5. The method for multi-modal pain assessment of an ICU patient as claimed in claim 4, wherein: In the face feature extraction unit, the input local image is processed by a face attention module composed of a feature extraction layer of ResNet-50 and a full connection layer to obtain the regional feature of each local image.

6. The method for multi-modal pain assessment of an ICU patient as claimed in claim 1, wherein, In the feature fusion module, the physiological features and the face features synchronized in time sequence are spliced in the order of the time axis, and each spliced feature arranged in time sequence is input into a time sequence attention module for processing to capture the rich relationship between the features at different frame moments. The time sequence attention module includes a plurality of attention layers connected in sequence and an MLP classification head; each attention layer includes a plurality of head self-attention modules, a multi-attention random inactivation module, a front addition and normalization layer, a multi-layer perceptron, and a rear addition and normalization layer connected in sequence; the input of the multi-head self-attention module is connected in residual with the output of the multi-attention random inactivation module, and serves as the input of the front addition and normalization layer; the output of the front addition and normalization layer is connected in residual with the output of the multi-layer perceptron, and serves as the input of the rear addition and normalization layer.

7. The method for multi-modal pain assessment of an ICU patient as claimed in claim 1, wherein: The individualization module of the pain recognition model further includes a voice auxiliary unit; the voice auxiliary unit is used for classifying the input voice data of the patient; and a weight coefficient is generated according to the classification result to weight the fusion feature; The voice classification result includes normal speaking, sighing and screaming; when the patient speaks normally, the fusion vector is multiplied by the weight coefficient 1; when the patient sighs, the fusion vector is multiplied by the weight coefficient 1.2; when the patient screams, the fusion vector is multiplied by the weight coefficient 1.

5.

8. The method for multi-modal pain assessment of an ICU patient as claimed in claim 1, wherein: In the training stage of the pain recognition model, when the pain assessment module is used to perform a classification task and implement pain grading; cross-entropy loss is used as a loss function to minimize the relative entropy of the observed value and the estimated value; when the pain assessment module is used to perform a regression task and generate a pain quantization value; mean square error loss is used as a loss function to minimize the mean of the square sum of the corresponding point errors of the predicted data and the original data.

9. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, creates a pain recognition model trained in the ICU patient multi-modal pain assessment method according to any one of claims 1-8, and further realizes the identification of the pain state of the ICU patient according to the input multi-modal data.

10. An ICU ward pain monitoring system characterized by, It includes: An audio and video component including a camera and / or a microphone; The camera is used to collect video stream data containing the patient's face; The microphone is used to collect the patient's voice data; a data acquisition component configured to acquire video stream data and / or voice data collected by the audio-video component, and to acquire basic information and physiological monitoring data of the patient; the synchronously acquired video stream data, physiological monitoring data, and / or voice data, together with the basic information, constitute multi-modal data of the patient; a pain recognition component including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, creates a pain recognition model trained in the method of claim 1-8, and further realizes recognition of the pain state of the ICU patient according to the input multi-modal data; a decision support component including an analysis unit, a decision unit, and a feedback unit; the analysis unit is configured to integrate the pain state of the patient, liver and kidney function indicators, and drug metabolism kinetics parameters, to predict the blood drug concentration-time curve of various sedative and analgesic drugs when metabolized in the current patient, and to generate individualized medication and dose adjustment suggestions for sedative and analgesic drugs; the decision unit dynamically adjusts the drug administration scheme of the current patient based on a reinforcement learning framework to balance the analgesic effect and adverse reaction risk; the feedback unit continuously acquires physiological indicators of the current patient after drug use through bedside monitoring equipment, and analyzes the drug efficacy decay trend using a sliding time window to trigger the start of dose reevaluation, thereby realizing closed-loop optimization of sedative and analgesic drug decision-making.

Images