A pain assessment model construction method and a non-contact continuous pain assessment method

Abstract

The application discloses a pain assessment model construction method and a non-contact continuous pain assessment method, and belongs to the field of image expression recognition. Through a non-contact mode, multi-modal time sequence data of a patient is collected, and after time-space synchronous processing to obtain a multi-modal frame package sequence, single-modal self-supervision pre-training is performed based on extracted single-modal time sequence data, and then a multi-modal projection layer is connected to obtain a front structure for cross-modal semantic alignment coding, so as to map output features of each mode to a large language model word vector dimension, and then a large language backbone model is connected to obtain an overall structure. The multi-modal time sequence data of the patient is input, fine-tuning training of the large language model is performed by indicating the overall structure to output a pain assessment result, a trained pain assessment model is obtained, and the trained pain assessment model is used for non-contact continuous pain assessment. The pain assessment result includes a pain level, a pain score, an interpretable heat map and the like. The application combines a large language model, so that the trained model can realize pain level division and interpretable continuous output.

Description

Technical Field

[0001] This invention belongs to the field of image expression recognition, specifically involving a method for constructing a pain assessment model and a non-contact continuous pain assessment method. Background Technology

[0002] Assessing and evaluating a patient's pain is an essential part of clinical care. Pain is a subjective experience, so its assessment is usually based on the patient's self-report. However, for patients who cannot communicate or express their pain, such as infants, comatose patients, or patients with certain cognitive impairments, clinicians and nurses may need to rely on alternative methods to assess pain.

[0003] One commonly used method is visual pain assessment, primarily using the Visual Analogue Scale (VAS). The VAS is a tool for qualitative and quantitative assessment of subjective feelings or attitudes, especially the intensity of pain.

[0004] According to the standards of the International Association for the Study of Pain (IASP), the Visual Analogue Scale (VAS) typically consists of a 10-centimeter horizontal line. Patients are asked to mark their pain level on a straight line without numbers. The two ends of the line represent the two extremes of a specific sensation, such as "no pain" and "most severe pain." Subjects must mark their current sensation on this line. Of course, the IASP standards also include the Numerical Rating Scale (NRS) and the Faces Rating Scale (FRS), which incorporates facial expressions. Currently, a major drawback of using the VAS in clinical practice to assess pain is its subjectivity; relying on purely manual assessment is labor-intensive, and there is a lack of standardized procedures and scoring methods.

[0005] Existing commonly used methods also include facial expression recognition technology. In recent years, vision-based facial expression recognition has largely relied on networks such as CNNs and Transformers to extract Action Units (AUs) from facial images and then classify them. However, these methods still remain within the "classification task" paradigm and have at least the following limitations: 1. Discretized labels: for example, they can only output a limited number of categories such as "happy," "sad," and "angry," lacking characterization of intensity and continuous changes, and failing to meet the clinical need for continuous pain curves; 2. Uncertainty of subjective expression: making inferences from a single static image makes it difficult to distinguish between context-similar expressions such as "level 7 pain" and "level 3 pain," easily leading to misjudgment and reduced credibility; 3. Insufficient interpretability of traditional classification methods: the model gives the conclusion of "pain," but cannot explain why the pain occurs, making it difficult for medical staff to trust the model. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a method for constructing a pain assessment model and a non-contact continuous pain assessment method. The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a method for constructing a pain assessment model, the method comprising:

[0008] S1, acquire the patient's multimodal time-series data; wherein, the multimodal time-series data includes electronic health record sequence, facial motion unit intensity sequence, body skeleton displacement sequence, remote photoplethysmography pulse wave and infrared thermal imaging data sequence;

[0009] S2, perform spatiotemporal synchronization processing on the multimodal time-series data to obtain a multimodal frame packet sequence, wherein each multimodal frame packet contains multimodal data of the frame after spatiotemporal synchronization processing;

[0010] S3, Based on the single-modal temporal data extracted from the multimodal frame packet sequence, perform single-modal self-supervised pre-training on the corresponding modal model;

[0011] S4. All modal models that have been pre-trained by single-modality self-supervised training are followed by a unified multimodal projection layer to obtain the pre-structure. The pre-structure is then encoded with cross-modal semantic alignment to map the output features of each modality to the word vector dimension of the large language model.

[0012] S5, the overall structure is obtained by connecting the obtained pre-structure with the large language backbone model, the patient's multimodal temporal data is input, and the large language model is fine-tuned by instructing the overall structure to output pain assessment results, so as to obtain the trained pain assessment model; wherein, the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is used to assess the patient's pain.

[0013] Secondly, embodiments of the present invention provide a non-contact continuous pain assessment method, the non-contact continuous pain assessment method comprising:

[0014] Acquire the target multimodal temporal data of the patient to be evaluated; the target multimodal temporal data includes electronic health record sequence, facial motion unit intensity sequence, body skeleton displacement sequence, remote photoplethysmography pulse wave and infrared thermal imaging data sequence;

[0015] The target multimodal time-series data is input into the trained pain assessment model to obtain the patient's pain assessment results; wherein, the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is trained according to the pain assessment model construction method described in the first aspect.

[0016] Thirdly, embodiments of the present invention provide a non-contact continuous pain assessment system, comprising:

[0017] A multimodal acquisition module is used to acquire the target multimodal time-series data of the patient to be evaluated; the target multimodal time-series data includes electronic health record sequence, facial motion unit intensity sequence, body skeleton displacement sequence, remote photoplethysmography pulse wave and infrared thermal imaging data sequence;

[0018] The pain assessment module is used to input the target multimodal time-series data into a trained pain assessment model to obtain the patient's pain assessment results; wherein, the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is trained according to the pain assessment model construction method described in the first aspect.

[0019] Fourthly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0020] The memory is used to store computer programs;

[0021] When the processor executes the program stored in the memory, it implements the steps of the pain assessment model construction method provided in the embodiments of the present invention, and / or implements the steps of the non-contact continuous pain assessment method provided in the embodiments of the present invention.

[0022] The beneficial effects of this invention are:

[0023] This invention discloses a method for constructing a pain assessment model. It collects multimodal temporal data, including facial expressions, body surface temperature, and skeletal posture, from patients using non-contact methods such as cameras and infrared devices. After spatiotemporal synchronization processing, a multimodal frame packet sequence is generated. Self-supervised pre-training is performed on each single-modal temporal data, and cross-modal semantic alignment is achieved through a multimodal projection layer, mapping multi-source features to the word vector space of a large language model. The model backbone uses a large language structure for joint encoding, outputting pain levels, continuous pain scores, and interpretable heatmaps. The pain assessment model obtained through self-supervised training enables non-contact, continuous, and interpretable pain assessment in medical scenarios, improving assessment accuracy and clinical usability. The provided pain assessment model construction method, by non-contactly collecting and integrating five channels of multimodal heterogeneous data, improves information utilization and enhances the accuracy and robustness of the model's pain assessment. Furthermore, this invention, combined with a large language model, enables the trained model to achieve pain level classification and interpretable continuous output, facilitating intuitive presentation and guidance for subsequent diagnosis.

[0024] The non-contact continuous pain assessment method provided in this invention acquires the patient's target multimodal temporal data to be assessed in a non-contact manner, performs spatiotemporal synchronization processing to obtain target multimodal frame packets, and then inputs the target multimodal frame packet sequence into a trained pain assessment model to obtain the patient's pain assessment results. The pain assessment results include pain level, pain score, and interpretable heatmap. The pain assessment model of this invention can not only provide a textual description of the pain level, but also generate corresponding image outputs (AU heatmap, skeletal displacement map, thermal infrared map), which facilitates intuitive display and understanding of the patient's pain state and has strong interpretability. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a method for constructing a pain assessment model according to an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating the time alignment process for multimodal time-series data according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram illustrating the principle of cross-modal alignment in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram illustrating the principle of instruction fine-tuning in an embodiment of the present invention;

[0029] Figure 5 This is a flowchart illustrating a non-contact continuous pain assessment method provided in an embodiment of the present invention.

[0030] Figure 6 This is an interpretable heatmap of rPPG according to an embodiment of the present invention;

[0031] Figure 7 This is a thermal infrared temperature difference diagram of an embodiment of the present invention;

[0032] Figure 8 This is a schematic diagram of the structure of a non-contact continuous pain assessment system provided in an embodiment of the present invention;

[0033] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0035] Firstly, embodiments of the present invention provide a method for constructing a pain assessment model. The executing entity of this method can be a pain assessment model construction system, which can run on an electronic device. This electronic device can be a server or a terminal device, more preferably a portable device placed near the patient, but is not limited thereto. Application scenarios for embodiments of the present invention include hospital ICUs (Intensive Care Units), where patients may have difficulties with movement or limited verbal expression, but are not limited thereto.

[0036] The pain assessment model construction system can include a multimodal acquisition module, a data synchronization and preprocessing module, a feature encoding module, a multimodal fusion and semantic alignment module, a large language model inference module, and a result visualization module. The multimodal acquisition module utilizes cameras, infrared imagers, depth sensors, and physiological signal acquisition devices to achieve non-contact synchronous acquisition of facial expressions, body surface temperature distribution, skeletal posture, and heart rate signals. Physiological signals are also acquired using a non-contact camera.

[0037] How Figure 1 As shown, the method for constructing this pain assessment model may include the following steps S1 to S5:

[0038] S1, acquire the patient's multimodal time-series data;

[0039] The multimodal time-series data includes electronic health record sequences, facial motion unit intensity sequences, body skeleton displacement sequences, remote photoplethysmography pulse waves, and infrared thermal imaging data sequences.

[0040] In this embodiment of the invention, multimodal time-series data of multiple patients can be acquired, thereby training a pain assessment model that is suitable for many patients.

[0041] This invention provides a non-contact acquisition method for obtaining multimodal time-series data from patients, eliminating the need for traditional patch-type physiological sensors. This improves the convenience and efficiency of data acquisition. Specifically:

[0042] An electronic health record (EHR) sequence comprises data from multiple points in time. An EHR can be simply understood as a medical record text, primarily containing data on pain assessments and medical history. Specifically, it may include basic patient information, medical history, allergy history, medication records, vaccination history, laboratory test results (such as blood tests), medical images (such as CT scans and X-ray reports), physician's records, discharge summaries, etc. It originates from hospital electronic health records and can also be obtained from some public datasets (such as MIMIC-III). An EHR is a digital, lifelong health record for an individual, a more comprehensive "health record" that can be shared across institutions. EHR sequences can be obtained through the interface of a Hospital Information System (HIS) and can use either the HL7v2 or FHIR standard.

[0043] The facial action unit intensity sequence includes facial action unit intensities at multiple moments. This invention utilizes self-collected facial video data of facial action units, as well as publicly available datasets (such as AffectNet or BioVid) to obtain facial expression data containing different levels of pain intensity. AffectNet alone has tens of thousands of facial data points with different expressions, ensuring sufficient samples for learning various expressions. Facial Action Unit (AU): This is not the everyday "expression" (such as happiness or sadness), but rather a more fundamental, basic facial muscle movement. It originates from the Facial Action Coding System (FACS), an objective standard that decomposes facial expressions into single or multiple muscle movement components. A human face can be divided into multiple facial action units; for example, AU25 represents the separation of the lips. Intensity here refers to the strength of a particular action unit. The FACS standard typically defines intensity as 5 levels (A~E) or 6 levels (0~5). For example, a small upward turn of the corners of the mouth is intensity 1, while a large upward turn is intensity 5. Facial Action Unit (AU) intensity sequences can be obtained by analyzing the intensity of facial muscle movements using computer algorithms from ordinary color (RGB) videos or images, and obtaining sequence data of these intensities changing over time. In short, a sequence of RGB video frames of a face is acquired. The first step is face detection and alignment. This part uses a deep learning model to first locate the face in the image, i.e., face detection. Then, key facial landmarks are located, such as the corners of the eyes, corners of the mouth, tip of the nose, and eyebrow contours (usually 68 or more points). This step is called "alignment," and its purpose is to normalize faces of different poses and sizes, reducing interference from irrelevant variables (such as head rotation). The second step is feature extraction. The deep learning model needs to extract features highly correlated with muscle movements from the aligned facial region. Since AUs involve subtle movements, the features must be very refined, including both spatial and temporal features. The third step involves using a deep learning model for AU intensity regression / classification. The goal is to predict an intensity value for each predefined AU (e.g., AU1, AU2, ..., AU45) based on the extracted rich (spatial + temporal) features. This can typically be modeled as a regression problem (directly outputting continuous values ​​from 0 to 5) or an ordinal regression problem (treating intensity levels as ordered labels). The fourth step involves repeating the above process for each frame of the video, ultimately generating an intensity-time series for each AU. During acquisition, an RGB camera can be used, and relevant parameters can be selected as needed, such as a spatial resolution of 1920. 1080 resolution, with a temporal resolution of @50fps, means that the camera can capture 50 images per second.

[0044] The body skeletal displacement sequence includes body skeletal displacement data at multiple time points. This invention employs both self-collected body motion data and publicly available datasets (NTU-RGB+D) to obtain labeled skeletal data. The self-collected data is annotated with accompanying painful behaviors and contains several thousand skeletal data points, including pose changes for different behaviors. The number of joints in the body skeletal displacement is determined by the body skeletal model used; for example, the BlazePose model includes 33 joints, and the OpenPose model includes 25 joints. The body skeletal displacement sequence can be obtained using different types of cameras. For example, an RGB camera can be used to capture human video streams, and the body skeletal displacement sequence can be estimated based on 2D image algorithms; alternatively, a depth camera (such as an RGB-D camera) can be used to capture image sequences, and the body skeletal displacement sequence can be obtained through point cloud generation and 3D skeleton fitting; or a professional skeleton camera (optical motion capture system) can be used to capture video, and the body skeletal displacement sequence can be obtained through multi-view infrared marker triangulation. The camera parameters used can be selected as needed, for example, the temporal resolution can be @30fps.

[0045] Remote photoplethysmography (rPPG) is a non-contact physiological signal acquisition technology that uses only a regular camera (such as a mobile phone camera or computer camera) to capture video of a face or skin from a certain distance. Without any contact, it can extract physiological information such as heart rate and respiratory rate. Its main processing steps include: acquiring RGB video, face detection and region of interest (ROI) selection, signal extraction and processing, pulse wave signal separation, generating and analyzing rPPG waveforms, thereby obtaining heart rate, heart rate variability, respiratory rate, etc. The rPPG system parameters can be selected according to needs. For example, an rPPG system deployed on an edge microprocessor (MCU) can be used, with an ROI data update frequency of 120Hz. An edge microprocessor-based rPPG system can achieve low power consumption, low cost, miniaturization, privacy, and real-time performance. The purpose of using 120Hz is: 1) Anti-aliasing: According to the Nyquist sampling theorem, to reconstruct a signal without distortion, the sampling frequency must be greater than twice the highest frequency of the signal. Even at a heart rate of 180 beats per minute (3Hz), the harmonic components can reach 10-20Hz. A sampling rate of 120Hz can easily cover these frequencies, avoiding signal aliasing and ensuring the accuracy of heart rate calculation. 2) Improved signal-to-noise ratio: A higher sampling rate means more data points. In subsequent signal processing (such as filtering and averaging), this helps to more effectively suppress random noise, resulting in a cleaner and more stable pulse wave signal. 3) Support for high-frequency physiological index analysis: For applications that require analysis of more detailed physiological phenomena, such as heart rate variability (HRV), it is necessary to accurately measure minute changes in the RR interval. A sampling rate of 120Hz means a time resolution of approximately 8.3 milliseconds, which is crucial for accurately capturing the RR interval.

[0046] Infrared thermal imaging data sequences, including infrared thermal imaging data from multiple moments. This can be obtained by acquiring video using a thermal infrared camera or thermal imager. The parameters of the thermal infrared camera can be selected as needed, for example, the spatial resolution can be 640. 480, with a temporal resolution of 50fps. This invention's embodiments include self-collected thermal infrared images, and additional datasets include NTU-RGB+D and USTC-NVIE. Similar to the body skeleton data, the data collected by this invention is labeled with accompanying pain behaviors, and the dataset contains several thousand frames of thermal infrared images, which can help the model of this invention learn the effect of pain on body temperature.

[0047] The process of obtaining the aforementioned multimodal time series data can be understood in conjunction with relevant technologies, and will not be explained in more detail here.

[0048] S2, perform spatiotemporal synchronization processing on the multimodal time series data to obtain a multimodal frame packet sequence, wherein each multimodal frame packet contains multimodal data of the frame after spatiotemporal synchronization processing;

[0049] In the data input process of multimodal large language models, the core technical challenge faced by each modality lies in the hardware clock skew. Without cross-device time drift alignment, the peak values ​​of facial expression detection cannot be accurately matched. Furthermore, spatial field of view (FoV) differences are also a key constraint; for example, data collected by thermal imagers and RGB-D cameras cannot completely overlap, making it difficult to accurately overlay action unit (AU) features with heatmaps. Finally, the asynchronous frame rate problem urgently requires a unified processing framework; for example, when inputting an AU sequence sampled at 25fps from a video into a Transformer model, frame rate synchronization alignment is necessary.

[0050] In one optional implementation, spatiotemporal alignment of multi-source information at a preset frame rate is achieved through timestamp links and adaptive frame interpolation technology. Specifically, in S2, spatiotemporal synchronization processing is performed on the multimodal time-series data to obtain a multimodal frame packet sequence, which may include:

[0051] S21, based on the nearest neighbor-interpolation-frame dropping rule, maps the patient's multimodal time series data to a common time axis to achieve time alignment at a preset frame rate;

[0052] The processing principle of this step can be combined with Figure 2 understand, Figure 2 This is a schematic diagram illustrating the time alignment process for multimodal time-series data according to an embodiment of the present invention. NTP stands for Network Time Protocol, and PTP stands for Precision Time Protocol, typically referring to IEEE 1588.

[0053] Since data acquisition is primarily based on cameras, and cameras typically operate at 25fps, from a hardware solution perspective, the preset frame rate can be 25fps, meaning the common clock is 25fps.

[0054] Among them, based on the nearest neighbor-interpolation-drop axis rule, the multimodal time series data of the same patient are all mapped to a common time axis to achieve time alignment at a preset frame rate, which is achieved by the following formula:

[0055] ;

[0056] The common timeline is represented as follows: ,total At a given moment, this unified time axis is the sole reference for calculating each peak moment; given mode The time series data is represented as , Representing modes Time series data The corresponding data, Representing modes Time series data The corresponding data, Representing modes Time in time series data , Represents moments in the common timeline , Representing modes Mapped to time in time series data As a result, This represents the first preset time threshold, which is the nearest neighbor sampling threshold. This represents the second preset time threshold, which is the maximum drift allowed by interpolation. , Indicates interpolation. Indicates empty;

[0057] For example, =2ms, =4ms. Frame missing rate ≤ 5%. Among them, The 4ms timeframe is because, across several sampling frequencies, rPPG heart rate typically requires bandpass filtering (0.7 – 4 Hz, corresponding to 42–240 bpm). Therefore, a base frequency of 80Hz is generally required.

[0058] It's important to note that in the rPPG processing flow, the original input is a video at a fixed frame rate (e.g., 60fps). This means that the original pulse signal extracted from the video has sampling points that are equally spaced in time. However, due to various reasons (such as algorithm processing latency, system load fluctuations, etc.), the final extracted rPPG signal time series (i.e., remote photoplethysmography pulse wave) may no longer be perfectly equally spaced. Non-uniform sampling introduces additional noise and causes spectral distortion, which can severely affect subsequent heart rate estimation (which relies on frequency domain analysis). Therefore, resampling is necessary, and its primary purpose is to convert the potentially non-uniform rPPG signal back into a uniformly sampled signal.

[0059] In this embodiment of the invention, the rPPG branch employs piecewise Sinc / spline resampling at the "interp" to suppress heart rate frequency band distortion. This means that this resampling is not arbitrary but has a specific fidelity target: it must retain information in the frequency range of 0.7 Hz to 4 Hz to the maximum extent possible. Since heart rate = 42 beats / minute to 240 beats / minute, converting heart rate / minute to Hertz yields the aforementioned 0.7 Hz to 4 Hz frequency band, which covers the extreme range of human heart rate. Inappropriate resampling methods (e.g., simple linear interpolation or nearest neighbor interpolation) can distort the frequency components of the signal, potentially introducing high-frequency noise and attenuating the energy of the heart rate frequency band of interest. Therefore, a resampling method that can maintain high fidelity within a specific frequency band is needed. To this end, the embodiments of the present invention employ segmented Sinc / spline resampling. This type of high-fidelity method can resample the signal into a uniform sequence, especially protecting the critical heart rate frequency band of 0.7-4 Hz to prevent distortion during the resampling process, thereby laying the foundation for the final calculation of accurate and robust heart rate values.

[0060] S22, Spatial registration is performed on the patient's time-aligned multimodal temporal data, including spatial registration between the first type of images and the second type of images, spatial registration between the first type of images and the third type of images, and spatial registration between different types of the first type of images; wherein, the first type of images includes RGB images of facial motion unit intensity and RGB images of remote photoplethysmography pulse waves, the second type of images are images corresponding to body skeletal displacement, and the third type of images are infrared images corresponding to infrared thermal imaging data;

[0061] Because the acquisition process utilizes multiple devices, the position coordinates of various modes may differ, thus spatial registration is required. The main registration objects, methods, and corresponding results for S22 spatial registration can be understood by referring to Table 1.

[0062] Table 1 Explanation of Spatial Registration

[0063]

[0064] Number 1 corresponds to the spatial registration between the first and second type of images. The purpose is to achieve spatial registration between the RGB image of facial motion unit intensity and the image corresponding to the depth body skeleton displacement (e.g., a depth image), and between the RGB image of remote photoplethysmography (LPG) and the image corresponding to the depth body skeleton displacement (e.g., a depth image). The spatial registration method used is the Kinect SDK extrinsic parameter (4...) The 4-matrix, in essence, utilizes the pre-calibrated coordinate transformation formula (4-matrix) built into the Kinect hardware. The 4-external parameter matrix automatically maps information seen by the depth camera (such as skeletal joints) onto the color camera image with high precision by calling ready-made functions in the official SDK. The result of this spatial registration is a mapping error of <0.5 pixels, or less than half a pixel. This extremely small error ensures that facial RGB information (used for rPPG) and body skeletal displacement information are strictly aligned spatially, providing a solid foundation for the fusion analysis of multimodal data. Specific spatial registration methods can be understood in conjunction with relevant technologies and will not be detailed here.

[0065] Number 2 corresponds to the spatial registration between the first type of image and the third type of image. The purpose is to achieve spatial registration between the RGB and infrared images of facial motion unit intensity, and between the RGB and infrared images of remote photoplethysmography (PPG). The spatial registration method used is a heated checkerboard pattern. SIFT + RANSAC to calculate Homography The essence of this method is to use a special calibration object that can be seen by both RGB and infrared cameras, allowing the computer to automatically find the perspective transformation relationship between two lenses. Its spatial registration process can be divided into the following steps: First, heating the checkerboard pattern. The heated checkerboard pattern will appear as clear black and white squares in the infrared thermal imager (the hot and cold areas create a contrast). This yields a common target that can be recognized by both RGB and infrared cameras (based on color / texture). This is the foundation for all subsequent calculations. Second, SIFT (Scale Invariant Feature Transform), which automatically finds unique and identifiable "key points" (such as corners, edge intersections, etc.) on the two images obtained in the previous step (one RGB image and one infrared image). Third, RANSAC (Random Sample Consensus), which eliminates some incorrect matches (noise) in the initial SIFT matching of feature point pairs, obtaining the most reliable transformation relationship. Fourth, calculating the Homography matrix. Homography is a 3D model of the homography matrix. A 3x3 matrix. It describes the perspective transformation relationship between two planar images. Using highly reliable matching point pairs selected by RANSAC, this optimal Homography matrix H can be calculated mathematically. Once H is obtained, for any point in the infrared image, its precise location in the RGB image can be found through matrix multiplication. This represents the optimal Homography matrix H. The result of this spatial registration is a facial ROI error of <0.6 px. This means that after registration, for key facial regions (such as cheeks, forehead, etc.), the average difference between the position of the same anatomical point in the transformed infrared image and its true position in the original RGB image is less than 0.6 pixels. This implies that after Homography transformation, the infrared and RGB images achieve sub-pixel alignment in the facial region. For example, the cheek region selected on an RGB image almost completely overlaps with the same physical region selected on the transformed infrared image. Such high precision ensures that the RGB texture regions used to analyze facial action units (AUs) strictly correspond to the facial temperature change regions reflected in the infrared image, and the RGB skin regions used to calculate rPPG strictly correspond to the blood flow change regions reflected in the infrared image. If the error is large (e.g., 3-5 pixels), it may incorrectly correlate forehead temperature data with eye texture and movement, causing multimodal data analysis to completely fail. Therefore, this invention can ensure that facial RGB information (for AU and rPPG) and infrared thermal imaging information are highly consistent in space, providing a reliable guarantee for subsequent fusion analysis.

[0066] Number 3 corresponds to spatial registration between different types of Class 1 images, specifically spatial registration between the RGB image of facial action unit intensity and the RGB image of remote photoplethysmography (RPG). The spatial registration method used is AU-68 keypoint real-time update of the cheek / forehead ROI. Essentially, instead of using a fixed, pre-drawn ROI to track facial skin color, the ROI is "attached" tightly to the moving skin like a "band-aid," adapting to head movement and facial expression changes. AU-68 keypoints are a general facial landmark detection model that can automatically identify and locate 68 predefined keypoints on a face image. These points outline the contours of the face, including eyebrows, eyes, nose, lips, and outer contours. For example, the cheek region can be defined by multiple points of the nose, corners of the mouth, and facial contours. Many early rPPG studies simply drew a fixed-size rectangle on the cheek or forehead in the first frame. When a person turns their head or speaks, the bounding box quickly deviates from the target skin area, causing the extracted signal to be filled with motion noise. This invention utilizes 68 key points to dynamically and frame-by-frame define the Region of Interest (ROI). The algorithm first detects these 68 points, then intelligently calculates the range of the cheek ROI (e.g., covering the cheekbone area) and the forehead ROI based on their positions. For each frame of the video, the algorithm re-executes the 68 facial key point detection. Then, based on the latest key point positions in the current frame, it recalculates and redraws the position and shape of the ROI. The result of this spatial registration is that the ROI adapts to facial expression drift. Here, "drift" refers to the displacement and deformation of the facial skin area in the image coordinate system caused by head movements (translation, rotation), speaking, and facial expressions. Adaptability means that the ROI has the ability to automatically follow and adapt to this drift. For example, regarding head translation / rotation, when a patient turns their head to the left, the right cheek will appear smaller and shift in the image. The adaptive ROI adjusts the patient's position and size according to the location of the key points on the right cheek, ensuring it always covers the skin of the right cheek. Regarding facial expressions, when the patient smiles, the cheek muscles bulge and the corners of the mouth pull upwards. At this time, the key points defining the cheek ROI (such as the corner points of the mouth) themselves move. The adaptive ROI moves and deforms accordingly, closely following the bulging skin area. This invention abandons static boxes and embraces dynamic points, dynamically and frame-by-frame redefining the facial region (ROI) used to extract pulse signals by tracking the positions of 68 key points on the face in real time. This ensures that the skin region for extracting rPPG signals is no longer severely affected by head movements and facial expression changes, and can always accurately attach to the target muscle / skin region. This greatly reduces motion artifacts, thus laying a solid foundation for subsequent calculations of more stable and accurate heart rate and facial motion unit strength. This is an advanced approach that upgrades from "static registration" to "dynamic tracking."

[0067] Section 4 supplements the spatial registration between the first and second types of images. With the same sensor output, there's a one-to-one correspondence between joint pixels and spatial coordinates. "Same sensor output" means that the skeleton data and RGB image data originate from the same integrated sensor device. In other words, the RGB stream, depth stream, and skeleton stream are all acquired and generated by the same hardware at the same time. They are inherently "of the same origin." The one-to-one correspondence between joint pixels and spatial coordinates refers to the significant advantage brought by "same sensor output." Here, "space" refers to 3D spatial coordinates, and "pixel" refers to 2D image coordinates. The "one-to-one correspondence" means that the device's internal SDK has already handled all the complex calibration and calculations. Since the skeleton data and RGB image come from the same hardware (such as Kinect), the device manufacturer has provided users with a "magic converter" through precise factory calibration. Simply inputting the 3D coordinates of a joint, this "converter" can immediately output its precise pixel position on the RGB image. Therefore, the skeleton and RGB image are "inherently registered," requiring no additional complex calculations from the developer. The end result of this method is that the skeleton and the RGB image achieve perfect, seamless, and high-precision spatial registration.

[0068] Here, in order to test the effectiveness of spatial registration, the peak consistency index can be used for quantification.

[0069] Record the three modes in the window The main peak times detected internally were respectively , , Among them, windows It is a closed interval selected on a unified time axis. (Unit: ms). Window length ( A fixed time interval, such as 2 seconds, is used to find the "main peak" within a local time period. Each window is calculated independently. The main peak is The local maximum with the largest internal amplitude; if they are tied, the one with the earliest time is selected (uniqueness rule). It is in the window Within, the main peak timestamps appearing after preprocessing the AU intensity sequence (e.g., normalized summation of the intensities of key AUs), in milliseconds, and defined domain. . It is in the window Internal, the main peak timestamp of thermal infrared temperature indicators (such as the temperature difference or temperature gradient norm of facial pain-related ROIs), in milliseconds. . It is in the window Within, the timestamp of the main peak of the envelope or power spectrum energy of the rPPG waveform, in milliseconds. The method for determining the dominant peak is not limited here. The cross-modal dominant peak can be obtained through the maximum lag of cross-correlation (cross-correlation of the mode pair with the AU reference and taking the maximum correlation) or the derivative zero-crossing / spectral peak method. The lag time is taken as the dominant peak (used for noise resistance and robust estimation of cross-modal phase difference).

[0070] , , ;

[0071] Peak Consistency Index for:

[0072] ;

[0073] It is the absolute time difference (ms) between AU and the main thermal infrared peak; It is the absolute time difference (ms) between the AU and the main peak of rPPG; It is a normalization constant used to linearly map time differences to... . ; This refers to the sampling resolution, which is the minimum temporal resolution (unit: ms) on a uniform time axis. It is determined by the modality with the highest frame rate / sampling rate; lower-speed modalities need to be upsampled to this resolution. Example: 25 fps video. 40 ms; 200 Hz rPPG 5 ms;

[0074] The physical meaning is: the smaller the time difference between two transmodal modes, the smaller the proportional term. The closer to 1. When This is considered as achieving the required alignment, meaning the cross-modal peaks are considered "relatively aligned". Threshold The optimal value is 0.9.

[0075] S23, each frame of multimodal data after patient spatial registration is used to form a multimodal frame packet, thereby obtaining a multimodal frame packet sequence according to the time order;

[0076] Each multimodal frame packet is represented as:

[0077] ;

[0078] in, Indicates time Multimodal frame packets, This represents facial motion unit intensity data. Indicates infrared thermal imaging data, This represents the displacement data of the body skeleton. This represents remote photoplethysmography (PPG) data. This refers to electronic health record data.

[0079] in, The dimension can be , The dimension can also be , The dimension can be , The dimension can be 512. The dimension can be 256.

[0080] Each It can be viewed as a token-level unified frame packet, which corresponds to a set of multimodal tokens in the multimodal projection layer and will be directly sent to the large language backbone model.

[0081] Due to hardware clock skew, asynchronous frame rates, and spatial field-of-view differences, the raw data of multimodal signals (AU, Thermal, rPPG) exhibit spatiotemporal misalignment. This can lead to discrepancies between, for example, the peak time of facial action units (AU) and the peak value of rPPG pulse waves or thermal infrared temperature changes, directly impacting the accuracy of subsequent fusion and analysis. This invention addresses this by performing spatiotemporal synchronization processing on multimodal time-series data to obtain a multimodal frame packet sequence. It employs a common 25 fps clock + token-level unified frame packets and uses an interpolation-frame dropping framework to maintain AU-Thermal-rPPG peak alignment within a ±4 ms drift. This is the first time that multimodal non-contact signals have been synchronized to a single time grid, providing strictly spatiotemporally consistent input for cross-modal inference in large language models while reducing hardware costs.

[0082] Understandably, S1~S2 are for data acquisition and preprocessing. The main processing steps can be summarized as follows: the acquired raw data is first timestamped to establish a multimodal temporal correspondence. For the video frame sequence, MTCNN or MediaPipe algorithms are used for face detection and keypoint extraction; rPPG is calculated by extracting photoplethysmography using a non-contact camera (which can reuse the camera used for expression auto-analysis); infrared images undergo non-uniformity correction and body surface region segmentation; skeleton data is used to obtain a 3D joint coordinate sequence through pose estimation algorithms. After temporal interpolation and denoising, a multimodal frame packet sequence is formed and input into the subsequent encoding network.

[0083] S3, based on the single-modal time series data extracted from the multimodal frame packet sequence, performs single-modal self-supervised pre-training on the corresponding modal model;

[0084] After completing the synchronous input of multiple signals, self-supervised pre-training of each modality is required. The goal is to train each modality into a "useful feature extractor," enabling each modality to learn to extract useful features from the raw data. At this stage, each modality needs to understand the spatial structure and statistical patterns of the image itself, rather than directly learning pain labels. The supervisory signals truly linked to pain are introduced in later stages. The reason for this is that labels are scarce; currently, clinically labeled pain intensities from 0 to 9 require scoring by anesthesiologists, which is very costly. However, self-supervision can learn the spatial structure and statistical patterns of the image itself from massive amounts of unlabeled video, heatmap, and medical record text data at zero cost.

[0085] Furthermore, in the subsequent stages of multimodal language learning, it is necessary to maintain the expertise of each modality. For example, thermal infrared excels at temperature textures, skeleton excels at mechanics, and rPPG excels at periodic physiological signals. Each modality needs to learn its own "native language" first before it can complete cross-modal learning, which will result in more stable performance.

[0086] Furthermore, to avoid overfitting, if multimodal joint training is performed with a small number of labels from the outset, the training load would be too large. The model could easily learn biases from accidental data such as facial flushing equaling high pain. Therefore, single-modal self-supervised pre-training first learns a more general embedding (Embedding is a technique that maps discrete data (such as text, images, and audio) to continuous vectors, widely used in machine learning and natural language processing (NLP). Its core function is to convert non-numerical data into a format that neural networks can process, while capturing the semantic information and potential relationships of the data), which can significantly reduce this risk.

[0087] The goal of unimodal self-supervised pre-training is to reconstruct pixels / sequences under high occlusion and learn robust spatiotemporal representations. The main updated parameters are the encoders for each modality. During unimodal self-supervised pre-training, each modality's data undergoes feature extraction by an independent encoder. Self-supervised pre-training tasks, such as masked temporal modeling and frame order prediction, are employed to enhance the model's understanding of temporal patterns. The output feature embedding vectors represent facial expression dynamics, heatmap changes, and skeletal motion trends, respectively.

[0088] In one optional implementation, S3 may include the following steps:

[0089] S31, construct an encoder for each modality as the corresponding modality model, and construct the corresponding single-modality self-supervised pre-training task and self-supervised pre-training loss function;

[0090] The multimodal time-series data in this embodiment of the invention can contain a total of four modes, including:

[0091] The first modality: facial motion unit intensity sequence and infrared thermal imaging data sequence, abbreviated as AU / thermal infrared; the second modality: body skeleton displacement sequence, abbreviated as 2.Skeleton; the third modality: remote photoplethysmography (rPPG); the fourth modality: electronic health record sequence, abbreviated as medical record text.

[0092] Specifically, this involves constructing modal models for each modality, a single-modal self-supervised pre-training task, and a loss function:

[0093] The modality model for each modality is a deep learning model, which can be a large language model. An exemplary construction scheme is given below:

[0094] For the first modality corresponding to the facial motion unit intensity sequence and infrared thermal imaging data sequence, the modality model used includes Video-ViT or Swin-T, and the corresponding single-modality self-supervised pre-training task is Video-MAE; the loss function used includes L2 loss or Sharpened-CE loss.

[0095] For the second modality corresponding to the body skeleton displacement sequence, the modality model used includes Masked ST-GCN, the corresponding single-modality self-supervised pre-training task is Skeleton-MAE+JTMD, and the loss function used includes L2 loss + KL loss.

[0096] For the third mode corresponding to the remote photoplethysmography pulse wave, the modal model used includes 1D convolution + time-only Transformer, the corresponding single-mode self-supervised pre-training task is RPPG-MAE, and the loss function used includes time domain L2 loss + frequency domain Log-STFT loss.

[0097] For the fourth modality corresponding to the electronic health record sequence, the modality model used includes Clinical-BERT / BioGPT, the corresponding single-modality self-supervised pre-training task is the standard Masked-LM, and the loss function used includes cross-entropy loss.

[0098] Please refer to Tables 2 to 5 to understand the encoder design, self-supervised pre-training task, loss function / trick, data, and implementation points for each modality.

[0099] Table 2. Relevant content of the first modality self-supervised pre-training task.

[0100]

[0101] The first modality uses an encoder that can be a Vision Transformer (ViT), or a Video-ViT or Swin-T. Video-ViT and Swin-T are two very important visual encoders, both based on the Transformer architecture. "Video-ViT" refers to a model that applies the Vision Transformer (ViT) to video tasks (such as video classification and action recognition). In short, Video-ViT (S / 16) is a smaller version of the ViT model for video tasks, which divides each frame of the video into 16 segments. A 16-tile image is used, and the Transformer is used to capture both spatial and temporal relationships. (S / 16) means dividing a complete image into fixed-size, non-overlapping tiles (e.g., 16). 16 pixels), meaning the tile size is 16. 16. Swin-T (Shifted Window Transformer) is a significant improvement over the original ViT, aiming to address the computational burden and inductive bias problems of ViT, such as the lack of inherent hierarchical structure and translation invariance of images. The core idea of ​​Swin-T is the introduction of "layering" and "sliding windows." In short, Swin-T is an efficient and powerful visual encoder that, through a layered, sliding-window Transformer structure, maintains the global modeling capabilities of the Transformer while possessing the multi-scale feature extraction efficiency of CNNs. Details of these two encoders can be found in related technical explanations and will not be elaborated upon here.

[0102] The first type of modal self-supervised pre-training task can employ two advanced paradigms, aiming to reconstruct facial and thermal imaging features using a modal model. The first paradigm is based on mask reconstruction (e.g., Video-MAE, video mask autoencoder). This method randomly masks a very high percentage (e.g., 90%-95%) of spatiotemporal patches in the input video, feeding only a small number of remaining visible patches into the encoder. The goal of the modal model is to reconstruct the original pixel values ​​(for RGB) or temperature values ​​(for thermal infrared) of the masked patches from these few cues using a lightweight decoder. The second paradigm is based on self-distillation (e.g., DINOv2 in DINO, a self-supervised distillation representation learning method for obtaining robust visual features). This method creates two different randomly augmented views of the same input sample (e.g., different cropping, color dithering, etc.). The two views are passed through two encoders with identical structures (an online network and a target network) but different parameter update methods. The training objective is to enable the online network to learn to predict the feature outputs of the target network for the same image from different perspectives, thereby achieving self-knowledge distillation. The two paradigms mentioned above can be combined. These two paradigms were chosen because VideoMAE is strong in spatiotemporal video representation, while DINOv2 is strong in general visual representation and has good transferability.

[0103] The loss function can be L2 (i.e., L2 loss, also known as mean squared error loss) / Sharpened-CE (Sharpened Cross-Entropy Loss).

[0104] The first modality uses ViT to train facial autoencoders (AUs) to learn the relationship between facial action units and pain. Through a self-supervised pre-training task, such as Masked Autoencoder (MAE), a portion of a facial image is randomly covered, allowing the modality model to reconstruct the covered portion and learn facial expression changes. For thermal infrared data, ViT or a similar convolutional neural network model is used to learn pain-related temperature change patterns in thermal infrared images. After training, spatio-temporal tokens are obtained, understanding "which part of the face is moving + temperature gradient." Here, a token is a basic unit representing input information in the Transformer model. For Video-ViT, one token represents a spatio-temporal cube in the video (e.g., a 16-bit cube). (16-pixel tiles, across several consecutive frames). Spatio-Temporal representation is spatiotemporal; after training, each Spatio-Temporal Token becomes a highly refined, semantically rich feature vector. It is no longer a raw pixel or temperature value, but an abstract representation learned by the modal model. Understanding "which part of the face is moving" means that after watching massive amounts of video, certain tokens learn to correspond to specific facial muscle groups. For example, some tokens pay particular attention to the corner of the mouth area. When the feature vectors of these tokens change drastically, the encoder knows that "the corner of the face is moving" (possibly corresponding to AU12, the corner of the mouth lifter). Similarly, tokens focusing on the eyebrow area will capture frowning (AU4) or raising eyebrows (AU2). This "understanding" comes from the spatial properties of the tokens and the position-action associations learned from large amounts of data. Understanding "temperature gradients" means that for thermal infrared data, the modal model also generates tokens. These tokens learn to represent temperature patterns, not just single temperature values. For example, a token might represent "a gentle temperature gradient from the bridge of the nose to the cheek." More importantly, through spatiotemporal learning, modal models can associate temperature change patterns with facial movements. For instance, it might learn that "when the corners of the mouth are turned up (AU12 activation), the temperature in the cheek area changes in a specific pattern due to increased blood flow." Ultimately, these spatio-temporal tokens collectively form a powerful encoding of the overall facial dynamics and thermodynamic state. Downstream tasks can be performed based on these tokens that already "understand" facial movements and temperature, requiring only a simple classification or regression head, without needing to learn basic concepts from scratch, thus achieving efficient and accurate analysis.

[0105] In the implementation details, enter 224 224 16 frames refers to the input data size and dimension, and a mask ratio of 0.9 is the mask ratio. Each batch of data processed consists of 256 images, using the AdamW optimizer with a learning rate of lr 1e-4 and a cosine learning rate decay strategy. The training epochs are 600. Suitable datasets include BioVid RGB-T (pain) and AffectNet (expression).

[0106] Table 3. Relevant content of the second modality self-supervised pre-training task.

[0107]

[0108] The second modality employs Masked ST-GCN (ST-GCN stands for Spatiotemporal Graph Convolutional Network), a learning model specifically designed for skeleton sequence data. It was chosen because ST-GCN directly learns the "space-time" pattern on the joint graph, serving as a classic baseline for skeleton motion modeling. Training skeleton data with ST-GCN aims to teach the model to extract motion features from the temporal changes of joints. The core idea of ​​Masked ST-GCN is to force the modality model to learn the essential laws of skeleton motion by occluding most of the data. The goal of self-supervised pre-training here is to allow the modality model to learn spatiotemporal features through joint displacement reconstruction or temporal masking, making the model's predicted coordinates as close as possible to the true coordinates before occlusion. Through this pre-training, the Masked ST-GCN encoder learns how to generate high-quality skeleton sequence feature representations. This "learned" encoder can be easily transferred to various downstream tasks, requiring only the addition of a simple task head for fine-tuning, thus greatly reducing the dependence on large amounts of labeled data.

[0109] The corresponding single-modal self-supervised pre-training task can employ a composite paradigm combining mask reconstruction and knowledge distillation: Skeleton-MAE+JTMD (JTMD is a joint temporal motion discriminator). Skeleton-MAE (skeleton mask autoencoder) forces the model to think by creating challenging problems, randomly masking the data, and then allowing the model to reconstruct it. Specifically, a high-intensity JTMD is applied to the input skeleton sequence, randomly masking 40% of the joints (spatial mask) and 30% of the frames (temporal mask). The student network (Masked ST-GCN) only sees the remaining "fragments," and its core task is to reconstruct the original 3D coordinates of the masked joints. JTMD (Joint-Temporal Mask Distillation) introduces a teacher (teacher network) to guide the student, preventing the student from "learning incorrectly" (i.e., overfitting) and only learning to fill in the blanks (i.e., reconstruct) without learning general knowledge. In this model, the teacher network receives complete, unmasked sequences and produces high-quality, stable feature outputs (Logits / soft targets). The student network, while attempting to complete the "fill-in-the-blank" task, also has the additional task of making its understanding of the "incomplete" input output features as close as possible to the teacher network's understanding of the complete input. The loss functions used include L2 loss and KL loss (i.e., KL divergence loss). Through training, the modal model learns a deep understanding of human motion. The resulting joint motion dynamic encoder transforms the original skeleton coordinate sequence into a low-dimensional, dense, semantically rich feature vector, significantly improving the performance of downstream tasks and reducing reliance on large amounts of labeled data.

[0110] In terms of implementation details, the skeleton is normalized to the human body's center of mass, with an FPS of 25. This reflects the spatiotemporal standardization of the original skeleton data. Spatially, by "normalizing to the center of mass," the centers of all skeletons are unified to the origin of the coordinate system, eliminating interference from absolute spatial positions. Temporally, by "unifying to 25 fps," the playback speed of all sequences is standardized, eliminating interference from the acquisition time frequency. The ultimate goal is to ensure that the modal model receives "pure" motion information, focusing only on how different parts of the body move relative to the core and how this movement unfolds over time. This significantly reduces the learning difficulty of the modal model, allowing it to focus more on learning meaningful motion patterns, thereby achieving better performance and generalization ability.

[0111] Table 4. Relevant content of the third modality self-supervised pre-training task.

[0112]

[0113] The third modality employs a modal model that combines 1D convolutions with a time-only Transformer, i.e., 1D-CNN + Transformer temporal coding, which can learn subtle heartbeat-related rhythms from facial videos. Of course, pure Transformers, such as PhysFormer / PhysFormer++, can also be used.

[0114] The third type of modal self-supervised pre-training task is Masked Signal Modeling (rPPG-MAE) or contrastive learning (different augmented views are positive samples), the purpose of which is to enable the modal model to reconstruct the pulse wave. Masked Signal Modeling can mask 50% of the time window, reconstruct the waveform and spectrum, and the loss function used includes time-domain L2 loss + frequency-domain Log-STFT loss; after training, heart rate and HRV sensitive features are obtained.

[0115] Table 5. Relevant content of the fourth modality self-supervised pre-training task.

[0116]

[0117] The fourth modality uses a medical domain model from the BERT family, as domain-specific BERT is significantly superior to general-purpose BERT for clinical texts. Specifically, this could include Clinical-BERT or BioClinicalBERT. The goal is for the modality model to learn to extract meaningful medical information from text, especially pain-related descriptions. The self-supervised pre-training task is Masked-LM (Masked Language Modeling, or MLM for short, meaning masked prediction), which involves randomly masking some words using the BERT model and then having the model predict the masked words, allowing the model to learn medical semantics. The training paradigm is BERT-style, that is, pre-training the text encoder according to BERT's "masked language modeling MLM" approach to achieve self-supervised pre-training of the case / EHR text modality. Specifically, a case text is first segmented into a token sequence; approximately 15% of the tokens are randomly selected as "masked positions"; 80% of these are directly replaced with [MASK], 10% are replaced with a random word, and 10% remain the original word (this is BERT). The classic strategy); predictions are only made at these selected positions: the model recovers the original words, so that the text encoder can learn transferable representations such as medical terminology, symptom causality, and treatment semantics; the pre-trained text vectors are then mapped to the same word vector dimension as LLM through a "unified multimodal projection layer", and used together with other modalities for cross-modal alignment and instruction fine-tuning.

[0118] The loss function is Cross-Entropy, calculated only for the occluded location. Training is performed using masked language modeling to obtain medical vocabulary and causal relationship representations (arXiv). The corresponding loss function used includes cross-entropy loss.

[0119] S32, for each modality, without using any pain labels, based on the single-modal temporal data extracted from the multimodal frame packet sequence and the corresponding single-modal self-supervised pre-training task, the corresponding self-supervised pre-training loss function value is minimized to complete the single-modal self-supervised pre-training, thereby determining the pre-training weights and obtaining the modal model completed by the single-modal self-supervised pre-training.

[0120] The unimodal self-supervised pre-training task is like "laying the foundation" to create high-quality, task-independent underlying representations for each modality. It enables the "dedicated encoders" of each modality to learn high-quality representations. They do not directly learn the knowledge that "pain = 7 points", but rather provide solid general features for subsequent cross-modal fusion and pain-supervised fine-tuning, and extract highly robust representations in unlabeled / weakly labeled scenarios.

[0121] In single-modal self-supervised pre-training, each modality's data is standardized to ensure a consistent data format input into the modal models. Self-supervised learning (such as MAE or DINO) is used to train each encoder, enabling it to automatically learn feature representations from the data without relying on the painful annotation information of the training samples. After single-modal self-supervised pre-training, each modality's encoder (i.e., the modal model) outputs a set of high-dimensional vectors (e.g., 768-dimensional or 4096-dimensional), representing the features of each modality.

[0122] S4 takes all the modal models that have been pre-trained by single-modality self-supervised training and then connects them to a unified multimodal projection layer to obtain the pre-structure. The pre-structure is then encoded with cross-modal semantic alignment to map the output features of each modality to the word vector dimension of the large language model.

[0123] After self-supervised pre-training, each modality obtains its "native language" expression in an independent feature space. The dimensions, temporal lengths, and statistical distributions of each modality are inconsistent. However, the inference of the large language backbone model requires the LLM (LLM is a large language model, referring to the large language backbone model in S5) to understand the five features as "the same paragraph". It is necessary to project them onto a consistent word vector coordinate system and assign them comparable semantic distances. However, existing methods have many shortcomings. Therefore, it is necessary to design a new cross-modal alignment mechanism for temporal and medical five modalities.

[0124] All modal models pre-trained under unimodal self-supervised conditions will be used for cross-modal alignment, interfaced with the LLM through a unified multimodal projection layer and LoRA parameters. During actual inference, the multimodal tokens output from the projection layer are directly input into the LLM in the form of word vectors, thus achieving "cross-modal alignment". Mapping and availability of the LLM word vector space.

[0125] Specifically, S4 achieves cross-modal alignment. This process involves setting connectors for various modalities to perform dimensionality transformation, mapping the output features of each modal model (after self-supervised pre-training) to a unified semantic space to adapt to the processing requirements of the large language backbone model. Specifically, each modal feature is mapped to a unified word vector space via a multimodal projection layer (the output features of each modal model, such as Au outputting the encoding of facial motion units and skeleton outputting the skeleton, are ultimately transformed into word vectors), achieving cross-modal semantic alignment. A cross-attention mechanism is used to aggregate relevant features from different modalities, forming a unified multimodal representation vector. This vector is then fed into the large language model backbone for inference.

[0126] A temporal consistency strategy is adopted in the process of cross-modal semantic alignment. Please refer to Table 6 for understanding.

[0127] Table 6. Temporal Consistency Strategies for Cross-Modal Semantic Alignment

[0128]

[0129] Frame alignment, already completed in time alignment in S21, allows for further detection and supplementary processing. Long sequence pruning involves randomly truncating every 128 tokens to control memory usage and enhance robustness. Furthermore, after spatial registration in S22, during cross-modal alignment, positional encoding sharing can be further performed at the multimodal projection layer entry point. A learnable geometric mapping matrix maps facial motion unit coordinates, skeletal joint displacement vectors, and physiological signal sampling points to a unified three-dimensional virtual coordinate system (e.g., a spherical coordinate system). This allows the large language backbone model to explicitly model cross-modal spatial-topological correlations in multimodal attention computation, significantly improving the spatial consistency of interpretable heatmaps and pain attribution accuracy. In short, it's a process of unifying body position across different sensors. "Positional encoding sharing" is equivalent to adding "same coordinate system" positional embeddings (Patch-XY / Joint-ID / Sample-Idx) to different modalities at the projection layer entry point. (Spherical coordinates) allow LLM attention to align across modalities within the same geometric reference frame. In the cross-modal semantic alignment stage, InfoNCE is used to align the mapped tokens across modalities (multimodal data from the same time period / person are positive samples, while others within the same batch are negative samples), and geometric consistency regularization is applied to the position embeddings (e.g., upper / lower bounds on the embedding similarity between adjacent joints and their face / thermal infrared neighborhood). During subsequent fine-tuning of instructions, the position embeddings are kept learnable but with weight decay to avoid overfitting the spatial coordinates.

[0130] It should be noted that there is also an anomaly handling process, which means that pure time signals without coordinates (such as the full-face average of rPPG) can be replaced with spatial coordinates by embedding ROI ID + temporal location encoding; when missing / occluded, an "invisible token" is used as a placeholder to avoid misfocusing of attention.

[0131] Please see Figure 3 The cross-modal alignment process of embodiments of the present invention can be understood in Table 7.

[0132] Table 7. Relevant information about multimodal projection layers

[0133]

[0134] Each modality has a corresponding connector. LoRA-Linear typically refers to a linear layer based on LoRA (Low-Rank Adaptation) technology. LoRA is an efficient model fine-tuning method that incrementally adjusts the original weights through a low-rank matrix, rather than directly fine-tuning the entire model. rank = 32 means the rank is 32. A 2-layer MLP refers to a multilayer perceptron containing two fully connected layers. 1-D Conv + Linear refers to a neural network structure that combines one-dimensional convolutional layers and fully connected layers.

[0135] Proj2, Proj3, and Proj4 refer to projection layers of different "granularities" or "levels" that map features from different modalities to a unified semantic space. The alignment granularities are global, transitional, and local / fine-grained, respectively.

[0136] Patch-ID+Time applies to video modalities. Patch-ID divides each frame of video image into several small patches, each with a unique spatial identifier (such as a sequence number or XY grid coordinates), representing spatial information. Time refers to the timestamp of the frame in which the patch is located, representing temporal information.

[0137] Joint-ID + Time is applicable to 3D skeleton modalities. Joint-ID identifies a specific joint in the skeleton (such as "left elbow" or "right knee"), which represents structural information. Time is the timestamp of the joint's position data.

[0138] Sample-Idx is applicable to the rPPG modality, which typically refers to the sample index within a batch. When a model needs to process multiple data samples (such as multiple video clips or multiple motion sequences), Sample-Idx is used to distinguish data from different samples. In advanced architectures, it can also serve as a "batch-level" location information, ensuring that the model does not confuse tokens from different samples.

[0139] Token-Pos is the most general and common positional encoding, derived from the Transformer architecture. It represents the absolute ordinal position of a token in the input sequence. Whether it's a text token, an image patch token, or a joint token, they will all obtain a Token-Pos after being flattened into a long sequence.

[0140] In this embodiment of the invention, the feature dimensions of each modality are mapped to 4096 (i.e., the word vector dimension of the selected LLM), and the text "directly reuses LLM word vectors (Token-Pos)," as shown in Table 6.

[0141] After cross-modal semantic alignment is completed, all modal features can be directly input into the LLM as 'multimodal sentences'. This means that "alignment has been completed in the word vector space and can be treated as tokens by the LLM".

[0142] The semantic alignment objective is to use the contrastive learning loss InfoNCE to make the projected visual token close to the synchronous text caption; at the same time, cross-modal distillation (KD) is performed to transfer the knowledge of unimodal teachers (such as CLS of Thermal-ViT / AU-ViT) to the projected token representation. This explains that "it is not just linear dimensionality" but that comparable semantic distances are learned in a 4096-dimensional word vector space.

[0143] In contrastive learning, the supervision signal is positive and negative pairings (same frame / patient vs. others in the batch). The training objective is to bring matching modal pairs closer together and mismatched pairs further apart in the LLM word vector space, achieving cross-modal semantic alignment. Cross-modal distillation refers to teacher-student distillation, where the supervision signal is the soft distribution of the teacher's output. The training objective is to transfer the strong single-modal representations learned after single-modal self-supervised pre-training to the projected student tokens, ensuring discriminative power before entering the LLM.

[0144] Specifically, the loss function for contrastive learning (InfoNCE) is as follows:

[0145] ;

[0146] in, Indicates the batch size of the sample. It can be 256; Indicates temperature. ; It is a visual token fused in the same frame (AU+Skel+Thermal+rPPG average pooling); It can be a doctor's description or an automatically generated caption (automatically generated captions use artificial intelligence models to automatically generate descriptive text for medical images).

[0147] Here, "Visual Token" refers to a unified feature unit formed by encoding and mapping data from different visual modalities (such as facial expressions, skeletons, infrared, and rPPG signals). It is equivalent to the model's "vocabulary" or "minimum semantic unit" in the visual domain, serving as the fundamental unit for large language models to understand visual information from multimodal inputs. The Visual Token is obtained by fusing four modal features—facial action units (AUs), skeletal joint features (Skel), infrared temperature features (Thermal), and photoplethysmography (rPPG) signals—through average pooling. It is used to construct multimodal semantic alignment input units, ensuring consistent representation of visual information from different modalities within a unified feature space. This is because direct concatenation of features with different dimensions, sampling rates, and time steps would lead to inconsistent feature dimensions. Therefore, average pooling compresses the temporal or spatial features of each modality into a fixed-length vector representation, thereby achieving intermodal alignment and pain weight integration.

[0148] The visual token is obtained by fusing four modal features—facial action unit (AU), skeletal joint features (Skel), infrared temperature features (Thermal), and photovolume wave signal (rPPG)—through average pooling. It is used to construct a multimodal semantic alignment input unit, so that visual information from different modalities can be expressed consistently in a unified feature space.

[0149] Cross-modal distillation (Teacher) The loss function for Token is as follows:

[0150] ;

[0151] in, Indicates the Kullback–Leibler divergence; This represents the Softmax function; It's a temperature parameter. ; This indicates that Thermal-ViT / AU-ViT freezes the teacher's CLS; This represents the corresponding CLS after projection.

[0152] Therefore, the total loss function is expressed as:

[0153] ;

[0154] yes The weights are optional. .

[0155] For an understanding of training and hyperparameters in cross-modal semantic alignment, please refer to Table 8.

[0156] Table 8. Relevant content regarding training and hyperparameters in cross-modal semantic alignment.

[0157]

[0158] This invention first uses LoRA-Linear / Q-Former to map four non-textual features to a shared space with the same dimensions as LLM word vectors; then, InfoNCE + KD consolidates cross-modal consistency and discriminative power; and the "common 25 fps time axis + unified spherical position encoding" ensures geometric alignment at the attention level. See Table 8, Optimizer=AdamW, "only update projection layer + LoRA", meaning that at this stage, most of the LLM and modal backbone are frozen, and only the LoRA adaptation parameters in the multimodal connector and LLM are trained. In other words, the learning process of "dimensional mapping + semantic alignment" is completed on the unified multimodal projection layer and the LoRA adaptation parameters in the LLM, while most parameters of the modal encoder and LLM remain frozen. It can be seen that this invention uses lightweight constraints, with total parameters <1% LLM weights, supporting single-card fine-tuning, and achieving stable and low-cost fine-tuning.

[0159] This invention is based on the CLIP-style contrastive learning framework, and utilizes "same-frame RGB" The invention employs the "InfoNCE" loss function for thermal imaging, combined with shared coordinate system position encoding (AU region / joint ID / hotspot image patch). It maps the feature vectors output by each modal encoder (a modal model pre-trained by a single modality) to a unified large language model word vector dimension (4096 dimensions). See the previous section on "Unified Frame Package Format" for further understanding. Each FramePackage corresponds to a set of multimodal tokens in the projection layer, directly fed into the LLM. It is evident that the multimodal projection layer outputs a token sequence in the LLM word vector dimension, thus enabling seamless integration into the LLM's embedding / attention channels. This invention, through cross-modal semantic alignment, ensures that features from different modalities can be compared in the same space, thereby solving the problem of multimodal embedding distribution differences. After cross-modal semantic alignment, all modal features can be directly input into the LLM as "multimodal sentences," laying a unified semantic foundation for subsequent fine-tuning and interpretable regression.

[0160] Overall, this process first employs a low-rank adaptive linear layer (LoRA-Linear) or a query transformer (Q-Former) to map four non-textual features to a shared semantic space (4096 dimensions) with the same dimensions as LLM word vectors. This enables LLM to seamlessly process visual / physiological features while preserving spatiotemporal information, supporting token-level differentiable positional encoding with minimal parameters and hot updates at the inference end. Secondly, InfoNCE pairs projected visual tokens with synchronized text captions, resolving cross-modal semantic disconnect. KD distillation of the discriminative power from a single-modal teacher preserves the fine-grained features of each modality's "native language." Finally, a common 25 fps timeline and unified spherical positional encoding ensure that LLM can directly align to a specific facial patch at the attention level. Temperature hotspots "Corresponding to skeletal joints". Compared to CLIP's single-image alignment and BLIP-2's single-image Q-Former solution, this solution is the first to support the unification of five temporal modalities into LLM, and introduces cross-modal distillation + temporal position sharing to improve robustness in medical scenarios.

[0161] S5, the overall structure is obtained by connecting the pre-structure to the large language backbone model, inputting the patient's multimodal temporal data, and outputting the pain assessment results by instructing the overall structure. Self-supervised learning is used to train the model; the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is used to assess the patient's pain.

[0162] In the previous step, cross-modal semantic alignment was completed, projecting the five outputs AU / Skeleton / rPPG / Thermal / EHR into a unified multimodal token sequence. Next, the LLM needs to first learn to understand instructions, such as "Please provide a continuous pain score and explain the reason based on the following multimodal information"; secondly, it needs to output actionable results: continuous pain intensity from 0-9, corresponding discrete levels, interpretable heatmap hints, etc. A unified large language backbone model and multi-head task output are chosen here because it needs to support question-and-answer responses, and it's for cross-modal heatmaps of continuous images, requiring the pain output to be a continuous regression interface. S5 involves instruction fine-tuning (SFT). This part is a supervised (or weakly supervised) step, the core of which is to use task instructions + target output to perform cross-entropy / regression loss; whether to use contrast / distillation is an optional auxiliary item. The supervision signal is the explicit target output (pain 0–9, JSON, report template, etc.), and the training objective is to teach the LLM to "speak in a human-like manner according to instructions / output structured results"; if necessary, the visualization task head will be trained. The main updated parameter is LoRA + (optional) task header, but the main branch is mostly frozen.

[0163] In brief, the fused multimodal features are input into a large language backbone model (represented in LLM format throughout). The model learns temporal and semantic dependencies through a multi-layer Transformer structure. The output includes three types of results:

[0164] 1. Pain level (discrete labels: no pain, mild, moderate, severe);

[0165] 2. Continuous Pain Score (quantified value from 0 to 100);

[0166] 3. Interpretable heat maps (reflecting areas of significant pain and pain-related areas).

[0167] In one optional implementation, in S5, the large language backbone model adopts an LLaMA-2 / 3 or a medical-grade GPT architecture of similar scale.

[0168] Please see Figure 4 Understanding the instruction fine-tuning of this embodiment of the invention, where stage 1 refers to steps S1 to S3, stage 2 refers to steps S4 to S5, and stage 3 is the output stage. S5, based on instruction fine-tuning data of "multimodal cues → pain scoring and interpretation", adopts a fine-tuning strategy of freezing the pre-layer + post-layer LoRA injection to endow the model with cross-modal reasoning and natural language interpretation capabilities.

[0169] In S5, the overall structure is obtained by connecting the pre-structure to the large language backbone model. The patient's multimodal temporal data is input, and the pain assessment results are output by instructing the overall structure to fine-tune the large language model. The resulting trained pain assessment model includes:

[0170] 1) The multimodal temporal data of the patient is used as training samples to input the overall structure, and after passing through the pre-structure, a multimodal token sequence is obtained that is mapped to the word vector dimension of the large language model;

[0171] 2) The large language backbone model receives the multimodal token sequence, processes it using a preset Prompt template and multi-task head, and outputs the pain assessment result; among which, the interpretable heatmap in the pain assessment result is to identify the pain contribution area in the image corresponding to the multimodal time series data used as training samples.

[0172] 3) Receive feedback information on the pain assessment results of the training samples, and use the method of updating only the multimodal projection layer + LoRA to learn and train, so as to optimize the local model parameters before the large language backbone model, and obtain the trained pain assessment model after multiple iterations of optimization.

[0173] It should be noted that the pain assessment model construction method for S1 to S5 adopts a two-stage training approach. The first stage refers to the self-supervised pre-training stage corresponding to S3, which uses unlabeled multimodal data to enhance feature representation capabilities. The second stage refers to the supervised fine-tuning stage corresponding to S5, which uses labeled pain level and text description samples for incremental training to achieve semantic alignment and interpretable output.

[0174] In the training process corresponding to S5, a two-stage fine-tuning strategy is adopted, specifically:

[0175] Phase 1 (Instruction Tuning) only uses the multimodal projection layer + LoRA to train the LLM to "understand the question → generate text".

[0176] Phase Two (Domain Fine-tune + Heads) unlocks regression / gradation heads, and combines continuous pain annotations for fine-tuning.

[0177] Phase 1 continues to use the "only enable projection layer + LoRA" strategy to achieve efficient alignment with low parameter count and low video memory. During this process, S4 semantic alignment encoding will be adjusted again. Phase 2 unlocks the regression / ordinal task head for domain fine-tuning.

[0178] During the Phase Two fine-tuning, the overall model corresponding to the overall structure is trained to generate pain scores from inputs of different modalities by giving explicit instructions (e.g., "Assess the pain level based on facial expressions and case text"). Through these instructions, the overall model learns how to generate correct outputs based on the task, such as pain scores (0–9) and related interpretations. The pain assessment results can achieve both pain scores and pain levels, such as a clinically compatible dual quantification system that supports precise numerical values ​​(e.g., 6.3 points) and grade classifications (e.g., Level IV moderate to severe). See Tables 9-11 for further explanation.

[0179] Table 9 contains information related to instruction fine-tuning (SFT).

[0180]

[0181] Table 10: Pain Monitoring Detailed Adjustment + Head Design Related Content for Multi-Task Heads

[0182]

[0183] A two-stage joint self-supervised learning training is performed, outputting continuous values ​​from 0 to 9 using classification (CLS) tokens. Ordinal logistic regression (CORAL) / ordered softmax hierarchical classification are implemented in parallel, compatible with both regression and hierarchical multi-task loss functions. The joint loss function used is:

[0184] ;

[0185] in, , , These are the MSE loss function (mean squared error loss function), the CORAL loss function (correlation alignment loss function), and the CE loss function (cross entropy loss function). , , These are the corresponding weights, and their values ​​can be respectively... : : =1:0.5:0.2.

[0186] The generation of the interpretable heatmap, after the multimodal large language model inference is completed, aims to visually present the model's focus areas and pain attribution locations, thereby achieving interpretability of the results. The interpretable heatmap generation process is as follows:

[0187] (1) Cross-Attention Extraction

[0188] From the fine-tuned Large Language Backbone Model (LLM), the cross-modal attention weight matrix of the last two layers is extracted. This weight matrix reflects the matching strength between the text query and the visual feature key, and has the following shape:

[0189] ;

[0190] in, These correspond to the height and width of the visual feature map, for example, 14×14 or 28×28 patch; This represents the original cross-modal attention weight matrix; It represents the space of real numbers.

[0191] (2) Normalization and Resizing

[0192] First, the attention weight matrix is ​​renormalized using the Softmax function, bringing all weight values ​​to the range [0,1], representing the attention intensity of the model at different spatial locations. The formula is as follows:

[0193] ;

[0194] Here, the attention weight matrix is ​​renormalized using the Softmax function, bringing all weight values ​​to the range [0,1], representing the attention intensity of the model at different spatial locations. Then, the normalized attention map is upsampled (interpolated) to a resolution of 224×224 to match the input image or modal feature space.

[0195] In the above formula, This indicates the attention intensity after Softmax normalization to [0,1]. These are all spatial indices, representing patch or pixel positions;

[0196] (3) Spatial Back-Projection

[0197] Based on the coordinate system of each modality, the upsampled heatmap is projected back to the original space. Specifically: for AU modality, it is mapped to the center point of the corresponding Action Unit Patch region of the face; for Skeleton modality, it is mapped to joint coordinates or limb connection lines; for Thermal modality, it is mapped to the coordinates of infrared image pixels.

[0198] This mapping ensures that regions of interest from different modalities can be superimposed on the same visualization plane, thereby achieving cross-modal interpretability.

[0199] (4) Visual synthesis -Blend Visualization)

[0200] Use transparency blending ( -Blend) overlays a heatmap onto the original image, expressed by the formula:

[0201] ;

[0202] in, This represents the overlaid image; Represents a heatmap; , which represents the fusion weight between the heatmap and the original image.

[0203] Different attention intensities are displayed using color gradients (e.g., blue-red gradients), with red areas representing key areas that the model identifies as pain-related.

[0204] (5) Output Results

[0205] The final interpretable heatmap output can: highlight muscle activity areas that the model considers "most relevant to pain" (such as AU4 / 6 / 9, etc.) in the face region; mark pain-related movement displacements in the skeleton view; present local temperature abnormality areas in the thermal infrared image; and support doctors or users to intuitively verify the model's decision-making basis.

[0206] In short, in interpretable heatmap generation, the cross-modal attention matrix of LLM is extracted, normalized using Softmax, and spatially upsampled. Then, the weight heats are mapped back to the coordinate space of each modality, and so on. -Blend overlays the image onto the original image, generating a uniform color heatmap overlay through multi-level transparency blending. This visualizes pain hotspots, body motion peaks, and temperature gradient changes, enabling spatial visualization and interpretation of pain attribution.

[0207] If described using the uniqueness of the algorithm, it can be represented as:

[0208] enter: From the last layer of the LLM, the output is: ;

[0209] ; ;

[0210]

[0211] in, This refers to the dimension of the key vectors, which is the feature dimension of each vector in the key matrix. It is usually the same as the dimension of the query vector, typically a value such as 64, 128, 256, or 512.

[0212] Table 11. Relevant content of the training-evaluation pipeline.

[0213]

[0214] Among them, BLEU stands for Bilingual Evaluation Understudy; MAE stands for Mean Absolute Error; CC stands for Concordance Correlation Coefficient; and Ordinal-Acc stands for Ordinal Accuracy.

[0215] The embodiments of the present invention have several innovations in steps S4 and S5:

[0216] Innovation 1: Employing LoRA + projection layer fine-tuning, this method efficiently connects multimodal token sequences with LLM instructions, achieving full-parameter fine-tuning with less than 30% memory usage.

[0217] Innovation 2 introduces a multi-task output head: simultaneously outputting continuous scores, hierarchical labels, and natural language explanations; joint loss ensures both quantitative accuracy and readability.

[0218] Innovation 3 utilizes Cross-Attention interpretable heatmaps to visually display pain contribution areas on the AU region, skeletal joints, and thermal infrared pixels, enabling doctor-patient collaboration.

[0219] The aforementioned innovations enable the overall model trained by this invention to not only "calculate scores" but also "speak and draw diagrams," thereby achieving end-to-end, interpretable, and continuous non-contact pain assessment, which is significantly superior to existing single-modal classification or static VLM methods.

[0220] In the above process, LoRA's lightweight trainable layer / adapter is used in two places: 1. As part of a unified multimodal projection layer, it maps the features of each modality to the same word vector dimension as LLM; 2. As an injected fine-tuning parameter of LLM, it updates only a small number of LoRA weights and freezes most of the base during the instruction fine-tuning stage, thereby achieving alignment and generation capabilities at low cost.

[0221] It is important to emphasize that in stages S4 and S5, this invention does not train all model parameters corresponding to the overall structure. Instead, it optimizes model performance for the task (painful expressions and multimodal pain assessment) through incremental training or fine-tuning. The LoRA + multimodal projection layer is the only part that needs to be updated, while most parameters of the modal encoder and the large language backbone model are frozen. The reason for adopting this training method is as follows:

[0222] 1) Computational efficiency: Large language models such as LLaMA typically have hundreds of billions or even trillions of parameters. Full parameter training requires a large amount of computational resources, while incremental training only updates a small portion of the parameters, which can significantly reduce computational and storage overhead and ensure minimal GPU memory usage.

[0223] 2) Utilization of pre-trained knowledge: Models such as LLaMA have been pre-trained on large-scale text data and have learned a great deal of general linguistic knowledge. In incremental training, we retain this knowledge and fine-tune (incremental training) the model to better adapt it to the specific task (pain expression evaluation). This approach not only improves efficiency but also better utilizes the capabilities of the pre-trained model.

[0224] 3) Task-specific adaptation: Through fine-tuning, the model can learn task-related knowledge (the relationship between facial expressions and pain intensity) while retaining its original language abilities. Therefore, it can perform better on multimodal tasks without needing to train the model from scratch.

[0225] During training, InfoNCE (contrast loss) and KL distillation loss were used to ensure the alignment and generation of each modality feature. After modality alignment and instruction fine-tuning, the overall model can accept case text, facial AU, skeletal data and thermal infrared images as input, and output pain level (0-9), pain explanation text, and three visualization results (AU heatmap, skeletal displacement map, thermal infrared temperature difference map).

[0226] Once a pain assessment model has been trained, it can be used to assess the pain of patients.

[0227] Table 12 lists some typical classification schemes for painful expressions.

[0228] Table 12 Comparison of Existing Technologies

[0229]

[0230] In summary, current methods for classifying expressions of pain have the following shortcomings:

[0231] 1. Insufficient modal coverage: Traditional systems have a maximum of RGB-DT three modes, and do not simultaneously integrate five heterogeneous data channels such as AU-RGB, Skeleton, rPPG, and EHR, resulting in limited information utilization.

[0232] 2. Lack of a unified feature space: Each modality often relies on early splicing or later voting, without a shared semantic coordinate system like "projection layer → large language model", resulting in weak cross-modal reasoning ability.

[0233] 3. Limited accuracy and robustness: Single-modal or adhesive solutions are susceptible to factors such as lighting, occlusion, and sensor detachment; clinical validation shows that multimodal solutions can improve the recognition rate by ≥ 6%, but lack an end-to-end framework (CVF open access).

[0234] 4. Lack of interpretable continuous output: Existing methods mostly output discrete levels or probabilities, and cannot provide continuous intensity + multimodal heatmaps from 0 to 9, making them difficult for medical staff to trust and intervene in.

[0235] 5. Real-time and privacy are difficult to coexist: Solutions that rely on hardware synchronization or cloud inference find it difficult to simultaneously meet bedside latency of <50 ms and GDPR / HIPAA privacy requirements.

[0236] 6. Lack of integration with large language models for clinical reasoning: The latest VLM / MLLM (such as LLaVA-Med) focuses on image question answering, but there is still no solution for time-series physiological signals + continuous regression tasks, nor has it achieved simultaneous output of natural language interpretation and quantitative indicators.

[0237] The pain assessment model construction method provided in this invention can collect and integrate five multimodal heterogeneous data in a non-contact manner, thereby improving information utilization and enhancing the accuracy and robustness of the model's pain assessment. Furthermore, by incorporating a large language model, the trained model can achieve pain level classification and interpretable continuous output, facilitating intuitive visualization and guidance for subsequent diagnosis.

[0238] Secondly, corresponding to the embodiments of the above-mentioned pain assessment model construction method, this invention also provides a non-contact continuous pain assessment method. The executing entity of this non-contact continuous pain assessment method can be a non-contact continuous pain assessment device, which can operate in an electronic device. This electronic device can be a server or a terminal device, more preferably a portable device placed near the patient, but it is not limited thereto. The application scenarios of this invention can be hospital ICUs, etc., where patients may have difficulties with movement, lack of verbal expression ability, etc., but it is not limited thereto. Figure 5 As shown, this non-contact continuous pain assessment method includes the following steps:

[0239] S100, acquire the target multimodal temporal data of the patient to be evaluated; the target multimodal temporal data includes electronic health record sequence, facial motion unit intensity sequence, body skeleton displacement sequence, remote photoplethysmography pulse wave and infrared thermal imaging data sequence;

[0240] S200, input the target multimodal time series data into the trained pain assessment model to obtain the patient's pain assessment results; wherein, the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is trained according to the pain assessment model construction method in the first aspect.

[0241] In S100, the method for collecting the target multimodal time-series data for patient evaluation can be understood by referring to the relevant content in the first section above. It is understood that this multimodal time-series data does not contain pain-related annotation information.

[0242] It should be noted that after the first stage of training, no further spatiotemporal synchronization processing is needed. The pain assessment model is already trained and does not require single-modal self-supervised pre-training during use. Each modality model already possesses the ability to learn and output, directly outputting features to the large language backbone model. At this point, the inference stage only requires inputting real-time multimodal video streams to obtain continuous pain assessment results. Specifically, the pain assessment model generates a pain score (0–9) and outputs explanatory text based on the input medical record text, facial AU (analogous area), skeletal data, and thermal infrared images, describing the cause and condition of the patient's current pain. Simultaneously, the pain assessment model generates relevant images (interpretable thermal images) through visualization (such as AU thermal maps, skeletal displacement maps, thermal infrared temperature difference maps, etc.) to help doctors more intuitively understand the patient's pain state. See also... Figure 6 and Figure 7 understand. Figure 6 This is an rPPG interpretable heatmap of an embodiment of the present invention, which superimposes an AU heatmap on an rPPG map to show pain-related pulse wave regions. Figure 7 This is a thermal infrared temperature difference map of an embodiment of the present invention.

[0243] The non-contact continuous pain assessment method provided in this invention acquires the patient's target multimodal temporal data to be assessed in a non-contact manner, performs spatiotemporal synchronization processing to obtain target multimodal frame packets, and then inputs the target multimodal frame packet sequence into a trained pain assessment model to obtain the patient's pain assessment results. The pain assessment results include pain level, pain score, and interpretable heatmap. The pain assessment model of this invention can not only provide a textual description of the pain level, but also generate corresponding image outputs (AU heatmap, skeletal displacement map, and thermal infrared map), which facilitates intuitive display and understanding of the patient's pain state and has strong interpretability.

[0244] Meanwhile, in terms of hardware implementation, this invention can provide real-time inference and human-computer interaction interfaces, such as a REST / WebSocket interface with a latency of ≤50ms, to support real-time confirmation / correction of output by medical staff (human-in-the-loop mechanism). Correction feedback automatically triggers LoRA incremental updates to enhance clinical safety and human-computer interaction experience.

[0245] Moreover, this invention provides a privacy and security compliance mechanism, such as using the FHIR standard to de-identify electronic medical records, locally encoding video data in real time and clearing the cache, transmitting data with TLS+AES256 encryption and deploying edge computing, etc., to meet hospital information security and GDPR / HIPAA compliance requirements.

[0246] Furthermore, this invention features a resource-efficient and scalable design, with projection layer parameters accounting for less than 1% of the LLM weight, supporting end-to-end operation on a single 24GB GPU, and employing a modular interface design for easy expansion into new modalities such as voiceprint / EEG. This ensures that the overall hardware device is lightweight, easily portable, and highly scalable.

[0247] In summary, this invention provides a non-contact, continuous pain assessment method based on a multimodal large language model, belonging to the interdisciplinary field of artificial intelligence and medical image recognition. This method simultaneously acquires non-contact, time-series data such as facial videos, infrared temperature fields, skeletal posture, and physiological signals from patients using multimodal sensors. It then utilizes self-supervised learning and cross-modal semantic alignment strategies to construct a unified pain assessment model, achieving continuous, objective, and interpretable assessment of the patient's pain level.

[0248] During the data acquisition phase, video cameras, infrared imaging modules, and depth sensors are used to simultaneously acquire patient facial expression dynamics, body surface thermal distribution, and limb posture information, forming a spatiotemporally aligned multimodal frame packet sequence. After temporal interpolation and noise correction, the data from each modality are input into the corresponding single-modal feature encoder for self-supervised pre-training to extract temporal feature vectors such as facial muscle action units, heat map texture distribution, and skeletal angle changes.

[0249] In terms of model structure design, this invention adopts a combination of multimodal projection layers and a large language model backbone structure. Features from each modality are mapped to a unified word vector space through the projection layer, achieving cross-modal semantic alignment and shared representation. Subsequently, the fused feature vectors are input into the large language model backbone for multi-layer temporal reasoning and attention modeling, generating corresponding pain descriptions and quantitative assessment outputs. The output results include pain levels (grading labels), pain scores (continuous numerical values), and corresponding interpretable heatmaps (visualization of highly relevant regions).

[0250] During model training, a strategy combining self-supervised learning and multimodal instruction fine-tuning is employed. In the self-supervised phase, the model's cross-temporal correlation ability is enhanced by predicting masked fragments of temporal features. In the instruction fine-tuning phase, manually labeled pain level and semantic description samples are used for incremental training to strengthen the model's clinical interpretability. In the inference phase, only real-time multimodal data from the patient needs to be input; the trained model then outputs continuous pain assessment results, achieving contactless, real-time intelligent analysis.

[0251] The beneficial effects of this invention are as follows: by introducing cross-modal semantic alignment and the unified representation capability of large language models, deep fusion of image, infrared, posture and physiological information is achieved, which effectively improves the robustness and generalization of pain recognition; at the same time, the output heat map results are interpretable and can help doctors understand the basis of model judgment.

[0252] In postoperative analgesia, rehabilitation training, and chronic pain management, this invention's system continuously outputs a pain level curve by capturing real-time changes in the patient's face and posture, assisting doctors in adjusting analgesic dosage in a timely manner. Compared to traditional assessment methods based on subjective scales, this invention achieves non-contact, continuous, and interpretable pain assessment, significantly improving assessment efficiency and objectivity. It can be widely applied in medical scenarios such as postoperative analgesia management, chronic pain monitoring, and rehabilitation efficacy evaluation, possessing significant clinical practical value and promising prospects for widespread application.

[0253] Thirdly, embodiments of the present invention also provide a non-contact continuous pain assessment system, such as... Figure 8 As shown, it includes:

[0254] The multimodal acquisition module is used to acquire the target multimodal time-series data of the patient to be evaluated; the target multimodal time-series data includes electronic health record sequence, facial motion unit intensity sequence, body skeleton displacement sequence, remote photoplethysmography pulse wave and infrared thermal imaging data sequence;

[0255] The pain assessment module is used to input the target multimodal time-series data into the trained pain assessment model to obtain the patient's pain assessment results. The pain assessment results include pain level, pain score, and interpretable heatmap. The pain assessment model is trained according to the pain assessment model construction method in the first aspect.

[0256] The functions and processes of each module in the non-contact continuous pain assessment system can be found in the description in the second part, and will not be detailed here.

[0257] Furthermore, the non-contact continuous pain assessment system provided in this embodiment of the invention corresponds to a pain assessment model construction system during the training of the pain assessment model. As described above, it may include a multimodal acquisition module, a data synchronization and preprocessing module, a feature encoding module, a multimodal fusion and semantic alignment module, a large language model inference module, and a result visualization module. The function and processing of each module can be found in the description of the first aspect, and will not be detailed here.

[0258] Fourthly, embodiments of the present invention also provide an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0259] Memory, used to store computer programs;

[0260] A processor, when executing a program stored in memory, implements the steps of any of the methods provided in the first and / or second aspects of the embodiments of the present invention.

[0261] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0262] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0263] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0264] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0265] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made here; any electronic device that can implement this invention falls within the protection scope of this invention. A preferred embodiment is to use a portable, miniaturized electronic device placed near the patient.

[0266] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for constructing a pain assessment model, characterized in that, include: S1, acquire the patient's multimodal time-series data; wherein, the multimodal time-series data includes electronic health record sequence, facial motion unit intensity sequence, body skeleton displacement sequence, remote photoplethysmography pulse wave and infrared thermal imaging data sequence; S2, perform spatiotemporal synchronization processing on the multimodal time-series data to obtain a multimodal frame packet sequence, wherein each multimodal frame packet contains multimodal data of the frame after spatiotemporal synchronization processing; S3. Based on the single-modal temporal data extracted from the multimodal frame packet sequence, perform single-modal self-supervised pre-training on the corresponding modal models. Specifically, for the first modality corresponding to the facial motion unit intensity sequence and infrared thermal imaging data sequence, the modal model used includes Video-ViT or Swin-T, and the corresponding single-modal self-supervised pre-training task is Video-MAE; the loss function used includes L2 loss or Sharpened-CE loss. For the second modality corresponding to the body skeleton displacement sequence, the modal model used includes Masked ST-GCN, and the corresponding single-modal self-supervised pre-training task is Skeleton-MAE+JTMD, with the loss function including L2 loss + KL loss. For the third modality corresponding to the remote photoplethysmography (LPG) pulse wave, the modal model used includes a 1D convolutional layer + a time-dimensional Transformer encoder, and the corresponding single-modal self-supervised pre-training task is RPPG-MAE, with the loss function including time-domain L2 loss + frequency-domain Log-STFT. Loss; For the fourth modality corresponding to the electronic health record sequence, the modality model used includes Clinical-BERT / BioGPT, the corresponding single-modality self-supervised pre-training task is the standard Masked-LM, and the loss function used includes cross-entropy loss; S4. All modal models that have been pre-trained by single-modality self-supervised training are followed by a unified multimodal projection layer to obtain the pre-structure. The pre-structure is then encoded with cross-modal semantic alignment to map the output features of each modality to the word vector dimension of the large language model. S5, the overall structure is obtained by connecting the obtained pre-structure with the large language backbone model, the patient's multimodal temporal data is input, and the large language model is fine-tuned by instructing the overall structure to output pain assessment results, so as to obtain the trained pain assessment model; wherein, the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is used to assess the patient's pain.

2. The method for constructing a pain assessment model according to claim 1, characterized in that, In S2, the multimodal time-series data undergoes spatiotemporal synchronization processing to obtain a multimodal frame packet sequence, including: S21, based on the nearest neighbor-interpolation-frame dropping rule, maps the patient's multimodal time series data to a common time axis to achieve time alignment at a preset frame rate; S22, Spatial registration is performed on the patient's time-aligned multimodal temporal data; the spatial registration includes spatial registration between the first type of images and the second type of images, spatial registration between the first type of images and the third type of images, and spatial registration between different types of the first type of images; wherein, the first type of images includes RGB images corresponding to facial motion unit intensity and RGB images corresponding to remote photoplethysmography pulse waves, the second type of images are images corresponding to body skeletal displacement; the third type of images are infrared images corresponding to infrared thermal imaging data. S23, each frame of multimodal data after patient spatial registration is used to form a multimodal frame packet, thereby obtaining a multimodal frame packet sequence according to the time order; Each multimodal frame packet is represented as: ； in, Indicates time Multimodal frame packets, This represents facial motion unit intensity data. Indicates infrared thermal imaging data, This represents the displacement data of the body skeleton. This represents remote photoplethysmography (PPG) data. This refers to electronic health record data.

3. The method for constructing a pain assessment model according to claim 2, characterized in that, In S3, based on the single-modal temporal data extracted from the multimodal frame packet sequence, single-modal self-supervised pre-training is performed on the corresponding modal model, including: For each modality, an encoder is constructed as the corresponding modality model, and a corresponding single-modality self-supervised pre-training task and self-supervised pre-training loss function are constructed. For each modality, without using any pain labels, based on the single-modal temporal data extracted from the multimodal frame packet sequence and the corresponding single-modal self-supervised pre-training task, the corresponding self-supervised pre-training loss function value is minimized to complete the single-modal self-supervised pre-training, thereby determining the pre-training weights of the modality model and obtaining the modality model completed by single-modal self-supervised pre-training.

4. The method for constructing a pain assessment model according to claim 1 or 3, characterized in that, In S4, cross-modal semantic alignment encoding is performed on the preceding structure, including: Connectors are set up for each modality to perform dimensional transformation, mapping the output data of each modality model that has been pre-trained by single modality self-supervised to a unified semantic space, so as to adapt to the processing needs of large language backbone models.

5. The method for constructing a pain assessment model according to claim 1, characterized in that, In S5, the large language backbone model adopts the LLaMA-2 / 3 or a similarly sized medical-grade GPT architecture.

6. The method for constructing a pain assessment model according to claim 1, characterized in that, In S5, the overall structure is obtained by connecting the pre-structure to the large language backbone model. The patient's multimodal temporal data is input, and the large language model is fine-tuned by instructing the overall structure to output pain assessment results, resulting in a trained pain assessment model, including: The patient's multimodal temporal data is input into the overall structure as training samples, and after passing through the pre-structure, a multimodal token sequence is obtained that is mapped to the word vector dimension of the large language model; The large language backbone model receives the multimodal token sequence, processes it using a preset Prompt template and a multi-task head, and outputs a pain assessment result; wherein, the interpretable heatmap in the pain assessment result is an image in which the pain contribution area is identified in the multimodal time-series data corresponding to the training sample. The system receives feedback information on the pain assessment results of the training samples, and uses the method of updating the multimodal projection layer + LoRA to learn and train, thereby optimizing the local model parameters before the large language backbone model, and obtaining the trained pain assessment model after multiple iterations of optimization.

7. A non-contact continuous pain assessment method, characterized in that, include: Acquire the target multimodal time-series data of the patient to be evaluated; The target multimodal time-series data includes electronic health record sequences, facial motion unit intensity sequences, body skeleton displacement sequences, remote photoplethysmography pulse waves, and infrared thermal imaging data sequences. The target multimodal time-series data is input into the trained pain assessment model to obtain the patient's pain assessment results; wherein, the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is trained by the pain assessment model construction method according to any one of claims 1-6.

8. A non-contact continuous pain assessment system, characterized in that, include: The multimodal acquisition module is used to acquire the target multimodal time-series data of the patient to be evaluated; The target multimodal time-series data includes electronic health record sequences, facial motion unit intensity sequences, body skeleton displacement sequences, remote photoplethysmography pulse waves, and infrared thermal imaging data sequences. The pain assessment module is used to input the target multimodal time-series data into a trained pain assessment model to obtain the patient's pain assessment results; wherein, the pain assessment results include pain level, pain score, and interpretable heatmap; the pain assessment model is trained by the pain assessment model construction method according to any one of claims 1-6.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the pain assessment model construction method according to any one of claims 1-6, or the steps of the non-contact continuous pain assessment method according to claim 7.

Images