Pain monitoring method based on multi-modal semantic flow

Abstract

The invention discloses a pain monitoring method based on multi-modal semantic flow. The method comprises the following steps: collecting and generating synchronous multi-modal data; based on the synchronous multi-modal data, extracting at least one visual semantic feature, audio semantic feature and physiological semantic feature; packaging the visual semantic feature, the audio semantic feature, the physiological semantic feature and a corresponding synchronization timestamp together into a time sequence synchronization multi-mode semantic feature packet; sending the time sequence synchronization multi-mode semantic feature packet to a central server in the form of a data stream; and the central server receives and analyzes the time sequence synchronous multi-mode semantic feature packet data stream, and analyzes and outputs a pain monitoring result in real time based on the multi-mode semantic features contained in the data stream. According to the method, semantic feature extraction and packaging are carried out at a data acquisition source, and the high-bandwidth original data stream is converted into the low-bandwidth structured semantic stream, so that the technical problems of network delay, bandwidth bottleneck and original data privacy security are fundamentally solved.

Description

Technical Field

[0001] This application relates to the field of medical information processing technology, and in particular to a pain monitoring method based on multimodal semantic flow. Background Technology

[0002] In clinical practice, accurate and continuous monitoring of patients' pain status is crucial for effective analgesia, assessing changes in their condition, and improving patient comfort. Traditional pain assessment relies primarily on subjective patient reports (such as the Numerical Rating Scale, NRS) or intermittent observation by healthcare professionals. However, subjective assessment methods are difficult to implement for patients who cannot communicate effectively (e.g., sedated patients in the intensive care unit, infants, or patients with cognitive impairment). Healthcare professional observation, on the other hand, is highly subjective, cannot provide continuous 24-hour monitoring, and is prone to missing critical pain events.

[0003] To achieve objective and continuous pain monitoring, existing technologies are exploring the fusion analysis of multimodal physiological and behavioral signals. These solutions typically deploy cameras, microphones, and physiological monitors at the bedside to collect raw data streams of the patient's facial expressions, voice, heart rate, and respiration, transmitting all of this data to a central server for centralized processing and analysis. However, this architecture based on raw data stream transmission faces significant technical challenges in practical clinical deployment. These challenges collectively constitute a pressing technical problem: how to achieve low-latency, high-reliability, and data privacy-preserving real-time multimodal pain monitoring in high-density, complex network environments of clinical settings. Specifically, the technical problems with existing technologies are mainly reflected in the following aspects: The massive amounts of raw data, especially high-definition video streams, put enormous pressure on the bandwidth of the hospital's internal network, easily leading to data congestion, packet loss, and significant network latency. This prevents the central server from acquiring the data required for analysis in real time, severely impacting the immediacy of monitoring. Furthermore, data streams from different devices have their own independent time bases and sampling frequencies. Performing high-precision timestamp alignment of multiple data streams on the central server is a computationally intensive task, consuming significant server computing resources and introducing additional processing latency. In addition, the transmission of raw data streams containing clear facial images and audio of patients over the network and their storage on servers involves significant risks to personal privacy and data security, posing a major challenge to the hospital's data compliance management.

[0004] Therefore, designing a new data pathway paradigm to fundamentally solve the problems of network latency, bandwidth bottlenecks, and raw data privacy and security has become an urgent technical challenge in this field. Summary of the Invention

[0005] The purpose of this application is to provide a pain monitoring method based on multimodal semantic streams, which aims to solve the technical problems in the prior art, such as high bandwidth consumption, large network latency, difficulty in data synchronization, and privacy leakage risks caused by transmitting raw multimodal data streams.

[0006] To achieve the above objectives, this application provides a pain monitoring method based on multimodal semantic streams. The method includes: real-time acquisition of video data, audio data, and physiological signal data at a bedside terminal; time-stamping and synchronizing the video data, audio data, and physiological signal data to generate synchronized multimodal data; extracting at least one visual semantic feature, audio semantic feature, and physiological semantic feature based on the synchronized multimodal data; encapsulating the visual semantic feature, audio semantic feature, physiological semantic feature, and a corresponding synchronization timestamp into a time-series synchronized multimodal semantic feature package; sending the time-series synchronized multimodal semantic feature package as a data stream to a central server via a low-latency message queue; the central server receiving and parsing the time-series synchronized multimodal semantic feature package data stream, and analyzing and outputting a pain monitoring result in real time based on the multimodal semantic features contained in the data stream.

[0007] Optionally, the time-stamp synchronization of the video data, the audio data, and the physiological signal data includes: setting the acquisition clock of the physiological signal data to a system master clock source; at the moment of acquiring each video data frame, attaching a first timestamp from the system master clock source to the video data frame; segmenting the audio data into audio segments corresponding to the duration of the video data frames, and attaching a second timestamp from the system master clock source corresponding to its start time to each audio segment; and combining the time-stamp-aligned video data frames, audio segments, and physiological signal data into the synchronized multimodal data based on the first timestamp and the second timestamp.

[0008] Optionally, the extraction of at least one visual semantic feature includes: performing face detection on video data frames in the synchronous multimodal data to locate facial regions; calculating a facial action unit vector based on facial key points in the facial regions as part of the visual semantic feature; and performing human pose estimation on the video data frames to extract the coordinates of upper body key points to form a skeletal key point vector as another part of the visual semantic feature.

[0009] Optionally, the extraction of audio semantic features includes: processing audio segments in the synchronous multimodal data to extract a set of acoustic features, wherein the acoustic features include at least one of Mel frequency cepstral coefficients, pitch and energy, to form an audio feature vector as the audio semantic features.

[0010] Optionally, the extraction of physiological semantic features includes: processing the physiological signal data in the synchronous multimodal data, extracting time-domain features and frequency-domain features, wherein the time-domain features and frequency-domain features include at least one of heart rate variability related indicators, root mean square value of electromyography signal and skin conductance response level, to form a physiological feature vector as the physiological semantic features.

[0011] Optionally, the time-synchronized multimodal semantic feature package has a predefined binary data structure, the binary data structure including: a metadata header, the metadata header including the synchronization timestamp and a feature mask for identifying the encapsulated feature type; and a data payload area, the data payload area storing the visual semantic features, the audio semantic features and the physiological semantic features quantized into low-precision data types in sequence according to the indication of the feature mask.

[0012] Optionally, after encapsulating the time-synchronous multimodal semantic feature packet, the method further includes: serializing the time-synchronous multimodal semantic feature packet using a binary serialization protocol to generate a byte stream for network transmission; and calculating a cyclic redundancy check (CRC) code on the byte stream and appending the CRC code to the end of the time-synchronous multimodal semantic feature packet.

[0013] Optionally, the central server receives and parses the time-series synchronous multimodal semantic feature packet data stream, including: the central server subscribing to and receiving the time-series synchronous multimodal semantic feature packet from the low-latency message queue; deserializing the received time-series synchronous multimodal semantic feature packet to restore the structured multimodal semantic features; and collecting the multimodal semantic features within a continuous time window according to the synchronization timestamp in the time-series synchronous multimodal semantic feature packet to construct a time-series feature sequence.

[0014] Optionally, the step of analyzing and outputting a pain monitoring result in real time based on the multimodal semantic features contained in the data stream includes: inputting the temporal feature sequence into a pre-trained multimodal fusion model; and the multimodal fusion model analyzing the temporal dependencies and cross-modal associations in the temporal feature sequence to output a continuous numerical pain score as the pain monitoring result.

[0015] Optionally, the multimodal fusion model is a Transformer network model based on an attention mechanism.

[0016] The technical solution of this application transforms raw data into structured semantic features at the source of data generation—the bedside terminal—and encapsulates these lightweight semantic features into unified TSSP data packets for transmission. This significantly reduces data transmission volume, compressing the Mbps-level raw data stream of traditional solutions into a kbps-level semantic feature stream. This effectively avoids the bandwidth bottleneck of hospital networks and ensures low latency and high reliability of data transmission. By forcibly performing timestamp synchronization on the terminal side, the complex synchronization task is pre-positioned and embedded in the data structure, reducing the computational load and processing latency of the central server and improving the overall response speed and scalability of the system. Furthermore, because anonymous, quantified semantic features are transmitted instead of raw audio and video data, the risk of leakage of patients' personal identity information is physically eliminated, fundamentally solving data privacy and security issues. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall technical architecture of a pain monitoring system provided in an embodiment of this application.

[0019] Figure 2 A flowchart of a pain monitoring method based on multimodal semantic flow provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the binary data structure of a timing synchronous multimodal semantic feature package (TSSP) provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] In this embodiment, the bedside device transmits only a low-bandwidth, synchronized structured semantic feature stream to the central server, thereby fundamentally solving a series of concurrent technical problems such as network latency, bandwidth bottlenecks, and the privacy and security of raw data. In a specific implementation, this method extracts and encapsulates unified semantic feature data packets in real time at the bedside terminal and transmits them using a low-latency message queue. This enables the multimodal fusion model on the central server to perform efficient real-time analysis of the feature stream and output accurate pain scores. This method solves the technical problems of high bandwidth pressure, high network latency, difficult data synchronization, and serious privacy and security risks caused by directly transmitting raw data streams in existing technologies.

[0024] Please see Figure 1 This is a schematic diagram of the overall technical architecture of a pain monitoring system provided in one embodiment of this application. The system 100 may include at least one bedside terminal 110 deployed next to the patient's bed, a low-latency message queue 120 as a communication middleware, and a central server 130 deployed in a hospital data center.

[0025] The bedside terminal 110 is responsible for raw data acquisition, timestamp synchronization, real-time extraction of multimodal semantic features, and standardized data encapsulation. In a specific implementation, the bedside terminal 110 can integrate multiple functional modules, including but not limited to: a multimodal data acquisition module 111, used to acquire data from multiple sensors in parallel; a time synchronization module 112, used to provide a unified time reference for all heterogeneous data streams; a multimodal feature extraction module 113, used to transform high-dimensional raw data into low-dimensional feature representations with clear semantics; and a TSSP encapsulation and transmission module 114, used to construct standardized data packets and send them over the network.

[0026] The low-latency message queue 120 can physically be a single server or a cluster of servers running message middleware software that supports a publish / subscribe pattern, such as an MQTT (Message Queuing Telemetry Transport) broker or a Redis server. Its role in the overall system is to decouple the bedside terminal 110 from the central server 130. The bedside terminal 110 acts as a publisher, publishing encapsulated TSSP data packets as messages to specified topics, while the central server 130 acts as a subscriber, asynchronously and in real-time receiving these messages by subscribing to the corresponding topics. This architecture features high throughput, low latency, and good scalability.

[0027] The central server 130 obtains standardized TSSP data streams covering the entire hospital from the low-latency message queue 120. In a specific implementation, the central server 130 may also integrate multiple functional modules, including but not limited to: a TSSP receiving and parsing module 131, which is responsible for handling network communication, message subscription, and data packet deserialization; a multimodal fusion analysis module 132, which is the intelligent core of the system, running complex deep learning models to understand multimodal features and perform pain assessment; and a results storage and presentation module 133, which is responsible for persisting the analysis results and providing them to upper-level clinical applications.

[0028] During system operation, the multimodal data acquisition module 111 of the bedside terminal 110 first acquires raw data in parallel from the camera, microphone, and physiological monitor. These raw data streams are immediately sent to the timing synchronization module 112, which assigns a high-precision timestamp to each data point (e.g., a video frame or an audio clip). The synchronized data is then sent to the multimodal feature extraction module 113, which transforms the data into structured semantic feature vectors. Subsequently, the TSSP encapsulation and transmission module 114 encapsulates these feature vectors along with the timestamps into a compact TSSP data packet and publishes it to the low-latency message queue 120. The TSSP receiving and parsing module 131 of the central server 130 continuously pulls these data packets from the message queue 120, parses them, and sends them to the multimodal fusion analysis module 132. This module performs in-depth analysis of the continuous feature sequences to calculate a real-time pain score. Finally, the result storage and presentation module 133 records the score and displays it in real-time on the central monitoring screen at the nurse station.

[0029] The following will combine Figure 2 The flowchart shown provides a detailed explanation of a pain monitoring method based on multimodal semantic flow provided in this application embodiment.

[0030] S100: At the bedside terminal, video data, audio data, and physiological signal data are collected in real time, and the video data, audio data, and physiological signal data are time-stamped and synchronized to generate synchronized multimodal data.

[0031] This step is the data source and time reference establishment stage of the entire method. Its core objective is to acquire raw data from different sensors that are precisely aligned in time, providing high-quality input for subsequent feature extraction. In one embodiment, the bedside terminal 110 performs this step through its built-in multimodal data acquisition module 111 and timing synchronization module 112.

[0032] Specifically, this step can be further broken down into S110, S120 and S130.

[0033] S110: Set the master clock source.

[0034] To ensure that the timestamps of all data streams originate from a unified and stable reference, the system must first establish a master clock. In clinical settings, physiological monitoring devices directly related to vital signs (e.g., electrocardiogram monitors) typically have high-precision internal clocks calibrated to medical device standards. Therefore, in a preferred embodiment, this method sets the acquisition clock of the physiological signal data as the system's sole master clock source. This setting ensures that all time information is referenced to the most clinically accepted time benchmark, resulting in extremely high reliability.

[0035] For example, suppose the bedside terminal 110 is connected to a multi-parameter monitor via an RS-232 serial port. This monitor outputs electrocardiogram (ECG) signal samples at a frequency of 250 Hz. The operating system kernel time of the bedside terminal 110 can be periodically synchronized with the monitor's clock (e.g., through a localized implementation of the Network Time Protocol (NTP)), or the local high-precision timer can be calibrated directly using the moment physiological data packets are received as the driving event. Thus, the bedside terminal 110 internally obtains a high-precision clock with millisecond or even microsecond resolution that is strictly synchronized with the physiological monitor; this clock serves as the system's master clock source.

[0036] S120: Parallel data acquisition and timestamp synchronization.

[0037] After the master clock source is established, the multimodal data acquisition module 111 begins to acquire data from each sensor in parallel. At the same time, the timing synchronization module 112 precisely adds a timestamp from the master clock source to each independent data unit.

[0038] In a specific implementation, this parallel process includes: S121: Video Data Acquisition and Synchronization. The camera integrated on the bedside terminal 110 is configured to continuously acquire video images of the patient's face and upper body at a preset frame rate (e.g., 15 frames per second, i.e., a frame interval of approximately 66.67 milliseconds). For each frame, at the exact moment it is fully exposed by the camera's photosensitive element and read into the memory buffer, the timing synchronization module 112 immediately calls the master clock source to obtain the current high-precision timestamp and binds this timestamp to the frame image data. This timestamp marks the precise time when the scene captured by that frame occurred.

[0039] For example, at a certain moment, the reading of the master clock source is In milliseconds, the camera completes exactly one frame of an image (denoted as 1 / 2). The collection of ) The timestamp will be recorded in the metadata. Approximately 66.67 milliseconds later, the next frame... Once collected, it will be assigned a new timestamp. millisecond.

[0040] S122: Audio Data Acquisition and Synchronization. In parallel with the camera, the microphone on the bedside terminal 110 continuously acquires audio signals from the environment. To correspond temporally with the video data, the continuous audio stream is divided into a series of short audio segments. The duration of each audio segment is set to be equal to the interval between video frames (e.g., 66.67 milliseconds). For each audio segment, the timing synchronization module 112 appends a master clock timestamp representing its start time.

[0041] For example, with video frames The audio data corresponding to the acquisition time period is organized into an audio segment. The start timestamp of this segment was also precisely marked as... millisecond.

[0042] S123: Physiological signal data acquisition and synchronization. The bedside terminal 110 continuously receives data from the physiological monitor through its interface (such as a serial port or network interface). Since the clock of the physiological monitor has been set as the master clock source, its output data itself carries a high-precision timestamp, or can be accurately calculated by the time the data packet is received.

[0043] For example, in timestamp Within a very small time window before and after (e.g.) (Milliseconds, because the sampling rate of physiological signals is higher) The bedside terminal 110 will receive a set of physiological signal data points from the monitor, such as an electrocardiogram signal sample. A sample of electromyography (EMG) signal A sample of skin conductance response The timestamps of these samples are naturally related to... Alignment.

[0044] S130: Construct a synchronization data frame.

[0045] The purpose of this step is to organize data from different modalities, collected simultaneously, into a unified, temporary composite data structure, known as a "synchronization data frame." This data frame serves as the atomic input unit for all subsequent feature extraction tasks. The bedside terminal 110 checks for timestamps on the same master clock (or within a very small tolerance window). All data units collected within a range of approximately 5 milliseconds.

[0046] For example, based on the above collection process, the system will assign the timestamp as... Millisecond video frames Audio clips and a set of physiological signal samples near that moment. Logically combined, they form a synchronous data frame. .this The unified timestamp is This process repeats continuously, with the bedside terminal 110 constantly generating a sequence of... The sequence of synchronous data frames constituted.

[0047] S200: Based on the synchronous multimodal data, extract at least one visual semantic feature, audio semantic feature, and physiological semantic feature.

[0048] After obtaining the raw data with precise time synchronization, this step transforms this raw data into structured semantic features containing pain-related information. This step is performed by the multimodal feature extraction module 113 of the bedside terminal 110, which typically embeds multiple optimized, lightweight artificial intelligence models and signal processing algorithm libraries suitable for edge computing.

[0049] Specifically, this step can be further broken down into S210, S220, and S230, which correspond to feature extraction for the visual, audio, and physiological modalities, respectively.

[0050] S210: Extract visual semantic features.

[0051] This sub-step processes video data frames in the synchronized data frames to extract pain-related facial expressions and body posture information.

[0052] S211: Perform face detection on the video data frames to locate facial regions. The multimodal feature extraction module 113 first runs an efficient face detection algorithm (e.g., a MobileNet-SSD-based detector) to quickly locate the bounding box of the patient's face in the input video frames.

[0053] For example, for the input video frame The face detection algorithm outputs a rectangular region with coordinates (x=300, y=150, width=200, height=250) (in pixels). This region will be cropped out for subsequent detailed analysis. If no face is detected, the subsequent facial feature extraction steps will be skipped and represented in the final feature package by a specific value (such as an all-zero vector) or a flag.

[0054] S212: Based on the facial key points within the facial region, a facial action unit vector is calculated as part of the visual semantic features. Within the located facial region, the system runs a lightweight facial key point detection model (also known as a face alignment model) to accurately locate the positions of dozens of facial key points (such as the corners of the eyes, eyebrows, the tip of the nose, and the corners of the mouth). Subsequently, based on the relative positional changes and geometric relationships of these key points (such as the distance between the center of the brow and the eyebrows, and the curvature of the corners of the mouth), the system calculates a quantized facial action unit (AU) vector according to the definition of the Facial Action Coding System (FACS). Each AU corresponds to the movement of one or a group of facial muscles, and its numerical value indicates the intensity of the movement.

[0055] For example, a typical AU vector It could be a 17-dimensional real vector, i.e. Hypothesis Analysis The resulting AU vectors are [AU1: 0.5, AU2: 0.0, AU4: 1.8, AU5: 0.2, AU6: 2.5, AU7: 2.1, ...] (the numerical range is usually a dimensionless floating-point number from 0 to 5). Among them, the high intensity values ​​of AU4 (frowning), AU6 (cheek lifting), and AU7 (eyelid closing) are typical indicators of pain expressions.

[0056] S213: Perform human pose estimation on the video data frames, extract the coordinates of key points in the upper body, and construct a skeletal keypoint vector as another part of the visual semantic features. To capture posture information related to pain, such as body tension and curling, the system runs a lightweight human pose estimation algorithm (e.g., MoveNet or Lite-HRNet) in parallel across the entire video frame. This algorithm outputs the two-dimensional coordinates of the patient's main upper body joints (such as shoulders, elbows, wrists, nose, and ears) in the image. These coordinate points together constitute the skeletal keypoint vector. .

[0057] For example, assuming the pose estimation model is configured to detect 15 upper body keypoints, the output skeletal keypoint vectors It is The matrix, i.e. Each row contains the (x, y) coordinates (in pixels) of a keypoint and its detection confidence (a floating-point number between 0 and 1). For example, [LeftShoulder: (320, 150, 0.95), RightShoulder: (480, 152, 0.96), LeftElbow: (290, 240, 0.88),...]. These coordinates can be used to calculate higher-order pose features such as joint angles and body symmetry.

[0058] It is important to note that the lightweight design aims to ensure that the model can be processed in real-time on resource-constrained bedside terminal hardware. In one specific embodiment, the lightweight model has a total number of parameters limited to less than 5 million (5M) and an inference latency of less than 30 milliseconds for processing a single frame of video image on the target embedded platform (e.g., a computing unit with a quad-core ARM Cortex-A72 processor). This performance metric ensures that the feature extraction speed can match or even exceed typical video capture frame rates (e.g., 15 fps, i.e., a processing time window of approximately 66 milliseconds per frame), thereby avoiding data processing bottlenecks.

[0059] To achieve the aforementioned lightweight performance metrics, the model can be implemented using various known model compression techniques. In one specific embodiment, the human pose estimation algorithm can employ a model with a ShuffleNetV2 network as its backbone. ShuffleNetV2 is a computationally efficient convolutional neural network architecture designed for mobile devices. Alternatively, knowledge distillation can be used to transfer knowledge from a large, high-precision human pose estimation model (teacher model) to a smaller, more compact model (student model), allowing the student model to maintain high prediction accuracy despite having a significantly smaller number of parameters than the teacher model. Those skilled in the art can select or construct a lightweight model that meets the aforementioned constraints on parameter count and inference latency, based on the computing power and memory size of the specific embedded hardware platform used.

[0060] S220: Extract audio semantic features.

[0061] This sub-step processes audio segments in the synchronized data frame to extract acoustic features that reflect changes in the patient’s groans, crying, or breathing.

[0062] In one embodiment, the system performs a series of standard speech signal processing operations on the input audio segment. First, the audio signal is pre-emphasized, framed, and windowed. Then, the Mel-Frequency Cepstral Coefficients (MFCCs) are calculated for each frame. MFCCs effectively characterize the timbre of speech. In addition, other acoustic features such as pitch, energy, and zero-crossing rate can be calculated. These features are combined into a single audio feature vector. .

[0063] For example, suppose an audio segment of 66.67 milliseconds... After processing, the system calculates 40 MFCC coefficients, as well as one scalar value each for pitch and energy. These values ​​are concatenated into a 42-dimensional audio feature vector. For example, we get [-250.5, 105.3, -20.1, ..., 150.2 Hz, 0.8 Pa^2*s]. The lower pitch and higher energy may be associated with vocalizations associated with pain.

[0064] S230: Extract physiological semantic features.

[0065] This sub-step processes physiological signal data points and their historical sequences in the synchronized data frame, aiming to extract physiological indicators that reflect changes in the autonomic nervous system caused by pain stress.

[0066] In one embodiment, the system maintains a sliding window containing physiological signal data over a past period (e.g., the last 5 minutes). Based on the data within this window, the system can calculate various time-domain and frequency-domain features. For example: For electrocardiogram (ECG) signals, the system detects the R wave peak, calculates the RR interval sequence, and calculates a series of heart rate variability (HRV) indicators based on this sequence, such as SDNN (standard deviation of all normal RR intervals), RMSSD (root mean square of the difference between adjacent RR intervals), and the LF / HF ratio in the frequency domain.

[0067] For electromyography (EMG) signals, the system can calculate their root mean square (RMS) value or integrated electromyography (iEMG) value to quantify the degree of muscle tension.

[0068] For the skin conductance response (GSR) signal, the system can extract its skin conductance level (SCL) or identify the frequency and amplitude of skin conductance responses (SCRs).

[0069] These calculated physiological indicators are combined into a physiological feature vector. .

[0070] For example, the system calculates a 10-dimensional physiological feature vector based on the current moment and historical data. For example, [HRVSDNN: 45.5 ms, EMGRMS: 350.2 μV, SCL: 5.2 μS, ...]. Pain typically leads to decreased HRV, increased EMG activity, and elevated GSR levels.

[0071] S300: The visual semantic features, the audio semantic features, the physiological semantic features, and a corresponding synchronization timestamp are encapsulated together into a time-synchronized multimodal semantic feature package.

[0072] After the semantic features of all modalities have been extracted, the core task of this step is to organize these heterogeneous feature data, along with their common time reference, into a standardized, compact data unit that is easy to transmit over the network. This data unit is defined as a "Time-Synchronized Multimodal Semantic Packet" (TSSP). This step is performed by the TSSP encapsulation and transmission module 114 of the bedside terminal 110.

[0073] This step can be further broken down into S310 and S320.

[0074] S310: Construct the TSSP data structure.

[0075] TSSP is not an abstract concept, but rather a well-defined, pre-defined binary data structure. See also... Figure 3 This demonstrates an exemplary structure for a TSSP packet 300. The structure is designed following the principles of efficiency and scalability.

[0076] In one embodiment, the structure of the TSSP packet 300 includes a metadata header 310, a data payload area 320, and an optional footer (e.g., for checksum).

[0077] The metadata header 310 is located at the beginning of the data packet and contains key information for parsing the entire data packet. For example, header 310 may include: A protocol version number field (e.g., 2 bytes) is used to identify the TSSP's structure version, ensuring backward compatibility of the system.

[0078] A packet total length field (e.g., 2 bytes) indicates the total number of bytes in the entire packet.

[0079] A 64-bit (8-byte) synchronization timestamp field is used to store the Unix timestamp accurate to milliseconds from the S100 step. This is the time base for the entire data packet.

[0080] A feature mask field (e.g., a 4-byte bitmask) is used to efficiently identify which types of features are actually contained in the data payload area 320. For example, a 1 in bit 0 of its binary representation indicates the presence of AU vectors, a 1 in bit 1 indicates the presence of skeletal point vectors, and so on. The receiver can check this mask to determine how to correctly parse subsequent data payloads.

[0081] The data payload area 320, which follows the header 310, is used to store the actual data of the various feature vectors extracted in S200. To compress the data volume to the maximum extent, all floating-point values ​​are quantized to a low-precision data type.

[0082] For example, an AU vector Its elements are typically 32-bit single-precision floating-point numbers (float32). During encapsulation, these values ​​can be converted to 16-bit half-precision floating-point numbers (float16). This reduces the space required to store the entire AU vector from... ByteDance has been reduced to The data size is halved while the loss of precision is within acceptable limits. Similarly, skeletal point coordinates can be converted from floating-point numbers to 16-bit integers (int16), and confidence scores can be converted to 8-bit unsigned integers (uint8). MFCC coefficients and physiological characteristic values ​​can also be represented using float16.

[0083] The internal layout of the data payload area 320 is dynamically determined based on the feature mask. If the mask indicates that it includes AU and skeletal point features, then the payload area will store the binary data of the AU vector and the binary data of the skeletal point vector in sequence.

[0084] S320: Serialize data packets.

[0085] After the TSSP data structure is constructed in memory, it needs to be converted into a continuous byte stream for transmission over the network. This process is called serialization.

[0086] In a preferred embodiment, the system employs an efficient, cross-language binary serialization protocol, such as Google's Protocol Buffers or MessagePack. Developers predefine a TSSP schema file, which formally describes the structure and fields of the TSSP. At runtime, the TSSP encapsulation and sending module 114 calls the corresponding library functions to automatically and efficiently encode the TSSP object (a C++ or Python object) in memory into a compact binary byte stream according to this schema file.

[0087] For example, suppose at time... Milliseconds, the system extracts , , , Four features. The TSSP wrapper module 114 creates a TSSP object, fills it with timestamps and four feature vectors, and then calls the Protocol Buffers serialization function. This function generates a binary array of approximately 400 bytes. This array is the final data to be sent.

[0088] Optionally, to ensure data integrity during transmission, a Cyclic Redundancy Check (CRC32) code can be calculated for the entire byte stream after serialization, and this 4-byte checksum can be appended to the end of the byte stream. Upon receiving the data, the receiving end will recalculate the checksum and compare it to detect whether the data has been corrupted during transmission.

[0089] S400: Transmit TSSP streams via a low-latency message queue.

[0090] This step is crucial for achieving efficient data transmission from the edge to the center. The TSSP encapsulation and transmission module 114 of the bedside terminal 110 generates a serialized TSSP byte stream and publishes it as the message body (Payload) to the low-latency message queue server 120 over the network.

[0091] In one embodiment, the bedside terminal 110 runs an MQTT client internally. This client is pre-configured with the address of an MQTT Broker (i.e., low-latency message queue 120) within the hospital's intranet. Each bedside terminal is assigned a unique topic with a hierarchical structure for easy management and subscription.

[0092] For example, a bedside terminal deployed in Ward 1, Bed 3, might have its published topic set to hospital / painmonitor / ward01 / bed_03. After S300 is completed, the terminal's MQTT client will immediately publish the generated TSSP byte stream as an MQTT message to this topic. Because TSSP packets are very small (approximately a few hundred bytes) and the MQTT protocol itself is designed for low overhead, the network latency throughout the publishing process is extremely low. The bedside terminal will continuously execute the S100 to S400 loop at a preset frequency (e.g., synchronized with the video frame rate, 15 times per second), thus forming a stable, real-time TSSP data stream flowing to the message queue.

[0093] S500: The central server receives and analyzes TSSP streams and outputs pain scores.

[0094] This step is performed on the central server 130, and its core objective is to transform the TSSP data stream collected from across the hospital into clinically meaningful pain monitoring results.

[0095] This step can be further broken down into S510, S520, S530 and S540.

[0096] S510: Receives and deserializes TSSP.

[0097] The TSSP receiving and parsing module 131 of the central server 130, acting as an MQTT client, pre-subscribes to all topics of interest. For example, it can subscribe to a wildcard topic hospital / pain_monitor / + / +, thereby receiving TSSP data streams from all beds in all wards.

[0098] When any bedside terminal publishes a TSSP message, the MQTT Broker immediately pushes the message to the central server 130. Upon receiving the MQTT message containing the TSSP byte stream, the TSSP receiving and parsing module 131 first (optionally) checks the CRC32 checksum at the end of the message to ensure data integrity. After successful verification, it calls a library function using the same binary serialization protocol (such as Protocol Buffers) as the sender to perform deserialization. This operation precisely restores the binary byte stream to a structured TSSP object in memory, allowing easy access to timestamps, feature masks, and various feature vectors.

[0099] S520: Construct time-series feature sequences.

[0100] Pain is a dynamic physiological and psychological phenomenon, and a snapshot of features at a single point in time is often insufficient for accurate judgment. Therefore, it is necessary to analyze the evolution trend of features over a period of time. After parsing individual TSSP objects, the TSSP receiving and parsing module 131 distributes them to their respective data buffers based on their internal synchronization timestamps and source topics (i.e., bed information).

[0101] Each bed has a corresponding data buffer, which is used to aggregate TSSP packets that arrive within a consecutive time window (e.g., the most recent 3 seconds).

[0102] For example, assume the analysis window is 3 seconds and the TSSP transmission frequency is 10 Hz. For bed number 3, its data buffer stores the 30 most recent TSSP packets. When a new TSSP packet arrives, it is added to the tail of the buffer, while the oldest TSSP packet is removed from the head. The 30 TSSP packets in the buffer are sorted by timestamp, and their individual feature vectors are extracted and concatenated to form a multimodal time-series feature sequence. For example, the AU feature sequence would be a... The matrix, the skeletal point feature sequence is a The tensor, and so on. This temporal feature sequence fully describes the patient's multimodal behavior and physiological changes over the past 3 seconds and serves as the input for the final analysis.

[0103] S530: Perform multimodal fusion and pain scoring.

[0104] This step is performed by the multimodal fusion analysis module 132. This module receives the time-series feature sequence constructed by S520 and inputs it into a pre-trained deep learning model that can understand and fuse multimodal time-series data.

[0105] In a preferred embodiment, the multimodal fusion model is a Transformer network model based on an attention mechanism. This model is particularly adept at capturing long-range dependencies and complex interactions between different features in sequence data.

[0106] The macroscopic architecture of the Transformer model can be deconstructed into the following parts: First, there is an input embedding layer. For feature sequences from different modalities (such as AU sequences, skeletal point sequences, audio sequences, and physiological sequences), the model first projects them into a unified high-dimensional feature space through their respective independent embedding layers (e.g., fully connected layers). Simultaneously, positional encoding is added to inject the temporal order information of the sequences into the model.

[0107] Secondly, there's the Transformer encoder core. The embedded multimodal feature sequences are concatenated and fed into a network composed of multiple stacked encoder layers. Each encoder layer contains two key sub-layers: a multi-head self-attention sub-layer and a position-wise feed-forward network sub-layer. The multi-head self-attention mechanism allows the model to simultaneously attend to all modal features at all other time points in the sequence when computing the representation at a certain time point, and dynamically assign different attention weights to features at different time points and different modalities. This enables the model to capture cross-modal temporal correlation patterns, such as "first muscle tension (physiological), then frowning (visual), and finally groaning (audio)."

[0108] Finally, there is an output layer. After deep processing through multiple encoder layers, the resulting feature sequence representation is fed into an output layer (e.g., a global average pooling layer connected to a fully connected layer), ultimately regressing a continuous, standardized pain score.

[0109] For example, the multimodal fusion analysis module 132 inputs the 3-second temporal feature sequence of bed number 3 into the pre-trained Transformer model. After forward computation, the model outputs a floating-point value of 8.7. This value represents the model's assessment of the patient's pain score at that moment as 8.7 (assuming a score range of 0-10), indicating severe pain.

[0110] In one exemplary embodiment, the multimodal fusion model employs a Transformer encoder composed of six identical encoder layers stacked together. The hidden layer dimension of its internal model... It is set to 256. Each multi-head self-attention sublayer contains Each attention head means the model can learn sequence dependencies in parallel from eight different representation subspaces. Each location-based feedforward network consists of two linear layers with internal hidden layer dimensions... The value is set to 1024. The multi-mode morphological feature sequence input to the encoder has a total sequence length fixed at 30 time steps (corresponding to 3 seconds of data, with a sampling frequency of 10 Hz). Those skilled in the art can adjust these hyperparameters according to the actual application scenario and available computing resources.

[0111] Those skilled in the art will understand that the multimodal fusion model can be obtained through supervised learning on a dataset containing multimodal data and corresponding pain rating labels. The training process can employ the Adam optimizer, using mean squared error as the loss function, and optimize the model parameters through backpropagation. The dataset can be constructed by collecting a large amount of synchronous multimodal data from clinical patients in different states, and having it labeled by professional medical personnel according to clinical pain scales (such as CPOT or BPS).

[0112] S540: Store and present results.

[0113] The pain score calculated by the multimodal fusion analysis module 132, along with its corresponding timestamp, source bed number, and other metadata, is sent to the result storage and presentation module 133.

[0114] This module first writes the structured results data into a database suitable for processing time series data (such as InfluxDB or Prometheus) for long-term storage, subsequent trend analysis, and clinical research.

[0115] In addition, this module provides a set of API interfaces. The hospital's central monitoring system or electronic medical record (EMR) system can call these interfaces to obtain the latest pain scores and historical change curves of a specified bed in real time, and display them graphically on the monitoring screen at the nurse station or on the doctor's workstation.

[0116] For example, on the large screen at the nurses' station, in the monitoring area of ​​bed number three, in addition to displaying traditional physiological parameters such as heart rate and blood pressure, a real-time updated pain score dashboard is also displayed, with its pointer pointing to 8.7, accompanied by a curve showing the trend of pain score changes over the past hour. When the score remains above a preset threshold (e.g., 4 points, i.e., moderate pain) for a certain period of time (e.g., 5 minutes), the system can also automatically trigger an alarm to remind the nurse to intervene in a timely manner.

[0117] Through the collaborative efforts of the above steps, the method proposed in this application constructs a closed-loop, efficient pain monitoring data pathway from the bedside edge to the central cloud. It fronts up computational load and semanticizes transmitted data, thereby ensuring real-time performance while overcoming many bottlenecks in existing technologies, providing a technological foundation for large-scale, high-precision automated clinical pain monitoring.

[0118] In the various embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A pain monitoring method based on multimodal semantic flow, characterized in that, include: At the bedside terminal, video data, audio data, and physiological signal data are collected, and the video data, audio data, and physiological signal data are time-stamped and synchronized to generate synchronized multimodal data. Based on the synchronous multimodal data, at least one visual semantic feature, audio semantic feature, and physiological semantic feature are extracted; The visual semantic features, the audio semantic features, the physiological semantic features, and a corresponding synchronization timestamp are collectively encapsulated into a time-series synchronized multimodal semantic feature package. The time-synchronous multimodal semantic feature package is sent to a central server in the form of a data stream through a low-latency message queue. The central server receives and parses the time-synchronized multimodal semantic feature packet data stream, and analyzes and outputs a pain monitoring result based on the multimodal semantic features contained in the data stream.

2. The method according to claim 1, characterized in that, The time-stamp synchronization of the video data, the audio data, and the physiological signal data includes: The acquisition clock for the physiological signal data is set as a system master clock source; At the moment of acquiring each video data frame, a first timestamp from the system master clock source is appended to the video data frame. The audio data is divided into audio segments corresponding to the duration of the video data frames, and a second timestamp corresponding to the start time of each audio segment is attached from the system master clock source. Based on the first timestamp and the second timestamp, the timestamp-aligned video data frames, audio segments, and physiological signal data are combined into the synchronous multimodal data.

3. The method according to claim 1, characterized in that, The extraction of at least one visual semantic feature includes: Face detection is performed on the video data frames in the synchronous multimodal data to locate facial regions; Based on the facial key points within the facial region, a facial motion unit vector is calculated as part of the visual semantic features; Human pose estimation is performed on the video data frames, and the coordinates of key points of the upper body are extracted to form a skeletal key point vector as another part of the visual semantic features.

4. The method according to claim 1, characterized in that, The extraction of audio semantic features includes: The audio segments in the synchronous multimodal data are processed to extract a set of acoustic features, including at least one of Mel frequency cepstral coefficients, pitch, and energy, to form an audio feature vector as the audio semantic feature.

5. The method according to claim 1, characterized in that, The extraction of physiological semantic features includes: The physiological signal data in the synchronous multimodal data are processed to extract time-domain features and frequency-domain features. The time-domain features and frequency-domain features include at least one of heart rate variability related indicators, root mean square value of electromyography signal and skin conductance response level, so as to form a physiological feature vector as the physiological semantic features.

6. The method according to claim 1, characterized in that, The time-synchronous multimodal semantic feature package has a predefined binary data structure, which includes: A metadata header, which includes the synchronization timestamp and a feature mask for identifying the encapsulated feature type; A data payload area, wherein the data payload area stores the visual semantic features, the audio semantic features and the physiological semantic features, which are quantized into low-precision data types, in sequence according to the indication of the feature mask.

7. The method according to claim 6, characterized in that, After being encapsulated into a time-synchronous multimodal semantic feature package, it also includes: The time-synchronous multimodal semantic feature packet is serialized using a binary serialization protocol to generate a byte stream for network transmission; In addition, a cyclic redundancy check (CRC) code is calculated for the byte stream, and the CRC code is appended to the end of the time-synchronized multimodal semantic feature packet.

8. The method according to claim 1, characterized in that, The central server receives and parses the time-synchronized multimodal semantic feature packet data stream, including: The central server subscribes to and receives the time-synchronized multimodal semantic feature package from the low-latency message queue; The received time-synchronized multimodal semantic feature packets are deserialized to restore the structured multimodal semantic features; Based on the synchronization timestamps in the time-series synchronized multimodal semantic feature package, the multimodal semantic features within a continuous time window are aggregated to construct a time-series feature sequence.

9. The method according to claim 8, characterized in that, The process of analyzing and outputting a pain monitoring result based on the multimodal semantic features contained in the data stream includes: The time-series feature sequence is input into a pre-trained multimodal fusion model; The multimodal fusion model analyzes the temporal dependencies and cross-modal associations in the temporal feature sequence to output a continuous numerical pain score as the pain monitoring result.

10. The method according to claim 9, characterized in that, The multimodal fusion model is a Transformer network model based on an attention mechanism.

Images