A speech micro-expression emotion state recognition system for patients with advanced tumors

Abstract

This invention relates to the fields of medical data processing and multimodal emotion computing, and discloses an emotion state recognition system for vocal micro-expressions in patients with advanced cancer. The system constructs a temporal state vector reflecting the patient's neural control ability by calculating the neural conduction lag rate and the physiological vocal energy index. A state-adaptive gating module generates a confidence mask to weight the multimodal features. Subsequently, an emotion decoupling and reconstruction module uses manifold orthogonal projection to separate the weighted features into physiological harm perception components and psychological emotional response components, and combines sparse dictionary reconstruction to remove noise. Finally, a fusion decision and grading module combines a temporal convolutional network with an adaptive threshold calibration mechanism based on the energy index to output pain grading results and decision signals. This invention can effectively distinguish between a patient's subjective masking and physiological exhaustion state, decouple the physical and psychological attributes of pain, and avoid missed detections due to weak reactions.

Description

Technical Field

[0001] This invention relates to the field of medical data processing and multimodal emotion computing technology, specifically to an emotion state recognition system for the voice micro-expressions of patients with advanced cancer. Background Technology

[0002] In palliative care and end-of-life care for patients with advanced cancer, pain assessment is a core basis for developing analgesia plans and adjusting opioid dosages. Currently, clinically accepted assessment methods mainly rely on patient self-reporting (such as the Visual Analogue Scale (VAS)) or observation scales used by healthcare professionals (such as the CPOT pain monitoring tool in intensive care). However, advanced-stage patients often experience confusion, delirium, or extreme weakness, making it difficult for them to accurately describe their pain verbally. Furthermore, manual observation and assessment suffer from discontinuity and subjective bias, failing to meet the needs of precision medicine.

[0003] With the development of computer vision and speech signal processing technologies, automatic pain recognition technology based on non-contact sensors is gradually being applied in clinical practice. Existing multimodal emotion recognition schemes typically extract facial action units and speech acoustic features, and establish a mapping relationship between feature intensity and pain level through machine learning models. These conventional schemes mainly assume that the intensity of external expressions is positively correlated with the degree of internal pain, that is, that large frowns or high-decibel groans correspond to severe pain.

[0004] However, existing technologies based on the linear mapping assumption have significant limitations in dealing with the complex pathological states of patients with advanced cancer. The pain expression mechanisms of patients with advanced cancer are nonlinearly regulated by both physiological and psychological factors. On the one hand, due to psychological defense mechanisms or social habits, patients often subjectively conceal their pain expressions when conscious, exhibiting a calm or even smiling face that does not match their actual pain. Existing technologies struggle to distinguish this cognitively controlled "masked signal" from genuine physiological reactions, easily leading to misjudgments. On the other hand, patients in cachexia or the terminal stage experience extreme exhaustion of physiological energy, decreased muscle tone, and insufficient lung capacity. Their pain responses often manifest as weak micro-expressions or low-energy breath sounds. Existing algorithms typically set fixed noise filtering thresholds, easily filtering out such weak pain signals with low signal-to-noise ratios as background noise, resulting in severe underreporting. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an emotional state recognition system for vocal micro-expressions in patients with advanced cancer, which solves the problem that patients with advanced cancer are often in an unstable pathophysiological state, and their pain expression is interfered with by multiple factors.

[0006] The first aspect of the present invention provides an emotion state recognition system for speech micro-expressions of patients with advanced tumors. The system includes a hysteresis and energy calculation module, a state adaptive gating module, an emotion decoupling and reconstruction module, and a fusion decision and classification module.

[0007] The hysteresis and energy calculation module is used to calculate the patient's neural control ability and physiological function reserves. This module processes the acquired audio stream and facial video stream in parallel, calculates the neural conduction hysteresis rate by calculating the phase difference of the audiovisual signals on the time axis, and calculates the physiological vocal energy index by combining the acoustic dynamics characteristics. The two are then combined to generate a temporal state vector.

[0008] When calculating the neural conduction lag rate, the system determines the background noise dynamic threshold of the audio stream and the facial response threshold of the video stream to locate the onset times of speech and facial responses. The system performs modal existence verification: when only facial responses are detected without effective speech, it is determined to be a painful aphasia state and the lag rate is set to a negative extreme value; when only speech is detected without facial responses, it is determined to be a forced vocalization state and the lag rate is set to a positive extreme value. If effective onset times exist in both the audiovisual and visual modalities, the system calculates the difference in onset time between the two and performs normalization and exponentially weighted moving average updates based on the standard response latency norm to characterize the synchronicity between the patient's cognitive control and muscle response.

[0009] When calculating the physiological vocal energy index, the system selects the short-time energy peak, average harmonic noise ratio, and vocal duration as feature parameters. The feature parameters are normalized, and the vocal duration is nonlinearly compressed using a logarithmic function. The energy index is obtained through weighted summation. The short-time energy peak and average harmonic noise ratio are given higher weights than the vocal duration, thus characterizing the patient's current physical reserve.

[0010] The state-adaptive gating module is used to dynamically adjust the feature fusion weights based on the patient's real-time state. This module receives a temporal state vector, generates confidence masks for different modal features through nonlinear mapping, and performs weighted gating processing on the extracted multimodal raw features to obtain a contextual feature vector reflecting the patient's current neural control ability.

[0011] Specifically, the system maps the temporal state vector to the probability distributions of three orthogonal baseline state prototypes using a multilayer perceptron. These three baseline state prototypes include: a social interaction state representing low hysteresis rate and medium-to-high energy; a cognitive masking state representing high hysteresis rate and medium energy; and a physiological exhaustion state representing low energy or abnormal hysteresis rate. The system pre-defines weight template vectors corresponding to these three baseline state prototypes. Based on the calculated probability distributions, the system performs linear weighted summation and lower-bound threshold truncation on the weight template vectors to generate a dynamic confidence mask. Subsequently, the system performs soft gating, multiplying the confidence mask element-wise with the original multimodal features, and inputting the weighted feature sequence into a bidirectional long short-term memory network. The weighted sum is calculated through a temporal attention layer to generate a contextual feature vector. This process reduces the weights of features that are inconsistent with the current pathological state while retaining high-confidence features.

[0012] The emotional decoupling and reconstruction module is used to separate the physiological and psychological components of pain. This module uses a preset pain baseline vector to perform manifold orthogonal projection on the context feature vector to separate the physiological harm perception component and the psychological emotional response component, and quantifies them into physiological pain score and emotional distress score, respectively.

[0013] The system uses the pain baseline vector as the principal direction of the pain manifold subspace and calculates the projection of the context feature vector onto this direction to obtain the physiological nociceptive component, which characterizes the intensity of the neural reflex. Simultaneously, the system calculates the residual vector after subtracting the physiological nociceptive component from the context feature vector and filters out high-frequency noise to obtain the psychological emotional response component. To further remove random noise from the physiological component, the system introduces a pre-trained pain atom dictionary containing noise-free pain micro-expression pattern primitives. Using the physiological nociceptive component as the target, the system solves an L1 regularization optimization problem based on the pain atom dictionary to obtain a sparse coefficient vector. The product of the sparse coefficient vector and the dictionary is then used to generate a denoised reconstructed pain feature vector. Finally, the system uses a sigmoid-calibrated nonlinear regression model to map the reconstructed pain feature vector to a physiological pain score and calculates the magnitude of the psychological emotional response component, normalizing it using a hyperbolic tangent function to obtain the emotional pain score.

[0014] The fusion decision and grading module is used to generate the final clinical assessment results. This module performs time-series trend analysis on physiological pain scores, emotional distress scores, neural conduction lag rate, and physiological vocal energy index, and outputs clinical pain grading results and classification decision signals.

[0015] This module employs a temporal convolutional network, receiving a multidimensional temporal feature tensor composed of the standardized four indicators mentioned above. The network extracts trend features through stacked residual blocks, which contain causal dilated convolutional layers with exponentially increasing dilation coefficients to cover historical time windows. Finally, a preliminary probability distribution of the clinical pain grade is output through a global average pooling layer and a fully connected layer. Furthermore, this module features an adaptive threshold calibration mechanism. The system acquires the average physiological vocal energy index within the current time window and dynamically adjusts the confidence threshold for determining severe pain based on this index. When the average physiological vocal energy index approaches zero, the system automatically lowers the confidence threshold required to determine severe pain. Finally, the system outputs a grade based on the calibrated results and compares the physiological pain score integral with the emotional distress score integral to generate a state indicator signal representing whether nociceptiveness or psychological anxiety is dominant.

[0016] This invention provides an emotional state recognition system for vocal micro-expressions in patients with advanced cancer. It has the following beneficial effects: 1. This invention constructs a prior state vector reflecting a patient's neural control ability by parallel computing neural conduction lag rate and physiological vocal energy index, and generates a confidence mask for multimodal features based on this vector. This mechanism can identify non-stationary states such as social camouflage or physiological exhaustion in patients, and automatically suppresses facial feature weights affected by subjective camouflage or enhances the weights of weak physiological tremor signals during the feature fusion stage. This improves the system's anti-interference ability in complex clinical scenarios and solves the problem that static fusion models cannot adapt to dynamic changes in patient states.

[0017] 2. This invention utilizes a feature decoupling technique based on manifold orthogonal projection. Using the pain baseline vector as the axis, it separates the mixed contextual feature vector into a physiological nociceptive component representing the intensity of nerve reflexes and a psychological emotional response component representing emotional fluctuations. Combined with a sparse dictionary reconstruction algorithm to remove random noise from the physiological component, this method achieves independent quantification of the physical and psychological attributes of pain signals. This provides an objective basis for clinically distinguishing between nociceptive pain and anxiety-related suffering, assisting medical staff in developing more targeted analgesia or psychological intervention plans. Furthermore, by combining a temporal convolutional network with an adaptive threshold calibration mechanism based on energy index, the pain assessment results are dynamically corrected. When a patient's physiological vocal energy index approaches zero, indicating extreme exhaustion, the system automatically lowers the confidence threshold required to determine severe pain. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system architecture according to an embodiment of the present invention; Figure 2 This is a flowchart of a method according to an embodiment of the present invention; Figure 3This is a schematic diagram of the pathological baseline calibration and parameter initialization process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the multimodal parallel feature extraction process according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the calculation process for neural conduction retardation and physiological energy according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the state adaptive gating and feature weighting logic in an embodiment of the present invention; Figure 7 This is a schematic diagram of the multidimensional emotion decoupling and holographic reconstruction process according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the multimodal fusion decision and clinical grading process according to an embodiment of the present invention; Figure 9 This is a simulation diagram of the time-series signal processing flow for pain reflex state in an embodiment of the present invention, wherein; A is a schematic diagram of multimodal signal time-series synchronization analysis, showing the physiological phenomenon that facial action signal (solid line) significantly precedes speech energy signal (dashed line) under sudden pain stimulus; B represents the nerve conduction lag rate. The curve shows the process of the hysteresis rate breaking through the negative threshold (-1.0); C represents the probability of the pathological state ( The dynamic gating graph illustrates the process by which the system identifies the confidence jump in the pain reflex state; D represents the final quantitative output of pain intensity. The graph shows the millisecond-level high-risk alarm response after the system triggers the bypass blocking mechanism; Figure 10 This is a bar chart comparing the pain assessment error (RMSE) of the embodiments of the present invention with that of the prior art in different pathological scenarios. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See attached document Figure 1 , Figure 1 This is a schematic diagram of a system architecture according to an embodiment of the present invention. The present invention provides a system for quantifying the emotional state of patients with advanced cancer, comprising: Data acquisition module: Acquires non-contact physiological and physical signals. Includes a visual sensor (capturing texture and subtle skin color changes) and a microphone array (acquiring speech and filtering out noise), and performs hardware timestamping synchronization of audio and video signals.

[0021] Baseline calibration module: Constructs pathological constraint parameters during the initialization phase. Performs anatomical motion limitation analysis to generate a cachexia weighted matrix; extracts principal components during non-interactive pain periods to generate a pain baseline vector, and stores both as static parameters.

[0022] Feature extraction module: Parallel processing of real-time signals. The visual unit extracts facial motion unit displacement vectors, subpixel-level fasciculation frequencies, and blood perfusion variability index; the acoustic unit extracts short-time energy envelopes, air-to-sound ratios, and fundamental frequency trajectories.

[0023] Hysteresis Calculation Module: The central logic control unit of the system. It calculates the time difference between the onset of speech and the onset of facial response (neural conduction hysteresis rate, reflecting the speed of neural control response), and calculates the physiological vocal energy index (reflecting active physical energy reserves) by combining the breath-to-voice ratio and pauses.

[0024] State gating module: Based on hysteresis rate and energy index, the current system state is divided into normal expression state, extreme weakness state, pain reflex state or cognitive masking state, and subsequent processing paths are selectively activated.

[0025] Decoupling and Reconstruction Module: Performs signal correction. In the weak state, it activates a microtremor and chromatographic complementary algorithm to reconstruct emotional intensity; in the masked state, it removes the pain component while retaining the emotional residual; in the pain reflex state, it outputs a blocking signal.

[0026] Fusion classification module: Input the corrected features into the classifier and output the final label and confidence level that reflects the true cognitive emotion.

[0027] See attached document Figure 2 , Figure 2 This is a flowchart of a method according to an embodiment of the present invention. The present invention provides a method for quantifying emotional states based on neural conduction delay verification, comprising the following steps: S10, perform system initialization and pathological baseline calibration, and construct the patient-specific cachexia weighted matrix and pain manifold subspace; S20: Real-time acquisition of facial image stream and audio stream of target object, and multimodal temporal alignment; S30, extracting facial geometric displacement, fasciculation frequency, blood perfusion variability index and speech acoustic dynamics features in parallel; S40, calculate the neural conduction delay rate between the onset of speech energy and the onset of facial response, and calculate the physiological vocal energy index; S50 determines whether the current state is a normal expression state, an extremely weak state, a pain reflex state, or a cognitive masking state based on the neural conduction lag rate and the physiological vocal energy index. S60 performs dynamic signal processing based on the determined state, corresponding to direct output, micro-vibration chromatographic complementary reconstruction, signal blocking, or nonlinear manifold constraint decoupling, respectively. S70 integrates corrected multimodal features and outputs quantified sentiment state labels through a classifier.

[0028] See attached document Figure 2 , Figure 2 This is a schematic diagram of a multimodal data acquisition and preprocessing process according to an embodiment of the present invention.

[0029] The data acquisition module, serving as the sensing front end, employs non-contact near-field sensing technology to accommodate the pathological characteristics of patients with cachexia and weak vocalization. The acquisition terminal is deployed at the bedside, 0.5 to 1.5 meters from the patient's face. A visual sensor captures an RGB image stream at a frame rate of at least 60 fps to satisfy the Nyquist sampling theorem, ensuring the capture of minute fasciculations of 8–12 Hz. The microphone array utilizes beamforming algorithms for directional sound pickup and noise reduction.

[0030] The system employs a facial landmark detection algorithm based on regression tree ensembles to identify regions of interest (ROIs). Anatomical landmarks are divided into rigid reference points (midpoint of the bridge of the nose, lateral edge of the orbit, highest point of the zygomatic arch) and soft tissue deformation points (brow ridge, eyelid edge, nasal alae, corner of the mouth). The system then uses a landmark detection confidence level... If the value is greater than a preset threshold (e.g., 0.85), it is included in the valid sequence; otherwise, frames are dropped or optical flow prediction compensation is performed.

[0031] Simultaneously, the audio signal processing link performs pre-emphasis and digitization operations. The analog speech signal acquired by the microphone is converted from analog to digital (A / D) with a sampling rate set to 16kHz or higher to meet the human voice frequency band coverage standard. Given that patient vocalization is often accompanied by high-frequency energy attenuation (manifesting as a low or muffled sound), the system introduces a first-order high-pass filter to pre-emphasize the digitized audio stream, aiming to flatten the spectrum and improve the clarity of high-frequency resonants. The transfer function of this pre-emphasis processing is defined as: ; In the formula, Delay operator representing the unit, This is the pre-emphasis coefficient. In this embodiment, considering the need to enhance the feature extraction of weak breath sounds (air sounds), The value is set in the range of 0.95 to 0.98, which can effectively counteract the spectral tilt effect of the glottal excitation pulse.

[0032] To address phase discrepancies caused by independent clock sources, the system establishes a unified time base based on hardware timestamps (PTS). Since the audio and video frame rates are not integer multiples, the system uses the high-precision timeline of the audio stream as the primary reference axis and performs linear interpolation alignment on the video stream.

[0033] Let the audio signal sequence be The video signal sequence is ,in Represents the respective hardware-generated timestamp. For any given audio sampling time... The system searches for the two frames with the closest timestamps in the video sequence. and The interpolated video features at the current moment are calculated using the following formula. : ; If a time deviation caused by a timestamp jump or transmission delay is detected... Exceeding the maximum allowable synchronization error threshold (For example, every 10ms) the system will trigger a reset logic, discarding abnormal data packets in the current buffer to prevent accumulated errors from affecting the accuracy of subsequent neural conduction lag rate calculations. The multimodal data that has undergone the above alignment process is encapsulated into data frames containing synchronized facial feature matrices and audio signal vectors, and transmitted to the subsequent feature extraction module.

[0034] See attached document Figure 3 , Figure 3 This is a schematic diagram of the pathological baseline calibration and parameter initialization process according to an embodiment of the present invention.

[0035] The system constructs a cachexia-weighted matrix based on anatomical movement limitations. During non-invasive monitoring, the system automatically captures occasional large-amplitude facial movements (such as yawning, swallowing, or even frowning in pain), and uses rigid reference points and soft tissue deformation points obtained from the aforementioned data acquisition module to calculate the actual contractile limit of facial muscles.

[0036] In this process, for the first Each facial action unit (AU) is used to calculate its corresponding soft tissue feature points. Relative to the anatomical rigid center The change in Euclidean distance (usually measured at the nasal bone or glabella). To eliminate the influence of head posture on the measurement, all coordinates are corrected for head posture. This patient is defined for the [specific measurement / measurement]. The measured maximum displacement of each action unit Considering the reduced emotional expression due to muscle atrophy, the system introduces a cachexia compensation gain coefficient. The coefficient is calculated based on measured displacement and the norm displacement of healthy individuals. The ratio of . Among them, It can be pre-loaded into the system database, which is derived from statistical data on the average facial movement amplitude of healthy people of the same age.

[0037] To prevent extreme muscle weakness (i.e.) To prevent the gain coefficient from diverging to infinity (approaching 0) and to avoid excessive amplification of noise, this embodiment introduces a smoothing parameter. With saturation threshold Construct the correction function. Compensate for the gain. The calculation formula is as follows; ; in, This is an intensity adjustment factor, typically ranging from 0.8 to 1.2, used to fine-tune the sensitivity of compensation. It needs to be adaptively adjusted based on the patient's subcutaneous fat thickness (the thinner the fat layer, the better). The smaller the value; To prevent division by zero errors, a small constant (e.g., 10) -5 mm); This is the maximum allowed gain (e.g., set to 5.0). Based on all The gain coefficients of each action unit are used to generate a diagonal weighted matrix for cachexia. This matrix will be used in the subsequent feature extraction module to linearly weight facial geometric features in order to "restore" the intensity of the patient's original expression.

[0038] The system synchronously performs pain manifold subspace extraction in a non-interactive state. To separate physiological pain features from mixed facial expressions in subsequent processing, the system needs to collect baseline data when the patient is in pure pain and has no social intention. The system continuously monitors the patient's heart rate variability (HRV) index. When the low-frequency / high-frequency ratio (LF / HF Ratio), which reflects sympathetic nerve excitation, exceeds a preset pain stress threshold (e.g., 2.5), and both vocal energy and facial interaction are silent, the pain baseline acquisition mode is triggered.

[0039] In this mode, the system collects a set of multimodal feature vectors within a time window (e.g., 30 seconds). , where each vector It includes facial micro-expression features and physiological signal features. Since physiological pain often manifests as a specific pattern of coordinated changes in the feature space (such as frowning accompanied by increased heart rate), this embodiment utilizes principal component analysis (PCA) to extract the principal direction of this pattern. The system first processes the set... The data is centered and its covariance matrix is ​​calculated. : ; in, The feature mean vector, This represents the sample size. Subsequently, for... Perform eigenvalue decomposition and select the corresponding largest eigenvalue. The feature vector is used as the pain baseline vector : ; unit vector Mathematically, the one-dimensional manifold direction of pure physiological pain in a multidimensional feature space is defined. In the subsequent feature decoupling step, this vector will serve as the normal direction of the orthogonal projection, used to remove the pain component from mixed emotions.

[0040] Furthermore, to address the need for micro-tremor analysis in extremely frail states, the system requires calibration of resting-state microphysiological characteristic benchmarks. During a resting period when the patient is confirmed to be breathing steadily and without significant movement, the system extracts photoplethysmography (rPPG) signals from the region of interest (ROI) on the face and calculates its signal quality index (SQl) to eliminate motion artifacts. Time-domain statistics are then performed on the selected effective rPPG signals to obtain the resting mean of blood perfusion intensity. with standard deviation Simultaneously, using the video stream processed with Euler video magnification technology, spectral analysis was performed on the frontalis and orbicularis oculi muscle regions to determine the patient's unique resting fasciculation frequency. This frequency is typically distributed in the 8Hz to 12Hz band, but frequency shifts may occur under certain neuropathologies. Establishment The aim is to provide the center frequency parameters of a bandpass filter for subsequent differentiation between physiological microfreaks and active microexpressions. These calibration parameters are stored in the system's non-volatile memory until the next recalibration is triggered.

[0041] See attached document Figure 4 , Figure 4 This is a schematic diagram of a multimodal parallel feature extraction process according to an embodiment of the present invention.

[0042] Extract pathologically weighted facial geometric features. Calculate feature points in the current frame using optical flow or keypoint difference methods. Reference frames for neutral facial expressions The difference in Euclidean distance between them.

[0043] Introducing a cachexia weighted matrix Perform non-uniform linear mapping gain correction: ; In the formula, Indicates the first The original displacement of each action unit (unit: millimeters). This represents the corresponding specificity gain coefficient. Through this matrix operation, the system can recover statistically significant facial expression feature vectors from subtle facial muscle contractions. This provides sufficient signal strength for subsequent sentiment classification.

[0044] Synchronous activation of the Euler video amplification (EVM) path was used to extract involuntary fasciculations of 8–12 Hz. Laplacian pyramid spatial decomposition was performed on the ROI image sequence, and bandpass filtering was applied to the pixel temporal signals of the spatial frequency band (passband). ,in The resting tremor frequency obtained during the baseline calibration phase. The bandwidth is defined as 1 Hz. The filtered signal is amplified and superimposed back onto the original image, but at the feature extraction level, the system directly calculates the energy density within the amplified bandwidth. (Location defined) At any moment micro-tremor intensity for: ; In the formula, This is the original pixel brightness signal. and These represent the Fourier transform and the inverse transform, respectively. This is the transfer function of the bandpass filter for the corresponding frequency band. The system statistically analyzes the values ​​of all pixels within the ROI region. The mean value is used to generate the fasciculation index, which reflects the intensity of involuntary contractions of subcutaneous muscle fibers.

[0045] The RGB channels of the ROI were separated, and the mixed signal was separated from the green channel using independent component analysis (ICA). The component with the highest energy proportion in the physiological heart rate frequency band (0.7-3.0Hz) was selected as the effective rPPG source signal by power spectral density (PSD) analysis. The perfusion variability index (PVI) was calculated to reflect the autonomic nerve response of microcirculation to pain stimuli.

[0046] In the audio branch, the speech signal is framed and windowed (Hamming window, window length 25ms, frame shift 10ms). Calculate the... Frame speech signal Its short-term energy The calculation is as follows: ; In the formula, For window functions. The trajectory will be used to subsequently determine the explosiveness and sustainability of the initial speech momentum.

[0047] To quantify the breathy sound caused by incomplete vocal cord closure in patients, the system calculates the harmonic noise ratio (HNR). The system uses the autocorrelation function method to estimate the fundamental period and separates the signal into harmonic components. With noise components To prevent computational overflow caused by noise energy approaching zero, a smoothing constant is introduced. (e.g., 10) -6 HNR value (unit: dB) is defined as: ; A low HNR value (e.g., below 12dB) typically indicates significant glottal leakage and weak vocalization. This feature serves as a key input for subsequent calculations of the physiological vocal energy index. Finally, the system performs temporal synchronization and concatenation of the features obtained from the parallel calculations. For each processing frame, the output feature vector... Includes: Pathologically corrected geometric displacement vector Fasciculation index, perfusion variability index, short-term energy and harmonic noise ratio After being standardized to the [0,1] interval, it is transmitted to the hysteresis calculation module.

[0048] See attached document Figure 5 , Figure 5 This is a schematic diagram of the calculation process for neural conduction delay and physiological energy according to an embodiment of the present invention.

[0049] A bidirectional sliding window algorithm based on dynamic threshold and persistence verification is used to detect the onset moment of audiovisual synergy.

[0050] For audio streams, the system calculates the background noise level in real time. and its standard deviation Set dynamic trigger threshold. When detected And this state lasts longer than the minimum vocalization period. (For example, at 50ms) the system backtracks to the moment when the energy first exceeds the threshold and marks it as the moment the speech momentum begins. This continuous verification logic can effectively filter out short-lived impact noises or device current noises.

[0051] For the video stream, the system calculates the total facial kinetic energy. Set facial response threshold. It is 1.5 times the resting-state baseline energy. When When the threshold is exceeded and a monotonically increasing trend is observed, this moment is marked as the onset moment of facial response. It should be noted that, in order to eliminate interference from muscle fasciculations, a low-pass filter (cutoff frequency 2Hz) was configured in this step to respond only to large macroscopic movements.

[0052] Then the neural conduction lag rate was calculated ( This reflects the time delay between the brain's motor cortex issuing a command and the effector (muscle / vocal cord) executing the command. In natural interaction, facial expressions typically precede or are synchronized with speech. However, given the extreme pathological states that advanced cancer patients may experience inability to speak (painful aphasia) or compulsive vocalization (expressionless groaning), the system performs a modality existence check before computation. If the system only detects the onset of facial response within the current time window... If no effective speech initiation is detected, the system determines it to be a painful aphasia state and forcibly sets [a certain condition]. A negative extreme value (e.g., -2.0) indicates that facial response significantly precedes speech (or speech absence); conversely, if only speech onset is detected without facial response, the value is set to... It is a positive extreme value (e.g., +2.0).

[0053] When both the audiovisual modalities have valid onset moments, the system calculates the time difference between the speech onset and the facial response. Considering individual differences and measurement errors, this difference needs to be normalized. Define the standard reaction latency norm. Typically taken as 200ms), then the neural conduction lag rate The calculation is as follows: ; In the formula, Allowable error for normal physiological synchronization (e.g., ±50ms).

[0054] like This indicates that facial expression significantly lags behind vocalization, which is clinically common in cognitive masking or compulsive social smiling; if This indicates that facial expression responses precede conscious vocalization and usually correspond to primitive reflexes triggered by acute pain.

[0055] To prevent state jumps caused by single measurement errors, the system uses an exponentially weighted moving average (EMA) to update the hysteresis rate. ; Among them, smoothing factor The value is set between 0.7 and 0.9 to ensure the stability of the indicator; initial value Set to 0. Simultaneously, the system quantifies the physiological vocal energy index ( This index combines acoustic dynamics characteristics with vocal sustain, and is used to distinguish between a weak state of exhaustion and a painful state of hoarseness.

[0056] Integrating short-time energy peaks Average harmonic noise ratio And the duration of this statement Considering the inconsistency of the dimensions of the various features, Min-Max standardization is first performed to the [0,1] interval. The calculation model for the physiological vocal energy index is as follows: ; In the formula, and These are the normalized eigenvalues; This is the weighting coefficient. In this embodiment, the weight is set to... This indicates that sound intensity and sound quality (especially HNR, which reflects the degree of breathiness) dominate the energy determination, while the duration factor is nonlinearly compressed through a logarithmic function to prevent long vocalizations (such as groans) from excessively increasing the index.

[0057] Ultimately, the rate of neural conduction lag Physiological vocal energy index Combined generation of time-series state vectors The data is transmitted to the state gating module to dynamically adjust the weights of the emotion classification strategy.

[0058] See attached document Figure 6 , Figure 6 This is a schematic diagram of state adaptive gating and feature weighting logic according to an embodiment of the present invention.

[0059] The system performs probability mapping in the ternary pathological state space. It receives the temporal state vector. Based on a multilayer perceptron (MLP), it is mapped to the probability distribution of three orthogonal baseline state prototypes: 1. Social interaction state: corresponds to a low latency rate ( Furthermore, the medium to high energy level indicates that the patient is in a normal communication situation; 2. Cognitive masking state: corresponds to a high lag rate ( It also has moderate energy, indicating a psychological defense mechanism in which the patient tries to hide their pain and appear strong. 3. Physiological exhaustion state: This corresponds to a normal lag rate but extremely low energy, or a negative lag rate (painful aphasia), which represents a dying period or severe cachexia scenario.

[0060] The MLP network structure consists of an input layer (2-dimensional), hidden layers (16-dimensional, ReLU activation function), and an output layer (3-dimensional, Softmax activation function). State probability distribution vector. The calculation is as follows: ; in, This is the weight matrix. This is the bias term. Output vector. Each term in the expression represents the posterior probability of belonging to one of the three states at the current time, and satisfies the following conditions: This probability distribution is not the final output, but rather serves as an intermediate variable for generating feature attention.

[0061] The system then generates a dynamic feature confidence mask. Three prior weight template vectors with the same dimensions as the fused feature vector are preset: (Social Template): Emphasize macro-facial geometric feature components (weight set to 0.8), suppress micro-tremor feature components (weight set to 0.2); (Mask Template): Based on the neuropsychological principle that micro-expressions and fasciculations are difficult to fake by subjective consciousness, the macro-facial feature components are significantly suppressed (weight reduced to 0.1), and the weights of the fasciculation index and rPPG feature components are significantly increased (weight increased to 0.9). (Exhaustion Template): Emphasizes acoustic aerophone features (HNR) and physiological signal components, ignoring facial movement components.

[0062] The system generates the current dynamic mask through linear weighted summation. : ; Introducing a lower threshold (For example, 0.05) is truncated. The final result is... The clinical reliability of each sensor channel at the current moment was quantified.

[0063] The system will integrate the original features With confidence mask Element-wise multiplication yields the weighted features: ; A soft gating mechanism is implemented to filter out untrusted feature components. The sequentially weighted features are input into a GatedRNN network containing two LSTM layers (256 hidden layers) and a temporal attention layer, and the attention weights are calculated. .

[0064] Furthermore, in order to achieve decoupling of pathological components, the system utilizes the pain baseline vector obtained from the baseline calibration module. (i.e., the principal direction of the pain manifold subspace, and satisfying) ) for context feature vectors Perform orthographic projection: ; The modulus directly characterizes the emotional intensity caused purely by physiological pain, eliminating orthogonal interference components caused by environmental factors or social anxiety.

[0065] For the aforementioned neural network model, this embodiment discloses its specific training strategy to ensure the reproducibility of the solution; Sample source: A clinical video library of advanced cancer patients that has undergone ethical review is used, and the labels are jointly annotated by three or more senior palliative care physicians based on the VAS and CPOT scales, with the average value taken as the Ground Truth; Loss function: Given the sparsity of high-intensity pain samples on the time axis (i.e., class imbalance problem), this embodiment uses FocalLoss instead of conventional cross-entropy loss: ; in, Predicting probabilities for the model; balancing factor (e.g., 0.25) is used to reduce the weight of the background class (painless); focus parameter (e.g., 2.0) is used to increase the model's attention to difficult-to-classify samples (such as micro-expression pain); Optimizer: AdamW optimizer is used, with an initial learning rate set to Furthermore, a cosine annealing strategy is employed to decay the learning rate, in order to prevent getting trapped in local optima.

[0066] See attached document Figure 7 , Figure 7 This is a schematic diagram of a multidimensional emotion decoupling and holographic reconstruction process according to an embodiment of the present invention.

[0067] The system performs feature decoupling based on manifold orthogonal projection. The system receives the context feature vector output by the state gating module. To separate physiological pain from psychological emotions, this embodiment utilizes the pain baseline vector obtained during the baseline calibration phase. (i.e., the principal direction of the pain manifold subspace, and satisfying) ( ) serves as the reference axis.

[0068] The system calculates the projection of the context vector onto the principal direction of the pain manifold and extracts the physiological nociceptive components. : ; This component Physically, it characterizes the intensity of a nerve reflex directly triggered by a noxious stimulus, is positively correlated with the degree of tissue damage, and is theoretically not subject to the patient's subjective will.

[0069] Based on this, the system extracts the residual vector perpendicular to the principal direction of pain, i.e., the psychological and emotional response component, through vector subtraction. : ; This component It includes non-specific negative emotions such as anxiety, fear, and depression, as well as facial expressions that patients use to mask their emotions due to social needs. To ensure that the residual truly corresponds to psychological fluctuations rather than random noise, the system further performs frequency domain analysis, retaining only slowly varying components with frequencies below 0.5 Hz (corresponding to continuous changes in emotional state) and filtering out high-frequency jitter noise.

[0070] Subsequently, the system performs sparse reconstruction based on dictionary learning. Although the projection operation separates the directions, The signal may still contain random noise such as involuntary muscle twitches. To further purify the signal, this embodiment employs sparse coding technology, using a pre-trained standard pain atom dictionary to clean and reconstruct the physiological components.

[0071] Define a pain atom dictionary ,in For feature dimension, The number of atoms (e.g., 1024). This dictionary. It was obtained by training a large number of clean feature samples from confirmed pain patients using the K-SVD algorithm in the offline stage. Each column represents a typical, noise-free pain micro-expression or micro-tremor pattern primitive.

[0072] The system solves the following L1 regularization optimization problem to obtain a sparse coefficient vector. : ; In the formula, The sparsity penalty factor (ranging from 0.01 to 0.1) is used to control the sparsity of the reconstruction. This embodiment uses the Iterative Shrink Thresholding Algorithm (ISTA) to solve the problem, forcing the system to use only the most typical pain primitives in the dictionary to fit the input signal, thereby automatically filtering out abnormal noise that does not conform to the general pain pattern.

[0073] Obtain the optimal sparsity coefficient Then, the reconstructed pure pain features are calculated. : ; This reconstructed vector Only statistically significant pain features were retained, effectively removing measurement noise introduced by sensor jitter or changes in lighting, thus improving the signal-to-noise ratio of subsequent quantization.

[0074] Furthermore, the system performs nonlinear mapping and multidimensional quantization output. In order to map the high-dimensional feature vector to the clinically applicable VAS score, this embodiment uses a nonlinear regression model calibrated with Sigmoid to adapt to the Weber-Fechner law characteristics of human pain perception, that is, sensitivity to low-to-medium intensity stimuli, while tending to saturate to high-intensity stimuli.

[0075] For physiological pain scores The calculation formula is as follows: ; In the formula, and The regression parameters are obtained by training on a labeled dataset by minimizing the mean squared error (MSE), and the label values ​​are the VAS scores given by clinical experts.

[0076] For emotional distress rating The system's emotional and psychological components Perform modulus calculation and normalization: ; In the formula, This is the sensitivity coefficient (e.g., 0.5). The function is used to restrict the value to a range of values. A range is defined to prevent score overflow.

[0077] Finally, the decoupling and reconstruction module outputs a holographic evaluation vector containing multidimensional information. : ; in: It directly reflects the degree of physiological damage and is used to guide the precise dosage adjustment of opioids; : Reflects the degree of psychological distress of the patient, and is used to prompt medical staff to intervene with psychological comfort or anti-anxiety treatment; : The confidence level of the result, defined as the reciprocal of the sparse reconstruction error. When this value is lower than a preset threshold (such as 0.5), the system will issue a warning, indicating that the current measurement may be subject to strong interference. : The main pathological states (such as cognitive masking states) identified by the state gating module.

[0078] Through the above decoupling and reconstruction mechanism, the system not only outputs a single pain value, but also deconstructs the texture and components of pain, solving the technical problem that traditional technologies cannot distinguish between physiological pain and social distress, and providing multi-dimensional diagnostic and treatment evidence for clinical practice.

[0079] See attached document Figure 8 , Figure 8 This is a schematic diagram of a multimodal fusion decision and clinical grading process according to an embodiment of the present invention.

[0080] The system extracts historical data segments of length L (e.g., 300 frames, approximately 10 seconds). This is used for physiological pain rating sequences. Emotional distress rating sequence Nerve conduction hysteresis sequence Physiological vocal energy index sequence Perform z-score normalization and stack as Input feature tensor .

[0081] The system then performs trend feature extraction based on causal dilated convolution. A classification model is constructed using a Temporal Convolutional Network (TCN): containing a receiving tensor. The input layer consists of three layers of residual blocks (causal convolution kernel size 3, number of channels 64, dilation coefficient d=1,2,4, configured with GELU and Dropout); and a fully connected output layer.

[0082] The forward propagation computation of the model is defined as follows: ; in, This represents the composite nonlinear transformation of the TCN network. These are the parameters for the fully connected layer; GAP is the global average pooling operation. Output vector. The vector has four dimensions, corresponding to the four levels of CPOT commonly used in clinical practice: no pain, mild discomfort, moderate pain, and severe pain. Each element in the vector represents the confidence probability of belonging to the corresponding level.

[0083] After the model outputs, the system performs adaptive threshold calibration based on the energy index. For extremely frail patients, this is done based on the average energy index within the current window. The baseline confidence threshold for determining severe pain. (For example, 0.8). Dynamically adjusted to... : ; In the formula: Adjustment coefficient (range of values) This determines the maximum extent to which the threshold is lowered; Energy decay sensitivity constant (range of values) When the patient's energy index As the confidence threshold approaches zero (i.e., in a state of extreme exhaustion), the confidence threshold required to determine severe pain will automatically decrease. This means that for patients in the dying stage, even if they only show slight frowning, the system will tend to issue a high-risk warning of severe pain, thereby avoiding insufficient analgesia due to the patient's inability to express their pain.

[0084] Finally, based on the calibrated classification results and decoupling features, the system generates structured clinical decision instructions. This instruction includes a property classification based on logical judgment: like Marking nociceptive dominance suggests adjusting opioid dosage; if emotional score is dominant and facial lag is significant ( ( ), indicating that psychological distress is dominant, suggesting psychological or pharmacological intervention.

[0085] To illustrate the working mechanism of the present invention more intuitively, a typical clinical monitoring scenario is described below.

[0086] Target patient: Patient A, male, 67 years old, with advanced lung cancer and multiple bone metastases, long-term bedridden, with severe cachexia (BMI<18.5) and cancer pain.

[0087] Scenario 1: Cognitive Disguise (During Family Visits): Input signal: During family visitation, Patient A attempted to reassure their family members, displaying a smiling expression (contraction of the zygomaticus major muscle). However, the system detected the following characteristics: Timing anomaly: The smiling emoji appeared approximately 1.2 seconds after the voice response "I'm fine," indicating a neural conduction delay rate. The value was calculated to be +1.5 (significantly lagging), which was determined to be an unnatural emotional expression; Microscopic features: Although there was a macroscopic smile on the face, the EVM algorithm extracted a 10Hz high-frequency micro-tremor in the orbicularis oculi muscle, and the PVI index was increased (sympathetic nerve excitation).

[0088] Processing logic: The state gating module determines the state to be a cognitive masking state. Activate. The template reduces the weight of macroscopic smile features to 0.1 and increases the weights of micro-tremors and blood flow features to 0.9. The decoupling module identifies "smile" as a psychological faking component and filters it out; output: despite the patient's outward smile, the system outputs a physiological pain score. (Moderate to severe pain) This indicates to healthcare professionals that the patient is enduring pain and the analgesia pump dosage needs to be monitored.

[0089] Scenario 2: Pain reflex state (when turning over at night): Input signal: When turning over during nighttime care, the patient's face momentarily constricts between the eyebrows (AU4), followed by a groan 0.3 seconds later.

[0090] Timing anomaly: calculated (The facial reaction occurred significantly earlier than the vocalization), and was accompanied by a burst of instantaneous energy. The processing logic was as follows: the state gating module determined that the patient was in a pain reflex state. The system triggers the blocking flag, skips the manifold decoupling step, and outputs the following result: the system outputs a high-risk alarm signal in milliseconds, records the instantaneous pain intensity VAS9.0, determines it to be breakthrough pain, and recommends the administration of an accelerated-acting opioid.

[0091] Experimental verification and effect comparison: To verify the effectiveness of the "emotional state quantification method based on neural conduction delay verification" proposed in this invention, the applicant constructed a clinical validation dataset containing 1200 video clips from 50 patients with advanced tumors.

[0092] Comparison baseline settings: This experiment set up three sets of comparison models: Baseline Model A: only facial images are used, and ResNet50 is used to extract features for emotion classification, without temporal hysteresis analysis; Baseline Model B: audiovisual multimodal features are directly concatenated, without state gating and decoupling modules.

[0093] The model of this invention is a complete system that includes hysteresis computation, state gating, and decoupling and reconfiguration.

[0094] Evaluation indicators: VAS-RMSE: Root mean square error between predicted pain scores and expert-labeled VAS scores (lower is better).

[0095] F1-Masking: The F1 score for classifying "forced smile" samples.

[0096] Response latency: The average time (in milliseconds) from the onset of pain to the system outputting an alarm.

[0097] Analysis of experimental results: The experimental results are shown in the table below (corresponding to the appendix). Figure 9 Appendix Figure 10 ): Regarding robustness in the masked state: baseline models A and B have extremely high errors and are easily misled by feigned expressions. This invention utilizes neural conduction lag rate... It identifies masquerading behaviors and reconstructs real pain through microtremor features, reducing RMSE by approximately 83%. In terms of response speed, thanks to the bypass blocking mechanism of pain reflex state, the response delay of this invention to severe pain is only 120ms, which is significantly better than the inference time of traditional deep learning models and meets the needs of clinical emergency care.

[0098] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A system for recognizing emotional states through vocal micro-expressions in patients with advanced cancer, characterized in that, The system includes: The hysteresis and energy calculation module is configured to process the acquired audio stream and facial video stream in parallel. It calculates the neural conduction hysteresis rate by detecting the phase difference of the audio-visual signals on the time axis, and calculates the physiological vocal energy index by combining the acoustic dynamics characteristics. The two are then combined to generate a temporal state vector. The state adaptive gating module is configured to receive the temporal state vector, generate confidence masks for different modal features through nonlinear mapping, and perform weighted gating processing on the extracted multimodal original features to obtain a context feature vector that reflects the patient's current neural control ability. The emotional decoupling and reconstruction module is configured to use a preset pain baseline vector to perform manifold orthogonal projection on the context feature vector, separate the physiological harm perception component and the psychological emotional response component, and quantify them into physiological pain score and emotional distress score, respectively. The integrated judgment and grading module is configured to perform time-series trend analysis on the physiological pain score, emotional distress score, nerve conduction lag rate, and physiological vocal energy index, and output the clinical grading results and classification decision signals for pain.

2. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 1, characterized in that, The hysteresis and energy calculation module is configured to perform the following operations when calculating the neural conduction hysteresis rate: Calculate the background noise dynamic threshold of the audio stream and the facial response threshold of the video stream respectively, and locate the speech initiation moment and the facial response initiation moment. Modal existence check: If only facial response is detected without effective speech, it is determined to be a painful aphasia state and the hysteresis rate is set to a negative extreme value; if only speech is detected without facial response, it is determined to be a forced vocalization state and the hysteresis rate is set to a positive extreme value. If both the audiovisual modalities have valid onset times, the time difference between the speech onset time and the facial response onset time is calculated and normalized based on the standard response latency norm to obtain the neural conduction lag rate, which is then updated using an exponentially weighted moving average.

3. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 1, characterized in that, The hysteresis and energy calculation module is configured to: When calculating the physiological vocal energy index, the hysteresis and energy calculation module is configured as follows: The short-time energy peak, average harmonic noise ratio, and duration of sound emission of the audio stream are selected as characteristic parameters. The feature parameters are normalized, and the duration of the vocalization is nonlinearly compressed using a logarithmic function. The physiological vocal energy index is obtained by weighted summation, where the weight of short-term energy peak and average harmonic noise ratio is higher than the weight of vocal duration.

4. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 1, characterized in that, The state adaptive gating module generates the confidence mask in the following way: Based on the temporal state vector, the patient's state is mapped to the probability distribution of three orthogonal baseline state prototypes through a multilayer perceptron. The baseline state prototypes include: a social interaction state characterized by low lag rate and medium to high energy, a cognitive masking state characterized by high lag rate and medium energy, and a physiological exhaustion state characterized by low energy or abnormal lag rate. The confidence mask is generated by pre-setting weight template vectors corresponding to the three baseline state prototypes, performing linear weighted summation and lower limit threshold truncation on the weight template vectors according to the probability distribution.

5. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 4, characterized in that, The state-adaptive gating module is also configured to: Perform a soft gating operation by multiplying the confidence mask element-wise with the original multimodal features; The gated feature sequence is input into a bidirectional long short-term memory network, and a weighted sum is calculated through a temporal attention layer to generate the context feature vector.

6. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 1, characterized in that, The specific operations performed by the emotion decoupling and reconstruction module include: The pain baseline vector is used as the principal direction of the pain manifold subspace; The projection of the context feature vector onto the direction of the pain baseline vector is calculated to obtain the physiological nociceptive component characterizing the intensity of the nerve reflex; The residual vector obtained by subtracting the physiological harm perception component from the context feature vector is calculated, and high-frequency noise is filtered out to obtain the psychological emotional response component.

7. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 6, characterized in that, The emotional decoupling and reconstruction module is also configured to perform sparse reconstruction of the physiological harm perception component: A pre-trained dictionary of pain atoms is introduced, which contains noise-free pain micro-expression pattern primitives; Taking the physiological nociceptive component as the target, an L1 regularization optimization problem is solved based on the pain atom dictionary to obtain a sparse coefficient vector; The denoised, clean pain feature vector is generated by multiplying the sparse coefficient vector with the pain atom dictionary.

8. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 7, characterized in that, The quantitative scoring method for the emotion decoupling and reconstruction module is as follows: The pure pain feature vector is mapped to the physiological pain score using a sigmoid-calibrated nonlinear regression model. The modulus of the psychological and emotional response components is calculated and normalized using the hyperbolic tangent function to obtain the emotional distress score.

9. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 1, characterized in that, The fusion decision and classification module includes a temporal convolutional network, which is configured as follows: Receive a multidimensional temporal feature tensor consisting of standardized physiological pain score, emotional distress score, neural conduction lag rate, and physiological vocal energy index; Trend features are extracted by stacked residual blocks, which contain causal dilated convolutional layers with exponentially increasing dilation coefficients to cover historical time windows. The preliminary probability distribution of clinical pain grading is output through a global average pooling layer and a fully connected layer.

10. The emotional state recognition system for voice micro-expressions in patients with advanced cancer according to claim 9, characterized in that, The fusion decision and classification module is also configured to perform adaptive threshold calibration: Obtain the average physiological vocal energy index within the current time window; The confidence threshold for determining severe pain is dynamically adjusted based on the average physiological vocal energy index. When the average physiological vocal energy index approaches zero, the confidence threshold is lowered. The final clinical pain grading result is output based on the calibrated confidence threshold, and the scores of physical pain and emotional distress are compared to generate a state indicator signal that represents whether nociceptiveness or psychological anxiety is dominant.

Images