Pain threshold estimation method and system based on multi-modal data fusion

Abstract

The invention is suitable for the field of pain assessment, and particularly provides a pain threshold estimation method and system based on multi-modal data fusion, and the method comprises the steps: constructing a cross-modal weak feature enhancement fusion network, carrying out the processing of collected multi-modal data, and extracting physiological weak features and behavior weak features; introducing a time-varying attention Transform model, dynamically allocating weights for the processed physiological weak features, behavior weak features and environment auxiliary data by using a pain dynamic tag, and outputting cross-modal fusion features; constructing a double-branch pain threshold estimation model, wherein the model comprises a supervised estimation branch and an unsupervised estimation branch; and introducing a pain feature element library, carrying out online calibration on parameters of the double-branch model in combination with a meta learning algorithm, and outputting a real-time pain threshold value and a short-term pain threshold value change trend of the target object. According to the embodiment of the invention, the clinical operability of pain assessment is remarkably improved, and ineffective intervention caused by inaccurate assessment is reduced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and particularly relates to the field of pain assessment technology, specifically a method and system for predicting pain thresholds based on multimodal data fusion. Background Technology

[0002] Pain assessment is a core technology in clinical diagnosis and treatment (such as postoperative analgesia and regulation, chronic pain management, and rehabilitation monitoring) and public health. Its accuracy directly affects the effectiveness of intervention programs and patients' quality of life. With the development of digital healthcare, pain assessment technology has gradually evolved from an early single-modality approach: initially relying on subjective ratings, but heavily influenced by individual subjective biases; subsequently shifting to physiological signal acquisition or behavioral feature recognition, but a single modality cannot fully cover the physiological nature, behavioral representation, and subjective perception dimensions of pain.

[0003] Currently, multimodal data fusion has become the mainstream development direction for pain threshold prediction, with the industry generally collecting physiological, behavioral, and subjective data simultaneously. However, clinical needs for pain assessment are further upgrading, and existing multimodal pain threshold prediction methods suffer from technical problems such as prediction lag and insufficient accuracy due to the mismatch between the dynamic temporal variability of pain signals and static fusion mechanisms, the suppression of weak cross-modal features by dominant features, and the lack of adaptation to populations without subjective expression. Summary of the Invention

[0004] The purpose of this invention is to provide a pain threshold prediction method and system based on multimodal data fusion, so as to at least solve the technical problems of prediction lag and insufficient accuracy in existing pain assessment methods.

[0005] To achieve the above objectives, the present invention provides the following technical solution.

[0006] According to one embodiment of this application, a pain threshold prediction method based on multimodal data fusion is provided, including: A lightweight LSTM network is used to perform real-time analysis on the preprocessed multimodal data, adaptively adjust the sampling parameters of each sensor in the multimodal sensor group, and label the collected multimodal data with dynamic pain tags. A cross-modal weak feature enhancement and fusion network is constructed to process the collected multimodal data and extract physiological weak features and behavioral weak features. A time-varying attention Transformer model is introduced, and dynamic pain labels are used to dynamically assign weights to the processed weak physiological features, weak behavioral features, and environmental auxiliary data, and output cross-modal fusion features. A bi-branch pain threshold prediction model is constructed, including a supervised prediction branch and an unsupervised prediction branch. The supervised prediction branch is used to learn the explicit correlation between cross-modal fusion features and pain threshold. The unsupervised prediction branch uses dynamic pain labels as supervision signals to learn the consistency pattern between weak physiological features and weak behavioral features. A pain feature meta-library is introduced, and the parameters of the bi-branch model are calibrated online using a meta-learning algorithm. The model outputs the real-time pain threshold and short-term pain threshold change trend of the target object.

[0007] Furthermore, it also includes steps to optimize the real-time pain threshold and short-term pain threshold change trends of the target object. Optimization methods include: The real-time pain threshold is compared with clinical real-time monitoring data to dynamically adjust the modal weight allocation rules of the time-varying attention Transformer; Accumulate and filter pain intervention effect data for target subjects, and update the pain feature library and training parameters of the bi-branch model.

[0008] Furthermore, the steps for real-time analysis of the preprocessed multimodal data using a lightweight LSTM network include: Construct a dual-branch lightweight LSTM main structure, including a physiological branch LSTM and a behavioral branch LSTM; In the physiological branch of the LSTM, depthwise separable convolutional layers are used to first compress the dimensionality of the physiological modality feature matrix and extract local features, and then input into a lightweight LSTM unit to output a physiological modality pain-related feature vector, represented as follows: ; Let D represent the set of real numbers, and let D represent the dimension. In the behavioral branch LSTM, a spatiotemporal attention module is introduced to first perform spatial attention weight allocation and temporal attention filtering on the behavioral modality grayscale temporal image sequence. Then, the attention-weighted temporal image features are input into a lightweight LSTM unit, and the output behavioral modality pain-related feature vector is expressed as follows: ; The feature vectors output from the physiological branch and the behavioral branch are concatenated to form a fused feature, represented as: The cross-modal attention weights are calculated using a multilayer perceptron (MLP) and are expressed as follows: The spliced ​​features are then weighted and optimized, and the final weighted fused features are represented as follows: Highlighting the consistency of pain correlation information between physiological and behavioral characteristics; The features fused by cross-modal attention are input into the Softmax classifier, which outputs the recognition probability of pain stage and pain type. The category corresponding to the maximum probability is taken as the final recognition result.

[0009] Furthermore, the steps for extracting weak physiological features include: using wavelet packet transform combined with an attention gating module to improve the signal-to-noise ratio of weak physiological features in the physiological modality data based on a differentiated strategy, and dynamically adjusting the noise suppression parameters in conjunction with environmental auxiliary data. The environmental auxiliary data includes the noise intensity of the scene in which the target object is located. When the environmental auxiliary data shows a noise intensity threshold in decibels, the number of decomposition layers of the wavelet packet transform is increased, and the noise suppression threshold of the attention gating module is lowered to enhance the screening of weak features. The physiological modality data includes electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, and electrodermal (ED) signals. The physiological weakness features include fluctuations in the gamma band of electroencephalography (EEG) and changes in the amplitude of skin conductance.

[0010] Further, the steps for extracting weak behavioral features include: using optical flow to extract weak behavioral features from facial expression videos in behavioral modality data, and combining this with environmental auxiliary data to remove false features. The environmental auxiliary data includes illumination intensity and shooting angle. When the environmental auxiliary data shows a low-light scene, brightness compensation is applied to the weak behavioral features extracted by optical flow, and the movement contours of the orbicularis oculi and orbicularis oris muscles are enhanced using an edge enhancement algorithm. If the environmental auxiliary data shows a deviation in shooting angle, the facial image is normalized to a frontal view using a pose correction algorithm, and then subpixel-level motion features are extracted. The behavioral modality data includes facial expression videos and body posture images; The weak behavioral features include subpixel-level motion features of the orbicularis oculi and orbicularis oris muscles.

[0011] Furthermore, the unsupervised prediction branch uses pain dynamic labels as supervision signals to learn the consistency patterns between physiological weak features and behavioral weak features, including the following steps: Construct a comparative sample based on dynamic pain labels; A dual-modal lightweight encoder is constructed, comprising a physiological feature encoder and a behavioral feature encoder, which map the preprocessed weak physiological and behavioral features into feature embedding vectors, respectively; the physiological feature embedding vectors are then calculated. With behavioral feature embedding vector The cosine similarity is used to output a consistency-enhanced bimodal embedding vector through the attention weight matrix. ; The triplet loss function is adopted, using the consistency-enhanced bimodal embedding vector of positive sample pairs as the anchor point, the embedding vector of positive sample pairs with the same label as the positive example, and the embedding vector of negative sample pairs as the negative example. The bimodal encoder parameters are optimized by minimizing the distance between the anchor point and the positive example and maximizing the distance between the anchor point and the negative example, as expressed in: In the formula, Indicates the interval threshold. Represents the L2 norm. Represents the anchor vector. Represents the positive example vector. Represents a negative instance vector. Represents the triplet loss function; The variance of the embedding vector after multiple forward propagations of the same weak feature segment by the encoder is calculated, and a stability loss function is introduced to enable the model to learn a fixed association pattern between physiological weak features and behavioral weak features under a specific pain label. Among them, the stability loss function Represented as: In the formula, M represents the embedding vector. The dimension; The stability loss function is measured by the variance of the embedding vector; K represents the number of times the same weak feature segment is forward-propagated through the encoder. This represents the enhanced embedding vector after the k-th forward propagation. The value taken in the m-th dimension; This represents the mean of the embedding vectors in the K forward propagation operations along the m-th dimension.

[0012] Furthermore, in the step of constructing a comparison sample based on pain dynamic labels, the comparison sample includes positive sample pairs and negative sample pairs; In the construction of positive sample pairs, for weak spatiotemporal alignment features under the same target object and the same label, feature fragments are randomly selected to form positive sample pairs. Positive sample pairs have feature consistency under the same pain state. The construction of negative sample pairs includes cross-label negative samples and misaligned negative samples with the same label; For cross-label negative samples, weak feature fragments under different dynamic labels but with the same target object are selected to form pairs; For negative samples with misaligned labels, weak feature segments with the same dynamic label and a time interval exceeding a threshold are selected to form pairs.

[0013] Furthermore, the steps of introducing a pain feature meta-database and combining it with a meta-learning algorithm to perform online calibration of the parameters of the bi-branch model include: A multi-dimensional pain feature meta-library is constructed, which is divided into multiple independent meta-tasks. Each meta-task stores the physiological weak feature statistics and behavioral weak feature statistics corresponding to that task, as well as the initial parameters of the dual-branch model adapted to the current task. The initial parameters are expressed as: In the formula, This represents the support set for the meta-task. This represents the learning rate of the original task; For the loss function with respect to the general parameters The gradient; A model-independent meta-learning algorithm is used to perform meta-training on the dual-branch model; For the target subjects to be evaluated, multimodal data were collected, and weak physiological and behavioral features were extracted. At the same time, the population characteristics and initial pain manifestations of the target subjects were recorded. The meta-task that best matches the target object is retrieved from the pain feature meta-database. The initial parameters of the bi-branch model stored in this meta-task are then used as the initial calibration parameters. Using the collected multimodal data of the target object, the initial calibration parameters are updated with a gradient to obtain calibration parameters adapted to the individual characteristics of the target object. , is represented as: In the formula, This indicates a new weak physiological characteristic of the target object; This represents a new weak behavioral feature of the target object. Indicates the initial pain threshold label of the target object; This represents the model parameters at the current m-th iteration; Indicates the online learning rate; Represents the loss function Regarding the current parameters The gradient.

[0014] Furthermore, the step of introducing a time-varying attention Transformer model, utilizing dynamic pain labels, to dynamically assign weights to processed weak physiological features, weak behavioral features, and environmental auxiliary data, and outputting cross-modal fusion features, includes: The system analyzes dynamic labels containing pain stages and pain types, and determines the baseline weight proportion of physiological, behavioral, and environmental characteristics under different labels based on the preset mapping rules between labels and modal weight priorities. Physiological and behavioral weak features are mapped to the same dimension through a fully connected layer, and environmental auxiliary data are concatenated into a vector of the same dimension after encoding and normalization. Initial attention weights are assigned to the three modalities according to preset rules, and the weights are adjusted based on cross-modal similarity. The final time-varying weights are obtained by normalization. The features of the three modalities are weighted and summed with the corresponding time-varying weights, and then enhanced by the Transformer feedforward network to output cross-modal fusion features.

[0015] According to another embodiment of this application, a prediction system is provided for implementing the pain threshold prediction method based on multimodal data fusion of the above embodiments. The prediction system includes the following modules: The multimodal data processing module is used to perform real-time analysis of preprocessed multimodal data using a lightweight LSTM network, adaptively adjust the sampling parameters of each sensor in the multimodal sensor group, and label the collected multimodal data with dynamic pain tags. The feature extraction module is used to construct a cross-modal weak feature enhancement and fusion network to process the collected multimodal data and extract physiological weak features and behavioral weak features. The feature fusion module is used to introduce the time-varying attention Transformer model, and uses the dynamic pain label to dynamically assign weights to the processed physiological weak features, behavioral weak features and environmental auxiliary data, and output cross-modal fusion features. The threshold prediction module is used to construct a two-branch pain threshold prediction model, including a supervised prediction branch and an unsupervised prediction branch. The supervised prediction branch is used to learn the explicit correlation between cross-modal fusion features and pain thresholds. The unsupervised prediction branch uses dynamic pain labels as supervision signals to learn the consistency patterns of weak physiological features and weak behavioral features. A pain feature meta-library is introduced, and the parameters of the two-branch model are calibrated online using a meta-learning algorithm. The module outputs the real-time pain threshold and short-term pain threshold change trend of the target object.

[0016] Compared with the prior art, the beneficial effects of the pain threshold prediction method and system based on multimodal data fusion in this application are: This invention uses a lightweight LSTM network to perform real-time analysis of multimodal data, which can adaptively adjust sensor sampling parameters and label the data with dynamic labels that include pain stage and pain type. Combined with a time-varying attention Transformer model, the weight allocation ratio of physiological weak features, behavioral weak features and environmental auxiliary data is adjusted in real time based on dynamic labels, so that the cross-modal fusion features always accurately match the current pain state, realizing the real-time output of pain threshold and effectively avoiding the prediction lag problem caused by rigid fusion mechanism. This invention constructs a cross-modal weak feature enhancement and fusion network. It improves the signal-to-noise ratio of physiological weak features through wavelet packet transform combined with a differentiated strategy of attention gating module. It removes spurious features from behavioral weak features by using optical flow method and environmental auxiliary data, thereby achieving accurate extraction and enhancement of weak features. Then, through collaborative learning of a dual-branch prediction model, the supervised branch focuses on the explicit correlation between cross-modal fusion features and pain threshold, while the unsupervised branch uses dynamic labels as supervision signals to mine the consistency pattern of physiological-behavioral weak features, forming a dual learning mechanism of explicit correlation and implicit pattern, which can accurately capture threshold changes of mild pain and occult pain. The unsupervised prediction branch of this invention does not rely on subjective labels. It learns from contrastive samples driven by dynamic labels to discover fixed association patterns between weak physiological features and weak behavioral features under specific pain states. Combined with a pain feature meta-library and meta-learning algorithm, it can quickly complete online calibration of model parameters on short-cycle multimodal data of target objects. In summary, the embodiments of this application significantly improve the clinical operability of pain assessment, reduce ineffective interventions caused by inaccurate assessments, and have broad technical promotion value. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] In the attached diagram: Figure 1 This is a flowchart illustrating the implementation of the pain threshold prediction method based on multimodal data fusion in an embodiment of the present invention. Figure 2 This is a sub-flowchart of the pain threshold prediction method based on multimodal data fusion according to an embodiment of this application; Figure 3 This is a structural block diagram of the pain threshold prediction system based on multimodal data fusion according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] The technical solutions in the exemplary embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described exemplary embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Currently, existing multimodal pain threshold prediction methods suffer from technical problems such as prediction lag and insufficient accuracy due to the mismatch between the dynamic temporal variability of pain signals and the static fusion mechanism, the suppression of weak cross-modal features by explicit features, and the lack of adaptation to populations without subjective expression.

[0022] To address the aforementioned issues, this invention provides a pain threshold prediction method based on multimodal data fusion. This method utilizes a lightweight LSTM network for real-time analysis of multimodal data, adaptively adjusting sensor sampling parameters and labeling the data with dynamic tags including pain stage and pain type. Combined with a time-varying attention Transformer model, the weight allocation ratio of physiological weak features, behavioral weak features, and environmental auxiliary data is adjusted in real-time based on the dynamic tags, ensuring that the cross-modal fusion features always accurately match the current pain state. This achieves real-time output of the pain threshold and effectively avoids prediction lag caused by rigid fusion mechanisms. Furthermore, this invention constructs a cross-modal weak feature enhancement fusion network, using wavelet packet transform combined with a differentiated strategy of the attention gating module to enhance physiological weak features. The signal-to-noise ratio is improved by using optical flow and environmental auxiliary data to remove spurious features from weak behavioral features, enabling accurate extraction and enhancement of weak features. Then, through collaborative learning of a dual-branch prediction model, the supervised branch focuses on the explicit correlation between cross-modal fusion features and pain thresholds, while the unsupervised branch uses dynamic labels as supervision signals to mine consistent patterns of physiological-behavioral weak features, forming a dual learning mechanism of explicit correlation and implicit patterns. This can accurately capture threshold changes in mild and latent pain. The unsupervised prediction branch of this invention does not rely on subjective labels; through dynamic label-driven comparative sample learning, it mines fixed correlation patterns between physiological and behavioral weak features under specific pain states. Combining a pain feature meta-library and meta-learning algorithm, online calibration of model parameters can be quickly completed using short-cycle multimodal data of target objects.

[0023] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart illustrating the implementation of the pain threshold prediction method based on multimodal data fusion in an embodiment of the present invention. like Figure 1 As shown in one embodiment of this application, the pain threshold prediction method based on multimodal data fusion includes the following steps: Step S101: Use a lightweight LSTM network to perform real-time analysis on the preprocessed multimodal data, adaptively adjust the sampling parameters of each sensor in the multimodal sensor group, and label the collected multimodal data with dynamic pain tags. In step S101 of this application embodiment, before performing real-time analysis of multimodal data, preprocessing of physiological modality data and behavioral modality data is included; The physiological modality data includes electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, and electrodermal (ED) signals. The behavioral modality data includes facial expression videos and body posture images; In the preprocessing of physiological modality data, electroencephalogram (EEG), electrocardiogram (ECG), and electrodermal (EDS) signals were acquired. After filtering to remove power frequency noise and using bandpass filtering to retain effective frequency bands, the data were segmented into feature matrices according to a preset time window, and Z-score normalization was used to eliminate dimensional differences. The resulting set of physiological modality feature matrices... , is represented as: In the formula, T represents the number of time windows, and each time window is preferably 1 second; In behavioral modality data preprocessing, facial expression videos and body posture images are acquired. Video frames are converted to grayscale images and cropped to facial and limb regions of interest (ROIs). Image pyramid downsampling is used to reduce resolution and computational load. The preprocessed behavioral modality feature matrix set... , is represented as: ; Furthermore, in this embodiment of the application, the step of performing real-time analysis of the preprocessed multimodal data using a lightweight LSTM network includes: Construct a dual-branch lightweight LSTM main structure, including a physiological branch LSTM and a behavioral branch LSTM; In the physiological branch of the LSTM, depthwise separable convolutional layers are used to first compress the dimensionality of the physiological modality feature matrix and extract local features, and then input into a lightweight LSTM unit to output a physiological modality pain-related feature vector, represented as follows: Specifically, the lightweight LSTM unit for the physiological branch receives matrices from the outputs of depthwise separable convolutions. The hidden layers of the LSTM unit employ a gating mechanism, with the forget gate sharing some parameters with the input gate. The dropout probability is set to 0.2. After temporal modeling using LSTM, the output is a physiological modality pain-related feature vector. The larger the value of a vector element, the stronger the correlation between the corresponding feature and pain, such as in electroencephalography (EEG). When the element value corresponding to the band is not less than 0.6, the improvement may be sharp and painful; In the behavioral branch LSTM, a spatiotemporal attention module is introduced to first perform spatial attention weight allocation and temporal attention filtering on the behavioral modality grayscale temporal image sequence. Then, the attention-weighted temporal image features are input into a lightweight LSTM unit, and the output behavioral modality pain-related feature vector is expressed as follows: ; Specifically, for the spatial attention submodule of the spatiotemporal attention module, the grayscale facial frame at time t is input, the number of channels is compressed through a 1×1 convolution, and then a 100×100 spatial attention weight map is generated through a sigmoid activation function. The weights range from 0 to 1. For example, if the weight of the orbicularis oculi and orbicularis oris muscles is ≥0.7, they are considered pain-related areas. The resulting weighted map... Multiply the original frame pixel by pixel to obtain the spatially weighted facial feature frame; In addition, for the temporal attention submodule of the spatiotemporal attention module, spatially weighted facial feature frames are input, the pain feature response values ​​of each frame are calculated, and then the temporal attention weights are generated through softmax. After weighted summation, the spatiotemporal fused facial feature frames are obtained; In this embodiment, the LSTM unit in the behavior branch LSTM concatenates the spatiotemporally fused facial feature frames with the limb pose feature vector to obtain the behavior modality pain association feature vector. .

[0025] Furthermore, in this embodiment, the feature vectors output by the physiological branch and the behavioral branch are concatenated into a fused feature, represented as: The cross-modal attention weights are calculated using a multilayer perceptron (MLP) and are expressed as follows: The spliced ​​features are then weighted and optimized, and the final weighted fused features are represented as follows: This process highlights the consistency of pain association information between physiological and behavioral characteristics. Finally, the features fused through cross-modal attention are input into a Softmax classifier, which outputs the recognition probabilities of pain stage and pain type. The category corresponding to the maximum probability is taken as the final recognition result.

[0026] This invention uses a lightweight LSTM network to perform real-time analysis of multimodal data, which can adaptively adjust sensor sampling parameters and label the data with dynamic labels that include pain stage and pain type. Combined with a time-varying attention Transformer model, the weight allocation ratio of physiological weak features, behavioral weak features and environmental auxiliary data is adjusted in real time based on dynamic labels, so that the cross-modal fusion features always accurately match the current pain state, realizing the real-time output of pain threshold and effectively avoiding the prediction lag problem caused by rigid fusion mechanism.

[0027] Please continue to refer to Figure 1 In this application embodiment, a pain threshold prediction method based on multimodal data fusion is provided, which further includes the following steps: Step S102: Construct a cross-modal weak feature enhancement and fusion network to process the collected multimodal data and extract physiological weak features and behavioral weak features; Furthermore, in this embodiment of the application, the step of extracting weak physiological features includes: A wavelet packet transform combined with an attention gating module is employed to enhance the signal-to-noise ratio of weak physiological features in physiological modality data based on a differentiated strategy. Furthermore, noise suppression parameters are dynamically adjusted in conjunction with environmental auxiliary data, which includes the noise intensity of the scene in which the target object is located. When the environmental auxiliary data shows a noise intensity threshold in decibels, the decomposition level of the wavelet packet transform is increased, and the noise suppression threshold of the attention gating module is lowered to strengthen the screening of weak features. The weak physiological features include EEG gamma band fluctuations and changes in skin conductance amplitude.

[0028] Furthermore, in the step of extracting weak behavioral features in this embodiment, the method includes: extracting weak behavioral features of facial expression videos from behavioral modality data using optical flow, and removing false features by combining environmental auxiliary data. The environmental auxiliary data includes light intensity and shooting angle. When the environmental auxiliary data shows a low-light scene, brightness compensation is applied to the weak behavioral features extracted by optical flow, and the motion contours of the orbicularis oculi and orbicularis oris muscles are enhanced by an edge enhancement algorithm. If the environmental auxiliary data shows a deviation in shooting angle, the facial image is normalized to a frontal view using a pose correction algorithm, and then subpixel-level motion features are extracted. The weak behavioral features include subpixel-level motion features of the orbicularis oculi and orbicularis oris muscles.

[0029] This application constructs a cross-modal weak feature enhancement and fusion network. By combining wavelet packet transform with a differentiated strategy of attention gating module, the signal-to-noise ratio of physiological weak features is improved. By using optical flow method and environmental auxiliary data, pseudo-features in behavioral weak features are removed, achieving accurate extraction and enhancement of weak features. Then, through collaborative learning of a dual-branch prediction model, the supervised branch focuses on the explicit correlation between cross-modal fusion features and pain threshold, while the unsupervised branch uses dynamic labels as supervision signals to mine the consistency pattern of physiological-behavioral weak features, forming a dual learning mechanism of explicit correlation and implicit pattern, which can accurately capture the threshold changes of mild pain and occult pain.

[0030] Please continue to refer to Figure 1 In this application embodiment, a pain threshold prediction method based on multimodal data fusion is provided, which further includes the following steps: Step S103: Introduce the time-varying attention Transformer model, utilize the dynamic pain label, dynamically assign weights to the processed weak physiological features, weak behavioral features, and environmental auxiliary data, and output cross-modal fusion features; Specifically, step S103 in this application embodiment includes: The system analyzes dynamic labels containing pain stages and pain types, and determines the baseline weight proportion of physiological, behavioral, and environmental characteristics under different labels based on the preset mapping rules between labels and modal weight priorities. Among them, the initial weights of physiological characteristics, behavioral characteristics and environmental characteristics are determined based on the dynamic pain label, the characteristics of the target population and the current environmental scene, and the sum of the three initial weights is 1; Physiological and behavioral weak features are mapped to the same dimension through a fully connected layer, and environmental auxiliary data are concatenated into a vector of the same dimension after encoding and normalization. Initial attention weights are assigned to the three modalities according to preset rules, and the weights are adjusted based on cross-modal similarity. The final time-varying weights are obtained by normalization. The features of the three modalities are weighted and summed with the corresponding time-varying weights, and then enhanced by the Transformer feedforward network to output cross-modal fusion features.

[0031] In this embodiment, the cross-modal similarity is calculated by comparing the similarity between physiological features and behavioral features, the similarity between behavioral features and environmental features, and the similarity between physiological features and environmental features. When adjusting the weights based on cross-modal similarity, if the average similarity between a certain modality and the other two modalities is not less than 0.6, the weight of that modality is increased based on the initial weight. If the average similarity is less than 0.3, the weight of that modality is decreased. As a preferred embodiment, in one implementation of this application, the weights are dynamically optimized through a two-factor adjustment. The two factors include not only cross-modal similarity but also real-time prediction error. Specifically, in the real-time prediction error, the temporary fusion features under the current weights are input into the error detection module of the dual-branch model, and the prediction error is calculated with the real-time clinical monitoring data. If the error rate exceeds the threshold, the modality with the largest error contribution is located through modal ablation experiments, and the weight of that modality is lowered, while the weight of the modality with the smallest error contribution is raised. Finally, in this embodiment, the adjusted weights are normalized using the Softmax function to obtain the final time-varying attention weights. The element-wise weighted sum of the three types of features is calculated according to the weights to obtain the preliminary fused features. Then, the preliminary fused features are input into a Transformer feedforward network, which contains two fully connected layers. The second layer introduces a clinical pain feature enhancement layer, which introduces enhancement coefficients to the feature values ​​of the pain strong correlation dimension based on clinical data mining. Furthermore, the Pearson correlation coefficient R between the enhanced fused features and the corresponding label templates in the pain label template feature library is calculated until R reaches a set threshold.

[0032] Step S104: Construct a two-branch pain threshold prediction model, including a supervised prediction branch and an unsupervised prediction branch; wherein, the supervised prediction branch is used to learn the explicit correlation between cross-modal fusion features and pain threshold; the unsupervised prediction branch uses pain dynamic labels as supervision signals to learn the consistency pattern of physiological weak features and behavioral weak features; introduce a pain feature meta-library, and combine meta-learning algorithms to calibrate the parameters of the two-branch model online, and output the real-time pain threshold and short-term pain threshold change trend of the target object.

[0033] In this invention, the unsupervised prediction branch does not rely on subjective labels. It learns from contrastive samples driven by dynamic labels to discover fixed association patterns between weak physiological and behavioral features under specific pain states. Combined with a pain feature meta-library and meta-learning algorithm, it can quickly complete online calibration of model parameters using multimodal data of target objects in a short period of time.

[0034] For further details, please refer to Figure 2 In step S104 of this application, the unsupervised prediction branch uses pain dynamic labels as supervision signals to learn the consistency pattern between physiological weak features and behavioral weak features. This involves preprocessing the physiological and behavioral weak features, including spatiotemporal alignment processing; it also includes: Step S201: Construct a comparative sample based on dynamic pain labels; The comparison samples include positive sample pairs and negative sample pairs; Specifically: In the construction of positive sample pairs, for the spatiotemporally aligned weak features under the same target object and the same label, feature fragments are randomly selected to form positive sample pairs, and the positive sample pairs have feature consistency under the same pain state. The construction of negative sample pairs includes cross-label negative samples and misaligned negative samples with the same label; For cross-label negative samples, weak feature fragments under different dynamic labels but with the same target object are selected to form pairs; For negative samples with misaligned labels, weak feature segments with the same dynamic label and a time interval exceeding a threshold are selected to form pairs.

[0035] Step S202: Construct a dual-modal lightweight encoder, including a physiological feature encoder and a behavioral feature encoder, mapping the preprocessed weak physiological features and weak behavioral features to feature embedding vectors respectively; calculate the physiological feature embedding vector. With behavioral feature embedding vector The cosine similarity is used to output a consistency-enhanced bimodal embedding vector through the attention weight matrix. ; In step S202 of this embodiment, both encoders adopt a structure of 1D convolution, batch normalization, ReLU activation and global average pooling. In addition, in the step of outputting the consistency-enhanced bimodal embedding vector through the attention weight matrix, the feature matrix with similarity of not less than 0.6 is strengthened by the attention weight matrix, and the features with similarity of less than 0.3 are weakened, so as to output the consistency-enhanced bimodal embedding vector. Step S203: Using the triplet loss function, with the consistency-enhanced bimodal embedding vector of positive sample pairs as the anchor point, the embedding vector of positive sample pairs with the same label as the positive example, and the embedding vector of negative sample pairs as the negative example, the bimodal encoder parameters are optimized by minimizing the distance between the anchor point and the positive example and maximizing the distance between the anchor point and the negative example, as expressed as: In the formula, Indicates the interval threshold. Represents the L2 norm. Represents the anchor vector. Represents the positive example vector. Represents a negative instance vector. Represents the triplet loss function; Step S204: Calculate the variance of the embedding vector after multiple forward propagations of the same weak feature segment by the encoder, and introduce a stability loss function to enable the model to learn the fixed association pattern between physiological weak features and behavioral weak features under a specific pain label. In this embodiment of the application, the stability loss function Represented as: In the formula, M represents the embedding vector. The dimension; The stability loss function is measured by the variance of the embedding vector; K represents the number of times the same weak feature segment is forward-propagated through the encoder. This represents the enhanced embedding vector after the k-th forward propagation. The value taken in the m-th dimension; This represents the mean of the embedding vectors in the K forward propagation operations along the m-th dimension.

[0036] Furthermore, this application embodiment introduces a pain feature meta-library and combines it with a meta-learning algorithm to perform online calibration of the parameters of the bi-branch model, including: A multi-dimensional pain feature meta-library is constructed, which is divided into multiple independent meta-tasks. Each meta-task stores the physiological weak feature statistics and behavioral weak feature statistics corresponding to that task, as well as the initial parameters of the dual-branch model adapted to the current task. The initial parameters are expressed as: In the formula, This represents the support set for the meta-task. This represents the learning rate of the original task; For the loss function with respect to the general parameters The gradient; A model-independent meta-learning algorithm is employed to meta-train the bi-branch model. Specifically, N independent meta-tasks are randomly sampled from a pain feature meta-database. Each meta-task includes statistical information on physiological and behavioral weak features, as well as initial parameters. For each meta-task, a support set and a query set are partitioned. The support set data is used to train the general parameters of the bi-branch model. Perform a gradient update and compute the query set at... The loss under the general parameters Perform meta-level optimization, repeat the above training steps until the number of iterations is no less than 100, and output meta-trained general parameters with fast adaptation capabilities. For the target subjects to be evaluated, multimodal data were collected, and weak physiological and behavioral features were extracted. At the same time, the population characteristics and initial pain manifestations of the target subjects were recorded. The meta-task that best matches the target object is retrieved from the pain feature meta-database. The initial parameters of the bi-branch model stored in this meta-task are then used as the initial calibration parameters. Using the collected multimodal data of the target object, the initial calibration parameters are updated with a gradient to obtain calibration parameters adapted to the individual characteristics of the target object. , is represented as: In the formula, This indicates a new weak physiological characteristic of the target object; This represents a new weak behavioral feature of the target object. Indicates the initial pain threshold label of the target object; This represents the model parameters at the current m-th iteration; Indicates the online learning rate; Represents the loss function Regarding the current parameters The gradient.

[0037] Furthermore, in this embodiment of the application, the calibrated parameters are... As input to the two-branch model, the output is the real-time threshold. and short-term pain threshold change trend ,in, In the formula, Indicates model parameters, Represents time-series feature sequences, including physiological weak feature vectors. and behavioral weak feature vectors .

[0038] Furthermore, the method also includes steps to optimize the real-time pain threshold and short-term pain threshold change trend of the target subjects. On the one hand, the real-time pain threshold is compared with clinical real-time monitoring data, and the modal weight allocation rules of the time-varying attention Transformer are dynamically adjusted. The clinical real-time monitoring data includes the observation records of medical staff and the limb response data of the target subjects. On the other hand, the pain intervention effect data of the target subjects is accumulated and screened, the pain feature library and the training parameters of the bi-branch model are updated, and the accuracy of pain threshold prediction is continuously optimized. The intervention effect data includes the time and dosage of analgesic drug administration and the change of pain threshold after administration.

[0039] In summary, the embodiments of this application significantly improve the clinical operability of pain assessment, reduce ineffective interventions caused by inaccurate assessments, and have broad technical promotion value.

[0040] Please refer to Figure 3 According to another embodiment of this application, a prediction system is provided for implementing the pain threshold prediction method based on multimodal data fusion of the above embodiments. The prediction system includes the following modules: The multimodal data processing module 301 is used to perform real-time analysis of preprocessed multimodal data using a lightweight LSTM network, adaptively adjust the sampling parameters of each sensor in the multimodal sensor group, and label the collected multimodal data with dynamic pain tags. Feature extraction module 302 is used to construct a cross-modal weak feature enhancement and fusion network to process the collected multimodal data and extract physiological weak features and behavioral weak features; The feature fusion module 303 is used to introduce a time-varying attention Transformer model, which uses pain dynamic labels to dynamically assign weights to the processed physiological weak features, behavioral weak features and environmental auxiliary data, and output cross-modal fusion features. The threshold prediction module 304 is used to construct a two-branch pain threshold prediction model, including a supervised prediction branch and an unsupervised prediction branch. The supervised prediction branch is used to learn the explicit correlation between cross-modal fusion features and pain threshold. The unsupervised prediction branch uses the dynamic pain label as a supervision signal to learn the consistency pattern between weak physiological features and weak behavioral features. A pain feature meta-library is introduced, and the parameters of the two-branch model are calibrated online in combination with a meta-learning algorithm. The module outputs the real-time pain threshold and short-term pain threshold change trend of the target object.

[0041] like Figure 4 As shown, in this embodiment of the invention, a computer device is provided. At the hardware level, the terminal includes a processor, and optionally also includes an internal bus, a network interface, and a memory.

[0042] The memory may include main memory, such as high-speed random access memory, or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its operations.

[0043] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA bus, PCI bus, or EISA bus, etc.

[0044] The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0045] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0046] The processor reads the corresponding computer program from non-volatile memory into memory and then runs it, forming a pain threshold prediction system based on multimodal data fusion at the logical level.

[0047] The processor executes the program stored in the memory, and specifically performs the pain threshold prediction method based on multimodal data fusion provided in the above embodiments.

[0048] The above is as stated in this application. Figure 1 The pain threshold prediction method based on multimodal data fusion disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed through integrated logic circuits in the processor's hardware or through software instructions. The processor can be a general-purpose processor, including a central processing unit (CPU), network processor, etc.; it can also be a digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0049] The steps of the method disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor.

[0050] The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0051] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The embodiment shown is a pain threshold prediction method based on multimodal data fusion.

[0052] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0053] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A pain threshold prediction method based on multimodal data fusion, characterized in that, include: A lightweight LSTM network is used to perform real-time analysis on the preprocessed multimodal data, adaptively adjust the sampling parameters of each sensor in the multimodal sensor group, and label the collected multimodal data with dynamic pain tags. A cross-modal weak feature enhancement and fusion network is constructed to process the collected multimodal data and extract physiological weak features and behavioral weak features. A time-varying attention Transformer model is introduced, and dynamic pain labels are used to dynamically assign weights to the processed weak physiological features, weak behavioral features, and environmental auxiliary data, and output cross-modal fusion features. A bi-branch pain threshold prediction model is constructed, comprising a supervised prediction branch and an unsupervised prediction branch. The supervised prediction branch learns the explicit association between cross-modal fusion features and the pain threshold; the unsupervised prediction branch uses dynamic pain labels as supervision signals to learn the consistency patterns between weak physiological and behavioral features. A pain feature metadata library is used, and a meta-learning algorithm is combined to calibrate the parameters of the dual-branch model online, outputting the real-time pain threshold and short-term pain threshold change trend of the target object.

2. The pain threshold prediction method based on multimodal data fusion according to claim 1, characterized in that, It also includes steps to optimize the real-time pain threshold and short-term pain threshold change trends of the target object. The optimization methods also include: The real-time pain threshold is compared with clinical real-time monitoring data to dynamically adjust the modal weight allocation rules of the time-varying attention Transformer; Accumulate and filter pain intervention effect data for target subjects, and update the pain feature library and training parameters of the bi-branch model.

3. The pain threshold prediction method based on multimodal data fusion according to claim 2, characterized in that, The steps for real-time analysis of preprocessed multimodal data using a lightweight LSTM network include: Construct a dual-branch lightweight LSTM main structure, including a physiological branch LSTM and a behavioral branch LSTM; In the physiological branch of the LSTM, depthwise separable convolutional layers are used to first compress the dimensionality of the physiological modality feature matrix and extract local features, and then input it into a lightweight LSTM unit to output a physiological modality pain-related feature vector. , is represented as: ; Let D represent the set of real numbers, and let D represent the dimension. In the behavior branch LSTM, a spatiotemporal attention module is introduced to first perform spatial attention weight allocation and temporal attention filtering on the grayscale temporal image sequence of the behavior modality. Then, the attention-weighted temporal image features are input into a lightweight LSTM unit, and the behavior modality pain association feature vector is output. , is represented as: ; The feature vectors output from the physiological branch and the behavioral branch are concatenated to form a fused feature. , is represented as: Cross-modal attention weights are calculated using a multilayer perceptron (MLP). , is represented as: The splicing features are weighted and optimized, and finally the weighted fused features are obtained. Represented as: ; The features fused by cross-modal attention are input into the Softmax classifier, which outputs the recognition probability of pain stage and pain type. The category corresponding to the maximum probability is taken as the final recognition result.

4. The pain threshold prediction method based on multimodal data fusion according to claim 3, characterized in that, The steps for extracting weak physiological features include: By employing wavelet packet transform combined with an attention gating module, a differentiated strategy is used to improve the signal-to-noise ratio of weak physiological features in physiological modality data, and noise suppression parameters are dynamically adjusted in conjunction with environmental auxiliary data.

5. The pain threshold prediction method based on multimodal data fusion according to claim 4, characterized in that, The steps for extracting weak behavioral features include: using optical flow to extract weak behavioral features from facial expression videos in behavioral modality data, and combining environmental auxiliary data to remove false features.

6. The pain threshold prediction method based on multimodal data fusion according to claim 5, characterized in that, The unsupervised prediction branch uses dynamic pain labels as supervision signals and learns the steps of consistency patterns between physiological and behavioral weak features, including: Construct a comparative sample based on dynamic pain labels; A dual-modal lightweight encoder is constructed, comprising a physiological feature encoder and a behavioral feature encoder, which map the preprocessed weak physiological and behavioral features into feature embedding vectors, respectively; the physiological feature embedding vectors are then calculated. With behavioral feature embedding vector The cosine similarity is used to output a consistency-enhanced bimodal embedding vector through the attention weight matrix. ; The triplet loss function is adopted, using the consistency-enhanced bimodal embedding vector of positive sample pairs as the anchor point, the embedding vector of positive sample pairs with the same label as the positive example, and the embedding vector of negative sample pairs as the negative example. The bimodal encoder parameters are optimized by minimizing the distance between the anchor point and the positive example and maximizing the distance between the anchor point and the negative example, as expressed in: In the formula, Indicates the interval threshold. Describing the L2 norm, Represents the anchor vector. Represents the positive example vector. Represents a negative instance vector. Represents the triplet loss function; The variance of the embedding vector after multiple forward propagations of the same weak feature segment by the encoder is calculated, and a stability loss function is introduced to enable the model to learn a fixed association pattern between physiological weak features and behavioral weak features under a specific pain label. Among them, the stability loss function Represented as: In the formula, M represents the embedding vector. The dimension; The stability loss function is measured by the variance of the embedding vector; K represents the number of times the same weak feature segment is forward-propagated through the encoder. This represents the enhanced embedding vector after the k-th forward propagation. The value taken in the m-th dimension; It represents the mean of the embedding vectors in the K forward propagation along the m-th dimension.

7. The pain threshold prediction method based on multimodal data fusion according to claim 6, characterized in that, In the step of constructing a comparison sample based on dynamic pain labels, the comparison sample includes positive sample pairs and negative sample pairs; In the construction of positive sample pairs, for weak spatiotemporal alignment features under the same target object and the same label, feature fragments are randomly selected to form positive sample pairs. Positive sample pairs have feature consistency under the same pain state. The construction of negative sample pairs includes cross-label negative samples and misaligned negative samples with the same label; For cross-label negative samples, weak feature fragments under different dynamic labels but with the same target object are selected to form pairs; For negative samples with misaligned labels, weak feature segments with the same dynamic label and a time interval exceeding a threshold are selected to form pairs.

8. The pain threshold prediction method based on multimodal data fusion according to claim 2, characterized in that, The steps of introducing a pain feature meta-library and combining it with a meta-learning algorithm to perform online calibration of the parameters of the bi-branch model include: A multi-dimensional pain feature meta-library is constructed, which is divided into multiple independent meta-tasks. Each meta-task stores the physiological weak feature statistics and behavioral weak feature statistics corresponding to that task, as well as the initial parameters of the dual-branch model adapted to the current task. The initial parameters are expressed as: In the formula, This represents the support set for the meta-task. This represents the learning rate of the original task; For the loss function with respect to the general parameters The gradient; A model-independent meta-learning algorithm is used to perform meta-training on the dual-branch model; For the target subjects to be evaluated, multimodal data were collected, and weak physiological and behavioral features were extracted. At the same time, the population characteristics and initial pain manifestations of the target subjects were recorded. The meta-task that best matches the target object is retrieved from the pain feature meta-database. The initial parameters of the bi-branch model stored in this meta-task are then used as the initial calibration parameters. Using the collected multimodal data of the target object, the initial calibration parameters are updated with a gradient to obtain calibration parameters adapted to the individual characteristics of the target object. , is represented as: In the formula, This indicates a new weak physiological characteristic of the target object; This represents a new weak behavioral feature of the target object. Indicates the initial pain threshold label of the target object; This represents the model parameters at the current m-th iteration; Indicates the online learning rate; Represents the loss function Regarding the current parameters The gradient.

9. The pain threshold prediction method based on multimodal data fusion according to claim 8, characterized in that, The step of using pain dynamic tags to dynamically assign weights to processed physiological weak features, behavioral weak features, and environmental auxiliary data, and outputting cross-modal fusion features includes: assigning initial attention weights to the three modalities according to preset rules, and adjusting the weights based on cross-modal similarity to obtain the final time-varying weights; weighting and summing the three modal features with the corresponding time-varying weights, and strengthening the output of cross-modal fusion features through a Transformer feedforward network.

10. A prediction system for implementing the pain threshold prediction method based on multimodal data fusion as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The multimodal data processing module is used to perform real-time analysis of preprocessed multimodal data using a lightweight LSTM network, adaptively adjust the sampling parameters of each sensor in the multimodal sensor group, and label the collected multimodal data with dynamic pain tags. The feature extraction module is used to construct a cross-modal weak feature enhancement and fusion network to process the collected multimodal data and extract physiological weak features and behavioral weak features. The feature fusion module is used to introduce the time-varying attention Transformer model, and uses the dynamic pain label to dynamically assign weights to the processed physiological weak features, behavioral weak features and environmental auxiliary data, and output cross-modal fusion features. The threshold prediction module is used to construct a two-branch pain threshold prediction model, including a supervised prediction branch and an unsupervised prediction branch. The supervised prediction branch learns the explicit association between cross-modal fusion features and pain thresholds; the unsupervised prediction branch uses dynamic pain labels as supervision signals to learn the consistency patterns between weak physiological and behavioral features. A pain feature metadata library is used, and a meta-learning algorithm is combined to calibrate the parameters of the dual-branch model online, outputting the real-time pain threshold and short-term pain threshold change trend of the target object.

Images