Facial pain assessment method based on comparative learning
Through comparative learning and adaptive weight update methods, multimodal facial information is extracted, and the problem of incomplete pain level distinction in the prior art is solved, achieving more accurate pain assessment.
Patent Information
- Application Number
- CN202410115841.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-26
- Publication Date
- 2025-07-29
AI Technical Summary
The existing facial pain assessment methods fail to effectively distinguish the differences in facial features of different pain levels, resulting in insufficient comprehensive pain recognition.
The comparative learning idea is adopted to extract multimodal information (geometric information, appearance information and timing information), and the comparison learning loss and mean square error loss are jointly optimized through the adaptive weight update method to conduct facial pain assessment.
It improves the accuracy of pain assessment, can better distinguish the differences in facial features of different pain levels, and improves the effect of pain recognition.
Smart Images
Figure BDA0004685015280000041 
Figure BDA0004685015280000051 
Figure HDA0004685015310000011
Abstract
Description
Technical Field
[0001] The present invention relates to the problem of pain intensity assessment in the field of deep learning, and particularly to a facial pain assessment method based on contrast learning. Background Art
[0002] Pain is a painful experience related to actual or potential tissue damage that encompasses sensory, emotional, cognitive, and social factors. At the same time, it is also one of the most common clinical symptoms and has great potential and medical significance in current healthcare and rehabilitation medicine. Currently, pain detection in clinical practice mainly relies on patients' subjective reports, but it lacks a unified judgment standard. Therefore, the technology of automatic pain recognition has become a hot research field. In the past few decades, researchers have studied pain recognition from aspects of physiological signals and facial images. Among them, facial pain recognition has evolved from early binary classification recognition of static images, pain played by subjects, and the presence or absence of pain to the current assessment of dynamic videos, spontaneous or externally stimulated pain, and the degree of pain.
[0003] Facial pain recognition methods are mainly divided into two types: using traditional handcrafted features and deep learning methods. Traditional handcrafted feature extraction methods are difficult to obtain deep spatial information and temporal information. Therefore, nowadays, people tend to use deep learning methods for facial pain recognition. Existing methods mostly focus on extracting facial pain features of different modalities and scales, without considering the differences in facial features corresponding to different degrees of pain and are unable to effectively distinguish pain levels. Therefore, this patent uses the idea of contrast learning for pain assessment. First, in order to obtain more comprehensive pain features, geometric information, facial appearance information, and temporal information composed of facial key points are respectively extracted to obtain multi-modal information of the samples. Secondly, the multi-modal features obtained above are used as positive samples for contrast learning with other samples to fully learn the facial differences under different pain levels. Finally, the contrast learning loss and mean square error loss are jointly optimized through an adaptive weight update method, which is beneficial to measuring the feature differences between the pain prediction value and the true value. Summary of the Invention
[0004] The purpose of the present invention is to provide a facial pain assessment method based on contrast learning, which introduces the idea of contrast learning to distinguish the facial feature differences corresponding to different pain levels, and uses an adaptive weight update method to jointly optimize the contrast learning loss and mean square error loss, which is beneficial to using the difference information between the predicted value and the true label and improves the effect of pain assessment.
[0005] For the convenience of description, the following concepts are first introduced:
[0006] Contrastive Learning: A learning technique whose core idea is to learn the representation of data by comparing the similarities and differences between different samples.
[0007] S3D-G Network (Separable 3D CNN with a Spatio-Temporal Gating Mechanism): A 3D convolutional network that achieves a balance between computational complexity and accuracy by replacing some of the 3D convolutions with low-cost 2D convolutions.
[0008] ResNet Network (ResNet): A convolutional neural network that deepens the network depth by introducing residual connections and solves the problems of gradient vanishing and gradient explosion at the same time.
[0009] The present invention specifically adopts the following technical solutions:
[0010] A facial pain assessment method based on contrastive learning, characterized in that:
[0011] a. Extract multi-modal information in the image sequence, including geometric information composed of facial key points, facial appearance information, and temporal information;
[0012] b. Use the features of the above three modalities as positive sample pairs for contrastive learning;
[0013] c. Use an adaptive weight update method to jointly optimize the contrastive learning loss and the traditional mean square error loss;
[0014] This method mainly includes the following steps:
[0015] (1) Data preprocessing: First, crop and align the face images in the dataset; then splice 15 frames with the same pain level as the current frame with the current frame to form a 16-frame image sequence; in addition, use the distance normalization method to process the facial key point information to obtain one-dimensional input data;
[0016] (2) Facial key point feature extraction: Use a one-dimensional convolutional neural network to process the one-dimensional input vector obtained in step (1) to obtain facial geometric features related to pain;
[0017] (3) Facial appearance feature extraction: Use the ResNet network to extract the facial appearance features in the static image cropped and aligned in step (1) to model the local pain information;
[0018] (4) Temporal feature extraction: Use the S3D-G network to extract the temporal information in the video sequence obtained in step (1) to capture more dynamic facial pain features;
[0019] (5) Pain feature contrast learning: Use the different modality information obtained in steps (2), (3), and (4) as positive sample pairs to perform contrast learning with other samples in the dataset, and learn the differences in facial features corresponding to different pain levels;
[0020] (6) Facial pain assessment: Concatenate the facial key points, appearance, and temporal features extracted in steps (2), (3), and (4), and send the concatenated multi-modal features into a fully connected layer for pain level prediction;
[0021] (7) Model training: In an end-to-end manner, use an adaptive weight update method to jointly optimize the contrast learning loss and the mean square error loss to achieve model training.
[0022] The beneficial effects of the present invention are:
[0023] (1) For a video sequence, use one-dimensional, two-dimensional, and three-dimensional convolutional neural networks to extract geometric information, facial appearance information, and temporal information composed of facial key points respectively, obtain facial pain features in different modalities, and form a complete feature set;
[0024] (2) Use the features of the above three modalities as positive samples to perform contrast learning with other samples, and fully learn the differences in facial features corresponding to different pain levels.
[0025] (3) Use an adaptive weight update method to jointly optimize two loss functions, namely the above-mentioned contrast learning loss and the traditional mean square error loss, and effectively utilize the difference information between the predicted value and the true value. Brief Description of the Drawings
[0026] Figure 1 It is a one-dimensional convolution structure diagram.
[0027] Figure 2 It is an overall model framework diagram.
[0028] Figure 3 It is a schematic diagram of the adaptive loss function weight update. Specific Embodiments
[0029] The present invention will be further described in detail below in conjunction with the drawings and embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the protection scope of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention according to the above-mentioned invention content and still fall within the protection scope of the present invention.
[0030] A facial pain assessment method based on contrast learning specifically includes the following steps:
[0031] (1) Data preprocessing
[0032] For a static image in the dataset, crop and align the face through the facial key point coordinates and affine transformation provided in the dataset; then splice 15 frames with the same pain level as the current frame with the current frame to form a 16-frame image sequence; in addition, use the distance normalization method to process the two-dimensional facial key point data to obtain a one-dimensional input vector.
[0033] (2) Facial key point feature extraction
[0034] The extraction process of facial key points is as Figure 1 shown. First, normalize the two-dimensional coordinate data of facial key points into one-dimensional data. Then use six layers of one-dimensional convolution for feature extraction of facial key points. After these six layers of convolution, the dimensions of the features are 256, 512, 1024, 2048, 4096, and 2048 respectively. This network with a structure of first increasing and then decreasing the dimension can better model the features.
[0035] (3) Facial appearance feature extraction
[0036] As Figure 2 shown, use two-dimensional convolution (2D CNN) to extract the facial appearance features in the static image. Specifically, use the ResNet-50 network pre-trained on ImageNet as the backbone network for feature extraction, and adjust the last classification layer to 2048 dimensions as the output of the facial appearance features. Due to the use of the residual structure, ResNet can deepen the network depth while avoiding the gradient problem, enhancing the performance of the network.
[0037] (4) Temporal feature extraction
[0038] As Figure 2 shown, use three-dimensional convolution (3D CNN) to extract the temporal features in the image sequence. Specifically, use the S3D-G network as the backbone network for feature extraction, and adjust the last classification layer to 2048 dimensions as the output of the temporal features. S3D-G replaces a part of the 3D convolution with low-cost 2D convolution, achieving a balance between computational complexity and accuracy and improving the recognition effect.
[0039] (5) Pain feature contrastive learning
[0040] As Figure 2 shown, after extracting the features of different modalities of a sample, use it as a positive sample for contrastive learning with other samples, and cluster the pain features by distinguishing samples with different labels in the feature embedding space to learn the differences in facial features corresponding to different pain levels. The formula of the used contrastive learning loss function (Contrastive Loss) is shown in Equation (1):
[0041]
[0042] For a batch of data with a sample size of N, where I is the sample set after data augmentation, i is the index of any sample in I, P(i) is the set with the same label as sample i and does not include i, |P(i)| is the number of samples in it, and A(i) ≡ I / {i} is the remaining samples in I after removing the current sample, z i 、z p and z a represent the extracted features.
[0043] (6) Facial pain assessment
[0044] As Figure 2 shown, splice the facial key points, appearance, and temporal features extracted in steps (2), (3), and (4), and send the multi-modal features obtained after splicing into the fully connected layer for pain level prediction, and calculate the mean squared error loss (MSELoss).
[0045] (7) Model training
[0046] Train in an end-to-end manner, and adaptively update the weights of the contrastive learning loss and the mean squared error loss using the method as Figure 3 shown to more rigorously constrain the training of the model. The process of learning the weights is shown in Equation (2):
[0047]
[0048] where r represents the weights of the two learned loss functions, σ(·) represents the Sigmoid activation function, W c 、b c 、W m 、W m are learnable parameters.
Claims
1. A facial pain assessment method based on contrastive learning, characterized by: a. Extracted multimodal information from image sequences, including geometric information composed of facial key points, facial appearance information, and temporal information; b. Treat the features of the above three modalities as positive sample pairs for comparative learning; c. Use adaptive weight update method to jointly optimize contrastive learning loss and traditional mean squared error loss; The method mainly includes the following steps: (1) Data preprocessing: First, the facial images in the dataset are cropped and aligned; then, 15 frames with the same pain level as the current frame are spliced with the current frame to form a 16-frame image sequence; in addition, the distance normalization method is used to process the facial key point information to obtain one-dimensional input data; (2) Facial key point feature extraction: A one-dimensional convolutional neural network is used to process the one-dimensional input vector obtained in step (1) to obtain facial geometric features related to pain; (3) Facial appearance feature extraction: The facial appearance features in the cropped and aligned static images in step (1) are extracted using the ResNet network to model local pain information; (4) Temporal feature extraction: The S3D-G network is used to extract the temporal information in the video sequence obtained in step (1) to capture more dynamic features of facial pain; (5) Pain feature comparison learning: The different modal information obtained in steps (2), (3), and (4) are used as positive samples for comparison learning with other samples in the dataset to learn the differences in facial features corresponding to different pain levels; (6) Facial pain assessment: The facial key points, appearance, and temporal features extracted in steps (2), (3), and (4) are concatenated, and the concatenated multimodal features are fed into the fully connected layer for pain level prediction; (7) Model training: In an end-to-end manner, the model training is achieved by jointly optimizing the contrastive learning loss and the mean square error loss using an adaptive weight update method.
2. A facial pain assessment method based on contrastive learning as claimed in claim 1, characterized in that In steps (2), (3) and (4), geometric information, facial appearance information and temporal information composed of facial key points are extracted. Since the objective pain level can be calculated through facial key points, one-dimensional convolution is first used to extract information from the processed facial key point data to obtain the geometric features of the face. Secondly, the dynamic changes in facial expressions are important distinguishing features of human expression of pain, so the S3D-G network is used to extract temporal features in the image sequence. Finally, in order to focus on more detailed pain information, the ResNet network is used to extract facial appearance features in static images.
3. The facial pain assessment method based on contrastive learning according to claim 1, characterized in that In step (5), the multimodal features obtained in steps (2), (3) and (4) are used as positive sample pairs for comparative learning with other samples in the dataset, and the multimodal pain data are fused when comparing the facial features corresponding to different pain levels.
4. The facial pain assessment method based on contrastive learning according to claim 1, characterized in that In step (7), the adaptive loss function weight update method is used to jointly optimize the contrastive learning loss and the mean square error loss. During the model training process, the proportion of the two loss functions is continuously updated to make the calculated total loss function more representative.