Pain expression evaluation method based on multi-task transformer

Abstract

The present application relates to a kind of pain expression evaluation methods based on multi-task transformer, belong to image processing and pattern recognition technique field, comprising the following steps: S1, expression data preprocessing;S2, establish multi-task transformer pain evaluation model;S3, obtain input image;S4, local feature extraction is carried out to input image;S5, weighted combination is carried out based on local feature of self-attention;S6, pain expression classification;S7, pain expression score regression;S8, update network parameter;S9, repeat step S3-S8, until loss function value tends to stable.The present application further exploits the relationship between local features using transformer structure, improves the quality of extracted features, uses a multi-task model of classification and regression, overcomes the boundary sample classification difficulty problem caused by emotional ambiguity, and improves the network's pain expression evaluation accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and pattern recognition, and particularly relates to a pain expression evaluation method based on a multi-task transformer. BACKGROUND

[0002] Automatic pain level evaluation based on facial expression analysis refers to evaluating the pain level of a patient by analyzing the facial expression of the patient. Current facial expression analysis technology often combines existing prior knowledge to explore the feature representation of facial expression information and mine the mapping relationship between different facial representations and expressions, so as to achieve the purpose of emotion recognition. Since facial expression is one of the intuitive ways for patients to express pain, the pain level evaluation method based on facial expression analysis has become one of the hot researches in the field of pain evaluation in recent years.

[0003] Feng Wang proposed a pain level regression model based on a regularized face verification network in the published paper "Regularizing face verification nets for pain intensity regression" (IEEE, 2018, pp. 1087-1091). The implementation steps of the model are: ① establishing a deep network model and pre-training in the face verification task; ② inputting the data of the task data set into the model; ③ using the center loss function to constrain the extracted features, reducing the distance between the features in the same class and increasing the distance between the features in different classes; ④ using the huber regression loss function to calculate the error between the model output and the real label. This method does not consider analyzing the locality in the patient's pain representation, and the prediction result accuracy is low.

[0004] Shaoxing Cui in his published paper "Multi-Scale Regional Attention Networks for Pain Estimation" (New York, NY, USA: Association for Computing Machinery, 2021, p. 1-8.) proposes a multi-scale attention network-based pain assessment model that focuses on local features in pain expressions. The specific implementation steps are: ① crop the input image to obtain k images of different scales; ② input the k images into k convolutional neural networks to extract features; ③ establish a self-attention module, input the extracted k features into k fully connected layers and sigmoid activation functions to calculate the attention coefficients of each feature, and weight the features through the attention coefficients; ④ establish a mutual attention module, concatenate the k features together and input them into k fully connected layers and sigmoid activation functions to calculate the attention coefficients of each feature, thereby mining the relationship between different scale features, and weighting the features through the mutual attention coefficients. Although this method considers weighting different scale features using attention mechanisms, it lacks analysis of the intensity and correlation of pain-related facial local motion units, and does not consider the ambiguity in expressions, making it difficult for classification models to classify boundary samples and easily leading to low prediction accuracy.

[0005] In summary, there is an urgent need to provide a pain level evaluation method based on facial expression analysis to solve the problems of lack of mining of the intensity and correlation of pain-related facial local motion units in patient expressions and difficulty in classifying boundary samples due to emotional ambiguity in the prior art. SUMMARY

[0006] To solve the above problems in the prior art, a multi-task transformer-based pain expression evaluation method is proposed. In the feature extraction stage, each pain expression image is segmented into blocks and local features are extracted, and the self-attention mechanism of the transformer is used to weight and combine the local features, which can overcome the problem of lack of mining of the intensity and correlation of pain-related facial local motion units in patient expressions in the prior art, and improve the accuracy of pain assessment. In the prediction stage, a multi-task model of regression and classification is adopted, where the classification task is used to constrain the similarity of intra-class features, and the regression task is used to obtain continuous and fine-grained pain expression labels, overcoming the difficulty in classifying boundary samples caused by facial emotional ambiguity.

[0007] The object of the present application can be achieved by the following technical solutions:

[0008] A pain expression evaluation method based on a multi-task transformer, comprising the following steps:

[0009] S1, expression data preprocessing: including uniformity of expression image size and conversion;

[0010] S2, establishing a multi-task transformer pain evaluation model;

[0011] S3, obtaining an input image: randomly sampling a group of data from an expression data set;

[0012] S4, local feature extraction on the input image;

[0013] S5, weighted combination based on self-attention local features;

[0014] S6, pain expression classification;

[0015] S7, pain expression score regression;

[0016] S8, updating network parameters;

[0017] S9, repeating steps S3-S8 until the loss function value tends to be stable.

[0018] Further, the multi-task transformer pain evaluation model is established, specifically comprising the following steps:

[0019] S21, establishing a convolutional neural network feature extractor E;

[0020] S22, establishing a transformer model T;

[0021] S23, establishing a multi-task pain evaluation module.

[0022] Further, the pain evaluation module includes a pain level classifier F and a pain level regressor R.

[0023] Further, the pain level classification specifically comprises the following steps:

[0024] S61, inputting the feature f of the input image x i into the pain level classifier F to obtain the classification result of the input image x i ;

[0025] S62, calculating the cross-entropy loss L c of the classification result of the input image x i .

[0026] Further, the calculation formula of the cross-entropy loss L c is:

[0027] L c = -log(p(y i |x i ))

[0028] wherein y i represents the corresponding category of the input image x i , p(y i |x i ) represents the prediction result of the pain level classifier F on the category of the input image x i , the cross-entropy loss value L c represents the gap between the prediction result of the pain level classifier F and the true result.

[0029] Further, the pain expression score regression specifically comprises the following steps:

[0030] S71, inputting the feature f of the input image x i to the pain level regressor R established in step S23 to obtain the regression result of the input image x i ;

[0031] S72, calculating the huber loss L i of the regression result of the input image x r .

[0032] Further, the calculation method of the huber loss L r is as follows:

[0033]

[0034] wherein y i represents the corresponding category of the input image x i , w T f represents the prediction result of the regression score of the pain level regressor R on the input image x i , sigma represents the sigmoid function, delta is the threshold value, the huber loss value L r represents the gap between the prediction result of the pain level regressor R and the true result.

[0035] Further, the updated network parameters specifically include the parameters in the convolutional neural network feature extractor E, the transformer model T, the pain level classifier F and the pain level regressor R.

[0036] The present application has the following advantages:

[0037] By utilizing the transformer structure, the relationships between local features were further explored, improving the quality of the extracted features and thus enhancing the network's accuracy in assessing pain expressions. The multi-task model of classification and regression not only broadens the application scenarios and makes the clinical significance more obvious, but also overcomes the difficulty of classifying boundary samples caused by emotional ambiguity, further improving the network's accuracy in assessing pain expressions. Attached Figure Description

[0038] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0039] Figure 1 This is a schematic diagram of the pain expression assessment method of the present invention;

[0040] Figure 2 This is a schematic diagram of the attention model structure in this invention. Detailed Implementation

[0041] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0042] A pain expression assessment method based on multi-task transformer, such as Figure 1 As shown, it includes the following steps:

[0043] S1. Facial expression data preprocessing:

[0044] The size of the facial expression images in the dataset was standardized to 224×224 pixels.

[0045] Convert the facial expression images in the dataset to grayscale images;

[0046] S2. Establish a multi-task transformer pain assessment model:

[0047] S21. Establish a convolutional neural network feature extractor E. This convolutional neural network feature extractor E is obtained by improving the existing ResNet18 neural network. Based on the conv1 and conv2.x of the ResNet18 neural network, a convolutional layer is added to form the convolutional neural network feature extractor E. The parameters of the convolutional neural network feature extractor E are initialized to be the same as the other network parameters of the ResNet18 model pre-trained on the large-scale face image dataset MS-Celeb-1M except for the added convolutional layer.

[0048] S22. Establish the transformer model T: This model is obtained by improving the existing swin-transformer neural network. The patch embedding layer of the original swin-transformer neural network is removed to form the transformer model. The convolutional neural network feature extractor E is initialized, and its parameters are the same as those of the swin-transformer model pre-trained on the natural image dataset ImageNet except for the patch embedding.

[0049] S23. Establish a multi-task pain assessment module. This module includes a pain level classifier F and a pain level regressor R.

[0050] The pain level classifier F consists of a fully connected layer and a softmax function. The network parameters are randomly initialized by sampling from a normal distribution with a mean of 0 and a variance of 0.02. The input of the fully connected layer is 1024-dimensional, and the output is 4-dimensional.

[0051] The softmax function is used to map the output of the fully connected layer of the pain level classifier F to the probability of belonging to each category. The sum of the probabilities of all categories is 1. Its calculation formula is as follows:

[0052]

[0053] Where z represents a 4-dimensional vector output by the fully connected layer. i Let z be the i-th element of z, where N represents the total number of categories. c Let c represent the c-th element in z, where c ∈ [1, N].

[0054] The pain level regressor R consists of a fully connected layer and a sigmoid function. The network parameters are randomly initialized from a normal distribution with a mean of 0 and a variance of 0.02. The input of the fully connected layer is 1024-dimensional, and the output is 1-dimensional.

[0055] The sigmoid function is used to map the output of the fully connected layer of the pain level regressor R to the patient's current pain level score, and its calculation formula is as follows:

[0056]

[0057] Where z represents a 1-dimensional output of the fully connected layer, and e is the base of the natural logarithm.

[0058] S3. Obtain input images: Randomly sample a set of data from the facial expression dataset, specifically including 32 images.

[0059] S4. Extract local features from the input image:

[0060] Input image x i (i∈[1,32]) are input into the convolutional neural network feature extractor E, divided into 32*32 blocks, and local features f are extracted. j :

[0061] f j =E(x) i )

[0062] Where i represents the image number, j represents the number of the segmented block and j∈[1,32*32].

[0063] S5. Weighted combination of local features based on self-attention:

[0064] Local feature f j The input is fed into the transformer model T to obtain the weighted combined features f:

[0065] f = T(f) j )

[0066] Where j∈[1,32*32], the transformer consists of multiple multi-head self-attention layers.

[0067] Because the multi-head self-attention mechanism of the transformer allows the model to learn the relationships between different local features in an image, it is adept at solving long-distance dependency problems. In this invention, the image is segmented into different blocks, such as... Figure 2 As shown, the model learns the features of different pain-related facial motor units and comprehensively considers the importance of each local area to the pain level. The formula for calculating the weighted combined feature f is as follows:

[0068]

[0069] Where, q j To query features, k j For index features, v j As a content feature, and q j k j v j From input f j d is obtained through three different fully connected layers. k This is the normalization coefficient.

[0070] S6. Pain Expression Classification:

[0071] S61, from the input image x i The feature f is input into the classifier F established in step S23 to obtain the input image x. iThe classification results;

[0072] S62. Calculate the input image x i The cross-entropy loss L of the classification results c :

[0073] L c =-log(p(y) i |x i ))

[0074] Where y i Indicates the input image x i The corresponding category, p(y) y |x i ) represents the pain level classifier F for the input image x. i The prediction results for the category, and the cross-entropy loss value L c This represents the difference between the predicted results of the pain level classifier F and the actual results.

[0075] S7, Pain Expression Score Regression:

[0076] S71, from the input image x i The feature f is input into the pain level regressor R established in step S23 to obtain the input image x. i The regression results;

[0077] S72, Calculate the input image x i The regression results of Huber loss L r :

[0078]

[0079] Where y i Indicates the input image x i The corresponding category, w T f represents the pain level regressor R for the input image x. i The predicted regression score, where σ represents the sigmoid function, δ is the threshold, and L is the Huber loss value. r This represents the difference between the predicted result and the actual result of the pain level regressor R. The closer the prediction is to the actual result, the smaller the loss value.

[0080] S8. Update network parameters:

[0081] S81, Obtain the input image x from step S62. i Cross-entropy loss L of classification results c Backpropagation is performed to bring the loss function value as close to zero as possible in order to update the parameters in the convolutional neural network feature extractor E, the transformer model T, and the pain level classifier F;

[0082] S82, Obtain the input image x from step S72. i Huber loss L in regression results c Perform backpropagation to bring the loss function value as close to zero as possible, in order to update the parameters in the convolutional neural network feature extractor E, transformer model T, and pain level regressor R;

[0083] S9. Repeat steps S3 to S8 until the loss function value stabilizes, thus completing the pain assessment based on facial expression analysis.

[0084] To verify the accuracy of the pain expression assessment method proposed in this invention, simulations were performed using the deep learning library PyTorch. The dataset used was the UNBC shoulder pain dataset, which includes video data from 25 patients, with at least two video clips for each patient. The videos were segmented frame by frame into images, and facial expressions were extracted, resulting in a total of 48,398 images. A 25-fold cross-validation was employed, with images from one patient used as the test set and images from the remaining patients used as the training set for each cross-validation.

[0085] The input patient facial expression image was 224×224 pixels in size. Parameter optimization was performed using stochastic gradient descent with 60 iterations. The results of each method in the simulation experiment were evaluated using four metrics: mean error (MAE), mean squared error (MSE), Pearson correlation coefficient (PCC), and intraclass correlation coefficient (ICC).

[0086] The formula for calculating MAE is as follows:

[0087]

[0088] In the above formula, pred i y represents the prediction result for sample i. i The label represents sample i, and N represents the value of MAE. The smaller the MAE value, the closer the prediction result is to the true label.

[0089] The formula for calculating MSE is as follows:

[0090]

[0091] In the above formula, pred i y represents the prediction result for sample i. i The MSE value represents the label of sample i. The smaller the MSE value, the closer the prediction result is to the true label.

[0092] The formula for calculating PCC is as follows:

[0093]

[0094] In the above formula, pred i The prediction result represents sample i. y represents the average of the predicted results. i The label representing sample i. PCC represents the mean of the sample labels. The larger the PCC value, the closer the distribution of the predicted results is to the true label distribution.

[0095] The formula for calculating ICC is as follows:

[0096]

[0097] In the above formula, pred i y represents the prediction result for sample i. i The ICC value represents the label of sample i, and avg represents the mean of the sample label and the prediction result. The larger the ICC value, the higher the reliability of the prediction result.

[0098] MAE MSE PCC ICC Prior art 0.4000 0.4600 0.7800 0.6300 The invention 0.2604 0.2594 0.7920 0.6476

[0099] Table 1 Comparison of Pain Assessment Performance

[0100] The evaluation results are shown in Table 1. The prior art refers to the index data of a convolutional neural network model based on a multi-scale attention mechanism proposed by Shaoxing Cui in his paper "Multi-Scale Regional Attention Networks for Pain Estimation" (New York, NY, USA: Association for Computing Machinery, 2021, pp. 1-8). As can be seen from the data in Table 1, compared with the prior art, the multi-task transformer pain assessment method proposed in this invention improves the pain assessment results.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A pain expression assessment method based on a multi-task transformer, characterized in that: Includes the following steps: S1. Facial expression data preprocessing: including facial expression image size unification and conversion; S2. Establish a multi-task transformer pain assessment model; The establishment of the multi-task transformer pain assessment model specifically includes the following steps: S21. Establish a convolutional neural network feature extractor. Based on the conv1 and conv2.x of the Resnet18 neural network, an additional convolutional layer is added to form the convolutional neural network feature extractor E; S22. Establish the transformer model Remove the patch embedding layer from the original Swin-transformer neural network to construct the transformer model; S23. Establish a multi-task pain assessment module; The pain assessment module includes a pain level classifier F and a pain level regressor R. The pain level classifier F consists of a fully connected layer and a softmax function, which maps the output of the fully connected layer of the pain level classifier F to the probability of belonging to each category. The pain level regressor R consists of a fully connected layer and a sigmoid function, which maps the output of the fully connected layer of the pain level regressor R to the patient's current pain level score. S3. Obtain the input image: Randomly sample a set of data from the facial expression dataset; S4. Extract local features from the input image; extract the input image... (i Input to the convolutional neural network feature extractor The middle is divided into 32 32 blocks, and extract local features. : ； Where i represents the image number, j represents the number of the segmented block, and j ; S5. Weighted combination of local features based on self-attention; combining local features The input is fed into the transformer model T to obtain the weighted combined features. ; S6. Pain Expression Classification; The classification of pain expressions specifically includes the following steps: S61, from the input image Features The input image is fed into the pain level classifier F to obtain the input image. The classification results; S62, Calculate the input image Cross-entropy loss of classification results ; S7, pain expression score regression; The regression of pain expression scores specifically includes the following steps: S71, from the input image Features The input image is obtained by inputting it into the pain level regressor R established in step S23. The regression results; S72, Calculate the input image Huber loss in the regression results ; S8. Update network parameters; S9. Repeat steps S3 to S8 until the loss function value tends to stabilize.

2. The pain expression assessment method based on multi-task transformer according to claim 1, characterized in that: The cross-entropy loss The calculation formula is: ； in Indicates the input image Corresponding categories The pain level classifier F represents the input image. The prediction results for the category, and the cross-entropy loss value. This represents the difference between the predicted results of the pain level classifier F and the actual results.

3. The pain expression assessment method based on multi-task transformer according to claim 1, characterized in that: The Huber loss The calculation method is as follows: ； in Indicates the input image Corresponding categories The pain level regressor R represents the input image. The predicted results of the regression scores, This represents the sigmoid function. The threshold is the Huber loss value. This represents the difference between the predicted results and the actual results of the pain level regressor R.

4. The pain expression assessment method based on multi-task transformer according to claim 1, characterized in that: The updated network parameters specifically include a convolutional neural network feature extractor. Transformer model Pain level classifier And the parameters in the pain level regressor R.

Images