A method for pain assessment of facial dynamic video images based on dynamic fusion module

Abstract

The present invention discloses a method for assessing pain in facial dynamic video images based on a dynamic fusion module. First, the UNBC-McMaster Shoulder Pain Expression Archive dataset is subjected to data balancing processing. Then, in the pre-trained model Resnet101 network, a dynamic fusion module is obtained based on the Non-local framework improvement. An image spatial feature extraction model is established. The spatial feature information of each frame of the image is extracted through the spatial feature extraction model. The extracted feature information is then input into a long short-term memory network to extract the corresponding timing information of the facial dynamic video image. The timing information between adjacent frames is further extracted to realize the recognition of facial pain, and finally realize the automatic assessment of pain. The method of the present invention is intended to assist doctors in long-term pain monitoring and assessment of patients, provide an auxiliary means for clinical pain assessment, and has potential application prospects for assessing the pain scenarios of special patients in clinical practice.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a facial dynamic video image pain assessment method based on a dynamic fusion module. Background Art

[0002] Pain is a complex phenomenon that involves the human body's sensory and emotional experience. It is also a self-protection mechanism of the human body. In clinical settings, pain intensity estimation is crucial for the diagnosis and treatment of patients, especially for patients who cannot express pain in words. Self-report and observation are commonly used as mainstream methods for pain assessment in clinical practice. However, these two methods are highly subjective and are not suitable for special groups of patients. Related studies have shown that the human face contains rich pain features, which makes it an effective way to automatically assess pain by extracting facial features.

[0003] So far, most of the pain assessment studies based on facial pain expressions are based on static images. However, static images lack the timing information between adjacent frames during feature extraction, which may lead to poor assessment performance. In addition, in a real clinical environment, the patient's pain needs to be monitored for a long time, which cannot be met by the pain assessment method based on static facial images. Pain assessment based on dynamic facial video images can meet clinical needs, whether it is based on offline or real-time conditions. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a facial dynamic video image pain assessment method based on a dynamic fusion module.

[0005] The technical solution adopted by the present invention is: a facial dynamic video image pain assessment method based on a dynamic fusion module, and the specific steps of the method are as follows:

[0006] S1. Perform data balancing on the UNBC-McMaster Shoulder Pain Expression Archive dataset.

[0007] S2. In the pre-trained model Resnet101 network, a dynamic fusion module is obtained based on the Non-local framework to establish an image space feature extraction model;

[0008] S3. Based on the spatial feature extraction model of step S2, the spatial feature information of each frame of the image is extracted, and then the extracted feature information is input into the long short-term memory network to extract the corresponding timing information of the facial dynamic video image, and further extract the timing information between adjacent frames to realize the recognition of facial pain, and finally realize the automatic assessment of pain.

[0009] Furthermore, the step S1 is specifically as follows:

[0010] The UNBC-McMaster Shoulder Pain Expression Archive dataset contains 25 subjects, 200 videos, and 48,398 frames of images. The PSPI method is used to mark the pain level of each frame.

[0011] The data balancing process is specifically as follows:

[0012] (1) Delete the complete sequence containing pain-free frames;

[0013] (2) If the beginning or end frame of a complete sequence contains a pain-free frame, it is partially deleted.

[0014] Furthermore, in step S2, the dynamic fusion module is intended to fuse the features extracted by the fourth block and the fifth block of the pre-trained model Resnet101 network, as follows:

[0015] The features extracted from the fourth and fifth blocks of the pre-trained model Resnet101 network are denoted as X h , X l , i.e., a high-level feature map and a low-level feature map

[0016] Among them, N h 、N l Respectively represent X h , X l The number of spatial locations, C h , C l Respectively represent X h , X l The number of channels, Represents a collection of features.

[0017] All the features extracted from the fourth and fifth blocks of the Resnet101 network are first fused, and then the fused features are input into the multi-scale channel attention module to calculate the weight value α, which is used to weight X h The input features of X are 1-α, and 1-α is used to weight X l The input features; the features obtained after weighting by weight values ​​α and 1-α are respectively represented by X' h , X l '; After the dynamic fusion module, the spatial feature information of each input image frame is extracted.

[0018] Furthermore, in step S2, the pre-trained model uses a five-fold cross-validation method to divide the data set, and then trains and tests the proposed network model, as follows:

[0019] 1) The ratio of training and testing data sets is 4:1. The Adam algorithm is used for training, the parameters in the network model are updated and the model is saved;

[0020] 2) Repeat step 1) for iterative training. When the training is completed, the model with the highest evaluation index is called the best model and is saved.

[0021] Beneficial effects of the present invention: The method of the present invention first performs data balancing processing on the UNBC-McMaster Shoulder PainExpression Archive dataset, and then in the pre-trained model Resnet101 network, a dynamic fusion module is improved based on the Non-local framework to establish an image spatial feature extraction model, and the spatial feature information of each frame of the image is extracted through the spatial feature extraction model, and then the extracted feature information is input into the long short-term memory network to extract the corresponding timing information of the facial dynamic video image, and further extract the timing information between adjacent frames to realize the recognition of facial pain, and finally realize the automatic assessment of pain. The method of the present invention is intended to assist doctors in long-term pain monitoring and assessment of patients, provide an auxiliary means for clinical pain assessment, and has potential application prospects for assessing the pain scenarios of special patients in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention is a flowchart of a method for assessing pain in dynamic facial video images based on a dynamic fusion module.

[0023] Figure 2 A distribution diagram of the number of frames of each subject's pain / no-pain video in the UNBC-McMaster Shoulder Pain Expression Archive dataset is provided in an embodiment of the present invention.

[0024] Figure 3 This is a distribution diagram of the number of frames of pain / nopain videos of each subject in the balanced UNBC-McMaster dataset in an embodiment of the present invention.

[0025] Figure 4 This is a diagram of the overall network model in an embodiment of the present invention.

[0026] Figure 5 This is a graph showing average performance comparison results of different evaluation indicators of the Resnet101-LSTM model with different input video frame numbers in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0028] like Figure 1 As shown, a flow chart of a facial dynamic video image pain assessment method based on a dynamic fusion module of the present invention, the specific steps are as follows:

[0029] S1. Perform data balancing on the UNBC-McMaster Shoulder Pain Expression Archive dataset.

[0030] S2. In the pre-trained model Resnet101 network, a dynamic fusion module is obtained based on the Non-local framework to establish an image space feature extraction model;

[0031] S3. Based on the spatial feature extraction model of step S2, the spatial feature information of each frame of the image is extracted, and then the extracted feature information is input into the long short-term memory network to extract the corresponding timing information of the facial dynamic video image, and further extract the timing information between adjacent frames to realize the recognition of facial pain, and finally realize the automatic assessment of pain.

[0032] In this embodiment, the step S1 is specifically as follows:

[0033] The UNBC-McMaster Shoulder Pain Expression Archive dataset contains 25 subjects, 200 videos, and 48,398 frames of images. The PSPI method is used to mark the pain level of each frame. Figure 2 As shown, it can be seen that there is an imbalance in the number of videos, video frames, and pain frames and non-pain frames contained in each subject.

[0034] Among them, the number of non-pain frames is far greater than the number of pain frames. Therefore, based on the data distribution characteristics of this data set, any model tends to predict non-pain frames when performing pain recognition. It is necessary to perform corresponding balancing processing on the data. The specific method of data balancing processing is as follows:

[0035] (1) Delete the complete sequence containing pain-free frames;

[0036] (2) If the beginning or end frame of a complete sequence contains a pain-free frame, it is partially deleted.

[0037] The image frames without pain were removed by the above method. After the data set was balanced, the number of pain frames and non-pain frames contained in each subject were as follows: Figure 3Secondly, the leave-one-out method and five-fold cross validation are used to train the model proposed in the present invention on the balanced data set.

[0038] In this embodiment, in step S2, the dynamic fusion module is intended to fuse the features extracted by the fourth block and the fifth block of the pre-trained model Resnet101 network, as follows:

[0039] In the pre-trained model Resnet101 network, a dynamic fusion module is proposed. The module uses Non-local as the main framework. However, due to the defect of high computational cost of the existing Non-local framework, the improved asymmetric fusion non-local block module based on Non-local is used as the overall framework of the dynamic fusion module. The proposed dynamic fusion module is mainly used to fuse all the extracted features of the fourth and fifth blocks of the Resnet101 network.

[0040] The features extracted from the fourth and fifth blocks of the pre-trained model Resnet101 network are denoted as X h , X l , i.e., a high-level feature map and a low-level feature map

[0041] Among them, N h 、N l Respectively represent X h , X l The number of spatial locations, C h , C l Respectively represent X h , X l The number of channels, Represents a collection of features.

[0042] Considering that the features extracted from different blocks of the Resnet101 network may have different contribution rates to the final recognition accuracy, all the features extracted from the fourth and fifth blocks of the Resnet101 network are first fused, and then the fused features are input into the multi-scale channel attention module to calculate the weight value α, which is used to weight X h The input features of X are 1-α, and 1-α is used to weight X l The input features of .

[0043] The features obtained after weighting by weight values ​​α and 1-α are X h ′、X l ′; It should be noted here that the multi-scale channel attention module used is as follows Figure 4As shown in the MS-CAM, MS-CAM uses X in the figure to represent input features; Y represents output features; C, H, and W represent the number of channels, height, and width of the feature map, respectively; r represents a variable parameter, and r=2 is set in this embodiment; after passing through the dynamic fusion module, the spatial feature information of each input image frame is extracted.

[0044] In this embodiment, in step S2, the pre-trained model divides the data set using a five-fold cross-validation method, and then trains and tests the proposed network model, as follows:

[0045] 1) The ratio of training and testing data sets is 4:1. The Adam algorithm is used for training, the parameters in the network model are updated and the model is saved.

[0046] 2) Repeat step 1) for iterative training. When the training is completed, the model with the highest evaluation index is called the best model and is saved.

[0047] like Figure 4 As shown, the pre-trained model Resnet101 network model uses the pre-trained model Resnet101 to extract the spatial feature information of each frame of the image, and then inputs the extracted spatial feature information into the long short-term memory network to extract the corresponding timing information of the facial dynamic video image, and finally realizes the recognition of facial pain.

[0048] In this embodiment, the image space feature extraction model proposed in the present invention is also tested and evaluated, as follows:

[0049] First, the optimal model parameters obtained by the pre-trained model training in step S2 are imported, and the test set images are input to test the performance of the model;

[0050] This embodiment takes into account that when the number of video frames input to the proposed model is too small, the timing information is small, which will result in insufficient timing information being extracted; and when the number of video frames input is too large, the timing information will be redundant. Therefore, the number of video frames selected for the model input is of great significance to the accuracy of pain recognition.

[0051] In this embodiment, the frame rates are selected as 10, 15, 20, 25, 30, 35 and 40 for training and testing the baseline model (Resnet101-LSTM model). The baseline results are shown in Table 1. Table 1 is the quantitative analysis result of the average performance comparison of the baseline model Resnet101-LSTM with different input video frame rates. The main evaluation indicators used include: Accuracy value, AUC value, F1-Score value, Recall value (recall rate), Precision (precision rate). In addition, this embodiment visualizes the different input frame rates and their corresponding model evaluation indicators. The visualization results are shown in Table 1. Figure 5 Shown

[0052] Table 1

[0053]

[0054] From Table 1 and Figure 5 It can be seen that as the number of input video frames increases, the accuracy of the model shows a trend of first increasing and then decreasing, and at 35 frames, the accuracy is the highest, reaching 81.92%. Other corresponding indicators are not much different. Therefore, the number of input video frames is selected as 35 as a standard for subsequent experiments related to the method of the present invention.

[0055] Then, this embodiment verifies the improvement of the proposed dynamic fusion module on the pain recognition performance and the comparison of model complexity and parameter quantity, as shown in Tables 2 and 3.

[0056] Table 2

[0057]

[0058] Table 3

[0059]

[0060] Among them, Table 2 is the average performance comparison of AFNB and DFNB fusion modules for pain recognition; Table 3 is the comparison of the complexity and parameter quantity of different models.

[0061] Finally, the pain recognition based on dynamic facial image video in the dynamic facial video image pain assessment method based on dynamic fusion module of the present invention is compared with the most advanced model results in the UNBC-McMaster shoulder pain dataset. It should be noted that when comparing with other models here, in order to ensure consistency, the leave-one-out method is used for data partitioning, that is, the data of 24 subjects out of 25 subjects are used for model training, and the data of the remaining subject is used for model testing, which is repeated 25 times and the data cannot be repeated each time. Finally, the 25 test results are averaged to obtain the corresponding results, as shown in Table 4.

[0062] Table 4

[0063]

[0064] In summary, the method of the present invention was tested on the public dataset UNBC-McMaster Shoulder Pain Expression Archive. The proposed model performed better than other existing methods, providing an auxiliary means for clinical pain assessment and having potential application prospects in assessing pain scenarios of special patients in clinical practice.

[0065] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A facial dynamic video image pain assessment method based on a dynamic fusion module, the specific steps are as follows: S1. Perform data balancing on the UNBC-McMaster Shoulder Pain Expression Archive dataset. S2. In the pre-trained model Resnet101 network, a dynamic fusion module is obtained based on the Non-local framework to establish an image space feature extraction model; S3, extracting spatial feature information of each frame of the image based on the spatial feature extraction model of step S2, then inputting the extracted feature information into the long short-term memory network to extract the corresponding timing information of the facial dynamic video image, further extracting the timing information between adjacent frames, realizing the recognition of facial pain, and finally realizing the automatic assessment of pain; The step S1 is specifically as follows: The UNBC-McMaster Shoulder Pain Expression Archive dataset contains 25 subjects, 200 videos, and 48,398 frames of images. The PSPI method is used to mark the pain level of each frame. The data balancing process is specifically as follows: (1) Delete the complete sequence containing pain-free frames; (2) if the beginning or end frame of a complete sequence contained a pain-free frame, it was partially deleted; In step S2, the dynamic fusion module is intended to fuse the features extracted by the fourth block and the fifth block of the pre-trained model Resnet101 network, as follows: The features extracted from the fourth and fifth blocks of the pre-trained model Resnet101 network are denoted as X h , X l , i.e., a high-level feature map and a low-level feature map in, N h 、N l Respectively represent X h , X l The number of spatial locations, C h , C l Respectively represent X h , X l The number of channels, Represents a set of features; All the features extracted from the fourth and fifth blocks of the Resnet101 network are first fused, and then the fused features are input into the multi-scale channel attention module to calculate the weight value α, which is used to weight X h The input features of X are 1-α, and 1-α is used to weight X l Input features of The features obtained after weighting by weight values ​​α and 1-α are X h ′、X l ′; After passing through the dynamic fusion module, the spatial feature information of each input image frame is extracted.

2. The facial dynamic video image pain assessment method based on dynamic fusion module according to claim 1 is characterized in that: In step S2, the pre-trained model uses a five-fold cross-validation method to divide the data set, and then trains and tests the proposed network model, as follows: 1) The ratio of training and testing data sets is 4:

1. The Adam algorithm is used for training, the parameters in the network model are updated and the model is saved; 2) Repeat step 1) for iterative training. When the training is completed, the model with the highest evaluation index is called the best model, and the best model is saved.

Images