Pain expression recognition method and system based on dual-branch neural network

Through the combination of the dual-branch neural network and attention feature extraction module, the problem of low accuracy in pain expression recognition in the prior art is solved, and high-accurate recognition of adjacent pain intensity is achieved, which is suitable for clinical pain analysis.

CN118522049BActive Publication Date: 2024-12-06QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES) +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410425506.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-10
Publication Date
2024-12-06
Estimated Expiration
2044-04-10

AI Technical Summary

Technical Problem

The prior art has difficulty capturing subtle differences between adjacent pain intensity in pain expression recognition, resulting in a decrease in recognition accuracy.

Method used

The pain expression recognition method based on a dual-branch neural network is adopted to extract features through global and local branches, and combine attention feature extraction modules to integrate global and local information, and use softmax classification to improve recognition accuracy.

Benefits of technology

It significantly improves the ability to distinguish adjacent pain intensity levels, improves the accuracy and robustness of pain intensity recognition, and is suitable for pain analysis in clinical medicine.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118522049B_ABST
    Figure CN118522049B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of pain expression recognition, and in particular to a method for pain expression recognition based on a dual-branch neural network, comprising the following steps: pre-processing an input image, segmenting the original image into four regions of a left eye EF, a right eye ER, a nose N, and a mouth M, and adjusting the four regions to the same size; inputting the original image into a global branch, and obtaining a global feature vector V1 after processing; inputting the segmented four regions into a feature extraction network of a local branch in sequence, obtaining a feature map corresponding to each region, splicing the feature map in a channel dimension, and then obtaining a feature vector V2 by global average pooling of the feature map; splicing V1 and V2 to obtain V, performing feature fusion through a fully connected layer to obtain #imgabs0#, applying softmax to #imgabs1# for classification, and obtaining the probability of each pain intensity level, wherein the level corresponding to the maximum probability is the recognition result. The present invention can improve the distinction between adjacent pain intensity levels and greatly improve the accuracy of pain intensity recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical equipment, and in particular to a method and system for recognizing pain expressions based on a double-branch neural network. Background Art

[0002] Pain is one of the most common manifestations of clinical patients. It is a manifestation of discomfort in patients. Pain can be strong or weak, and it needs to be treated accordingly according to the patient's pain condition. Therefore, the assessment of pain intensity has become an important task for current medical institutions and has broad development prospects in the field of clinical medicine. At present, artificial intelligence algorithms represented by deep learning have achieved outstanding results in fields such as expression recognition by virtue of powerful computing resources and large-scale data sets. In the field of facial pain expression recognition, many scholars have proposed a large number of excellent pain intensity recognition network structures for the recognition of pain expressions. These network structures can achieve relatively high recognition accuracy while maintaining a fast speed. However, there are still some problems in this field. Since adjacent pain intensities have similar characteristics, there are only slight differences in the movement of their facial motor units. It is difficult for the network to recognize such differences. It is difficult to capture such small differences through existing methods, which makes it difficult for the network to distinguish adjacent pain intensities, thereby affecting the accuracy of pain intensity recognition for pain expressions.

[0003] Therefore, the present invention provides a pain expression recognition method based on a dual-branch neural network to solve the above problems. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention develops a method for pain expression recognition based on a double-branch neural network, which can achieve relatively high recognition accuracy while maintaining a relatively fast speed.

[0005] The technical solution of the present invention to solve the technical problem is: on the one hand, the present invention provides a method for pain expression recognition based on a double-branch neural network, which is characterized by comprising the following steps:

[0006] a) Preprocess the input image, segment the original image into four regions: left eye EF, right eye ER, nose N, and mouth M, and adjust the four regions to the same size;

[0007] b) Input the original image into the global branch and obtain the global feature vector V1 after processing;

[0008] c) Input the four segmented regions into the feature extraction network of the local branch in turn to obtain the feature map corresponding to each region, concatenate the feature maps in the channel dimension, and then obtain the feature vector V2 through global average pooling of the feature maps;

[0009] d) Concatenate V1 and V2 to get V, and perform feature fusion through the fully connected layer to get ,right Apply softmax for classification and obtain the probability of each pain intensity level, where the level corresponding to the maximum probability is the recognition result.

[0010] Step a) comprises the following steps:

[0011] a-1) Use the face detector MTCNN to perform face detection and facial key point detection, and locate the position of each facial key point;

[0012] a-2) According to the detected facial key points, four regions are segmented from the face image, namely: left eye EF, right eye ER, nose N, mouth M, and the size of the cropped area image is adjusted to 112×112.

[0013] Step b) comprises the following steps:

[0014] b-1) The original image is extracted by the simple feature extraction module of the global branch to obtain features T1, T2 and T3, and the original image is transmitted to the first network layer to obtain features , and then the feature Transmit to the second network layer to obtain feature F;

[0015] b-2) Input feature T1 into the first network layer of the first residual block R1 to obtain feature , for the features Perform batch normalization operation to obtain features ,feature After ReLU activation function, we get ;

[0016] The characteristics Input the second network layer of the first residual block R1, and the convolution operation obtains the feature ;

[0017] The characteristics Input the third network layer of the first residual block R1, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ;

[0018] The characteristics Input the fourth network layer of the first residual block R1, and the convolution operation obtains the feature ;

[0019] The characteristics Input the fifth network layer of the first residual block R1, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ;

[0020] The characteristics Input the sixth layer network of the first residual block R1 and convolve to obtain features , for the features Perform batch normalization operation to obtain features Finally, the feature Add to input feature T1 and execute Operation, execute to obtain the output features of the first residual block R1 ;

[0021] b-3) Input feature T2 into the first network layer of the second residual block R2, and convolve to obtain feature , for the features Perform batch normalization operation to obtain features ,feature After ReLU activation function, we get ;

[0022] The characteristics Input the second network layer of the second residual block R2, and the convolution operation obtains the feature ;

[0023] The characteristics Input the third network layer of the second residual block R2, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ;

[0024] The characteristics Input the fourth network layer, and the convolution operation obtains the features ;

[0025] The characteristics Input the fifth network layer, and the convolution operation obtains the features , for the features Perform batch normalization to obtain features ,feature The features are obtained by ReLU activation function , the feature Add to input feature T2 and execute Operation, get the output features of the second residual block R2 ;

[0026] b-4) Input feature T3 into the four branches X1, X2, X3, and X4 of Inception module I respectively;

[0027] In X1, feature T3 is convolved to obtain feature ; In X2, feature T3 is input into the first layer network to obtain feature ,feature Input the second layer network and convolve to get features ,feature Input the third layer network and convolve to get features ;

[0028] In X3, feature T3 is input into the first layer network to obtain feature ,feature Input the second layer network and convolve to get features ,feature Input the third layer network and convolve to get features ,feature Input the fourth layer network and convolve to get features ,feature Input the fifth layer network and convolve to get features ;

[0029] In X4, feature T3 is input into the first layer network and the feature is obtained by the maximum pooling operation. ,feature Input the second layer network and obtain the features through 1×1 convolution ;

[0030] In the channel dimension , , , Splice and get features

[0031] ,

[0032] The characteristics Input 1×1 convolution layer, the number of channels is reduced to One quarter of , and finally Perform the softmax operation. The specific operations are as follows:

[0033] ,

[0034] The output obtained by the above formula is ;

[0035] b-5) Input feature F into the first attention feature extraction module A1 of the attention mechanism of the global branch;

[0036] After the first residual block in A1 Get intermediate features ;

[0037] Will Input the second residual block in A1 Get intermediate features ;

[0038] Will Input the second residual block in A1 Get Features ,Will Input Inception module I to get the attention score , calculate the attention feature map ,

[0039] ;

[0040] Attention feature map Input the second residual block R2 in A1 to get the feature ,implement The output of attention module A1 is obtained by calculation ;

[0041] b-6) Input the second attention feature extraction module A2 of the global branch;

[0042] After the first residual block in A2 Get intermediate features ;

[0043] Will Input the second residual block in A2 'Get the intermediate features , the intermediate features The second residual block R2 in input A2 obtains the intermediate features ;

[0044] Will Input the second residual block in A2 Get Features ,Will Input Inception module I to get the attention score , calculate the attention feature map ,

[0045] ;

[0046] The obtained attention feature map Input to the second residual block in A2 Get Features ,implement The output of attention module A2 is obtained by calculation ;

[0047] b-7) Input the third attention feature extraction module A3 of the global branch,

[0048] After the first residual block in A3 Get intermediate features ;

[0049] Will Input the second residual block in A3 Get intermediate features ,Will The second residual block R2 in input A2 obtains the feature ;

[0050] Will Input Inception module I to get the attention score , calculate the attention feature map ,

[0051] ;

[0052] Attention feature map Input the second residual block in A3 Get Features ,implement The output of attention module A3 is obtained by calculation ;

[0053] b-8) The final feature obtained by the above attention feature extraction module Through the global average pooling operation, the feature vector V1 is obtained.

[0054] Step c) comprises the following steps:

[0055] c-1) Input the segmented local images of left eye EF, right eye ER, nose N, and mouth M into the local branch feature extraction network respectively;

[0056] c-2) Input the first network layer L1 of the local branch feature extraction network, and the convolution operation obtains the features ;feature Features obtained by batch normalization ,feature The features are obtained by ReLU ,feature Input to the second layer of L1 and get the features through pooling ;

[0057] c-3) Characteristics Input to the second network layer L2 of the local branch feature extraction network, convolution obtains the feature ,feature Features obtained by batch normalization ;feature Input to the second layer of L2 and obtain features through convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ;feature Input to L2, pooling to get features ;

[0058] c-4) Characteristics Input to the third network layer L3 of the local branch feature extraction network, and obtain the features through convolution ,feature Features obtained by batch normalization ,feature Features obtained by convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ,feature Input to the third layer of L3 and get the features through pooling ;

[0059] c-5) Characteristics Input to the fourth network layer L4 of the local branch feature extraction network and obtain the features through convolution ,feature Features obtained by batch normalization ,feature Features obtained by convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ,feature Input to the third layer of L4 and get the features through pooling ;

[0060] c-6) Yes Apply global average pooling to obtain the corresponding feature vector V2.

[0061] Step d) comprises the following steps:

[0062] Connect the eigenvector V1 with the eigenvector V2 to get the eigenvector V, , the fully connected layer maps the connected vector V to another space to obtain a new feature vector :

[0063]

[0064] in is the feature vector output by the fully connected layer, W is the weight matrix, and b is the bias vector.

[0065] After getting the feature vector output by the fully connected layer After that, the original connection vector V is combined with Fusion, the final feature vector is generated by weighted average :

[0066] ,

[0067] in, is the fused feature vector, is the feature vector output by the fully connected layer, V is the original feature vector after connection, is the learnable fusion coefficient, and the feature vector is obtained After that, Send it to the fully connected layer activated by softmax to get the output probability corresponding to each pain intensity level , and perform the following operations:

[0068]

[0069] The output dimension of this fully connected layer is the same as the number of pain intensity levels E, where the item with the highest probability is , , E represents the highest level of pain intensity, the highest probability The corresponding pain intensity level e is the final result of pain expression pain intensity level recognition.

[0070] On the other hand, the present invention also provides a pain expression recognition system based on a dual-branch neural network, comprising the following modules:

[0071] Image processing module: used to pre-process the input image and segment it into four areas;

[0072] Global branch module: used to process the preprocessed image globally to obtain the global feature vector V1;

[0073] Local branch module: used to extract features from the four segmented regions to obtain corresponding feature maps, concatenate the feature maps in the channel dimension, and obtain feature vector V2 through global average pooling of the feature maps;

[0074] Pain expression recognition module: used to concatenate the global feature vector V1 and the feature vector V2, perform feature fusion, and apply softmax for classification to obtain the probability of each pain intensity level, where the level corresponding to the maximum probability is the recognition result.

[0075] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0076] The present invention proposes an attention feature extraction module, which uses the residual structure designed by the present invention to extract image features inside the module, and uses the Inception structure designed by the present invention to calculate and obtain an attention score, so that the network accurately pays attention to the area where the painful movement is generated, while ignoring other irrelevant noises; in addition, a separate network branch is designed to extract features from the pain movement-related area, and finally the features extracted by the attention module are fused with the features extracted directly from the pain-related area to capture more subtle features, which can improve the distinction between adjacent pain intensity levels and greatly improve the accuracy of pain intensity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0078] Figure 1 The network flow chart of the present invention;

[0079] Figure 2 This is a diagram of the attention feature extraction module described in the present invention;

[0080] Figure 3 is the residual block R1 graph of the present invention;

[0081] Figure 4 is the residual block R2 graph of the present invention;

[0082] Figure 5 This is the Inception graph described in the present invention. DETAILED DESCRIPTION

[0083] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0084] Example 1

[0085] A method for pain expression recognition based on a dual-branch neural network comprises the following steps:

[0086] a) Preprocess the input image, segment the original image into four regions: left eye EF, right eye ER, nose N, and mouth M, and adjust the four regions to the same size;

[0087] b) Input the original image into the global branch and obtain the global feature vector V1 after processing;

[0088] c) Input the four segmented regions into the feature extraction network of the local branch in turn to obtain the feature map corresponding to each region, concatenate the feature maps in the channel dimension, and then obtain the feature vector V2 through global average pooling of the feature maps;

[0089] d) Concatenate V1 and V2 to get V, and perform feature fusion through the fully connected layer to get ,right Apply softmax for classification and obtain the probability of each pain intensity level, where the level corresponding to the maximum probability is the recognition result.

[0090] In this embodiment, the global branch includes a simple feature extraction module and an attention feature extraction module; wherein the simple feature extraction module consists of two network layers, the first network layer is a convolution layer, the convolution kernel size is 7, the step size is 2, and the padding is 3; the second network layer is a pooling layer, the convolution kernel size is 3, the step size is 2, and the padding is 1; the attention feature extraction module includes a first residual block R1, a second residual block R2 and an Inception module I, the residual block R1 contains six network layers, the first network layer is a convolution layer, the convolution kernel size is 1, the step size is 1, and the padding is 0, and the first network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to data transformations, and the second is a ReLU activation function number; the second network layer is a convolutional layer, the convolution kernel size is 3×1, the stride is 1, the horizontal padding is 0, and the vertical padding is 1; the third network layer is a convolutional layer, the convolution kernel size is 1×3, the stride is 1, the horizontal padding is 1, and the vertical padding is 0. The third network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, and the second is the ReLU activation function; the fourth network layer is a convolutional layer, the convolution kernel size is 3×1, the stride is 1, the horizontal padding is 0, and the vertical padding is 1; the fifth network layer is a convolutional layer, the convolution kernel size is 1×3, the stride is 1, the horizontal padding is 1, and the vertical padding is 0. The fifth network layer contains two network components, the first is batch normalization, which The model is allowed to learn to adapt to the transformation of the data, and the second is the ReLU activation function; the sixth network layer is a convolutional layer with a convolution kernel size of 1, a step size of 1, and a padding of 1. The sixth convolutional layer is followed by batch normalization, which allows the model to learn to adapt to the transformation of the data; the second residual block R2 contains five network layers, the first network layer is a convolutional layer with a convolution kernel size of 1, a step size of 1, and a padding of 1. The first network layer is followed by two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, and the second is the ReLU activation function; the second network layer is a convolutional layer with a convolution kernel size of 3×1, a step size of 1, a horizontal padding of 0, and a vertical padding of 1; the third network layer is a convolutional layer with a convolution kernel size of 1× 3, the step size is 1, the horizontal padding is 1, and the vertical padding is 0. The third network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, and the second is the ReLU activation function; the fourth network layer is the convolution layer, the convolution kernel size is 3×1, the step size is 1, the horizontal padding is 0, and the vertical padding is 1; the fifth network layer is the convolution layer, the convolution kernel size is 1×3, the step size is 1, the horizontal padding is 1, and the vertical padding is 0. The fifth network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, and the second is the ReLU activation function; Inception module I contains four branches, namely X1, X2, X3, and X4.X1 contains one convolutional layer with a kernel size of 1, a stride of 1, and padding of 0; X2 contains three network layers, the first network layer is a convolutional layer with a kernel size of 1, a stride of 1, and padding of 0, the second network layer is a convolutional layer with a kernel size of 3×1, a stride of 1, horizontal padding of 0, and vertical padding of 1, the third network layer is a convolutional layer with a kernel size of 1×3, a stride of 1, horizontal padding of 1, and vertical padding of 0; X3 contains five network layers, the first network layer is a convolutional layer with a kernel size of 1, a stride of 1, and padding of 0, the second network layer is a convolutional layer with a kernel size of 3×1, The stride is 1, the horizontal padding is 0, and the vertical padding is 1. The third network layer is a convolution layer, the convolution kernel size is 1×3, the stride is 1, the horizontal padding is 1, and the vertical padding is 0. The fourth network layer is a convolution layer, the convolution kernel size is 3×1, the stride is 1, the horizontal padding is 0, and the vertical padding is 1. The fifth network layer is a convolution layer, the convolution kernel size is 1×3, the stride is 1, the horizontal padding is 1, and the vertical padding is 0; X4 contains two network layers, the first network layer is the maximum pooling, the convolution kernel size is 3, the stride is 1, and the padding is 1. The second network layer is a convolution layer, the convolution kernel size is 1, the stride is 1, and the padding is 0. The last four branches are connected in the channel dimension. After the connection, the next layer is a convolution layer, the convolution kernel size is 1, the stride is 1, and the padding is 0. The convolution layer is followed by a softmax operation.

[0091] In this embodiment, the local branch feature extraction network is composed of four network layers L1, L2, L3, and L4. L1 includes two network layers, the first network layer is a convolution layer, the convolution kernel size is 3, the step size is 2, and the padding is 1. The first network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, and the second is the ReLU activation function. The second network layer is a pooling layer with a convolution kernel size of 3, a step size of 1, and a padding of 1; L2 includes three network layers, the first network layer is a convolution layer, the convolution kernel size is 3, the step size is 2, and the padding is 1. The first network layer is connected to the ReLU activation function, the second network layer is a convolution layer, the convolution kernel size is 3, the step size is 1, and the padding is 1. The second network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, the second is the ReLU activation function, the third network layer is a pooling layer, the convolution kernel size is 3, the step size is 1, and the padding is 1; L3 includes three network layers, the first The network layer is a convolutional layer with a convolution kernel size of 3, a step size of 2, and a padding of 1. The first network layer is connected to the ReLU activation function, the second network layer is a convolutional layer with a convolution kernel size of 3, a step size of 1, and a padding of 1. The second network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, the second is the ReLU activation function, and the third network layer is a pooling layer with a convolution kernel size of 3, a step size of 1, and a padding of 1; L4 contains three network layers, the first network layer is a convolutional layer with a convolution kernel size of 3, a step size of 2, and a padding of 1. The first network layer is connected to the ReLU activation function, the second network layer is a convolutional layer with a convolution kernel size of 3, a step size of 1, and a padding of 1. The second network layer contains two network components, the first is batch normalization, which allows the model to learn to adapt to the transformation of the data, the second is the ReLU activation function, which allows the model to learn to adapt to the transformation of the data, the third network layer is a pooling layer with a convolution kernel size of 3, a step size of 1, and a padding of 1.

[0092] In this embodiment, step a) includes the following steps:

[0093] a-1) Use the face detector MTCNN to perform face detection and facial key point detection, and locate the position of each facial key point;

[0094] a-2) According to the detected facial key points, four regions are segmented from the face image, namely: left eye EF, right eye ER, nose N, mouth M, and the size of the cropped area image is adjusted to 112×112.

[0095] In this embodiment, step b) includes the following steps:

[0096] b-1) The original image is extracted by the simple feature extraction module of the global branch to obtain features T1, T2 and T3, and the original image is transmitted to the first network layer to obtain features , and then the feature Transmit to the second network layer to obtain feature F;

[0097] b-2) Input feature T1 into the first network layer of the first residual block R1 to obtain feature , for the features Perform batch normalization operation to obtain features ,feature After ReLU activation function, we get ;

[0098] The characteristics Input the second network layer of the first residual block R1, and the convolution operation obtains the feature ;

[0099] The characteristics Input the third network layer of the first residual block R1, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ;

[0100] The characteristics Input the fourth network layer of the first residual block R1, and the convolution operation obtains the feature ;

[0101] The characteristics Input the fifth network layer of the first residual block R1, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ;

[0102] The characteristics Input the sixth layer network of the first residual block R1 and convolve to obtain features , for the features Perform batch normalization operation to obtain features Finally, the feature Add to input feature T1 and execute Operation, execute to obtain the output features of the first residual block R1 ;

[0103] b-3) Input feature T2 into the first network layer of the second residual block R2, and convolve to obtain feature , for the features Perform batch normalization operation to obtain features ,feature After ReLU activation function, we get ;

[0104] The characteristics Input the second network layer of the second residual block R2, and the convolution operation obtains the feature ;

[0105] The characteristics Input the third network layer of the second residual block R2, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ;

[0106] The characteristics Input the fourth network layer, and the convolution operation obtains the features ;

[0107] The characteristics Input the fifth network layer, and the convolution operation obtains the features , for the features Perform batch normalization to obtain features ,feature The features are obtained by ReLU activation function , the feature Add to input feature T2 and execute Operation, get the output features of the second residual block R2 ;

[0108] b-4) Input feature T3 into the four branches X1, X2, X3, and X4 of Inception module I respectively;

[0109] In X1, feature T3 is convolved to obtain feature ; In X2, feature T3 is input into the first layer network to obtain feature ,feature Input the second layer network and convolve to get features ,feature Input the third layer network and convolve to get features ;

[0110] In X3, feature T3 is input into the first layer network to obtain feature ,feature Input the second layer network and convolve to get features ,feature Input the third layer network and convolve to get features ,feature Input the fourth layer network and convolve to get features ,feature Input the fifth layer network and convolve to get features ;

[0111] In X4, feature T3 is input into the first layer network and the feature is obtained by the maximum pooling operation. ,feature Input the second layer network and obtain the features through 1×1 convolution ;

[0112] In the channel dimension , , , Splice and get features

[0113]

[0114] The characteristics Input 1×1 convolution layer, the number of channels is reduced to One quarter of , and finally Perform the softmax operation. The specific operations are as follows:

[0115]

[0116] The output obtained by the above formula is ;

[0117] b-5) Input feature F into the first attention feature extraction module A1 of the attention mechanism of the global branch;

[0118] After the first residual block in A1 Get intermediate features ;

[0119] Will Input the second residual block in A1 to get the intermediate features ;

[0120] Will Input the second residual block in A1 Get Features ,Will Input Inception module I to get the attention score , calculate the attention feature map ,

[0121] ;

[0122] Attention feature map Input the second residual block R2 in A1 to get the feature ,implement The output of attention module A1 is obtained by calculation ;

[0123] b-6) Input the second attention feature extraction module A2 of the global branch;

[0124] After the first residual block in A2 Get intermediate features ;

[0125] Will Input the second residual block in A2 'Get the intermediate features , the intermediate features The second residual block R2 in input A2 obtains the intermediate features ;

[0126] Will Input the second residual block in A2 Get Features ,Will Input Inception module I to get the attention score , calculate the attention feature map ,

[0127] ;

[0128] The obtained attention feature map Input to the second residual block in A2 Get Features ,implement The output of attention module A2 is obtained by calculation ;

[0129] b-7) Input the third attention feature extraction module A3 of the global branch,

[0130] After the first residual block in A3 Get intermediate features ;

[0131] Will Input the second residual block in A3 Get intermediate features ,Will The second residual block R2 in input A2 obtains the feature ;

[0132] Will Input Inception module I to get the attention score , calculate the attention feature map ,

[0133] ;

[0134] Attention feature map Input the second residual block in A3 Get Features ,implement The output of attention module A3 is obtained by calculation ;

[0135] b-8) The final feature obtained by the above attention feature extraction module Through the global average pooling operation, the feature vector V1 is obtained.

[0136] In this embodiment, step c) includes the following steps:

[0137] c-1) Input the segmented local images of left eye EF, right eye ER, nose N, and mouth M into the local branch feature extraction network respectively;

[0138] c-2) Input the first network layer L1 of the local branch feature extraction network, and the convolution operation obtains the features ;feature Features obtained by batch normalization ,feature The features are obtained by ReLU ,feature Input to the second layer of L1 and get the features through pooling ;

[0139] c-3) Characteristics Input to the second network layer L2 of the local branch feature extraction network, convolution obtains the feature ,feature Features obtained by batch normalization ;feature Input to the second layer of L2 and obtain features through convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ;feature Input to L2, pooling to get features ;

[0140] c-4) Characteristics Input to the third network layer L3 of the local branch feature extraction network, and obtain the features through convolution ,feature Features obtained by batch normalization ,feature Features obtained by convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ,feature Input to the third layer of L3 and get the features through pooling ;

[0141] c-5) Characteristics Input to the fourth network layer L4 of the local branch feature extraction network and obtain the features through convolution ,feature Features obtained by batch normalization ,feature Features obtained by convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ,feature Input to the third layer of L4 and get the features through pooling ;

[0142] c-6) Yes Apply global average pooling to obtain the corresponding feature vector V2.

[0143] In this embodiment, step d) includes the following steps:

[0144] Connect the eigenvector V1 with the eigenvector V2 to get the eigenvector V, , the fully connected layer maps the connected vector V to another space to obtain a new feature vector :

[0145]

[0146] in is the feature vector output by the fully connected layer, W is the weight matrix, and b is the bias vector;

[0147] After getting the feature vector output by the fully connected layer After that, the original connection vector V is combined with Fusion is performed to generate the final feature vector by weighted average :

[0148]

[0149] is the fused feature vector, is the feature vector output by the fully connected layer, V is the original feature vector after connection, Represents the learnable fusion coefficient, which is used to control the weight of each feature vector during fusion to obtain the feature vector After that, Send it to the fully connected layer activated by softmax to get the output probability corresponding to each pain intensity level , and perform the following operations:

[0150]

[0151] The output dimension of this fully connected layer is the same as the number of pain intensity levels E, where the item with the highest probability is , , E represents the highest level of pain intensity, the highest probability The corresponding pain intensity level e is the final result of pain expression pain intensity level recognition.

[0152] Example 2

[0153] A pain expression recognition system based on a dual-branch neural network includes the following modules:

[0154] Image processing module: used to pre-process the input image and segment it into four areas;

[0155] Global branch module: used to process the preprocessed image globally to obtain the global feature vector V1;

[0156] Local branch module: used to extract features from the four segmented regions to obtain corresponding feature maps, concatenate the feature maps in the channel dimension, and obtain feature vector V2 through global average pooling of the feature maps;

[0157] Pain expression recognition module: used to concatenate the global feature vector V1 and the feature vector V2, perform feature fusion, and apply softmax for classification to obtain the probability of each pain intensity level, where the level corresponding to the maximum probability is the recognition result.

[0158] Example 3

[0159] It can be seen from Table 1 that the comparison results of the method in the present invention and other methods in the pain expression recognition task are compared in terms of average accuracy, F1 score, and AUC. Compared with other methods, the present invention proposes the concept of combining local information with global information, which solves the problem that traditional methods fail to make full use of local detail information in the image. In addition, the present invention proposes an attention feature extraction module, which helps the network to better extract global features in the image through the attention mechanism. At the same time, the local branch proposed by the present invention segments the facial features according to the key points of the face, which can more effectively extract the local detail information of the image, further improve the recognition ability of the network and the ability to extract tiny features, thereby improving the recognition accuracy of the network. The method of the present invention shows a high robustness, and can extract important features from images taken by non-professional cameras, keeping the network stable and efficient. The present invention has shown obvious advantages in the task of formulating treatment plans based on analyzing the degree of pain in clinical medicine, and has a wide range of applications in the field of pain intensity recognition by analyzing pain expressions.

[0160] Table 1 Comparison of the results of the method in the present invention with other methods

[0161] .

[0162] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A method for pain expression recognition based on a dual-branch neural network, characterized in that: The steps include: a) Preprocess the input image, segment the original image into four regions: left eye EF, right eye ER, nose N, and mouth M, and adjust the four regions to the same size; b) Input the original image into the global branch and obtain the global feature vector V1 after processing; b-1) The original image is extracted by the simple feature extraction module of the global branch to obtain features T1, T2 and T3, and the original image is transmitted to the first network layer to obtain features , and then the feature Transmit to the second network layer to obtain feature F; b-2) Input features T1, T1 into the first network layer of the first residual block R1 to obtain features , for the features Perform batch normalization operation to obtain features ,feature After ReLU activation function, we get ; The characteristics Input the second network layer of the first residual block R1, and the convolution operation obtains the feature ; The characteristics Input the third network layer of the first residual block R1, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ; The characteristics Input the fourth network layer of the first residual block R1, and the convolution operation obtains the feature ; The characteristics Input the fifth network layer of the first residual block R1, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ; The characteristics Input the sixth layer network of the first residual block R1 and convolve to obtain features , for the features Perform batch normalization operation to obtain features Finally, the feature Add to input feature T1 and execute Operation, execute to obtain the output features of the first residual block R1 ; b-3) Input features T2 and T2 into the first network layer of the second residual block R2, and convolve to obtain features , for the features Perform batch normalization operation to obtain features ,feature After ReLU activation function, we get ; The characteristics Input the second network layer of the second residual block R2, and the convolution operation obtains the feature ; The characteristics Input the third network layer of the second residual block R2, and the convolution operation obtains the feature , for the features Perform batch normalization to obtain features ,feature After ReLU activation function, we get ; The characteristics Input the fourth network layer, and the convolution operation obtains the features ; The characteristics Input the fifth network layer, and the convolution operation obtains the features , for the features Perform batch normalization to obtain features ,feature The features are obtained by ReLU activation function , the feature Add to input feature T2 and execute Operation, get the output features of the second residual block R2 ; b-4) Input features T3, T3 into the four branches X1, X2, X3, and X4 of the Inception module I respectively; In X1, feature T3 is convolved to obtain feature ; In X2, feature T3 is input into the first layer network to obtain feature ,feature Input the second layer network and convolve to get features ,feature Input the third layer network and convolve to get features ; In X3, feature T3 is input into the first layer network to obtain feature ,feature Input the second layer network and convolve to get features ,feature Input the third layer network and convolve to get features ,feature Input the fourth layer network and convolve to get features ,feature Input the fifth layer network and convolve to get features ; In X4, feature T3 is input into the first layer network and the feature is obtained by the maximum pooling operation. ,feature Input the second layer network and obtain the features through 1×1 convolution ; In the channel dimension , , , Splice and get features , The characteristics Input 1×1 convolution layer, the number of channels is reduced to One quarter of , and finally Perform the softmax operation. The specific operations are as follows: , The output obtained by the above formula is ; b-5) Input feature F into the first attention feature extraction module A1 of the attention mechanism of the global branch; After the first residual block in A1 Get intermediate features ; Will Input the second residual block in A1 Get intermediate features ; Will Input the second residual block in A1 Get Features ,Will Input Inception module I to get the attention score , calculate the attention feature map , ; Attention feature map Input the second residual block R2 in A1 to get the feature ,implement The output of attention module A1 is obtained by calculation ; b-6) Input the second attention feature extraction module A2 of the global branch; After the first residual block in A2 Get intermediate features ; Will Input the second residual block in A2 'Get the intermediate features , the intermediate features The second residual block R2 in input A2 obtains the intermediate features ; Will Input the second residual block in A2 Get Features ,Will Input Inception module I to get the attention score , calculate the attention feature map , ; The obtained attention feature map Input to the second residual block in A2 Get Features ,implement The output of attention module A2 is obtained by calculation ; b-7) Input the third attention feature extraction module A3 of the global branch, After the first residual block in A3 Get intermediate features ; Will Input the second residual block in A3 Get intermediate features ,Will The second residual block R2 in input A2 obtains the feature ; Will Input Inception module I to get the attention score , calculate the attention feature map , ; Attention feature map Input the second residual block in A3 Get Features ,implement The output of attention module A3 is obtained by calculation ; b-8) The final feature obtained by the above attention feature extraction module Through the global average pooling operation, the feature vector V1 is obtained; c) Input the four segmented regions into the feature extraction network of the local branch in turn to obtain the feature map corresponding to each region, concatenate the feature maps in the channel dimension, and then obtain the feature vector V2 through global average pooling of the feature maps; c-1) Input the segmented local images of left eye EF, right eye ER, nose N, and mouth M into the local branch feature extraction network respectively; c-2) Input the first network layer L1 of the local branch feature extraction network, and the convolution operation obtains the features ;feature Features obtained by batch normalization ,feature The features are obtained by ReLU ,feature Input to the second layer of L1 and get the features through pooling ; c-3) Characteristics Input to the second network layer L2 of the local branch feature extraction network, convolution obtains the feature ,feature Features obtained by batch normalization ;feature Input to the second layer of L2 and obtain features through convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ;feature Input to L2, pooling to get features ; c-4) Characteristics Input to the third network layer L3 of the local branch feature extraction network, and obtain the features through convolution ,feature Features obtained by batch normalization ,feature Features obtained by convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ,feature Input to the third layer of L3 and get the features through pooling ; c-5) Characteristics Input to the fourth network layer L4 of the local branch feature extraction network, and obtain the feature through convolution ,feature Features obtained by batch normalization ,feature Features obtained by convolution ,feature Features obtained by batch normalization ,feature The features are obtained by ReLU activation function ,feature Input to the third layer of L4 and get the features through pooling ; c-6) Yes Apply global average pooling to obtain the corresponding feature vector V2; d) Concatenate V1 and V2 to get V, and perform feature fusion through the fully connected layer to get ,right Apply softmax for classification and obtain the probability of each pain intensity level, where the level corresponding to the maximum probability is the recognition result.

2. The method for pain expression recognition based on a dual-branch neural network according to claim 1, characterized in that: Step a) comprises the following steps: a-1) Use the face detector MTCNN to perform face detection and facial key point detection, and locate the position of each facial key point; a-2) According to the detected facial key points, four regions are segmented from the face image, namely: left eye EF, right eye ER, nose N, mouth M, and the size of the cropped area image is adjusted to 112×112.

3. The method for pain expression recognition based on a dual-branch neural network according to claim 2, characterized in that: Step d) comprises the following steps: Connect the eigenvector V1 with the eigenvector V2 to get the eigenvector V, , the fully connected layer maps the connected vector V to another space to obtain a new feature vector : in is the feature vector output by the fully connected layer, W is the weight matrix, and b is the bias vector; After getting the feature vector output by the fully connected layer After that, the original connection vector V is combined with Fusion is performed to generate the final feature vector by weighted average : is the fused feature vector, is the feature vector output by the fully connected layer, V is the original feature vector after connection, Represents the learnable fusion coefficient and obtains the feature vector After that, Send it to the fully connected layer activated by softmax to get the output probability corresponding to each pain intensity level , and perform the following operations: The output dimension of this fully connected layer is the same as the number of pain intensity levels E, where the item with the highest probability is , , E represents the highest level of pain intensity, the highest probability The corresponding pain intensity level e is the final result of pain expression pain intensity level recognition.

4. A pain expression recognition system based on a dual-branch neural network, executing a pain expression recognition method based on a dual-branch neural network as claimed in claim 1, characterized in that: include: An image processing module is used to pre-process the input image and segment it into four regions; A global branch module, used for globally branching the preprocessed image to obtain a global feature vector V1; The local branch module is used to extract features from the four segmented regions to obtain corresponding feature maps, concatenate the feature maps in the channel dimension, and obtain feature vector V2 through global average pooling of the feature maps; The pain expression recognition module is used to concatenate the global feature vector V1 and the feature vector V2, perform feature fusion, and apply softmax for classification to obtain the probability of each pain intensity level, where the level corresponding to the maximum probability is the recognition result.