A pain intensity estimation method based on knowledge distillation
By introducing knowledge distillation and attention modules into the pain intensity estimation model, the feature extraction and prediction capabilities of the student network are improved, the problem of difficulty in deploying lightweight equipment is solved, and efficient pain intensity recognition and reduced calculation costs are achieved.
Patent Information
- Application Number
- CN202411248077.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-06
AI Technical Summary
The existing pain intensity estimation model is difficult to deploy on lightweight devices, with high storage and computing costs for model, and insufficient potential representation capabilities, affecting accuracy.
A knowledge distillation-based method is adopted to perform feature extraction and layer knowledge distillation through the attention module of the teacher network and the student network, to improve the feature extraction and prediction capabilities of the student network and reduce the irrelevant noise impact.
Improves the accuracy of pain intensity recognition and generalization capabilities of the model, reduces the storage and computing costs of the model, making it suitable for lightweight device deployment.
Smart Images

Figure CN118781644B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of expression recognition, and in particular relates to a pain intensity estimation method based on knowledge distillation. Background Art
[0002] Current facial pain expression assessment technologies mainly use computer vision and machine learning algorithms to judge the degree of pain by analyzing changes in facial expressions. These technologies have important applications in the fields of medicine, nursing, and rehabilitation, helping medical staff to assess patients' pain conditions more objectively and promptly.
[0003] At present, artificial intelligence algorithms represented by deep learning have achieved outstanding results in the fields of expression classification and expression prediction by relying on various algorithms and powerful computing resources. For the estimation of pain expressions, many scholars have proposed a large number of excellent evaluation algorithms, including feature extraction (such as facial key point positioning), feature representation (such as using convolutional neural networks to extract deep features), and classification and regression models (such as support vector machines and deep neural networks) to predict pain scores. These algorithms can evaluate the intensity of pain according to the patient's facial expression state, which can reduce the workload of medical staff to a certain extent. However, there are still some problems in this field. Training a model with high accuracy often requires a network that is too wide or too deep and a lot of calculations, which greatly limits the deployment of the model on some lightweight devices. Reducing the storage and computing costs of the model is very important for the specific application of pain intensity assessment. Although some lightweight networks can achieve good results, their potential representation capabilities still need to be developed. For this reason, in-depth research on representation capabilities while compressing models is a necessary development. Summary of the invention
[0004] In order to further improve the accuracy of the model in identifying pain intensity while compressing the model, the present invention provides a pain intensity estimation method based on knowledge distillation.
[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:
[0006] In a first aspect, the present invention provides a method for estimating pain intensity based on knowledge distillation, comprising the following steps:
[0007] Acquire an original image sequence, preprocess the original image sequence to obtain a facial image sequence, and divide the preprocessed facial image sequence into a training set and a test set;
[0008] A pain intensity estimation model based on knowledge distillation is constructed, wherein the model includes a teacher network and a student network, wherein the teacher network is a ResNet50 network architecture with an integrated attention module, and the student network is a ResNet18 network architecture with an integrated attention module, wherein the teacher network and the student network each have five network layers, wherein the first three network layers of the teacher network and the student network are each followed by an attention module att1, and the last two network layers of the teacher network and the student network are each followed by an attention module att2, and the facial image sequences in the training set are respectively input into the teacher network and the student network for image analysis;
[0009] Performing knowledge distillation between the layers of the teacher network and the student network to obtain a layer distillation loss of each network layer;
[0010] The final output results of the teacher network and the student network are subjected to knowledge distillation to obtain the total loss of the output result distillation;
[0011] The layer distillation loss and the total distillation loss of the output result are added together to obtain the total model loss. The final pain intensity estimation model based on knowledge distillation is trained according to the total model loss. The images to be tested in the test set are predicted by the final pain intensity estimation model based on knowledge distillation to obtain the pain intensity estimation result.
[0012] Furthermore, the preprocessing of the original image sequence specifically includes:
[0013] For the input original image sequence , use yolov5-face network to detect facial key points and locate the face position, Represents the nth image in the original image sequence;
[0014] The facial image is cropped according to the located facial position, and the size of the cropped facial image is resized to Size, get the facial image sequence , Represents the nth face image in the face image sequence.
[0015] Furthermore, the attention module att1 includes a first network layer, a second network layer and a third network layer:
[0016] The first network layer includes: a convolution layer, batch normalization and a ReLU activation function, wherein the convolution kernel size of the convolution layer is 3×3, the step size is 1, and the padding is 1;
[0017] The second network layer includes two branches: branch Y1 and branch Y2, wherein branch Y1 includes in sequence: the first convolution layer, batch normalization, ReLU activation function, the second convolution layer, batch normalization, ReLU activation function and the third convolution layer, and branch Y2 includes a convolution layer; the convolution kernel size of the first convolution layer and the second convolution layer is 3×3, the step size is 1, and the padding is 1, and the convolution kernel size of the third convolution layer and the convolution layer in Y2 is 1×1, the step size is 1, and the padding is 0;
[0018] The third network layer includes: a convolution layer and a softmax operation, wherein the convolution kernel size of the convolution layer is 3×3, the step size is 1, and the padding is 1;
[0019] The images in the facial image sequence are sequentially passed through the first three attention modules att1 in the teacher network to obtain the first teacher feature 2. Characteristics of the Second Teacher and third teacher characteristics , after the first three attention modules att1 in the student network, the first student features are obtained in turn 2. Student Characteristics and third student characteristics .
[0020] Furthermore, the attention module att2 is composed of two branches and two network layers, specifically including:
[0021] The two branches are a spatial attention branch YS and a channel attention branch YA, the spatial attention branch YS includes a convolution layer, batch normalization, a ReLU activation function and a softmax operation, the channel attention branch YA includes a convolution layer and a global average pooling layer gap, and the two network layers are two convolution layers; the convolution kernel size of the convolution layer in YS and YA is 3×3, the step size is 1, and the padding is 1, and the convolution kernel size of the two network layers is 1×1, the step size is 1, and the padding is 0;
[0022] The outputs of the two attention modules att2 after the teacher network are respectively the fourth teacher feature and the fifth teacher characteristic The outputs of the last two attention modules att2 of the student network are the fourth student features and the fifth student characteristics .
[0023] Furthermore, obtaining the layer distillation loss of each network layer specifically includes:
[0024] Calculate the layer distillation loss of the first network layer of the teacher network and the student network: convert the first student feature After a convolution layer with a convolution kernel size of 1×1, a stride of 1, and padding of 0, it is upsampled and adjusted to the first teacher feature Same size, adjusted The feature representation ;calculate and The mean square error loss , the calculation formula is as follows:
[0025]
[0026]
[0027] in is the feature map height, is the feature map width, is the number of feature map channels, is the multiplication operation of the three, h, w, c represent the positions in the corresponding dimensions respectively, (h, w, c) represents the feature points on the feature map, To calculate feature points, calculate and The contrast loss , the calculation formula is as follows:
[0028] ,
[0029] in is the number of all locations in the feature map, The window size used to measure the similarity of features by comparing local areas of feature maps, u represents a specific position, v represents the position within the local window, and and The mean square error loss and contrast loss are weightedly added to get the first network layer distillation loss , the calculation formula is as follows:
[0030]
[0031] in, is the weighted coefficient of the mean square error loss in the first network layer, is the weighting coefficient of the contrast loss in the first network layer;
[0032] Similarly, the layer distillation loss of the second network layer of the teacher network and the student network is: , represents the mean squared error loss of the second network layer, represents the contrastive loss of the second network layer, is the weighted coefficient of the mean square error loss in the second network layer, is the weighting coefficient of the contrast loss in the second network layer;
[0033] The layer distillation loss of the third network layer of the teacher network and the student network is: , represents the mean square error loss of the third network layer, represents the contrast loss of the third network layer, is the weighted coefficient of the mean square error loss in the third network layer, is the weighted coefficient of the contrast loss in the third network layer;
[0034] The layer distillation loss of the fourth network layer of the teacher network and the student network is: , represents the mean square error loss of the fourth network layer, represents the contrast loss of the fourth network layer, is the weighted coefficient of the mean square error loss in the fourth network layer, is the weighted coefficient of the contrast loss in the fourth network layer;
[0035] The layer distillation loss of the fifth network layer of the teacher network and the student network is: , represents the mean square error loss of the fifth network layer, represents the contrast loss of the fifth network layer, is the weighted coefficient of the mean square error loss in the fifth network layer, is the weighting coefficient of the contrast loss in the fifth network layer.
[0036] Furthermore, the output result of the total distillation loss specifically includes:
[0037] The fifth teacher characteristics The final prediction result is obtained through the fully connected layer in the teacher network , the fifth student characteristic The final prediction result is obtained through the fully connected layer of the student network ; Calculate the soft labels of the final prediction results of the teacher network and obtain the soft label distribution of each category of the teacher network , the calculation formula is as follows:
[0038] ,
[0039] in, represents the output score of the teacher network on the i-th category, T represents the distillation temperature, which is used to control the smoothness of softmax. It means to divide the output score of the teacher network on the i-th category by the temperature parameter and take the exponential, where , j is the total number of categories, Indicates the sum of the output scores of all categories in the teacher network after the same processing; calculate the soft label of the final prediction result of the student network, and obtain the soft label distribution of each category of the student network , the calculation formula is as follows:
[0040] ,
[0041] in, represents the output score of the student network on the i-th category, It means that the output score of the student network on the i-th category is divided by the temperature parameter and then taken as the exponent. It means the output scores of all categories in the student network are summed after the same processing;
[0042] The distillation loss between the soft label of the final prediction result of the teacher network and the soft label of the final prediction result of the student network is calculated as follows:
[0043] ,
[0044] in, It represents the sum of all categories and calculates the cross entropy loss between the final prediction result output of the student network and the true label. The calculation formula is as follows:
[0045] ,
[0046] in, represents the label of the true label on the i-th category, represents the softmax probability of the student network on the i-th category, It means summing all categories;
[0047] According to the distillation loss between the soft label of the final prediction result of the teacher network and the soft label of the final prediction result of the student network and the cross entropy loss between the final prediction result output of the student network and the true label, the total loss of the output result distillation is obtained, and the calculation formula is as follows:
[0048] ,
[0049] Among them, λ is a hyperparameter used to balance the weights of distillation loss and cross entropy loss.
[0050] Furthermore, the total loss of the model is: .
[0051] In a second aspect, the present invention provides a system for applying a pain intensity estimation method based on knowledge distillation, comprising:
[0052] Image acquisition module: used to acquire image sequences, preprocess the input image sequences, and obtain preprocessed facial image sequences;
[0053] Model building module: used to build a pain intensity estimation model based on knowledge distillation, the model includes a teacher network and a student network, and the preprocessed image sequence is input into the teacher network and the student network respectively for feature extraction;
[0054] Layer knowledge distillation module: used to perform knowledge distillation between the layers of the teacher network and the student network to obtain the layer distillation loss of each network layer;
[0055] Output result knowledge distillation module: used to perform knowledge distillation on the final output results of the teacher network and the student network to obtain the total loss of output result distillation, and add the layer distillation loss and the total loss of output result distillation to obtain the total loss of the model;
[0056] Pain intensity estimation module: used to obtain the final pain intensity estimation model based on knowledge distillation according to the total loss training of the model. The images to be tested in the test set are predicted by the final pain intensity estimation model based on knowledge distillation to obtain the pain intensity estimation result.
[0057] In a third aspect, the present invention provides a pain intensity estimation device based on knowledge distillation, comprising a processor and a storage medium:
[0058] The storage medium is used to store instructions;
[0059] The processor is used to operate according to the instruction to execute the pain intensity estimation method based on knowledge distillation described in the first aspect.
[0060] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed to implement a pain intensity estimation method based on knowledge distillation as described in the first aspect.
[0061] The advantages of the present invention are:
[0062] The present invention proposes a novel attention module and knowledge distillation method, which embeds different attention modules between the network layers of the teacher network and the student network, greatly reducing the influence of irrelevant noise and making the network pay attention to the key area where facial pain expression is produced; at the same time, layer knowledge distillation is performed between each network layer, so that the features extracted by the student network are close to the teacher network, and through the influence of the teacher network, the feature extraction ability of the student network is improved, and the student network learns the rich knowledge of the teacher network, and performs soft label distillation of the final output result, so that the student network learns more comprehensive knowledge about the prediction results through the output of the teacher network, further improving the prediction ability of the student network, thereby improving the student network's ability to distinguish different pain intensity levels, and then improving the accuracy of pain intensity recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0064] Figure 1 is a network flow chart of the present invention;
[0065] Figure 2 This is the attention module att1 diagram of the present invention;
[0066] Figure 3 This is the attention module att2 diagram of the present invention;
[0067] Figure 4 It is a visualization effect diagram of the experimental results of the present invention. DETAILED DESCRIPTION
[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0069] Example 1
[0070] In this embodiment, Figure 1 As shown, the present invention provides a pain intensity estimation method based on knowledge distillation, and the specific steps include:
[0071] a) obtaining an image sequence, preprocessing the input image sequence to obtain a preprocessed facial image sequence, and dividing the preprocessed facial image sequence into a training set and a test set;
[0072] a-1) For the input image sequence , use yolov5-face network to detect facial key points and locate the face position.
[0073] a-2) Crop the facial images according to the positioning, and resize all the cropped facial images to Size, get the preprocessed face image sequence .
[0074] b) constructing a pain intensity estimation model based on knowledge distillation, the model comprising a teacher network and a student network, the teacher network being a ResNet50 convolutional neural network architecture integrated with an attention module, the student network being a ResNet18 convolutional neural network architecture integrated with an attention module, the teacher network and the student network each having five network layers, the first three network layers of the teacher network and the student network each being followed by an attention module att1, the last two network layers of the teacher network and the student network each being followed by an attention module att2, the preprocessed image sequence being respectively input into the teacher network and the student network for feature extraction;
[0075] b-1) The teacher network is a ResNet50 network that integrates the attention modules att1 and att2, and the student network is a ResNet18 network that integrates the attention modules att1 and att2.
[0076] b-2) In this embodiment, if Figure 2 As shown in the figure, the attention module att1 consists of three network layers. The first network layer is a convolutional layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1. It contains two network components, the first is batch normalization, and the second is the ReLU activation function; the second network layer contains two branches Y1 and Y2. Branch Y1 contains three convolutional layers. The convolution kernel size of the first convolutional layer is 3×3, the step size is 1, and the padding is 1. It contains two network components, the first is batch normalization, and the second is the ReLU activation function. ; The convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the padding is 1. It contains two network components, the first is batch normalization, and the second is the ReLU activation function; the convolution kernel size of the third convolution layer is 1×1, the stride is 1, and the padding is 0; branch Y2 contains a convolution layer, the convolution kernel size of this convolution layer is 1×1, the stride is 1, and the padding is 0; the third network layer is a convolution layer, the convolution kernel size is 3×3, the stride is 1, and the padding is 1. This convolution layer is connected to a softmax operation. Image sequence When inputting the teacher network, it first passes through the first network layer of the teacher network ResNet50 to obtain the feature Next, the features Input attention module att1, in att1, first pass through the first network layer to get the features ,feature Input the second network layer, in branch Y1, the features are obtained through the first convolution layer ,feature The features are obtained through the second convolution layer of Y1 ,feature The features are obtained through the third convolution layer of Y1 , where the characteristics The size of H and W remains unchanged, and the number of channels becomes the characteristic of ; In branch Y2, the feature Features obtained by convolution ,feature The size of H and W remains unchanged, and the number of channels becomes the characteristic of ; The feature and Features Splicing is performed in the channel dimension,
[0077]
[0078] Get the features after splicing ;feature The features are obtained through the convolution layer of the third network layer ,feature The attention map is obtained by softmax operation ,
[0079]
[0080] Then the input features of this module With attention map Multiply,
[0081]
[0082] Get the attention feature map .
[0083] Input the second network layer of the teacher network ResNet50 to get the features ,feature After the attention module att1 behind the second network layer of the teacher network, the feature ; Input the third network layer of the teacher network ResNet50 to get the feature ,feature After the attention module att1 behind the third network layer of the teacher network, the feature .
[0084] In the image sequence While inputting the teacher network, the image sequence Enter the student network. In the student network, first pass through the first network layer of the student network ResNet18 to obtain the feature ,feature The features are obtained by the attention module att1 after the first network layer of the student network ,feature After the second network layer of the student network ResNet18, the feature ,feature The features are obtained by the attention module att1 after the second network layer of the student network ,feature After the third network layer of the student network ResNet18, the feature ,feature The features are obtained by the attention module att1 after the third network layer of the student network .
[0085] b-3) In this embodiment, if Figure 3 As shown, the attention module att2 includes two branches, namely the spatial attention branch YS and the channel attention branch YC. In the spatial attention branch YS, it contains a convolution layer with a convolution kernel size of 3×3, a stride of 1, and a padding of 1. It is followed by two network components. The first is batch normalization, and the second is the ReLU activation function. The ReLU activation function is followed by a softmax operation; in the channel attention branch YA, the first network layer is a convolution layer with a convolution kernel size of 3×3, a stride of 1, and a padding of 1. The second network layer is the global average pooling layer gap. The first network layer outside the two branches is a convolution layer with a convolution kernel size of 1×1, a stride of 1, and a padding of 0; the second network layer outside the two branches is a convolution layer with a convolution kernel size of 1×1, a stride of 1, and a padding of 0. The features obtained by the attention module att1 after the third network layer of the teacher network mentioned in b-2) , after passing through the fourth network layer of the teacher network ResNet50, the feature ,feature Input the attention module att2 after the fourth network layer of the teacher network, and pass through the convolution layer in the spatial attention branch YS to obtain the feature ,right Perform the softmax operation,
[0086]
[0087] Get the spatial attention map ,right and Multiply them to get the spatial attention feature map ,
[0088]
[0089] The spatial attention feature map With features Add,
[0090]
[0091] Get Features .
[0092] feature After the first network layer in the channel attention branch YA, the feature , for the features Perform the global average pooling operation of the second network layer of the channel attention branch,
[0093]
[0094] Get channel attention map ,right and Multiply them to get the channel attention feature map ,
[0095]
[0096] The channel attention feature map With features Add,
[0097]
[0098] Get Features .
[0099] The characteristics With features Perform weighted fusion to obtain features The calculation formula is as follows:
[0100]
[0101] in, =1, and , Both are greater than 0.
[0102] The characteristics With features Concatenate in the channel dimension,
[0103]
[0104] Get Features ,feature After the first network layer outside the two branches, the feature ,in The H and W of the channel remain unchanged, and the number of channels becomes the number of its input .
[0105] The characteristics With features Concatenate on the channel dimension to get features ,
[0106]
[0107] feature After the second network layer outside the two branches, the feature ,in The H and W of the channel remain unchanged, and the number of channels becomes the number of its input .feature After the fifth network layer of the teacher network ResNet50, the feature ,feature Input the attention module att2 after the fifth network layer of the teacher network to get the feature .
[0108] The features obtained by the attention module att1 after the third network layer of the student network mentioned in b-2) , after passing through the fourth network layer of the student network ResNet18, the feature ,feature Input the attention module att2 after the fourth network layer of the student network to get the feature ,feature After the fifth network layer of the student network ResNet18, the feature ,feature Input the attention module att2 after the fifth network layer of the student network to get the feature .
[0109] c) performing knowledge distillation between the layers of the teacher network and the student network to obtain a layer distillation loss of each network layer;
[0110] c-1) The output features of the attention module att1 after the first network layer of the teacher network ResNet50 are features , the output feature of the attention module att1 behind the first network layer of the student network ResNet18 is the feature , first feature The features are transformed through a convolution layer with a kernel size of 1×1, a stride of 1, and a padding of 0. The number of channels is adjusted to the characteristic The number of channels is increased, and the output features of the student network are converted to The size is adjusted to match the output features of the teacher network Same height and width, adjusted It can be expressed as ,calculate and When the distillation loss between them is calculated, the mean square error loss MSE of the two is first calculated.
[0111]
[0112]
[0113] Get the MSE loss ,in is the feature map height, is the feature map width, is the number of feature map channels, is the multiplication operation of the three, where h, w, and c represent the positions in the corresponding dimensions respectively, and (h, w, c) represents the feature points on the feature map.
[0114] It is calculated for each feature point; then the contrast loss between the two is calculated.
[0115]
[0116] Get the contrast loss ,in is the number of all locations in the feature map, The window size used to compare local areas of feature maps to measure feature similarity, u represents a specific position, and v represents the position within the local window.
[0117] The final distillation loss function is:
[0118]
[0119] in , It is a weighting factor used to balance the impact of various losses.
[0120] c-2) The output features of the attention module att1 after the second network layer of the teacher network ResNet50 are features , the output feature of the attention module att1 behind the second network layer of the student network ResNet18 is the feature , first feature The features are transformed through a convolution layer with a kernel size of 1×1, a stride of 1, and a padding of 0. The number of channels is adjusted to the characteristic The number of channels is increased, and the output features of the student network are converted to The size is adjusted to match the output features of the teacher network Same height and width, adjusted It can be expressed as ,calculate and When the distillation loss between them is calculated, the mean square error loss MSE of the two is first calculated.
[0121]
[0122]
[0123] Get the MSE loss ,in is the feature map height, is the feature map width, is the number of feature map channels, It is a multiplication operation of the three, where h, w, and c represent the positions in the corresponding dimensions respectively, and (h, w, c) represents the feature points on the feature map. Then the contrast loss of the two is calculated.
[0124]
[0125] Get the contrast loss ,in is the number of all locations in the feature map, The window size used to compare local areas of feature maps to measure feature similarity, u represents a specific position, and v represents the position within the local window.
[0126] The final distillation loss function is:
[0127]
[0128] in , It is a weighting factor used to balance the impact of various losses.
[0129] c-3) The output features of the attention module att1 after the third network layer of the teacher network ResNet50 are features , the output features of the attention module att1 behind the third network layer of the student network ResNet18 are features , first feature The features are transformed through a convolution layer with a kernel size of 1×1, a stride of 1, and a padding of 0. The number of channels is adjusted to the characteristic The number of channels is increased, and the output features of the student network are converted to The size is adjusted to match the output features of the teacher network Same height and width, adjusted It can be expressed as ,calculate and When the distillation loss between them is calculated, the mean square error loss MSE of the two is first calculated.
[0130]
[0131]
[0132] Get the MSE loss ,in is the feature map height, is the feature map width, is the number of feature map channels, It is a multiplication operation of the three, where h, w, and c represent the positions in the corresponding dimensions respectively, and (h, w, c) represents the feature points on the feature map. Then the contrast loss of the two is calculated.
[0133]
[0134] Get the contrast loss ,in is the number of all locations in the feature map, The window size used to compare local areas of feature maps to measure feature similarity, u represents a specific position, and v represents the position within the local window.
[0135] The final distillation loss function is:
[0136]
[0137] in , It is a weighting factor used to balance the impact of various losses.
[0138] c-4) The output features of the attention module att2 after the fourth network layer of the teacher network ResNet50 are features , the output feature of the attention module att2 behind the fourth network layer of the student network ResNet18 is the feature , first feature The features are transformed through a convolution layer with a kernel size of 1×1, a stride of 1, and a padding of 0. The number of channels is adjusted to the characteristic The number of channels is increased, and the output features of the student network are converted to The size is adjusted to match the output features of the teacher network Same height and width, adjusted It can be expressed as ,calculate and When the distillation loss between them is calculated, the mean square error loss MSE of the two is first calculated.
[0139]
[0140]
[0141] Get the MSE loss ,in is the feature map height, is the feature map width, is the number of feature map channels, It is a multiplication operation of the three, where h, w, and c represent the positions in the corresponding dimensions respectively, and (h, w, c) represents the feature points on the feature map. Then the contrast loss of the two is calculated.
[0142]
[0143] Get the contrast loss ,in is the number of all locations in the feature map, The window size used to compare local areas of feature maps to measure feature similarity, u represents a specific position, and v represents the position within the local window.
[0144] The final distillation loss function is:
[0145]
[0146] in , It is a weighting factor used to balance the impact of various losses.
[0147] c-5) The output features of the attention module att2 after the fifth network layer of the teacher network ResNet50 are features , the output feature of the attention module att2 behind the fifth network layer of the student network ResNet18 is the feature , first feature The features are transformed through a convolution layer with a kernel size of 1×1, a stride of 1, and a padding of 0. The number of channels is adjusted to the characteristic The number of channels is increased, and the output features of the student network are converted to The size is adjusted to match the output features of the teacher network Same height and width, adjusted It can be expressed as ,calculate and When the distillation loss between them is calculated, the mean square error loss MSE of the two is first calculated.
[0148]
[0149]
[0150] Get the MSE loss ,in is the feature map height, is the feature map width, is the number of feature map channels, It is a multiplication operation of the three, where h, w, and c represent the positions in the corresponding dimensions respectively, and (h, w, c) represents the feature points on the feature map. Then the contrast loss of the two is calculated.
[0151]
[0152] Get the contrast loss ,in is the number of all locations in the feature map, The window size used to compare local areas of feature maps to measure feature similarity, u represents a specific position, and v represents the position within the local window.
[0153] The final distillation loss function is:
[0154]
[0155] in , It is a weighting factor used to balance the impact of various losses.
[0156] d) performing knowledge distillation on the final output results of the teacher network and the student network to obtain the total loss of output result distillation, and adding the layer distillation loss and the total loss of output result distillation to obtain the total loss of the model;
[0157] d-1) The output of the attention module att2 after the fifth network layer of the teacher network ResNet50 is , and then the final prediction result is obtained through the subsequent fully connected layer ; The output of the attention module att2 after the fifth network layer of the student network ResNet18 is , and then the final prediction result is obtained through the subsequent fully connected layer .
[0158] The final prediction result of the teacher network And the final prediction results of the student network To perform knowledge distillation, first calculate the soft labels of the final prediction results of the teacher network.
[0159]
[0160] Get the soft label distribution for each category ,in represents the output score of the teacher network on the i-th category, T represents the distillation temperature, which is used to control the smoothness of softmax. It means to divide the output score of the teacher network on the i-th category by the temperature parameter and take the exponential, where , j is the total number of categories, It means that the output scores of all categories are summed after the same processing, which is used to normalize the probability distribution so that the sum of the probabilities of all categories is 1.
[0161] Then calculate the soft label of the final prediction result of the student network,
[0162]
[0163] Get the soft label distribution for each category ,in represents the output score of the student network on the i-th category, T represents the distillation temperature, which is used to control the smoothness of softmax. It means that the output score of the student network on the i-th category is divided by the temperature parameter and then taken as the exponent, where , j is the total number of categories, It means that the output scores of all categories are summed after the same processing, which is used to normalize the probability distribution so that the sum of the probabilities of all categories is 1.
[0164] d-2) Calculate the distillation loss between the soft labels of the teacher network’s final predictions and the soft labels of the student network’s final predictions,
[0165]
[0166] in It represents the sum of all categories.
[0167] Then calculate the cross entropy loss between the final prediction output of the student network and the true label,
[0168]
[0169] in represents the label of the true label on the i-th category, represents the softmax probability of the student network on the i-th category, It represents the sum of all categories.
[0170] d-3) Combining the distillation loss and the cross entropy loss, we can get the total loss function for distilling the output results:
[0171]
[0172] Among them, λ is a hyperparameter used to balance the weights of distillation loss and cross entropy loss.
[0173] d-4) The distillation loss of the entire model is the sum of the layer distillation loss and the output distillation loss, which is:
[0174]
[0175] L is the total loss of the model.
[0176] e) The final pain intensity estimation model based on knowledge distillation is obtained according to the total loss training of the model. The image to be detected is predicted by the final pain intensity estimation model based on knowledge distillation to obtain the pain intensity estimation result.
[0177] Example 2
[0178] Table 1 Comparison of the results of the method of the present invention with other existing pain intensity estimation methods
[0179]
[0180] In this embodiment, Table 1 is a comparison of the results of the method in the present invention with other methods. When we experimentally compare the method of the present invention with other existing pain intensity estimation methods, we compare the mean absolute error MAE and the mean square error MSE. These two indicators are the most commonly used evaluation indicators in the field of pain intensity assessment. The lower the values of these two indicators, the better. The method of the present invention achieves the best results in the comparison of these two indicators. Compared with other methods, the present invention proposes the concept of attention knowledge distillation, which solves the problem of deployment difficulties on some lightweight devices. In addition, the present invention proposes an attention module, which helps the network to focus on important information and suppress unnecessary information through the attention mechanism, thereby improving the evaluation ability of the model. The proposed attention mechanism is concentrated in ResNet50 and ResNet18 respectively, so that the model can focus on the key parts of the extracted features while performing knowledge distillation, which helps to improve the efficiency of knowledge distillation. The present invention shows a strong generalization ability and can evaluate pain intensity well. The present invention has a wide range of applications in clinical medicine, can reduce the workload of medical staff, and helps medical staff to formulate treatment plans as needed according to the patient's pain state.
[0181] In this embodiment, Figure 4 The visualization effect of the experimental results of the present invention. The pain intensity score PSPI is divided into four levels: level 0 represents no pain. Level 1, 0-3 points, mild pain. Level 2, 4-6 points, moderate pain. Level 3, 7-10 points, severe pain. It can be seen that the method of the present invention can predict the pain intensity score PSPI based on facial expressions.
[0182] Example 3
[0183] In this embodiment, a system for applying a pain intensity estimation method based on knowledge distillation is provided, including:
[0184] Image acquisition module: used to acquire image sequences, preprocess the input image sequences, and obtain preprocessed facial image sequences;
[0185] Model building module: used to build a pain intensity estimation model based on knowledge distillation, the model includes a teacher network and a student network, and the preprocessed image sequence is input into the teacher network and the student network respectively for feature extraction;
[0186] Layer knowledge distillation module: used to perform knowledge distillation between the layers of the teacher network and the student network to obtain the layer distillation loss of each network layer;
[0187] Output result knowledge distillation module: used to perform knowledge distillation on the final output results of the teacher network and the student network to obtain the total loss of output result distillation, and add the layer distillation loss and the total loss of output result distillation to obtain the total loss of the model;
[0188] Pain intensity estimation module: used to obtain the final pain intensity estimation model based on knowledge distillation according to the total loss training of the model. The images to be tested in the test set are predicted by the final pain intensity estimation model based on knowledge distillation to obtain the pain intensity estimation result.
[0189] Example 4
[0190] In this embodiment, a pain intensity estimation device based on knowledge distillation is provided, including a processor and a storage medium:
[0191] The storage medium is used to store instructions;
[0192] The processor is used to operate according to the instruction to execute the pain intensity estimation method based on knowledge distillation described in the first aspect.
[0193] Example 5
[0194] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. The computer program can be executed to implement the pain intensity estimation method based on knowledge distillation described in the first aspect.
[0195] Example 6
[0196] In this embodiment, a specific application of lightweight deployment is provided:
[0197] 1. Portable pain monitoring devices: Portable pain intensity monitoring devices can be used to monitor the pain levels of patients with chronic diseases for a long time, such as patients recovering from surgery or elderly people who are bedridden for a long time. By embedding the lightweight pain intensity recognition model after knowledge distillation into such devices, medical staff can remotely obtain the patient's real-time pain data and adjust the treatment plan based on this data, reducing the need for frequent face-to-face consultations.
[0198] 2. Mobile medical devices: In mobile medical care, lightweight pain intensity recognition models can be deployed on smartphones or tablets, allowing doctors to remotely assess the patient's current condition. Patients can use their mobile phones to take facial videos or photos, and the model in the application will analyze the pain level in real time and send the results to the doctor. The doctor can flexibly analyze the patient's current condition based on the patient's pain status and formulate corresponding medical strategies. This is of great value for remote areas or areas with limited resources, and can significantly improve the accessibility of medical resources.
[0199] 3. Edge computing devices: In the emergency room or ambulance, a lightweight pain intensity recognition model can be deployed on an edge computing device to analyze the patient's pain condition in real time and help medical staff make treatment decisions in the first place. This instant analysis can improve the efficiency of emergency treatment and provide important reference data for subsequent treatment.
[0200] In the above application, the application of the present invention mainly includes the following steps:
[0201] 1. Synchronous training of the teacher model and the student model: The teacher model and the student model of the present invention are trained simultaneously on a high-performance server. The student model continuously learns key features and classification information from the teacher model to achieve similar or even better recognition accuracy.
[0202] 2. Real-time model compression: compress the parameters of the student model in real time during the training process to adapt it to the resource constraints of lightweight devices. This method does not require the teacher model to be trained in advance and is more suitable for dynamically adjusting the model.
[0203] 3. Lightweight deployment: Deploy the student model after knowledge distillation and compression to the target lightweight device, such as portable pain monitoring device, mobile medical device or edge computing device, to achieve efficient pain expression recognition function.
[0204] Through this method, the size of the model and the computing resource requirements can be greatly reduced while maintaining the model's recognition capabilities, truly achieving efficient deployment on lightweight devices.
[0205] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A pain intensity estimation method based on knowledge distillation, characterized in that: The following steps are involved: S1. Obtaining an original image sequence, and preprocessing the original image sequence to obtain a facial image sequence, and dividing the preprocessed facial image sequence into a training set and a test set; S2. Construct a pain intensity estimation model based on knowledge distillation, the model comprising a teacher network and a student network, the teacher network is a ResNet50 network architecture with an integrated attention module, the student network is a ResNet18 network architecture with an integrated attention module, the teacher network and the student network each have five network layers, the first three network layers of the teacher network and the student network are each followed by an attention module att1, and the last two network layers of the teacher network and the student network are each followed by an attention module att2, and the facial image sequences in the training set are respectively input into the teacher network and the student network for image analysis; The attention module att1 includes a first network layer, a second network layer and a third network layer: The first network layer includes: a convolution layer, batch normalization and a ReLU activation function, wherein the convolution kernel size of the convolution layer is 3×3, the step size is 1, and the padding is 1; The second network layer includes two branches: branch Y1 and branch Y2, wherein branch Y1 includes in sequence: the first convolution layer, batch normalization, ReLU activation function, the second convolution layer, batch normalization, ReLU activation function and the third convolution layer, and branch Y2 includes a convolution layer; the convolution kernel size of the first convolution layer and the second convolution layer is 3×3, the step size is 1, and the padding is 1, and the convolution kernel size of the third convolution layer and the convolution layer in Y2 is 1×1, the step size is 1, and the padding is 0; The third network layer includes: a convolution layer and a softmax operation, wherein the convolution kernel size of the convolution layer is 3×3, the step size is 1, and the padding is 1; The images in the facial image sequence are sequentially passed through the first three attention modules att1 in the teacher network to obtain the first teacher feature 2. Characteristics of the Second Teacher and third teacher characteristics , after the first three attention modules att1 in the student network, the first student features are obtained in turn 2. Student Characteristics and third student characteristics ; The attention module att2 consists of two branches and two network layers, specifically including: The two branches are a spatial attention branch YS and a channel attention branch YA, the spatial attention branch YS includes a convolution layer, batch normalization, a ReLU activation function and a softmax operation, the channel attention branch YA includes a convolution layer and a global average pooling layer gap, and the two network layers are two convolution layers; the convolution kernel size of the convolution layer in YS and YA is 3×3, the step size is 1, and the padding is 1, and the convolution kernel size of the two network layers is 1×1, the step size is 1, and the padding is 0; The outputs of the two attention modules att2 after the teacher network are respectively the fourth teacher feature and the fifth teacher characteristic The outputs of the last two attention modules att2 of the student network are the fourth student features and the fifth student characteristics ; S3. performing knowledge distillation between the layers of the teacher network and the student network to obtain a layer distillation loss of each network layer; The layer distillation loss of each network layer is obtained, specifically including: Calculate the layer distillation loss of the first network layer of the teacher network and the student network: convert the first student feature After a convolution layer with a convolution kernel size of 1×1, a stride of 1, and padding of 0, it is upsampled and adjusted to the first teacher feature Same size, adjusted The feature representation ;calculate and The mean square error loss , the calculation formula is as follows: in is the feature map height, is the feature map width, is the number of feature map channels, is the multiplication operation of the three, h, w, c represent the positions in the corresponding dimensions respectively, (h, w, c) represents the feature points on the feature map, To calculate feature points, calculate and The contrast loss , the calculation formula is as follows: , in is the number of all locations in the feature map, The window size used to measure the similarity of features by comparing local areas of feature maps, u represents a specific position, v represents the position within the local window, and and The mean square error loss and contrast loss are weightedly added to get the first network layer distillation loss , the calculation formula is as follows: in, is the weighted coefficient of the mean square error loss in the first network layer, is the weighting coefficient of the contrast loss in the first network layer; Similarly, the layer distillation loss of the second network layer of the teacher network and the student network is: , represents the mean squared error loss of the second network layer, represents the contrast loss of the second network layer, is the weighted coefficient of the mean square error loss in the second network layer, is the weighting coefficient of the contrast loss in the second network layer; The layer distillation loss of the third network layer of the teacher network and the student network is: , represents the mean square error loss of the third network layer, represents the contrast loss of the third network layer, is the weighted coefficient of the mean square error loss in the third network layer, is the weighted coefficient of the contrast loss in the third network layer; The layer distillation loss of the fourth network layer of the teacher network and the student network is: , represents the mean square error loss of the fourth network layer, represents the contrast loss of the fourth network layer, is the weighted coefficient of the mean square error loss in the fourth network layer, is the weighted coefficient of the contrast loss in the fourth network layer; The layer distillation loss of the fifth network layer of the teacher network and the student network is: , represents the mean square error loss of the fifth network layer, represents the contrast loss of the fifth network layer, is the weighted coefficient of the mean square error loss in the fifth network layer, is the weighting coefficient of the contrast loss in the fifth network layer; S4. Perform knowledge distillation on the final output results of the teacher network and the student network to obtain the total loss of the output result distillation; S5. The layer distillation loss and the total distillation loss of the output result are added together to obtain the total model loss. The final pain intensity estimation model based on knowledge distillation is trained according to the total model loss. The images to be detected in the test set are predicted by the final pain intensity estimation model based on knowledge distillation to obtain the pain intensity estimation result.
2. The pain intensity estimation method based on knowledge distillation according to claim 1, characterized in that: The step S1 specifically includes: For the input original image sequence , use yolov5-face network to detect facial key points and locate the face position, Represents the nth image in the original image sequence; The facial image is cropped according to the located facial position, and the size of the cropped facial image is resized to Size, get the facial image sequence , Represents the nth face image in the face image sequence.
3. The pain intensity estimation method based on knowledge distillation according to claim 1, characterized in that: The output result of the total distillation loss obtained in step S4 specifically includes: The fifth teacher characteristics The final prediction result is obtained through the fully connected layer in the teacher network , the fifth student characteristic The final prediction result is obtained through the fully connected layer of the student network ; Calculate the soft labels of the final prediction results of the teacher network and obtain the soft label distribution of each category of the teacher network , the calculation formula is as follows: , in, represents the output score of the teacher network on the i-th category, T represents the distillation temperature, which is used to control the smoothness of softmax. It means to divide the output score of the teacher network on the i-th category by the temperature parameter and take the exponential, where , j is the total number of categories, Indicates the sum of the output scores of all categories in the teacher network after the same processing; calculate the soft label of the final prediction result of the student network, and obtain the soft label distribution of each category of the student network , the calculation formula is as follows: , in, represents the output score of the student network on the i-th category, It means that the output score of the student network on the i-th category is divided by the temperature parameter and then taken as the exponent. It means the output scores of all categories in the student network are summed after the same processing; The distillation loss between the soft label of the final prediction result of the teacher network and the soft label of the final prediction result of the student network is calculated as follows: , in, It represents the sum of all categories and calculates the cross entropy loss between the final prediction result output of the student network and the true label. The calculation formula is as follows: , in, represents the label of the true label on the i-th category, represents the softmax probability of the student network on the i-th category, It means summing all categories; According to the distillation loss between the soft label of the final prediction result of the teacher network and the soft label of the final prediction result of the student network and the cross entropy loss between the final prediction result output of the student network and the true label, the total loss of the output result distillation is obtained, and the calculation formula is as follows: , Among them, λ is a hyperparameter used to balance the weights of distillation loss and cross entropy loss.
4. The pain intensity estimation method based on knowledge distillation according to claim 3, characterized in that: The total loss of the model described in step S5 is: .
5. A system using the pain intensity estimation method based on knowledge distillation as claimed in claim 1, characterized in that: include: Image acquisition module: used to acquire the original image sequence and pre-process the original image sequence to obtain the facial image sequence; Model construction module: used to construct a pain intensity estimation model based on knowledge distillation, the model includes a teacher network and a student network, the teacher network is a ResNet50 network architecture integrated with an attention module, the student network is a ResNet18 network architecture integrated with an attention module, the teacher network and the student network each have five network layers, the first three network layers of the teacher network and the student network are each followed by an attention module att1, and the last two network layers of the teacher network and the student network are each followed by an attention module att2, and the facial image sequences in the training set are respectively input into the teacher network and the student network for image analysis; Layer knowledge distillation module: used to perform knowledge distillation between the layers of the teacher network and the student network to obtain the layer distillation loss of each network layer; Output result knowledge distillation module: used to perform knowledge distillation on the final output results of the teacher network and the student network to obtain the total loss of output result distillation; Pain intensity estimation module: used to obtain the final pain intensity estimation model based on knowledge distillation according to the total loss training of the model. The images to be tested in the test set are predicted by the final pain intensity estimation model based on knowledge distillation to obtain the pain intensity estimation result.
6. A pain intensity estimation device based on knowledge distillation, characterized in that: Including processor and storage medium: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute a pain intensity estimation method based on knowledge distillation according to any one of claims 1-4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program can be executed to implement a pain intensity estimation method based on knowledge distillation as described in any one of claims 1-4.
Citation Information
Patent Citations
Attention mechanism-based mobilenet-v1 knowledge distillation method, memory and terminal equipment
CN112784999A
Image deblurring algorithm based on knowledge distillation and deep neural network
CN114677304A
System and method for knowledge-preserving neural network pruning
US11200497B1
