Neonatal pain expression recognition method based on multi-head multi-level feature sparse transformer

Abstract

The present invention discloses a method for recognizing neonatal pain expressions based on a multi-head, multi-level feature sparse transformer. The method comprises the following steps: preprocessing images of neonatal pain expressions; constructing a multi-head, multi-level feature sparse transformer network model, wherein the network model includes a feature selection and compression module and a multi-level feature fusion module; and performing emotion recognition on the preprocessed neonatal pain expression images based on the constructed network model. The present invention prunes and compresses tokens using the multi-head feature selection and compression module, while simultaneously using the multi-level feature fusion module to compensate for the loss of information due to pruning. This method reduces computational complexity while improving classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method for recognizing neonatal pain expressions based on a multi-head, multi-level feature sparse transformer. Background Art

[0002] The International Association for the Study of Pain (IASP) defines pain as "an unpleasant sensory and emotional experience associated with actual or potential tissue damage or described in terms of such damage." Contrary to the previous misconception that newborns cannot feel pain due to their underdeveloped nervous systems, numerous studies have shown that newborns can indeed feel pain, and that prolonged pain experience can lead to long-term physiological and psychological effects. At the same time, the current standards for pain assessment are subjective, inconsistent, and intermittent. Intermittent and inconsistent pain assessments may lead to inappropriate treatment, which in turn may cause serious long-term consequences. Therefore, it is necessary to develop an automated system for assessing neonatal pain expressions.

[0003] Early studies on neonatal pain expressions were based on manually extracted features combined with existing machine learning methods, such as Local Binary Pattern (LBP) and Histogram of Oriented Gradients (HOG) combined with Support Vector Machine (SVM).

[0004] In recent years, with the advancement of deep learning, the accuracy of convolutional neural network models in neonatal pain recognition has significantly improved. Currently, a variety of methods, such as AlexNet, VGGNet, ResNet, and DenseNet, and their variants, are able to better extract deep features from images and improve the accuracy of pain classification. Following the tremendous success of the Transformer in NLP, researchers have proposed a ViT model suitable for image processing, which has also achieved great success in this field.

[0005] However, the high computational cost of ViT limits its practical application. Although pruning and compressing the tokens can reduce the computational complexity to a certain extent, the information lost due to pruning will inevitably lead to a decrease in accuracy.

[0006] In view of this, it is necessary to design a method based on a multi-head and multi-level feature sparse transformer network and a system for implementing the method to solve the above problems. Summary of the Invention

[0007] The main purpose of this invention is to introduce the ViT network model to handle the task of neonatal pain expression recognition, and reduce the computational cost of ViT by compressing and pruning the tokens in ViT, accurately select the tokens that need to be pruned through a multi-head feature selection module, and use a multi-level feature fusion module to fuse the information lost due to pruning and compression, so as to achieve a balance between performance and computational cost.

[0008] To achieve the above objectives, the present invention provides a method for neonatal pain emotion recognition based on a multi-head multi-level feature sparse transformer, comprising the following steps:

[0009] Preprocessing of neonatal pain expression images;

[0010] Constructing a multi-head multi-level feature sparse transformer network model, wherein the network model includes a feature selection and compression module and a multi-level feature fusion module;

[0011] Emotion recognition is performed on pre-processed neonatal pain expression images based on the constructed network model.

[0012] Furthermore, the steps of pre-processing the neonatal pain expression picture include:

[0013] Rotate the images of neonates with pain expressions that are tilted to the same angle to ensure they are at the same angle.

[0014] Remove the background area of ​​the newborn's pain expression picture and retain the newborn's facial area;

[0015] The images of neonatal pain expressions were normalized in scale, grayscale and equalized.

[0016] Furthermore, a feature selection and compression module is constructed, and the specific steps include: selecting and classifying tokens, pairing tokens, and fusing tokens.

[0017] Furthermore, the specific steps of token selection and classification include: selecting tokens to be pruned, that is, dividing all tokens into a retained set and a discarded set; the input vector x passes through a multi-head attention mechanism, and the specific formula is:

[0018] x'=MultiHead(x) (1)

[0019] in, Represents the input vector, x' represents the output vector, N represents the number of blocks the image is divided into, and C represents the dimension of the input feature;

[0020] The features are encoded by multi-layer perceptron to obtain the local feature z local, the specific formula is:

[0021]

[0022] Where C' represents the retained dimension;

[0023] Perform average pooling on the retained tokens to obtain the global feature z global , the specific formula is:

[0024]

[0025] in, Represents the decision matrix, which is used to identify whether the token is divided into the retention set or the discard set;

[0026] The global feature z global and local feature z local Combine them and input them into the multi-layer perceptron to get the probability of each token. The specific formula is:

[0027] z=[z local ,z global ],1≤i≤N (4)

[0028]

[0029] Among them, z represents the vector of local and global features, π i,0 Indicates the probability of discarding the i-th token, π i,1 Indicates the probability of retaining the i-th token;

[0030] The probability π of token retention based on Gumbel-Softmax technology i,1 Sampling is performed, and the specific formula is:

[0031] D=Gumbel-Softmax(π) i,1 ∈{0,1} N (6)

[0032] Where D represents the one-hot code vector;

[0033] Set the token pruning rate to ε, and divide all tokens into two subsets: the reserved set S r and discard set S a , the top 1-ε tokens with the highest retention probability are divided into the retention set, and the remaining tokens are divided into the discard set.

[0034] Furthermore, the specific steps of pairing tokens include: calculating the discard set S a All tokens in the reserved set S rThe similarity of tokens in is calculated as follows:

[0035]

[0036] Among them, c i,j Represents the similarity matrix, x i Indicates the i-th token in the discarded set, x j represents the jth token in the retained set;

[0037] For each token in the discard set, select the token with the highest similarity to the token in the retained set. The specific formula is:

[0038]

[0039] in, Indicates the target token corresponding to the token in the discarded set and the token in the retained set;

[0040] Define the mask matrix Indicates the result of token pairing. The specific formula is:

[0041]

[0042] Among them, N a and N r Represents the number of elements in the discard set and the retain set respectively.

[0043] Furthermore, the specific steps of fusing tokens include: according to the mask matrix M i,j and the similarity matrix c i,j Calculate the weight w of each discarded set label to be fused i and the holdout set label weights w j , the specific formula is:

[0044]

[0045]

[0046] According to w i and w j The weight matrix updates the reserved set labels, and the tokens that are not selected as targets remain unchanged. The specific formula is:

[0047]

[0048] Among them, y j Indicates the updated holdout set label.

[0049] Furthermore, a multi-level feature fusion module is constructed. The specific steps include:

[0050] The multi-level features are averaged pooled in the spatial dimension. The specific formula is:

[0051] F i,avg =AvgPool(F i ) (13)

[0052] Among them, F i represents the feature of the i-th input, F i,avg represents the result of average pooling, where i = 1, 2, ..., n;

[0053] F i,avg Use 1×1 convolution to model the correlation between channels. The specific formula is:

[0054] F i,conv =Conv(F i,avg ) (14)

[0055] F i,conv The sigmoid activation function is used to generate the channel descriptor. The specific formula is:

[0056] F i '=sigmoid(F i,conv ) (15)

[0057] F i 'Splice in the second dimension and obtain the weight representation of each input through the softmax function. The specific formula is:

[0058] w(i)=softmax(Concat(F0',…,F i ',…,F n ')) (16)

[0059] The obtained weight is multiplied by the corresponding input and then added to obtain the fused feature representation Y. The specific formula is:

[0060]

[0061] Here, ⊙ represents the Hadamard product.

[0062] Furthermore, the multi-head multi-level feature sparse transformer network model includes: 12 layers of transformer modules are connected sequentially, a multi-head feature selection and compression module is inserted after the transformer modules of the 3rd, 6th and 9th layers, and the output features of the 3rd, 6th and 9th layers are fused with the output features of the 11th layer through a multi-level feature fusion module and then input into the 12th layer transformer module.

[0063] According to one aspect of the present invention, a storage medium is provided, wherein instructions are stored in the storage medium. When a computer reads the instructions, the computer is caused to execute any one of the above-mentioned methods for neonatal pain emotion recognition based on a multi-head, multi-level feature sparse transformer.

[0064] According to another aspect of the present invention, an electronic device is provided, comprising a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. This paper introduces VisionTransformer to handle the task of neonatal pain expression recognition. At the same time, it prunes and compresses the tokens in the Transformer to reduce the computational complexity required by VisionTransformer. In the token selection process, a multi-head attention mechanism is introduced to enable selection in multiple subspaces, thereby improving the effectiveness of selection.

[0067] 2. The present invention proposes to use a multi-level feature fusion module to overcome the disadvantage of information loss caused by pruning and compression, and fuse the information in tokens before pruning and compression at different stages, thereby improving the accuracy of neonatal pain expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0069] Figure 1 This is a flow chart of a method for recognizing neonatal pain expressions according to an embodiment of the present invention;

[0070] Figure 2 This is a structural block diagram of a multi-head, multi-level, feature-sparse transformer network according to an embodiment of the present invention;

[0071] Figure 3 It is a multi-head feature selection and compression module according to an embodiment of the present invention;

[0072] Figure 4 This is a multi-level feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] like Figure 1As shown in FIG, a method for neonatal pain emotion recognition based on a multi-head multi-level feature sparse transformer includes the following steps:

[0075] Preprocessing of neonatal pain expression images;

[0076] Build a multi-head feature selection and compression module;

[0077] Construct a multi-level feature fusion module;

[0078] Construct a multi-head, multi-level feature sparse transformer network model. The network model consists of a transformer module, a multi-head feature selection and compression module, and a multi-level feature fusion module.

[0079] Train a network model and use it to identify pain emotions in newborns.

[0080] The pretreatment comprises the following steps:

[0081] Newborn images with angled deflection were manually rotated to ensure the face was straight;

[0082] Remove the background area and keep the newborn's face area;

[0083] The rotated and cropped neonatal images were scale normalized, grayscale normalized, and equalized.

[0084] like Figure 3 As shown in the figure, building a multi-head feature selection and compression module includes:

[0085] Select the tokens that need to be pruned: that is, all tokens are divided into a retention set and a discard set. The input vector x first passes through a multi-head attention mechanism, which enables the selection model to learn in parallel on multiple subspaces; the local feature z local To encode a specific feature, a simple multi-layer perceptron can be used. In order to obtain the global information contained in the retained token, we choose to perform average pooling on all retained tokens to obtain the global feature z. global ; Then combine the global and local information and send them to another multi-layer perceptron to get the weight of each token and divide the token into the retention set and the discarded set. In order to make the sampling strategy π to the binary mask decision matrix D trainable in the network, this paper uses the Gumbel-Softmax technique to train the retention probability π i,1 Sampling is performed; finally, according to the set token pruning rate ε, all tokens are divided into two subsets and the retention set S r and discard set S a. The top 1-ε tokens with the highest retention probability are divided into the retention set, and the remaining tokens are divided into the discard set;

[0086] To pair tokens: First, for the discarded set S a Calculate the difference between the token and the retention set S r Then for each tokenx in the discarded set i , select the closest tokenx based on the similarity with the token in the retained set j , then x j is x i Goal The target token and the discarded tokens have a one-to-many relationship. This means that multiple discarded tokens may correspond to the same target token, and not all tokens in the retained set can be considered the target token. Finally, a mask matrix is ​​used to represent the matching results. This allows the elements in the two sets to be used properly for subsequent matrix operations and eliminates the influence of unmatched tokens.

[0087] To fuse tokens: First, according to the mask matrix m i,j and the similarity matrix c i,j Calculate the weight w of each fused discarded set label i and the holdout set label weights w j ; According to these two weight matrices, the retention set label can be updated to y j , while the tokens that are not selected as targets remain unchanged.

[0088] Specifically, building a multi-head feature selection and compression module includes the following steps:

[0089] Select the tokens that need to be pruned: that is, all tokens are divided into a retain set and a discard set. The input vector x first passes through a multi-head attention mechanism, which enables the selection model to learn in parallel on multiple subspaces. The formula is as follows

[0090] x'=MultiHead(x) (1)

[0091] in is the input vector, and N=HW is the number of blocks the image is divided into.

[0092] Local feature z local To encode a specific feature, a multi-layer perceptron can be used. The formula is as follows:

[0093]

[0094] Where C' is the retained dimension, which is usually C'=0.7C in experiments.

[0095] In order to obtain the global information contained in the retained token, we choose to perform average pooling on all retained tokens to obtain the global feature z global , the formula is as follows:

[0096]

[0097] in is a decision matrix, which is used to indicate whether to classify this token into the retention set or the discard set.

[0098] Then, by combining the global and local information and sending them to another multi-layer perceptron, we can get the weight of each token and divide the token into the retention set and the discard set. The formula is as follows:

[0099] z=[z local ,z global ],1≤i≤N (4)

[0100]

[0101] where π i,0 Indicates the probability of discarding the i-th token, π i,1 Indicates the probability of retaining the i-th token.

[0102] In order to make the sampling strategy π to the binary mask decision matrix D be trainable in the network, the Gumbel-Softmax technique is used here to retain the probability π i,1 For sampling, the formula is as follows:

[0103] D=Gumbel-Softmax(π) i,1 ∈{0,1} N (6)

[0104] The output of Gumbel-Softmax is a one-hot code vector, and the expected value is exactly equal to π, and Gumbel-Softmax is differentiable, so it can be trained in a neural network.

[0105] Finally, according to the set token pruning rate ε, all tokens are divided into two subsets and the reserved set S r and discard set S a The top 1-ε tokens with the highest retention probability are divided into the retention set, and the remaining tokens are divided into the discard set.

[0106] To pair tokens: First, for the discarded set S a Calculate the difference between the token and the retention set S r The similarity of the elements in , the calculation formula of the similarity matrix is ​​as follows:

[0107]

[0108] Where i is the number of the discarded token, and j is the number of the retained token.

[0109] Then for each tokenx in the discarded set i , select the closest tokenx based on the similarity with the token in the retained set j , then x j is x i Goal The selection is as follows:

[0110]

[0111] The target token and the discarded token have a one-to-many relationship, that is, multiple discarded tokens may correspond to the same target token, and not all tokens in the retained set can be called target tokens.

[0112] Finally, a mask matrix Represents the matching results, the value of each element in the matrix is ​​as follows;

[0113]

[0114] where N a and N r Represents the number of elements in the discard set and the retained set respectively. Using the mask matrix allows the elements in the two sets to perform subsequent matrix operations normally and eliminates the influence of unmatched tokens.

[0115] Perform token fusion: First, according to the mask matrix m i,j and the similarity matrix c i,j Calculate the weight w of each fused discarded set label i and the holdout set label weights w j , the formula is as follows:

[0116]

[0117]

[0118] According to these two weight matrices, the retention set label can be updated to yj , while the tokens that are not selected as targets remain unchanged. The formula is as follows:

[0119]

[0120] like Figure 4 As shown in Figure 2, building a multi-level feature fusion module includes the following steps:

[0121] For multiple stage features X i First, average pooling is performed in the spatial dimension. The formula is as follows:

[0122] F i,avg =AvgPool(F i ) (13)

[0123] For the global feature representation after average pooling, 1×1 convolution is used to model the correlation between channels. The formula is as follows:

[0124] F i,conv =Conv(F i,avg ) (14)

[0125] The channel descriptor is generated by the sigmoid activation function. The formula is as follows:

[0126] F i '=sigmoid(F i,conv ) (15)

[0127] The global features of N scales are concatenated in the second dimension, and then the softmax function is used to obtain their respective weight representations. The formula is as follows:

[0128] w(i)=softmax(Concat(F0',…,F i ',…,F n ')) (16)

[0129] The obtained weight is multiplied by the corresponding input and then added to obtain the fused feature representation Y. The formula is as follows:

[0130]

[0131] where ⊙ is the Hadamard product.

[0132] like Figure 2As shown in the figure, the multi-head multi-level feature sparse transformer network model contains 12 layers of sequentially connected transformer modules. Multi-head feature selection and compression modules are inserted after the 3rd, 6th and 9th layers, and the features of the 3rd, 6th and 9th layers are fused with the features of the 11th layer through the multi-level feature fusion module and then input into the 12th layer transformer module.

[0133] Initialize with pre-trained weights and select the AdamW optimizer and cosine learning rate scheduler. Input the images in the dataset into the network for training and update the parameters in the model.

[0134] The computer-readable storage medium of this embodiment may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal; the computer-readable storage medium of this embodiment may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.

[0135] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0136] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0137] The examples described in the present invention are merely descriptions of the preferred embodiments of the present invention and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should fall within the scope of protection of the present invention.

Claims

1. A method for neonatal pain emotion recognition based on multi-head multi-level feature sparse transformer, characterized by: The following steps are involved: Preprocessing of neonatal pain expression images; Constructing a multi-head multi-level feature sparse transformer network model, wherein the network model includes a feature selection and compression module and a multi-level feature fusion module; Construct a feature selection and compression module. The specific steps include: selecting and classifying tokens, pairing tokens, and fusing tokens. The specific steps of token selection and classification include: selecting the tokens to be pruned, that is, dividing all tokens into a retention set and a discarded set; input vector Through a multi-head attention mechanism, the specific formula is: (1) Among them, x Represents the input vector represents the output vector, Indicates the number of blocks the image is divided into, Represents the dimension of the input features; The features are encoded through a multi-layer perceptron to obtain local features , the specific formula is: (2) in, Indicates the retained dimension; Perform average pooling on the retained tokens to obtain global features , the specific formula is: (3) in, Represents the decision matrix, which is used to identify whether the token is divided into the retention set or the discard set; Global Features With local features Combine them and input them into the multi-layer perceptron to get the probability of each token. The specific formula is: (4) (5) in, Represents the concatenated vector of local and global features, Indicates that the The probability of token abandonment, Indicates that the The probability of a token being retained; The probability of token retention based on Gumbel-Softmax technology Sampling is performed, and the specific formula is: Gumbel-Softmax (6) in, represents a one-hot encoding vector; Set the token pruning rate to , divide all tokens into two subsets and retain them and discarded sets , the top one with the highest probability will be retained The tokens are divided into the retention set, and the remaining tokens are divided into the discard set; The specific steps for pairing tokens include: calculating the discard set All tokens in the reserved set The similarity of tokens in is calculated as follows: (7) in, represents the similarity matrix, Indicates discarding the first tokens, represents the first tokens; For each token in the discard set, select the token with the highest similarity to the token in the retained set. The specific formula is: (8) in, Indicates the target token corresponding to the token in the discarded set and the token in the retained set; Define the mask matrix Indicates the result of token pairing. The specific formula is: (9) in, and Represents the number of elements in the discarded set and the retained set respectively; Emotion recognition is performed on pre-processed neonatal pain expression images based on the constructed network model.

2. The method according to claim 1, characterized in that The steps for preprocessing the neonatal pain expression images include: Rotate the images of neonates with pain expressions that are tilted to the same angle to ensure they are at the same angle. Remove the background area of ​​the newborn's pain expression picture and retain the newborn's facial area; The images of neonatal pain expressions were normalized in scale, grayscale and equalized.

3. The method according to claim 1, characterized in that The specific steps of fusing tokens include: and similarity matrix Calculate the weight of each discarded set label to be fused and the holdout set label weights , the specific formula is: (10) (11) according to and The weight matrix updates the reserved set labels, and the tokens that are not selected as targets remain unchanged. The specific formula is: (12) in, Indicates the updated holdout set label.

4. The method according to claim 1, wherein Construct a multi-level feature fusion module. The specific steps include: The multi-level features are averaged pooled in the spatial dimension. The specific formula is: (13) in, Indicates the The input features, represents the result of average pooling, where ; right Use 1 1 convolution is used to model the correlation between channels. The specific formula is: (14) right The sigmoid activation function is used to generate the channel descriptor. The specific formula is: = (15) Will Splicing is performed on the second dimension, and the weight representation of each input is obtained through the softmax function. The specific formula is: (16) The obtained weight is multiplied by the corresponding input and then added to obtain the fused feature representation Y. The specific formula is: (17) in, represents the Hadamard product.

5. The method according to claim 1, wherein The multi-head multi-level feature sparse transformer network model includes: 12 layers of transformer modules are connected sequentially, a multi-head feature selection and compression module is inserted after the transformer modules of the 3rd, 6th and 9th layers, and the output features of the 3rd, 6th and 9th layers are fused with the output features of the 11th layer through a multi-level feature fusion module and then input into the 12th layer transformer module.

6. A storage medium, characterized in that The storage medium stores instructions, and when a computer reads the instructions, the computer executes the neonatal pain emotion recognition method based on a multi-head multi-level feature sparse transformer as described in any one of claims 1 to 5.

7. An electronic device, characterized in that: The device comprises a processor and the storage medium according to claim 6, wherein the processor executes instructions in the storage medium.

Images