Facial expression-based pain recognition method, device, equipment and storage medium

Abstract

The application provides a facial-expression-based pain recognition method, device, equipment and storage medium, and the facial-expression-based pain recognition method comprises the following steps: acquiring a to-be-recognized face video, and extracting an AU image from the to-be-recognized face video; and inputting the AU image into a pre-trained pain recognition model to obtain a pain level of the to-be-recognized face image. The AU image related to pain is input into the pre-trained pain recognition model, so that other regions of the face unrelated to pain are avoided from being input into the pre-trained pain recognition model, and the accuracy of pain recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a pain recognition method and device based on facial expressions, equipment and a storage medium. BACKGROUND

[0002] Pain is an important indicator of human health, which is a highly unpleasant feeling caused by disease, injury or mental distress. It is crucial for the medical diagnosis and treatment of patients and is one of the indicators of medical diagnosis and treatment. Therefore, pain recognition is very important for pain control in intensive care and postoperative wound healing.

[0003] At present, people mainly determine the degree of pain through visual analog system, digital rating scale, pain measuring instrument and McGill pain questionnaire, etc. However, the above-mentioned pain recognition methods are completed through self-report of patients, which is greatly affected by the subjective factors of patients. Moreover, not all patients can complete pain recognition through the above-mentioned self-reporting method, such as infants, patients with nerve damage diseases and patients who need to use breathing equipment to assist breathing, which seriously affects the recognition of pain.

[0004] Therefore, deep learning is gradually applied to pain recognition. However, due to the difference in pain degree, the facial action may be small when pain occurs, and it is difficult for deep learning method to distinguish the action features from the facial features. Moreover, deep learning often extracts features from the whole facial image, which easily ignores the semantic information and correlation information between each part of the face, and has poor robustness. Moreover, the pain recognition method based on deep learning does not make good use of the relationship between data in space and time, which affects the accuracy of pain recognition. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a pain recognition method and device based on facial expressions, which has the advantage of high recognition accuracy.

[0006] The technical solution adopted by the present application is as follows:

[0007] The pain recognition method based on facial expressions comprises:

[0008] Obtaining a to-be-recognized face video, and extracting an AU image from the to-be-recognized face video;

[0009] Inputting the AU image into a pre-trained pain recognition model to obtain the pain level of the to-be-recognized face image.

[0010] Further, the extraction of the AU image from the to-be-recognized face video comprises:

[0011] extracting the n frames of face images to be recognized from the face video to be recognized frame by frame by using OpenCV, wherein n is the number of frames of the face video to be recognized;

[0012] detecting and locating faces in the n frames of face images to be recognized by using a PIPNET network to obtain n frames of face images to be recognized;

[0013] extracting an AU image of each frame of the face image to be recognized.

[0014] Further, the extraction of the AU image of each frame of the face image to be recognized comprises:

[0015] extracting a plurality of feature points of the face image to be recognized by using a PIPNET network;

[0016] locating and cutting the AU region of the face image to be recognized by using the plurality of feature points to obtain the AU image.

[0017] Further, the pain recognition model comprises a spatial feature extraction module, a time sequence feature extraction module and a first full connection layer;

[0018] inputting the AU image into a pre-trained pain recognition model to obtain the pain level of the face image to be recognized comprises:

[0019] inputting the AU image of the t-th frame of the face image to be recognized into the spatial feature extraction module to obtain the spatial feature of the t-th frame of the face image to be recognized, wherein t = 1, 2, …, n;

[0020] inputting the spatial features of the n frames of the face image to be recognized into the time sequence feature extraction module to obtain the time sequence feature;

[0021] inputting the time sequence feature into the full connection layer to obtain the pain level of the face image to be recognized.

[0022] Further, the spatial feature extraction module comprises m dynamic DenseGCN graph convolution units connected in series;

[0023] inputting the AU image of the face image to be recognized into the spatial feature extraction module to obtain the spatial feature of the face image to be recognized is:

[0024] inputting the AU image of the face image to be recognized into the first dynamic DenseGCN graph convolution unit to obtain the output result of the dynamic DenseGCN graph convolution unit;

[0025] The AU image of the to-be-identified face image and at least one of the output results of the first (i-1) dynamic DenseGCN graph convolution units are input into the ith dynamic DenseGCN graph convolution unit to obtain an output result of the ith dynamic DenseGCN graph convolution unit, and the output result of the mth dynamic DenseGCN graph convolution unit is the spatial feature of the to-be-identified face image, where i=2, 3,..., m.

[0026] Further, the time sequence feature extraction module comprises a GRU network, a global attention layer and a second full connection layer.

[0027] The spatial feature of the to-be-identified face image is input into the time sequence feature extraction module to obtain the time sequence feature of the to-be-identified face image, which comprises:

[0028] The spatial feature of the to-be-identified face image is input into the time sequence feature extraction module to obtain the time sequence feature of the to-be-identified face image, which comprises:

[0029] The global attention layer calculates the weight a of the last hidden state of the to-be-identified face image in the tth frame by using the last hidden state and other hidden states of the to-be-identified face image in the tth frame. t , and wherein: h t is the last hidden state of the to-be-identified face image, h i is the other hidden state of the to-be-identified face image, w t is a matrix for assisting in analyzing the correlation between h t and h i , which helps to confirm the to-be-identified face image related to pain.

[0030] The second full connection layer calculates the time sequence feature h f by using the last hidden state of the to-be-identified face image in n frames and the weight thereof, and: wherein a t represents the weight of the last hidden state of the to-be-identified face image in the tth frame; h t represents the last hidden state of the to-be-identified face image in the tth frame.

[0031] Further, the first full connection layer comprises a first full connection layer unit and a second full connection layer unit, the size of the first full connection layer is (2048, 1024), and the size of the second full connection layer is (1024, 1).

[0032] Input the time sequence feature into the full connection layer, and obtain the pain level of the to-be-identified face image as:

[0033] The time sequence feature h is sequentially processed by the first full connection layer unit and the second full connection layer unit. f The pain level is obtained by performing nonlinear processing and linear processing.

[0034] The application further provides a pain recognition device based on facial expression, comprising:

[0035] An acquisition module is configured to acquire a to-be-identified face video and extract an AU image from the to-be-identified face video.

[0036] An identification module is configured to input the AU image into a pre-trained pain recognition model to obtain a pain level of a to-be-identified face image.

[0037] The application further provides a pain recognition device based on facial expression, comprising:

[0038] A processor;

[0039] A memory is configured to store a computer program executed by the processor.

[0040] When the processor executes the computer program, the pain recognition method based on facial expression is realized.

[0041] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed to realize the pain recognition method based on facial expression.

[0042] Compared with the prior art, the application has the following beneficial effects:

[0043] 1. The pain recognition method based on facial expression disclosed in the application can avoid inputting other facial regions unrelated to pain into a pre-trained pain recognition model, thereby improving the accuracy of pain recognition.

[0044] 2. The face expression-based pain recognition method disclosed in the application, the spatial feature extraction module comprises m dynamic DenseGCN graph convolution units connected in sequence, the dynamic DenseGCN graph convolution unit is adopted, the spatial feature extraction module can perform dynamic edge learning according to the feature information of the to-be-recognized face image, and the efficiency of acquiring the spatial feature of the to-be-recognized face image is improved; and when the spatial feature extraction module is used to extract the spatial feature of the to-be-recognized face image, the input of the i-th dynamic DenseGCN graph convolution unit is at least one of the AU image of the to-be-recognized face image and the output result of the first (i-1) dynamic DenseGCN graph convolution unit, so that the output result of different dynamic DenseGCN graph convolution units can be fully utilized to improve the accuracy of extracting the spatial feature of the to-be-recognized face image.

[0045] 3. The face expression-based pain recognition method disclosed in the application, the time sequence feature extraction module uses the weight alpha of the last hidden state of the t-th frame of the to-be-recognized face image calculated by the global attention layer t The pain-related frame can be better reflected, and the accuracy of pain recognition is further improved.

[0046] In order to better understand and implement, the application is described in detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The flowchart of the face expression-based pain recognition method in an embodiment of the application is shown in the figure;

[0048] Figure 2 The schematic diagram of the AU image is shown in the figure;

[0049] Figure 3 The distribution diagram of the face feature points is shown in the figure;

[0050] Figure 4 The structural block diagram of the pain recognition model is shown in the figure;

[0051] Figure 5 The structural schematic diagram of the spatial feature extraction module in the embodiment is shown in the figure; Figure 4 The structural schematic diagram of the spatial feature extraction module in the embodiment is shown in the figure;

[0052] Figure 6 The structural schematic diagram of the time sequence feature extraction module in the embodiment is shown in the figure; Figure 4 The structural schematic diagram of the time sequence feature extraction module in the embodiment is shown in the figure;

[0053] Figure 7 The structural schematic diagram of the GRU network in the embodiment is shown in the figure; Figure 6 The structural schematic diagram of the GRU network in the embodiment is shown in the figure; The structural schematic diagram of the GRU network in the embodiment is shown in the figure;

[0054] The structural schematic diagram of the GRU network in the embodiment is shown in the figure; Figure 8 The structural block diagram of the face expression-based pain recognition device in an embodiment of the application is shown in the figure; The structural block diagram of the face expression-based pain recognition device in an embodiment of the application is shown in the figure;

[0055] Wherein, the reference numerals in each figure are:

[0056] 1, acquisition module; 11, video acquisition unit; 12, first image extraction unit; 13, second image extraction unit; 14, third image extraction unit; 2, recognition module; 21, spatial feature extraction unit; 22, time sequence feature extraction unit; 23, recognition unit. DETAILED DESCRIPTION

[0057] It should be clear that the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0058] The terms used in the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more associated listed items.

[0059] The following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not necessarily describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0060] In addition, in the description of the present application, "multiple" means two or more, unless otherwise specified. The "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents a "or" relationship between the associated objects before and after it.

[0061] It should be understood that the present application is not limited to the precise structures already described and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

[0062] Referring to Figure 1 The embodiment provides a facial expression-based pain recognition method, which comprises the following steps:

[0063] S1: acquiring a to-be-recognized face video and extracting an AU image from the to-be-recognized face video.

[0064] In the embodiment, the extraction of the AU image from the to-be-recognized face video comprises the following sub-steps:

[0065] S11: frame-by-frame extraction of the to-be-recognized face video by using OpenCV to obtain n to-be-recognized face images, wherein n is the frame number of the to-be-recognized face video.

[0066] S12: face detection and positioning of the n to-be-recognized face images by using a PIPNET network to obtain n to-be-recognized face images. In the embodiment, a nose is a relatively prominent feature in a face, and when a facial expression changes, the related features of the nose can also be well retained. Therefore, in order to improve the data quality, the face in the to-be-recognized face image is normalized with the nose as the center to obtain an image with a size of 33x224x224, that is, the size of each to-be-recognized image is 33x224x224.

[0067] The PIPNET network is a high-efficiency face feature extraction model, which comprises three important modules. First, a novel detection head, called nested regression, which is the key of the method, decomposes the face positioning task into a heat map regression based on a low-resolution feature map and a coordinate regression on a local feature map, which enables the PIPNET network to have a high progress without relying on a high-resolution feature map, thereby saving the calculation amount. Moreover, the detection head comprises a proximity regression module, which can determine the proximity of a key point by training each key point, so that the shape constraint of a local region can be obtained at the same time of positioning, thereby improving the robustness of the model. Finally, when the pseudo-label is estimated without a standard sample, the task can be started from a simple task, and the difficulty of the task can be gradually increased in the subsequent iteration process until it becomes a standard self-training task, thereby effectively alleviating the noise problem introduced by the standard self-training method in the pseudo-label.

[0068] S13: extraction of an AU image of each to-be-recognized face image.

[0069] In the embodiment, the AU image is an image of a facial movement unit (namely, an AU) in the pain index scale PSPI, which has the ability to express the semantic information of pain. Referring to Figure 2 four AU images are extracted from each to-be-recognized face image, and the four AU images are obtained through the following process: (1) extraction of a plurality of feature points of the to-be-recognized face image by using a PIPNet network, referring toFigure 3 The feature points include facial contour feature points, nose feature points, eyebrow feature points, mouth feature points and eye feature points, etc.; (2) The AU region of the face image to be identified is located and cropped using multiple of the feature points to obtain the AU image. The coordinates of the relevant feature points of each AU region are used to locate each AU region, and then the located AU region is cropped from the face image to be identified to obtain the AU image.

[0070] S2: Input the AU image into a pre-trained pain recognition model to obtain the pain level of the face image to be recognized.

[0071] In this embodiment, refer to Figure 4 The pain recognition model includes a spatial feature extraction module, a temporal feature extraction module, and a first fully connected layer. Therefore, by inputting the AU image into the pre-trained pain recognition model, the pain level of the face image to be recognized is obtained, including:

[0072] S21: Input the AU image of the face image to be identified in the t-th frame into the spatial feature extraction module to obtain the spatial features of the face image to be identified in the t-th frame, where: t = 1, 2, ..., n.

[0073] Reference Figure 5 The spatial feature extraction module includes m sequentially connected dynamic DenseGCN graph convolutional units. Furthermore, the spatial features of the face image to be identified are obtained by inputting the AU image of the face image to the spatial feature extraction module as follows: First, the AU image of the face image to be identified is input into the first dynamic DenseGCN graph convolutional unit to obtain its output. Then, at least one of the AU image of the face image to be identified and the outputs of the first (i-1) dynamic DenseGCN graph convolutional units is input into the i-th dynamic DenseGCN graph convolutional unit to obtain its output. The output of the m-th dynamic DenseGCN graph convolutional unit is the spatial feature of the face image to be identified, where i = 2, 3, ..., m.

[0074] In this embodiment, the spatial feature extraction module includes m dynamic DenseGCN graph convolution units connected in series. The dynamic DenseGCN graph convolution unit can enable the spatial feature extraction module to dynamically learn edges according to the feature information of the to-be-identified face image, thereby improving the efficiency of obtaining the spatial features of the to-be-identified face image. In addition, when the spatial feature extraction module extracts the spatial features of the to-be-identified face image, the input of the i-th dynamic DenseGCN graph convolution unit is at least one of the AU image of the to-be-identified face image and the output of the previous (i-1) dynamic DenseGCN graph convolution units, which improves the information flow between the m dynamic DenseGCN graph convolution units, realizes the effective reuse of different dynamic DenseGCN graph convolution units, and facilitates the extraction of the spatial features of the to-be-identified face image using the information flow of different dynamic DenseGCN graph convolution units, thereby improving the accuracy of extracting the spatial features of the to-be-identified face image.

[0075] S22: input the spatial features of n frames of the to-be-identified face image into the time sequence feature extraction module to obtain the time sequence features.

[0076] With reference to Figure 6 , the time sequence feature extraction module includes a GRU network, a global attention layer, and a second full connection layer. The input of the spatial features of the to-be-identified face image into the time sequence feature extraction module to obtain the time sequence features of the to-be-identified face image includes the following sub-steps: (1) input the spatial features of the t-th frame of the to-be-identified face image into the GRU network respectively to obtain a plurality of hidden states of the t-th frame of the to-be-identified face image, wherein t=1, 2, …, n; (2) the global attention layer calculates the weight of the last hidden state of the t-th frame of the to-be-identified face image using the last hidden state and other hidden states of the t-th frame of the to-be-identified face image, and the weight is represented as α t , and wherein: h t is the last hidden state of the to-be-identified face image, h i is the other hidden state of the to-be-identified face image, w t is a matrix for assisting in analyzing the correlation between h t and h i , which helps to identify the to-be-identified face image related to pain; (3) the second full connection layer calculates the time sequence features h f of the to-be-identified face image using the last hidden state and the weight of the n frames of the to-be-identified face image, and: wherein: α t represents the weight of the last hidden state of the t-th frame of the to-be-identified face image; ht This represents the last hidden state of the face image to be identified in frame t.

[0077] In this embodiment, the GRU network is a recurrent neural network structure, which is simpler than the traditional RNN network structure, as shown below. Figure 7 The GRU network includes an input gate, a reset gate, and an output gate, which are used to control the flow of information to better capture dependencies in the time series. The GRU network outputs all hidden states at the output gate. Currently, the last hidden state of the GRU network is generally selected for prediction. However, important pain-related information in the middle of the sequence may be hidden in peak frames, i.e., frames with significant facial expression changes. Not every frame is worth deep learning, and overlearning of other frames may lead to overfitting, affecting the results. Therefore, in this embodiment, for each frame of the face image to be identified, the last hidden state of the GRU network and other hidden states are input into the score function of the global attention layer to calculate the weight of the last hidden state. This is because the score function can be calculated using a trainable matrix w. t Go to assist h t and h i The correlation is used to help determine the peak frame, i.e., the pain-related frame; then, the correlation reflected by the score function is normalized using the softmax function to obtain the weight α of the last hidden state of the face image to be identified in frame t. t .

[0078] S23: Input the temporal features into the fully connected layer to obtain the pain level of the face image to be identified.

[0079] In this embodiment, the first fully connected layer includes a first fully connected layer unit and a second fully connected layer unit. The size of the first fully connected layer is (2048, 1024), and the size of the second fully connected layer is (1024, 1). Therefore, by inputting the temporal feature into the fully connected layer, the pain level of the face image to be identified is obtained as follows: the first fully connected layer unit and the second fully connected layer unit are used to sequentially process the temporal feature h. f The pain level is obtained by performing nonlinear and linear processing.

[0080] It should be noted that in this embodiment, the pain level is a value between 1 and 5, and the higher the value, the more severe the pain.

[0081] The face expression-based pain recognition method disclosed in the application can input the AU image related to pain into the pre-trained pain recognition model, can avoid inputting other facial regions unrelated to pain into the pre-trained pain recognition model, and can improve the accuracy of pain recognition.

[0082] In addition, in the embodiment, the pre-trained pain recognition model is obtained through the following steps:

[0083] (1) input the AU image of the tth frame of the to-be-recognized face image obtained in step S1 into the spatial feature extraction module to obtain the spatial feature of the tth frame of the to-be-recognized face image, wherein t = 1, 2, …, n, and n is the number of frames of the to-be-recognized face video;

[0084] (2) input the spatial features of the n frames of the to-be-recognized face image into the time sequence feature extraction module respectively to obtain the time sequence feature h f ;

[0085] (3) input the time sequence feature h f into the first full connection layer to obtain the predicted value Y i of the pain level of the to-be-recognized face image;

[0086] (4) manually label the pain level of the to-be-recognized face image to obtain the true value Y

[0087] (5) calculate the loss function MSE by using the predicted value Y i and the true value Y of the pain level of the to-be-recognized face image, and

[0088] (6) judge whether the loss function MSE meets the threshold value, when the loss function MSE does not meet the threshold value, optimize the spatial feature extraction module, the time sequence feature extraction module and the first full connection layer by using the loss function MSE to obtain the improved pain recognition model, and repeat steps (1)-(5) until the loss function MSE meets the threshold value, and the improved pain recognition model when the loss function meets the threshold value is the pre-trained pain recognition model.

[0089] Based on the face expression-based pain recognition provided in the embodiment, referring to Figure 8 , the embodiment further provides a face expression-based pain recognition device, which comprises:

[0090] An acquisition module 1 is configured to acquire a to-be-recognized face video and extract an AU image from the to-be-recognized face video.

[0091] In this embodiment, the acquisition module 1 comprises a video acquisition unit 11, a first image extraction unit 12, a second image extraction unit 13 and a third image extraction unit 14, wherein: (1) the video acquisition unit 11 is configured to acquire a to-be-identified face video; (2) the first image extraction unit 12 is configured to extract the to-be-identified face video frame by frame using OpenCV to obtain n to-be-identified face images, wherein n is the frame number of the to-be-identified face video; (3) the second image extraction unit 13 is configured to perform face detection and positioning on the n to-be-identified face images using a PIPNET network to obtain n to-be-identified face images; and (4) the third image extraction unit 14 is configured to extract an AU image of each to-be-identified face image.

[0092] Moreover, the third image extraction unit 14 is configured to extract an AU image of each to-be-identified face image by the following steps: (1) extracting a plurality of feature points of the to-be-identified face image using a PIPNet network, wherein the feature points include facial contour feature points, nose feature points, eyebrow feature points, mouth feature points and eye feature points, etc.; and (2) positioning and cutting an AU region of the to-be-identified face image using the plurality of feature points to obtain the AU image, wherein each AU region is positioned using the coordinates of the feature points related to the AU region, and then the positioned AU region is cut from the to-be-identified face image to obtain the AU image. Figure 3

[0093] The recognition module 2 is configured to input the AU image into a pre-trained pain recognition model to obtain a pain level of the to-be-identified face image.

[0094] In this embodiment, the pain recognition model comprises a spatial feature extraction module, a time sequence feature extraction module and a first full connection layer, and the recognition module 2 comprises a spatial feature extraction unit 21, a time sequence feature extraction unit 22 and a recognition unit 23, wherein:

[0095] The spatial feature extraction unit 21 is configured to input the AU image of the t-th to-be-identified face image into the spatial feature extraction module to obtain a spatial feature of the t-th to-be-identified face image, wherein t = 1, 2, …, n.

[0096] ​In this embodiment, the spatial feature extraction module includes m dynamic DenseGCN graph convolution units connected in series; and the spatial feature extraction unit 21 is configured to input the AU image of the t-th frame of the to-be-identified facial image into the spatial feature extraction module to obtain the spatial feature of the t-th frame of the to-be-identified facial image through the following process: first, input the AU image of the to-be-identified facial image into the first dynamic DenseGCN graph convolution unit to obtain the output result of the dynamic DenseGCN graph convolution unit; then, input the AU image of the to-be-identified facial image and at least one of the output results of the first (i-1) dynamic DenseGCN graph convolution units into the i-th dynamic DenseGCN graph convolution unit to obtain the output result of the i-th dynamic DenseGCN graph convolution unit, and the output result of the m-th dynamic DenseGCN graph convolution unit is the spatial feature of the to-be-identified facial image, where i = 2, 3, …, m.

[0097] The temporal feature extraction unit 22 is configured to input the spatial features of n frames of the to-be-identified facial image into the temporal feature extraction module to obtain the temporal feature.

[0098] In this embodiment, the temporal feature extraction module includes a GRU network, a global attention layer and a second full connection layer, therefore, the temporal feature extraction unit 22 is configured to input the spatial features of n frames of the to-be-identified facial image into the temporal feature extraction module to obtain the temporal feature, including the following sub-steps: (1) input the spatial feature of the t-th frame of the to-be-identified facial image into the GRU network respectively to obtain a plurality of hidden states of the t-th frame of the to-be-identified facial image, where t = 1, 2, …, n; (2) the global attention layer uses the last hidden state and other hidden states of the t-th frame of the to-be-identified facial image to calculate the weight a t of the last hidden state of the t-th frame of the to-be-identified facial image, and a t = wherein: h t is the last hidden state of the to-be-identified facial image, h i is the other hidden state of the to-be-identified facial image, w t is a matrix for assisting in analyzing the correlation between h t and h i , which is helpful to confirm the to-be-identified facial image related to pain; (3) the second full connection layer uses the last hidden state and its weight of n frames of the to-be-identified facial image to calculate the temporal feature h f , and: wherein a ta weight representing a last hidden state of the face image to be identified in the t-th frame; h t a weight representing a last hidden state of the face image to be identified in the t-th frame.

[0099] The recognition unit 23 is configured to input the time sequence feature into the full connection layer to obtain the pain level of the face image to be identified.

[0100] In this embodiment, the first full connection layer includes a first full connection layer unit and a second full connection layer unit, the size of the first full connection layer is (2048, 1024), and the size of the second full connection layer is (1024, 1). Therefore, the recognition unit 23 is configured to input the time sequence feature h f into the full connection layer to obtain the pain level of the face image to be identified, which is: performing nonlinear processing and linear processing on the time sequence feature h

[0101] Based on the face expression-based pain recognition method provided in this embodiment, the face expression-based pain recognition device is a terminal device, which can be a server, a desktop computing device, or a mobile computing device (for example, a laptop computer, a handheld computing device, a tablet computer, a netbook, etc.), and includes:

[0102] a processor;

[0103] a memory configured to store a computer program executed by the processor;

[0104] When the processor executes the computer program, the terminal repeated data determination method provided in this embodiment is implemented.

[0105] Based on the face expression-based pain recognition method provided in this embodiment, the face expression-based pain recognition device is a terminal device, which can be a server, a desktop computing device, or a mobile computing device (for example, a laptop computer, a handheld computing device, a tablet computer, a netbook, etc.), and includes:

[0106] The application can take the form of a computer program product accessible from a storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code for use by or in connection with a computer. Computer-usable storage media include permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0107] The above-described embodiments only express the preferred embodiments of the present application, which are described in detail and specifically, but cannot be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, and the present application also intends to include these modifications and improvements.

Claims

1. A pain recognition method based on facial expressions, characterized in that, include: Acquire a video of the face to be identified, and extract an AU image from the video of the face to be identified; The AU image is input into a pre-trained pain recognition model to obtain the pain level of the face image to be recognized; The pain recognition model includes a spatial feature extraction module, a temporal feature extraction module, and a first fully connected layer; The AU image is input into a pre-trained pain recognition model to obtain the pain level of the face image to be recognized, including: The AU image of the face image to be identified in the t-th frame is input into the spatial feature extraction module to obtain the spatial features of the face image to be identified in the t-th frame, where: t=1,2,…,n; The spatial features of the n frames of the face image to be identified are input into the temporal feature extraction module to obtain the temporal features; The temporal features are input into the fully connected layer to obtain the pain level of the face image to be identified; The spatial feature extraction module includes m dynamically connected DenseGCN graph convolutional units. The AU image of the face image to be identified is input into the spatial feature extraction module to obtain the spatial features of the face image to be identified: The AU image of the face image to be identified is input into the first dynamic DenseGCN graph convolutional unit to obtain the output result of the dynamic DenseGCN graph convolutional unit; The AU image of the face to be identified and at least one of the outputs of the first (i-1) dynamic DenseGCN graph convolutional units are input into the i-th dynamic DenseGCN graph convolutional unit to obtain the output of the i-th dynamic DenseGCN graph convolutional unit. The output of the m-th dynamic DenseGCN graph convolutional unit is the spatial feature of the face to be identified, where i = 2, 3, ..., m. The temporal feature extraction module includes a GRU network, a global attention layer, and a second fully connected layer. The spatial features of the face image to be identified are input into the temporal feature extraction module to obtain the temporal features of the face image to be identified, including: The spatial features of the face image to be identified in frame t are input into the GRU network to obtain multiple hidden states of the face image to be identified in frame t, where: t = 1, 2, ..., n; The global attention layer uses the last hidden state of the face image to be identified in frame t and other hidden states to calculate the weight α of the last hidden state of the face image to be identified in frame t. t ,and in: h t h represents the last hidden state of the face image to be identified. i For other hidden states of the face image to be identified, w t For auxiliary analysis h t and h i A correlation matrix, which helps to identify face images to be identified that are associated with pain; The second fully connected layer uses the last hidden state and its weights of the n frames of the face image to be identified to calculate the temporal feature h. f ,and: Where: α t h represents the weight of the last hidden state of the face image to be identified in frame t; t This represents the last hidden state of the face image to be identified in frame t.

2. The pain recognition method based on facial expression according to claim 1, characterized in that, Extracting the AU image from the video of the face to be identified includes: The video of the face to be identified is extracted frame by frame using OpenCV to obtain n frames of face images to be identified, where n is the number of frames in the video of the face to be identified. The PIPNET network is used to detect and locate faces in n face images to be identified, resulting in n frames of face images to be identified; Extract the AU image of each frame of the face to be identified.

3. The pain recognition method based on facial expressions according to claim 2, characterized in that, Extracting the AU image of each frame of the face to be identified includes: Multiple feature points of the face image to be identified are extracted using the PIPNet network; The AU region of the face image to be identified is located and cropped using multiple feature points to obtain the AU image.

4. The pain recognition method based on facial expression according to claim 3, characterized in that, The first fully connected layer includes a first fully connected layer unit and a second fully connected layer unit. The size of the first fully connected layer is (2048, 1024), and the size of the second fully connected layer is (1024, 1). The temporal features are input into the fully connected layer to obtain the pain level of the face image to be identified: The timing feature h is processed sequentially using the first fully connected layer unit and the second fully connected layer unit. f The pain level is obtained by performing nonlinear and linear processing.

5. A pain recognition device based on facial expressions, characterized in that, The facial expression-based pain recognition device uses the facial expression-based pain recognition method as described in claim 1.

6. A pain recognition device based on facial expressions, characterized in that, include: processor; A memory for storing computer programs executed by the processor; The processor executes the computer program to implement the pain recognition method based on facial expression as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed, implements the pain recognition method based on facial expression as described in any one of claims 1-4.

Images