Neonatal pain expression recognition method based on generative adversarial network

Abstract

A method for recognizing neonatal pain expressions based on a generative adversarial network, constructing a generative adversarial network to learn how to recover unobstructed and correct facial images from neonatal facial images with different postures and occlusions; treating the hidden variables of the generator in the generative adversarial network as modified facial pain features; constructing a residual network combined with an attention mechanism to screen and analyze the facial pain features modified by the zero-sum game, so as to further get rid of the influence of occlusion and posture changes, and output accurate pain level results. The present invention is committed to improving the accuracy of neonatal pain expression recognition in real environments, enhancing the robustness of neonatal pain recognition methods to occlusion and adaptability to posture changes, optimizing the extraction of pain features based on a generative adversarial network, and implementing the screening of pain features through an attention mechanism, effectively solving the problem of neonatal pain expression recognition in environments with occlusion and posture changes.

Description

Technical Field

[0001] The invention relates to the field of neonatal pain recognition, and in particular to a neonatal pain recognition method based on facial expressions. Background Art

[0002] Neonatal pain not only causes physiological reactions, but also leads to a series of short-term or long-term adverse reactions, such as growth retardation, permanent central nervous system damage, emotional disorders, and even increases the risk of future diseases. Therefore, developing an automatic neonatal pain recognition algorithm to achieve continuous and objective pain assessment is of great significance for neonatal pain management and healthy growth. There have been some studies on automatic recognition of neonatal pain, such as the Chinese patent application "Neonatal Pain Expression Recognition Method Based on Dual-Channel Three-Dimensional Convolutional Neural Network" (patent application number CN201810145292.6, publication number CN108363979A), "Neonatal Pain Expression Recognition Method and System Based on Deep 3D Residual Network" (patent application number CN201810346075.3, publication number CN108596069A), "Neonatal Pain Expression Recognition Method Based on Two-Stream Convolutional Neural Network" (patent application number CN201910748936.5, publication number CN111401117A). However, existing studies including the above patent applications have only considered the recognition of neonatal pain expressions in ideal environments (controlled environments). These studies have made breakthrough progress in the use of deep learning-based methods for unobstructed and well-posed facial images of newborns. However, in real environments (uncontrolled environments), the existence of occlusions and variable head postures poses a huge challenge to the recognition of neonatal pain expressions. How to improve the accuracy of neonatal pain expression recognition in real environments and enhance the robustness of neonatal pain recognition methods to occlusion and adaptability to posture changes are key issues that need to be urgently addressed. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, considering the current lack of a neonatal facial expression pain method that is robust to occlusion and posture changes in real scenarios, the present invention provides a neonatal facial expression pain method that is robust to facial occlusion and posture changes, and uses a generative adversarial network to learn how to restore unobstructed and correct-posture facial images from neonatal facial images with different postures and occlusions, so as to get rid of the influence of occlusion and posture changes, obtain the intrinsic facial features of pain in neonates to complete the final pain level classification, and effectively solve the problem of neonatal pain recognition in an environment of occlusion and posture changes.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A method for recognizing neonatal pain expressions based on a generative adversarial network, the method comprising the following steps:

[0006] Step S1: Recovering an unobstructed and correct facial image from facial images of newborns with different postures and possible occlusions based on a generative adversarial network;

[0007] Step S2: using the latent vector of the generator in the generative adversarial network as the modified facial pain feature for subsequent pain analysis;

[0008] Step S3: Use the residual network combined with the attention mechanism to screen and analyze the corrected facial pain features to further get rid of the interference of occlusion and posture changes and output accurate pain level results.

[0009] Further, the process of step S1 is:

[0010] The generative adversarial network consists of two parts: a generator and a discriminator. The generator is responsible for generating a corrected facial image based on the input facial image; the discriminator is responsible for learning to distinguish between the image generated by the generator and the ideal facial image with no occlusion and correct posture in the guidance set. The guidance set g consists of all the ideal facial images in the training set. Through the zero-sum game between the generator and the discriminator, the generator will continuously improve its ability to convert the input image into an image with no occlusion and correct face; in the process of the zero-sum game, four loss functions are responsible for the training of the generator and the discriminator, and the parameters of the generator and the discriminator are adjusted to the optimal through the error back propagation algorithm. The four loss functions are described as follows:

[0011] (1) Symmetric loss function

[0012] Symmetry is an inherent feature of normal faces, and the symmetry loss is calculated as:

[0013]

[0014] Where H and W represent the height and width of the image, (n, m) represents the pixels of the image, and |·| represents the absolute value. Real-world images may not have absolute symmetry at the pixel level, so it is decided to minimize the symmetry loss in the Laplace space;

[0015] (2) Adversarial Loss Function

[0016] The discriminator network acts as a supervisor, responsible for distinguishing the generated face images from the ideal images, and is trained simultaneously with the generator. The discriminator is trained using the following cross entropy loss function:

[0017] L GAN-Dis (g i ,x′ j)=-log(Dis(g i ))-log(1-Dis(x′ j ))

[0018] where g i represents the guidance set image, GAN-Dis represents the discriminator, and x' j is the image generated by the generator;

[0019] For the generator, the adversarial loss function is calculated as:

[0020] L GAN-Gen (x′ j )=-log(Dis(x′ j ))

[0021] Among them, GAN-Gen represents the generator and Dis represents the discriminator;

[0022] (3) Identity Preservation Loss Function

[0023] Preserving identity is a key part of ideal face generation. Perceptual loss is used to maintain perceptual similarity to help the pain feature correction module obtain identity preservation capabilities. The loss function is calculated based on the feature maps output by the last two layers in the open source Light CNN:

[0024]

[0025] Among them, H l ,W l is the height and width of the last l-th layer feature map, Ω represents the feature map, |·| represents the absolute value, and the identity preservation loss aims to make the generated image have a smaller distance with the original image in the deep feature space. Considering that LightCNN can classify thousands of identities after pre-training, it is believed that it can capture the most important facial structure or features for identity recognition;

[0026] (4) Total Variation Regularization

[0027] In order to improve the spatial smoothness of the generated image and reduce spike artifacts, a Total Variation regularizer is used, which is defined as follows:

[0028]

[0029] Where H and W represent the height and width of the image, (n,m) represents the pixels of the image, x' is the image generated by the generator, and |·| represents the absolute value.

[0030] The process of step S3 is as follows:

[0031] A residual network combined with an attention mechanism is used to screen and analyze the facial pain features corrected by zero-sum game. For this part of the network structure, a conventional residual structure is used as the backbone, and the core part is the attention mechanism combined with it. An attention branch parallel to the residual branch is constructed. It is based on the bottom-up top-down structure and can output an attention mask with the same size as the residual structure feature map to soft-weight the facial features in the residual structure. In the bottom-up top-down structure, "downsampling" is achieved through a series of convolution and pooling operations, while "upsampling" is achieved through deconvolution operations. The attention mask output by this self-attention mechanism can not only be used as a feature selector in forward propagation, but also as a filter in backward gradient update. The gradient calculation method of facial features under the action of the attention mask is:

[0032]

[0033] Where M represents the attention branch, T represents the residual branch, σ represents the attention branch parameter, and φ is the residual branch parameter.

[0034] The beneficial effects of the present invention are as follows: in a real environment (uncontrolled environment), the face of a newborn is often occluded or changes in posture, resulting in invisible facial areas, which poses a huge challenge to the recognition of pain in newborns. Existing methods currently only consider the recognition of pain in unobstructed and well-posed facial images of newborns. In this regard, we train a generative adversarial network through semi-supervised learning to learn how to restore unobstructed and well-posed facial images from facial images of newborns with different postures and occlusions, thereby obtaining facial pain expression features that are less affected by occlusion and posture changes, namely the eigenvectors of the generator in the generative adversarial network. In addition, the subsequent screening and filtering of the attention mechanism finally obtains the intrinsic information of newborn pain expression to complete the recognition of newborn pain expression, effectively solving the problem of newborn pain expression recognition in an environment of occlusion and posture changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flowchart of the method for recognizing neonatal pain expressions based on generative adversarial networks;

[0036] Figure 2 It is a diagram of the pain feature modification module in the pain recognition model;

[0037] Figure 3 is an illustration of the pain level classification module in the pain recognition model. DETAILED DESCRIPTION

[0038] The technical scheme in the method of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0039] Reference Figure 1 to Figure 3 A method for recognizing neonatal pain expressions based on a generative adversarial network, the method comprising the following steps:

[0040] Step S1: Recovering an unobstructed and correct facial image from newborn facial images with different postures and possible occlusions based on a generative adversarial network; the process is:

[0041] The generative adversarial network consists of two parts: a generator and a discriminator. The generator is responsible for generating a corrected facial image based on the input facial image; the discriminator is responsible for learning to distinguish between the image generated by the generator and the ideal facial image with no occlusion and correct posture in the guidance set. The guidance set g consists of all the ideal facial images in the training set. Through the zero-sum game between the generator and the discriminator, the generator will continuously improve its ability to convert the input image into an image with no occlusion and correct face; in the process of the zero-sum game, four loss functions are responsible for the training of the generator and the discriminator, and the parameters of the generator and the discriminator are adjusted to the optimal through the error back propagation algorithm. The four loss functions are described as follows:

[0042] (1) Symmetric loss function

[0043] Symmetry is an inherent feature of normal faces, and the symmetry loss is calculated as:

[0044]

[0045] Where H and W represent the height and width of the image, (n, m) represents the pixels of the image, and |·| represents the absolute value. Real-world images may not have absolute symmetry at the pixel level, so it is decided to minimize the symmetry loss in the Laplace space;

[0046] (2) Adversarial Loss Function

[0047] The discriminator network acts as a supervisor, responsible for distinguishing the generated face images from the ideal images, and is trained simultaneously with the generator. The discriminator is trained using the following cross entropy loss function:

[0048] L GAN-Dis (g i ,x′ j )=-log(Dis(g i ))-log(1-Dis(x′j ))

[0049] where g i represents the guidance set image, GAN-Dis represents the discriminator, and x' j is the image generated by the generator;

[0050] For the generator, the adversarial loss function is calculated as:

[0051] L GAN-Gen (x′ j )=-log(Dis(x′ j ))

[0052] Among them, GAN-Gen represents the generator and Dis represents the discriminator;

[0053] (3) Identity Preservation Loss Function

[0054] Preserving identity is a key part of ideal face generation. Perceptual loss is used to maintain perceptual similarity to help the pain feature correction module obtain identity preservation capabilities. The loss function is calculated based on the feature maps output by the last two layers in the open source Light CNN:

[0055]

[0056] Among them, H l ,W l is the height and width of the last l-th layer feature map, Ω represents the feature map, |·| represents the absolute value, and the identity preservation loss aims to make the generated image have a smaller distance with the original image in the deep feature space. Considering that LightCNN can classify thousands of identities after pre-training, it is believed that it can capture the most important facial structure or features for identity recognition;

[0057] (4) Total Variation Regularization

[0058] In order to improve the spatial smoothness of the generated image and reduce spike artifacts, a Total Variation regularizer is used, which is defined as follows:

[0059]

[0060] Where H and W represent the height and width of the image, (n,m) represents the pixels of the image, x' is the image generated by the generator, and |·| represents the absolute value.

[0061] Step S2: using the latent vector of the generator in the generative adversarial network as the modified facial pain feature for subsequent pain analysis;

[0062] Step S3: Use the residual network combined with the attention mechanism to screen and analyze the corrected facial pain features to further get rid of the interference of occlusion and posture changes and output accurate pain level results. The process is as follows:

[0063] A residual network combined with an attention mechanism is used to screen and analyze the facial pain features corrected by zero-sum game. For this part of the network structure, a conventional residual structure is used as the backbone, and the core part is the attention mechanism combined with it. An attention branch parallel to the residual branch is constructed. It is based on the bottom-up top-down structure and can output an attention mask with the same size as the residual structure feature map to soft-weight the facial features in the residual structure. In the bottom-up top-down structure, "downsampling" is achieved through a series of convolution and pooling operations, while "upsampling" is achieved through deconvolution operations. The attention mask output by this self-attention mechanism can not only be used as a feature selector in forward propagation, but also as a filter in backward gradient update. The gradient calculation method of facial features under the action of the attention mask is:

[0064]

[0065] Where M represents the attention branch, T represents the residual branch, σ represents the attention branch parameter, and φ is the residual branch parameter.

[0066] The implementation process of the method for recognizing neonatal pain expression based on a generative adversarial network in this embodiment is as follows:

[0067] 1): Obtain a set of neonatal images in a real environment, wherein the set of neonatal images contains pain level labels, and the process is as follows:

[0068] A one-minute video was recorded in the neonatal ward using a handheld electronic device as the original data. To ensure the authenticity and clinical value of the data, the video was recorded only when clinical pain-inducing procedures occurred, and it was ensured that the handheld electronic device was not moved to follow the changes in the neonate's posture, and no occlusion occurred. After obtaining enough raw video data, nurses with professional training and pain assessment experience were selected to conduct pain level assessment in line with clinical standards; key frames were selected from each video to form a neonatal image set, and the pain level of the neonate was assessed according to the Neonatal Pain Scale (NIPS). Four pain states were generated based on the NIPS score, which were used as pain level labels for the neonatal image set, namely no pain (NIPS score: 0-1 points), mild pain (NIPS score: 2-3 points), moderate pain (NIPS score: 4-5 points) and severe pain (NIPS score: 6-7 points). Depending on whether the face was occluded by other objects (such as medical equipment or neonatal limbs), the dataset was divided into two subsets: "occluded" and "unoccluded". Then, the open source method in OpenCV was used to estimate the facial posture of the neonate and obtain Tait Bryan angles (pitch, yaw, roll). When the “pitch” angle is greater than 30° or the “yaw” angle is greater than 45°, the facial posture of the newborn is considered to be a non-ideal type. Based on this, the dataset is further divided into four subsets, and the test set in each subset is determined according to the common ratio of 7:3, and they are combined into the complete test set;

[0069] 2): Perform preprocessing operations on the newborn image set, including face detection, cropping and alignment; the process is:

[0070] Use open source ZFace to crop newborn facial images from newborn images. ZFace can output 49 facial landmarks and facial boundary points. After obtaining the newborn facial images, uniform alignment is performed. Specifically, compared with the yaw angle and pitch angle, the rolling angle of the face is easy to calibrate. Therefore, affine transformation is used to unify the facial images into a vertical state. Under the action of the affine matrix M, the two-dimensional facial images are aligned through linear transformation. Specifically, the coordinates of the key points in the original image are calculated, and the corresponding relationship with the coordinates of the key points in the reference face is used to dynamically obtain the affine transformation matrix M. The calculation process is:

[0071]

[0072] Among them, a1, b1, c1, a2, b2, c2 represent the values ​​to be determined in the three-dimensional affine matrix M, (а, β) are the original coordinates, and (u, v) are the transformed coordinates. The three steps of face detection, cropping and alignment in the preprocessing operation can be replaced by other related algorithms.

[0073] 3): Construct a pain recognition model, which includes a pain feature correction module and a pain level classification module; the process is:

[0074] like Figure 2 As shown in the figure, the pain feature correction module is mainly built based on the generative adversarial network. The generative adversarial network mainly consists of two parts: the generator and the discriminator. The discriminator is responsible for learning to distinguish the images generated by the generator from the guidance set. The guidance set g consists of all ideal facial images in the training set. Inspired by TP-GAN, the generator in the designed generative adversarial network has two paths, focusing on global shape and local details respectively. For the design of the local path, the open source MTCNN is first used to detect five facial landmarks related to pain, namely the left eye, the right eye, the nose, the upper part of the mouth, and the lower part of the mouth. The five regions cropped from these five facial landmarks as the center are used as the input of the five generators in the local path; the design of the global path is relatively conventional, using only one generator. The information fusion strategy between the two paths is similar to TP-GAN. The denoising auto-encoder (DAE) is selected as the generator. The denoising auto-encoder is characterized by receiving images that have been damaged by some form of "noise" and removing noise by requiring the output image to be similar to the ideal version of the original image. The input x (face image) can be regarded as an ideal face image that has been damaged by posture changes and partial occlusion. "Denoising" is achieved by requiring DAE to learn how to make the output image as close to the guidance set as possible. At the same time, four loss functions are used to give full play to the advantages of the generative adversarial network and DAE. The four loss functions are described as follows:

[0075] (1) Symmetric loss function

[0076] Symmetry is an inherent feature of normal faces, and the symmetry loss is calculated as:

[0077]

[0078] Where H and W represent the height and width of the image, (n, m) represents the pixels of the image, and |·| represents the absolute value. Real-world images may not have symmetry at the pixel level, so it is decided to minimize the symmetry loss in the Laplace space;

[0079] (2) Adversarial Loss Function

[0080] The discriminator network acts as a supervisor, responsible for distinguishing the generated face images from the ideal images, and is trained simultaneously with the generator. The discriminator is trained using the following cross entropy loss function:

[0081] L GAN-Dis (g i ,x′ j )=-log(Dis(g i ))-log(1-Dis(x′ j ))

[0082] where g i represents the guidance set image, GAN-Dis represents the discriminator, and x' j is the image generated by the generator;

[0083] For the generator, the adversarial loss function is calculated as:

[0084] L GAN-Gen (x′ j )=-log(Dis(x′ j ))

[0085] Among them, GAN-Gen represents the generator and Dis represents the discriminator;

[0086] (3) Identity Preservation Function

[0087] Preserving identity is a key part of ideal face generation. Perceptual loss is used to maintain perceptual similarity to help the pain feature modification module obtain identity preservation capabilities. Specifically, the loss function is calculated based on the feature maps output by the last two layers in the open source Light CNN:

[0088]

[0089] Among them, H l ,W l is the height and width of the last l-th layer feature map, Ω represents the feature map, |·| represents the absolute value, and the identity preservation loss aims to make the generated image have a smaller distance with the original image in the deep feature space. Considering that LightCNN can classify thousands of identities after pre-training, we believe that it can capture the most important face structure or features for identity recognition;

[0090] (4) Total Variation Regularization

[0091] In order to improve the spatial smoothness of the generated image and reduce spike artifacts, a Total Variation regularizer is used, which is defined as follows:

[0092]

[0093] Where H and W represent the height and width of the image, (n,m) represents the pixels of the image, x' is the image generated by the generator, and |·| represents the absolute value;

[0094] The pain level classification module is responsible for analyzing the corrected pain features to output the final pain level classification result, and the latent vector z (the output variable of the encoder in the generator) of the pain feature correction module is regarded as the corrected pain feature and input into the pain level classification module. Figure 3 As shown in the figure, for this part of the network structure, the residual structure is used as the backbone. In addition, an attention mechanism is added to construct an attention branch parallel to the residual branch. It is based on the bottom-up top-down structure and can output attention masks of the same size to softly weight facial features. In the bottom-up top-down structure, "downsampling" is achieved through a series of convolutions and pooling, while "upsampling" is achieved through deconvolution. The attention mask output by this self-attention mechanism can not only be used as a feature selector in forward propagation, but also as a filter in backward gradient update. The gradient calculation method of facial features under the action of the attention mask is:

[0095]

[0096] Where M represents the attention branch, T represents the residual branch, σ represents the attention branch parameter, and φ is the residual branch parameter;

[0097] 4): Use the neonatal pain image set to train and test the pain recognition model constructed; the process is:

[0098] The constructed neonatal pain recognition model is trained using the training set of the neonatal image set. For the training of the pain correction module, the loss function of the generator is calculated as follows:

[0099] L Gen =λ1L tv +λ2L id +λ3L sym +ηL GAN-Gen

[0100] Among them, λ1, λ2 and λ3 are the weights of the generator’s Total Variation regularization loss function, identity preservation loss function and symmetry loss function respectively. Since the generator also receives the error signal from the discriminator, the parameter η is used as the weight of the adversarial loss function. The specific values ​​of the above parameters are as follows: λ1 = 5 × 10 -3 ;λ2=3×10 -2;λ3=0.3;η=0.1. For the training of the discriminator, only the adversarial loss function is used, that is, L GAN-Dis . In addition, considering the difficulty of completely removing occlusions in real scenes and the importance of five facial landmarks to pain assessment, the occlusion removal of the local path output image is strengthened. Specifically, after the pain correction module is trained once, a new generative adversarial network is constructed for each of the five DAEs in the local path, that is, the output image of the local path and the image at the corresponding position in the guidance set are input into a new discriminator to achieve adversarial training of the DAE in the local path. After the training of the pain correction module is completed, the pain level classification module is trained, and the parameters of the module are adjusted to the optimal value through the error back propagation algorithm. The loss function of the pain level classification module is based on the mean square error loss function with a constraint based on the attention weight added. The gradient calculation method under the action of the attention mask is described in detail in step S3. For the test samples in the neonatal image set, the trained neonatal pain recognition model is used to perform neonatal pain recognition on them to obtain their corresponding pain levels.

[0101] Based on the above method, the present invention was verified on the newborn expression set, and the performance comparison between the present invention and other newborn pain expression recognition methods is shown in Table 1. As shown in Table 1, our invention shows significant performance advantages in the face of occlusion and posture changes. In addition, an ablation experiment was conducted on the effect of the attention branch on improving the accuracy of newborn pain recognition, and the results are shown in Table 2.

[0102]

[0103] Table 1

[0104]

[0105] Table 2

[0106] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for recognizing neonatal pain expressions based on generative adversarial networks, characterized in that: The method comprises the following steps: Step S1: Recovering an unobstructed and correct facial image from facial images of newborns with different postures and possible occlusions based on a generative adversarial network; Step S2: using the eigenvector of the generator in the generative adversarial network as the modified facial feature for subsequent pain analysis; Step S3: Use the residual network combined with the attention mechanism to screen and analyze the corrected facial features to further get rid of the interference of occlusion and posture changes and output accurate pain level results; The process of step S1 is as follows: The generative adversarial network consists of two parts: a generator and a discriminator. The generator is responsible for generating a corrected facial image based on the input facial image; the discriminator is responsible for learning to distinguish between the image generated by the generator and the ideal facial image with no occlusion and correct posture in the guidance set. The guidance set g consists of all the ideal facial images in the training set. Through the zero-sum game between the generator and the discriminator, the generator will continuously improve its ability to convert the input image into an image with no occlusion and correct face; in the process of the zero-sum game, four loss functions are responsible for the training of the generator and the discriminator, and the parameters of the generator and the discriminator are adjusted to the optimal through the error back propagation algorithm. The four loss functions are described as follows: (1) Symmetric loss function Symmetry is an inherent feature of normal faces, and the symmetry loss is calculated as: Where H and W represent the height and width of the image, (n, m) represents the pixels of the image, and |·| represents the absolute value. Real-world images do not have absolute symmetry at the pixel level, so it is decided to minimize the symmetry loss in the Laplace space; (2) Adversarial Loss Function The discriminator network acts as a supervisor, responsible for distinguishing the generated face images from the ideal images, and is trained simultaneously with the generator. The discriminator is trained using the following cross entropy loss function: L GAN-Dis (g i ,x j ′)=-log(Dis(g i ))-log(1-Dis(x j ′)) where g i represents the guidance set image, GAN-Dis represents the discriminator, and x j , is the image generated by the generator; For the generator, the adversarial loss function is calculated as: L GAN-Gen (x j ′)=-log(Dis(x j ′)) Among them, GAN-Gen represents the generator and Dis represents the discriminator; (3) Identity Preservation Loss Function Preserving identity is a key part of ideal face generation. Perceptual loss is used to maintain perceptual similarity to help the pain feature correction module obtain identity preservation capabilities. The loss function is calculated based on the feature maps output by the last two layers in the open source Light CNN: Among them, H l ,W l is the height and width of the last l-th layer feature map, Ω represents the feature map, |·| represents the absolute value, and the identity preservation loss aims to make the generated image have a smaller distance with the original image in the deep feature space. Considering that Light CNN has been pre-trained to classify thousands of identities, it is believed that it captures the most important facial structure or features for identity recognition; (4) Total Variation Regularization In order to improve the spatial smoothness of the generated image and reduce spike artifacts, a Total Variation regularizer is used, which is defined as follows: Where H and W represent the height and width of the image, (n,m) represents the pixels of the image, x is the image generated by the generator, and |·| represents the absolute value.

2. The method for recognizing neonatal pain expression based on a generative adversarial network according to claim 1, characterized in that: The process of step S3 is as follows: A residual network combined with an attention mechanism is used to screen and analyze the facial pain features corrected by zero-sum game. An attention branch parallel to the residual branch is constructed. It is based on a bottom-up top-down structure and outputs an attention mask with the same size as the feature map in the residual structure, which softly weights the facial features in the residual structure. In the bottom-uptop-down structure, "downsampling" is achieved through a series of convolution operations and pooling operations, while "upsampling" is achieved through deconvolution operations; the attention mask output by this self-attention mechanism is not only used as a feature selector in the forward propagation, but also as a filter in the backward gradient update. The gradient calculation method of facial features under the attention mask is: Where M represents the attention branch, T represents the residual branch, σ represents the attention branch parameter, and φ is the residual branch parameter.

Images