Newborn pain data labeling enhancement method based on deep learning
Through deep learning ScorER model and Grad-CAM algorithm, the labeling bias in neonatal pain videos is identified and corrected, which improves the quality and consistency of pain data labeling, and supports the construction of large-scale pain databases.
Patent Information
- Application Number
- CN202510555788.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-29
- Publication Date
- 2025-08-29
AI Technical Summary
The prior art is difficult to effectively identify and correct labeling deviations caused by facial occlusion, posture changes and other people's movement interference in newborn pain videos or images, resulting in low data quality and difficult to meet the needs of large-scale and high-quality pain databases.
Using a newborn pain data labeling enhancement method based on deep learning, the ScorER model combines multi-scale feature fusion and Grad-CAM algorithm to identify labeling deviations and generate heat maps to guide medical staff to improve labeling quality, including steps S1 to S7.
It realizes automatic identification of visual annotation deviations, improves label consistency and accuracy, and supports the construction of large-scale high-quality pain databases.
Smart Images
Figure CN120565101A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical artificial intelligence, and specifically relates to a deep learning-based neonatal pain data annotation enhancement method, which aims to improve the quality of data annotation in pain management. Background Art
[0002] Newborns undergo multiple painful medical procedures during their hospitalization, and pain has both short-term and long-term adverse effects on their brain development. Therefore, neonatal pain management is crucial, and accurate pain assessment is an important prerequisite. Because newborns are unable to self-report pain, current assessments rely mainly on medical staff using pain scales. However, this method is susceptible to individual experience differences and is time-consuming and labor-intensive, making it difficult to meet the needs of long-term, continuous monitoring.
[0003] Currently, the application of artificial intelligence in the field of automatic pain assessment of neonates has attracted much attention. The neonatal pain database is the basis of related research, but it is difficult for expert teams to achieve large-scale on-site annotation (OS-NPA), so annotation based on visual data (VB-NPA) has become a more efficient and convenient alternative. However, videos or images of neonatal pain taken in real scenes may contain various interferences, such as facial occlusion, posture changes, body occlusion, and movement interference from others. These information interferences often lead to annotation deviations, affecting data quality and the development of automated pain assessment systems. Existing solutions lack the ability to automatically identify annotation deviations and explore the causes of annotation deviations, making it difficult to meet the needs of large-scale, high-quality clinical databases. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies, the present invention provides a deep learning-based neonatal pain data annotation enhancement method to solve the deviation problem in visual pain annotation (VB-NPA), improve the annotation quality of medical staff, and support the development of automated pain assessment systems.
[0005] The technical solution adopted by the present invention to solve its technical problem is:
[0006] A method for enhancing the annotation of neonatal pain data based on deep learning, comprising the following steps:
[0007] Step S1, obtaining video data of a neonatal pain response and extracting a key frame facial image;
[0008] Step S2: Using the ScorER model, the local details and global structure of facial features are captured through multi-scale feature fusion technology;
[0009] Step S3, integrating the visual pain annotation VB-NPA score and training the ScorER model;
[0010] Step S4: input the facial image and VB-NPA score, and use the trained model to derive;
[0011] Step S5: Generate a heat map using the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm to locate the key deviation areas;
[0012] Step S6: combining the pain scale and facial key points to realize facial key area division, and mapping each area to a corresponding sub-item of the Neonatal Facial Coding System (NFCS);
[0013] Step S7: Generate annotation prompts based on the regional distribution of the heat map to guide the improvement of annotation quality.
[0014] Furthermore, the process of step S1 is as follows:
[0015] Step S1.1, using a handheld electronic device to record a video of a neonatal pain response in the neonatal intensive care unit, using the NFCS to assess pain, and extracting key frames representing the highest pain intensity from the video;
[0016] Step S1.2: Extract the newborn's facial area through the facial detection system and divide the dataset into a training set and a test set.
[0017] Furthermore, the process of step S2 is as follows:
[0018] Step S2.1: Convolve the newborn's facial image with a 3x3 convolution kernel, then perform batch dimension normalization, activate the feature map using the ReLU activation function, perform convolution and batch normalization again, and obtain a preliminary extracted feature map;
[0019] Step S2.2: Divide the input facial feature map into multiple scale blocks. Each scale block is processed by a convolutional layer and a nonlinear activation function. The formula is:
[0020] F1,F2,...,F n =Split(F in )
[0021] Among them, F1, F2, ..., F n Represents the feature maps of n different scale blocks, For the input feature map, each feature block is fused with the feature block of the next scale after a convolution layer and a nonlinear activation function.
[0022] F i ′=Conv(F i )+F i+1
[0023] The converted multi-scale features are finally fused by weighted summation.
[0024]
[0025] Among them, α i is the weight of the i-scale block, which is learned through model training to generate fusion features.
[0026] Furthermore, the process of step S3 is as follows:
[0027] Step S3.1: Fuse the VB-NPA scores and map them to a high-dimensional feature space through affine transformation:
[0028] vb′=W·vb+b
[0029] in, is the weight matrix of the fully connected layer, is the bias vector, It is the high-dimensional feature after transformation;
[0030] Step S3.2: If the dataset contains the on-site pain annotation OS-NPA, the deviation value ScB between VB-NPA and OS-NPA is used as the label; if the dataset only contains the visual pain annotation VB-NPA, VB-NPA is used as the label;
[0031] Step S3.3: Use the focal loss function to train the ScorER model to deal with the imbalanced distribution of annotation biases.
[0032] Step S3.4: Test and optimize the ScorER model using the test set to obtain the optimal ScorER model.
[0033] In step S4, the VB-NPA score data and the neonatal pain facial image are input into the trained ScorER model, and a decision is made through model reasoning. When trained with ScB, it determines whether there is a deviation in the labeling. When trained with VB-NPA, it outputs the pain assessment result instead of the deviation judgment.
[0034] In step S5, the Grad-CAM algorithm is used to generate a heat map to identify the key areas of the annotation deviation. The global average weight of the k channels of the last convolutional feature map corresponding to the c category is first calculated. Then the feature maps are weighted and combined and activated using ReLU to obtain Grad-CAM, the formula is:
[0035]
[0036] Among them, y c is the output score of the model for the target category c, which is the linear value before the Softmax layer. is the value of the kth channel at position (i, j) in the feature map of the last convolutional layer, H and W are the height and width of the feature map respectively, K is the number of channels of the last convolutional layer, and ReLU is used to remove negative areas that do not contribute to the target category.
[0037] In step S6, the facial key points include the left eye, the right eye, the nose, the left corner of the mouth, and the right corner of the mouth. The key points of the left and right eyes and the left and right corners of the mouth are connected, and two vertical lines perpendicular to the two connecting lines are drawn through the nose key point respectively, and two dividing lines are made through the midpoints of the two vertical lines. The dividing lines are parallel to the connecting lines of the eye and mouth key points. The area above the upper dividing line is the eye area, which reflects the pain response of the eyebrows and eyes (NFCS sub-items "winking" and "eyebrow bulging"); the area between the two dividing lines is the nasolabial area, which reflects the pain response of the nasolabial groove (NFCS sub-item "deepening of nasolabial groove"); the area below the lower dividing line is the oral area, which reflects the pain response of the oral cavity and tongue (NFCS sub-items "oral stretching" and "tongue curling").
[0038] The process of step S7 is as follows:
[0039] Step S7.1: Use perspective mapping technology to align the input facial image to the standard facial coordinate system. Use perspective mapping to project the newborn's face from different shooting angles onto the standard face and quantitatively calculate the heat map distribution of each facial key area. In the perspective mapping transformation, the isomorphism matrix H is minimized by optimizing the input key point p. i =(x i ,y i ) and the standard key point q i =(x′ i ,y′ i ) is calculated using the registration error between the two, and the formula is:
[0040]
[0041] The input image is then converted to a standard coordinate system using the relationship:
[0042] q=Hp
[0043] Where q = (x′, y′) represents the alignment coordinate.
[0044] Step S7.2: Calculate the distribution ratio of the heat map in the eye area, nasolabial area, and oral area using the following formula:
[0045]
[0046] Among them, i represents the index of three regions, namely the eye area, nose and labial area and oral area, k iis the area normalization factor, Heatmap(x,y) represents the heatmap intensity value at the coordinate (x,y);
[0047] Generate annotation prompts based on the heat map regional distribution. The heat map distribution reflects the potential causes of annotation deviation and maps them to the corresponding relevant scale sub-items to guide the improvement of annotation quality.
[0048] Among them, when trained with VB-NPA, the heat map highlights the attention of key facial areas, assisting annotators to assess pain more accurately without directly reflecting the source of bias.
[0049] The beneficial effects of the present invention are mainly reflected in: 1. Automatic identification of visual annotation deviations and improved annotation consistency; 2. Data-driven analysis of the causes of deviations to provide accurate improvement suggestions; 3. Adaptation to clinical scenarios and support for the construction of large-scale high-quality pain databases. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 This is the overall flow chart of the deep learning-based neonatal pain data annotation enhancement method, which is used to show the complete steps of pain annotation deviation identification and prompting;
[0051] Figure 2 This is a block diagram of the neonatal pain data annotation enhancement system based on deep learning, illustrating the structure and relationship of the data preprocessing, ScorER model architecture, and annotation prompt generation modules;
[0052] Figure 3 This is a schematic diagram of the facial region segmentation of a newborn, showing the facial regions corresponding to the NFCS sub-items, the facial segmentation method, and the method for prompting the error region; DETAILED DESCRIPTION
[0053] The present invention will be further described below with reference to the accompanying drawings.
[0054] Reference Figure 1 , a deep learning-based neonatal pain data annotation enhancement method, comprising the following steps:
[0055] Step S1: Obtain video data of the newborn's pain response and extract key frame facial images. The process is as follows:
[0056] Step S1.1: 585 pain response videos were recorded in the neonatal intensive care unit using a handheld electronic device. Pain was assessed using the simplified neonatal facial coding system (NFCS), and key frames reflecting the highest pain intensity were extracted.
[0057] Step S1.2: Use the facial detection system to extract the newborn's facial region. Use the deviation ScB between the VB-NPA and the on-site pain annotation OS-NPA as the label to divide the training set and the test set. The distribution of the samples is shown in Table 1:
[0058] Table 1 Sample data distribution table
[0059]
[0060]
[0061] Step S2: Using the ScorER model, we capture the local details and global structure of facial features through multi-scale feature fusion technology. The process is as follows:
[0062] Step S2.1. Preferably, the batch size is selected as 32, and a 3*3 convolution kernel is used to convolve the newborn face image, and then the batch dimension is normalized once, the feature map is activated using the ReLU activation function, and convolution and batch normalization are performed again to obtain a preliminary extracted feature map.
[0063] Step S2.2: Divide the input facial feature map into multiple scale blocks. Each scale block is processed by a convolutional layer and a nonlinear activation function. The specific formula is:
[0064] F1,F2,...,F n =Split(F in )
[0065] Among them, F1, F2, ..., F n Represents the feature maps of n different scale blocks, For the input feature map, each feature block is fused with the feature block of the next scale after a convolution layer and a nonlinear activation function.
[0066] F i ′=Conv(F i )+F i+1
[0067] The converted multi-scale features are finally fused by weighted summation.
[0068]
[0069] Among them, α i is the weight of the i-scale block, which is learned through model training to generate fusion features.
[0070] Step S3: Integrate the visual data-based neonatal pain annotation (VB-NPA) score information to train the ScorER model. The process is as follows:
[0071] Step S3.1: Fuse the VB-NPA scores and map them to a high-dimensional feature space through affine transformation:
[0072] vb′=W·vb+b
[0073] in, is the weight matrix of the fully connected layer, is the bias vector, It is the high-dimensional feature after transformation;
[0074] Step S3.2: Use the focal loss function to train the ScorER model, with the preferred parameters α = 0.25 and γ = 2 to solve the problem of unbalanced distribution of annotation bias.
[0075] Step S3.3: Test and optimize the ScorER model using the test set. Preferably, this example is trained on 8 Nvidia RTX 8000 GPUs, using the Adam optimizer (learning rate 0.0001), a batch size of 32, and 30 training cycles to obtain the optimal ScorER model. The classification accuracy of ScorER under five different random seeds on the dataset of this example is shown in Table 2. The test results show that the average classification accuracy of ScorER is 74.79%, which is better than models such as VGG16 and ResNet.
[0076] Table 2 Classification accuracy statistics of different backbone networks
[0077]
[0078] Step S4: input the facial image and VB-NPA score, and use the trained model to identify the annotation deviation;
[0079] Step S5: Use the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm to locate the key areas of the annotation deviation. Preferably, the Grad-CAM algorithm is used to generate a heat map to identify the key areas of the annotation deviation. First, the global average weight of the k channels of the last convolutional feature map corresponding to the c category is calculated. Then the feature maps are weighted and combined and activated using ReLU to obtain Grad-CAM, the formula is:
[0080]
[0081] Among them, y c is the output score of the model for the target category c, which is the linear value before the Softmax layer. is the value of the kth channel at position (i, j) in the feature map of the last convolutional layer, H and W are the height and width of the feature map respectively, K is the number of channels of the last convolutional layer, and ReLU is used to remove negative areas that do not contribute to the target category.
[0082] Step S6: Combine the pain scale and facial key points to achieve facial key area division, the facial key points include the left eye, right eye, nose, left corner of the mouth and right corner of the mouth, connect the key points of the left and right eyes and the left and right corners of the mouth, and draw two vertical lines perpendicular to the two connecting lines through the nose key point. Figure 3 Schematic diagram of facial region division. The distances h1 and h2 from the upper dividing line to the line connecting the eye key points and the lower dividing line to the line connecting the mouth key points are adjustable according to the scale type. In this embodiment, they are the distances from the nose key point to the midpoint of the two vertical lines, and the two dividing lines are parallel to the lines connecting the eye and mouth key points.
[0083] The area above the upper dividing line is the periocular region, reflecting pain responses in the eyebrows and eyes (NFCS sub-items "winking" and "brow raising"). The area between the two dividing lines is the nasolabial region, reflecting pain responses in the nasolabial folds (NFCS sub-item "deepening of the nasolabial folds"). The area below the lower dividing line is the oral region, reflecting pain responses in the mouth and tongue (NFCS sub-items "oral extension" and "tongue lifting"). Each area is mapped to the corresponding sub-item of the simplified neonatal facial coding system (NFCS).
[0084] Step S7: Generate annotation tips based on the distribution of the heat map in each area to guide the annotator to improve the annotation quality. The process is as follows:
[0085] Step S7.1: Use perspective mapping technology to align the input facial image to the standard facial coordinate system. Use perspective mapping to project the newborn's face from different shooting angles onto the standard face and quantitatively calculate the heat map distribution of each facial key area. In the perspective mapping transformation, the isomorphism matrix H is minimized by optimizing the input key point p. i =(x i ,y i ) and the standard key point q i =(x′ i ,y′ i ) is calculated using the registration error between the two, and the formula is:
[0086]
[0087] The input image is then converted to a standard coordinate system using the relationship:
[0088] q=Hp
[0089] Where q = (x′, y′) represents the alignment coordinate;
[0090] Step S7.2: Calculate the distribution ratio of the heat map in the eye area, nasolabial area, and oral area. The specific formula is:
[0091]
[0092] Among them, i represents the index of three regions, namely the eye area, nose and labial area and oral area, k i is the area normalization factor, Heatmap(x,y) represents the heatmap intensity value at the coordinate (x,y);
[0093] The sources of labeling bias were analyzed using heatmap distribution. Regions with higher intensity in the heatmap indicate areas of high model attention. Regions of highest attention indicate the areas most likely to contribute to labeling bias. The heatmap distribution reflects the potential causes of labeling bias and is mapped to the corresponding sub-items on the scale. In this approach, the heatmap provides information to guide nurses in focusing their attention on relevant areas during pain assessment. In the absence of labeling bias, the model's attention regions cover the entire face almost evenly, indicating that the model focuses on the entire face during inference. However, in the presence of labeling bias, the model's attention is focused on more localized areas. For example, the attention region is located in the mouth. This is because the newborn's mouth is particularly susceptible to obstruction, such as from an endotracheal tube, gauze, or the hands of medical personnel. This significantly impacts the assessment of the "tongue curl" feature in the NFCS. The heatmap information indicates facial regions that medical personnel should pay more attention to, facilitating more accurate pain assessment.
[0094] The above technical solution uses a dataset built by the present invention. In the design of an automatic neonatal pain model based on deep learning, the dataset may only contain VB-NPA information. For data without OS-NPA information, the present invention directly uses VB-NPA as the label to train the ScorER model. The trained model in step S4 will not discriminate against labeling deviations, but will instead output pain assessment results. The heat map obtained in step S7 will highlight the model's focus area, assisting the annotator in more accurately assessing the NFCS sub-item. The other steps remain unchanged.
[0095] Reference Figure 2 The deep learning-based neonatal pain data annotation enhancement system includes three major modules, namely data preprocessing, ScorER model architecture and annotation prompt generation module.
[0096] The data preprocessing module is used to preprocess the data, extract key facial information from neonatal pain videos or images in real clinical scenarios, and normalize them to a unified input format for the model.
[0097] The ScorER model architecture module provides the model's detailed architecture and consists of three submodules: feature extraction, multi-scale feature fusion, and the decision block. Combining facial features with VB-NPA and OS-NPA information, ScorER can determine whether there is labeling bias. For datasets where only VB-NPA information is available, this module uses VB-NPA as the label to train the ScorER model. The decision block does not detect labeling bias and instead outputs a pain assessment result.
[0098] The annotation prompt module is used to output prompt information to assist medical staff in achieving higher-quality data annotation. The heat map will indicate the possible areas causing deviations and map the faces to the sub-items of the scoring scale. Manual reference is made to the auxiliary information and the final decision is made.
[0099] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.
Claims
1. A method for enhancing the annotation of neonatal pain data based on deep learning, characterized in that: The method comprises the following steps: Step S1, obtaining video data of a neonatal pain response and extracting a key frame facial image; Step S2: Using the ScorER model, the local details and global structure of facial features are captured through multi-scale feature fusion technology; Step S3, integrating the visual pain annotation VB-NPA score and training the ScorER model; Step S4: input the facial image and VB-NPA score, and use the trained model to derive; Step S5: Generate a heat map using the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm to locate the key deviation areas; Step S6: combining the pain scale and facial key points to realize facial key area division, and mapping each area to a corresponding sub-item of the Neonatal Facial Coding System (NFCS); Step S7: Generate annotation prompts based on the regional distribution of the heat map to guide the improvement of annotation quality.
2. A method for enhancing the annotation of neonatal pain data based on deep learning according to claim 1, characterized in that: The process of step S1 is as follows: Step S1.1: Use a handheld electronic device to record a video of a neonatal pain response in the neonatal intensive care unit, assess pain using the NFCS, and extract key frames representing the highest pain intensity from the video; Step S1.2: Extract the newborn's facial area through the facial detection system and divide the dataset into a training set and a test set.
3. A method for enhancing the annotation of neonatal pain data based on deep learning according to claim 1 or 2, characterized in that: The process of step S2 is as follows: Step S2.1: Convolve the newborn's facial image with a 3x3 convolution kernel, then perform batch dimension normalization, activate the feature map using the ReLU activation function, perform convolution and batch normalization again, and obtain a preliminary extracted feature map; Step S2.2: Divide the input facial feature map into multiple scale blocks. Each scale block is processed by a convolutional layer and a nonlinear activation function. The formula is: F1,F2,...,F n =Split(F in ) Among them, F1, F2, ..., F n Represents the feature maps of n different scale blocks, For the input feature map, each feature block is fused with the feature block of the next scale after a convolution layer and a nonlinear activation function. F i ′ =Conv(F i )+F i+1 The converted multi-scale features are finally fused by weighted summation. Among them, α i is the weight of the i-scale block, which is learned through model training to generate fusion features.
4. A method for enhancing the annotation of neonatal pain data based on deep learning according to claim 1 or 2, characterized in that: The process of step S3 is as follows: Step S3.1: Fuse the VB-NPA scores and map them to a high-dimensional feature space through affine transformation: vb′=W·vb+b in, is the weight matrix of the fully connected layer, is the bias vector, It is the high-dimensional feature after transformation; Step S3.2: If the dataset contains the on-site pain annotation OS-NPA, the deviation value ScB between VB-NPA and OS-NPA is used as the label; if the dataset only contains the visual pain annotation VB-NPA, VB-NPA is used as the label; Step S3.3: Use the focal loss function to train the ScorER model to deal with the imbalanced distribution of annotation biases. Step S3.4: Test and optimize the ScorER model using the test set to obtain the optimal ScorER model.
5. A method for enhancing the annotation of neonatal pain data based on deep learning according to claim 1 or 2, characterized in that: In step S4, the VB-NPA score data and the neonatal pain facial image are input into the trained ScorER model, and a decision is made through model reasoning. When trained with ScB, it is determined whether there is a deviation in the labeling. When trained with VB-NPA, the pain assessment result is output instead of the deviation judgment.
6. A method for enhancing the annotation of neonatal pain data based on deep learning according to claim 1 or 2, characterized in that: In step S5, the Grad-CAM algorithm is used to generate a heat map to identify the key areas of the annotation deviation. The global average weight of the k channels of the last convolutional feature map corresponding to the c category is first calculated. Then the feature maps are weighted and combined and ReLU activation is used to obtain the Grad-CAM heat map The formula is: Among them, y c is the output score of the model for the target category c, which is the linear value before the Softmax layer. is the value of the kth channel at position (i, j) in the feature map of the last convolutional layer, H and W are the height and width of the feature map respectively, K is the number of channels of the last convolutional layer, and ReLU is used to remove negative areas that do not contribute to the target category.
7. A method for enhancing the annotation of neonatal pain data based on deep learning according to claim 1 or 2, characterized in that: In step S6, the facial key points include the left eye, the right eye, the nose, the left corner of the mouth and the right corner of the mouth. The key points of the left and right eyes and the left and right corners of the mouth are connected, and two vertical lines perpendicular to the two connecting lines are drawn through the nose key point. Two dividing lines are made through the midpoints of the two vertical lines. The dividing lines are parallel to the connecting lines of the eye and mouth key points. The area above the upper dividing line is the eye area, which reflects the pain response of the eyebrows and eyes; the area between the two dividing lines is the nasolabial area, which reflects the pain response of the nasolabial groove; the area below the lower dividing line is the oral area, which reflects the pain response of the mouth and tongue.
8. A method for enhancing the annotation of neonatal pain data based on deep learning according to claim 1 or 2, characterized in that: The process of step S7 is as follows: Step S7.1: Use perspective mapping technology to align the input facial image to the standard facial coordinate system. Use perspective mapping to project the newborn's face from different shooting angles onto the standard face and quantitatively calculate the heat map distribution of each facial key area. In the perspective mapping transformation, the isomorphism matrix H is minimized by optimizing the input key point p. i =(x i ,y i ) and the standard key point q i =(x i ′,y i The registration error between ′) is calculated as follows: The input image is then converted to a standard coordinate system using the relationship: q=Hp Where q = (x′, y′) represents the alignment coordinate; Step S7.2: Calculate the distribution ratio of the heat map in the eye area, nasolabial area, and oral area using the following formula: Among them, i represents the index of three regions, namely the eye area, nose and labial area and oral area, k i is the area normalization factor, Heatmap(x,y) represents the heatmap intensity value at the coordinate (x,y); Generate annotation prompts based on the heat map regional distribution. The heat map distribution reflects the potential causes of annotation deviation and maps them to the corresponding relevant scale sub-items to guide the improvement of annotation quality. Among them, when trained with VB-NPA, the heat map highlights the attention of key facial areas, assisting annotators to assess pain more accurately without directly reflecting the source of bias.