Model training method of pain level evaluation model and pain level evaluation method

Through multi-view video stream capture and uterine contraction interval interception technology, multi-dimensional heat map volume is generated, which solves the problem that pain assessment systems in the prior art is difficult to capture dynamic changes in the face, and improves the accuracy of pain level assessment and individual correlation of evaluation results. sex.

CN119091491APending Publication Date: 2024-12-06JIANGSU APON MEDICAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411221939.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing pain assessment system is difficult to accurately capture dynamic changes in facial expressions during the maternal labour process, resulting in a low accuracy rate of pain level assessment.

Method used

Multi-view video stream capture technology and uterine contraction intervals are used to capture video clips, generate node heat maps of facial feature points and stack them along the time dimension to form multi-dimensional heat map volumes, which are used to train pain level evaluation models to ensure that the model can capture facial expressions. Continuous change.

Benefits of technology

It significantly improves the accuracy of pain level assessment, overcomes the problem of incomplete capture of facial features in a single perspective, and ensures the comprehensive collection of facial expression information of the maternal mother and the individual correlation of evaluation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091491A_ABST
    Figure CN119091491A_ABST
Patent Text Reader

Abstract

The invention provides a model training method of a pain grade evaluation model and a pain grade evaluation method. The model training method comprises the following steps: acquiring a plurality of first video clips and a pain grade label corresponding to each first video clip; for any first video clip, performing image segmentation on each video frame in the first video clip based on the facial feature points to obtain a second video clip; for any sample video frame, generating a node heat map of the sample video frame; stacking a plurality of node heat maps in one-to-one correspondence with the plurality of sample video frames according to the time dimensions of the plurality of sample video frames to generate a multi-dimensional heat map volume of the second video clip; and based on the plurality of multi-dimensional heat map volumes and the pain grade label corresponding to each multi-dimensional heat map volume, performing model training on the pain grade evaluation model to obtain a trained pain grade evaluation model. According to the embodiment of the invention, the accuracy of pain assessment can be greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of video processing, and in particular to a model training method of a pain level assessment model and a pain level assessment method. Background Art

[0002] Existing pain assessment systems for pregnant women in labor generally predict pain levels by extracting various forms of features from single-frame images, but this prediction method lacks temporal information and ignores the dynamic and subtle changes in facial expressions in the time dimension. Painful expressions are usually continuous and gradual, and it is difficult for a single-frame image to capture these important temporal features. In addition, the facial features of a person in a single-frame image or multi-frame input video segment may be affected by factors such as instantaneous expression changes, lighting, and occlusion, resulting in incomplete information, which in turn leads to a low accuracy rate in pain assessment for pregnant women. Summary of the invention

[0003] In view of this, the present disclosure proposes a model training method for a pain level assessment model and a pain level assessment method to solve the problem of low pain assessment accuracy existing in the related art.

[0004] The first embodiment of the present disclosure proposes a model training method for a pain level assessment model, comprising:

[0005] Acquire multiple first video clips and a pain level label corresponding to each first video clip; each first video clip is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval;

[0006] For any first video segment among the plurality of first video segments, image segmentation is performed on each video frame in the first video segment based on facial feature points to obtain a second video segment; the second video segment includes a plurality of sample video frames including a human face region; the human face region is composed of a plurality of facial feature points;

[0007] For any sample video frame, a node heat map of the sample video frame is generated; the node heat map is used to characterize the position and probability distribution of each facial feature point in the sample video frame;

[0008] According to the time dimension of the plurality of sample video frames, a plurality of node heat maps corresponding one to one to the plurality of sample video frames are stacked to generate a multi-dimensional heat map volume of the second video segment; the multi-dimensional heat map volume is used to characterize the dynamic change of each facial feature point in the time dimension;

[0009] Based on multiple multidimensional heat map volumes and the pain level label corresponding to each multidimensional heat map volume, a pain level assessment model is trained to obtain a trained pain level assessment model; the trained pain level assessment model is used to evaluate the first video clip to be evaluated to obtain the corresponding pain level.

[0010] The disclosed embodiment generates a node heat map for each sample video frame, and stacks multiple node heat maps corresponding to multiple sample video frames one by one according to the time dimension of the multiple sample video frames to generate a multidimensional heat map volume for the second video clip. It can be combined with objective facts, that is, the main characteristics of facial expression changes come from facial feature points, and the change of expression is a continuous and small / local process. Facial feature points are understood as heat maps of expression changes, and are stacked along the time dimension to form a multidimensional heat map volume. Each node heat map represents the position and probability distribution of the key point in the facial image, and the multidimensional heat map volume retains the complete spatiotemporal information of facial expression changes. Therefore, the pain level assessment model trained by the multidimensional heat map volume can more effectively capture the characteristics of local pain expression changes between frames, thereby greatly improving the prediction accuracy of the pain level assessment model for the pain level.

[0011] In the embodiment of the present disclosure, before acquiring the plurality of first video segments, the method further includes:

[0012] Obtain multiple facial video clips from different camera angles;

[0013] For any facial video segment among the multiple facial video segments, intercepting the facial video segment based on the uterine contraction interval to obtain a third video segment corresponding to each uterine contraction interval;

[0014] Filtering out a plurality of target third video segments under different camera angles within the same uterine contraction interval from a plurality of third video segments corresponding to the plurality of facial video segments;

[0015] Merging the multiple target third video clips to obtain the first video clip;

[0016] A corresponding pain level label is added to the first video clip.

[0017] The disclosed embodiment obtains multiple facial video clips from different camera angles and adopts multi-view video stream capture technology to effectively overcome the defect of incomplete capture of facial features under a single view. This technology ensures that even if the parturient moves her head during the delivery process, the front and adjusted side of the face are fully captured at any time. It ensures that the comprehensive facial expression information of the parturient can be captured at any time, thereby improving the efficiency of pain assessment.

[0018] The disclosed embodiment intercepts facial video segments by contraction intervals to obtain a third video segment corresponding to each contraction interval. Each facial video segment can be intercepted in conjunction with a key time period of maternal pain, thereby ensuring a high correlation between the third video segment obtained after interception and maternal pain, thereby significantly improving the effectiveness and reliability of the model training data.

[0019] In the embodiment of the present disclosure, the multiple facial video clips include a first sub-video clip, a second sub-video clip, and a third sub-video clip; acquiring multiple facial video clips at different camera angles includes:

[0020] Obtaining the first sub-video segment by using a first camera disposed at a first position;

[0021] obtaining the second sub-video segment by a second camera arranged at a second position;

[0022] Obtaining the third sub-video segment by a third camera arranged at a third position;

[0023] The first position, the second position and the first position surround the face of the target user.

[0024] In the embodiment of the present disclosure, performing image segmentation on each video frame in the first video segment based on facial feature points to obtain a second video segment includes:

[0025] Performing facial feature point detection on each video frame in the first video clip to obtain a plurality of facial feature points;

[0026] Performing facial alignment and adjusting the positions of facial feature points on multiple video frames in the first video clip, so that the positions of each facial feature point in the multiple video frames after adjustment are the same;

[0027] Image segmentation is performed on each adjusted video frame based on multiple facial feature points to obtain a sample video frame containing a face area.

[0028] In an embodiment of the present disclosure, the probability distribution is existence confidence; before generating the multi-dimensional heat map volume of the second video segment, the method further includes:

[0029] Extracting a preset number of sample video frames from the multiple sample video frames; the preset number of sample video frames are used for the multi-dimensional heat map volume of the second video segment;

[0030] The preset number of sample video frames are continuous, each sample video frame contains the multiple facial feature points, and the existence confidence of each facial feature point is a preset confidence.

[0031] The disclosed embodiment extracts a preset number of sample video frames from the multiple sample video frames, and limits the preset number of sample video frames to be continuous, each sample video frame contains the multiple facial feature points, and the existence confidence of each facial feature point is a preset confidence. This can effectively avoid the situation where facial feature points may not be detected in several video frames of the multiple sample video frames containing face areas due to the movement of the parturient's head, thereby destroying the continuity of expression changes in the subsequent multi-dimensional heat map volume, thereby affecting the accuracy of pain assessment.

[0032] In the embodiment of the present disclosure, the plurality of facial feature points include facial contour feature points, mouth feature points, nose feature points, eye feature points and eyebrow feature points.

[0033] The embodiment of the second aspect of the present disclosure provides a method for assessing pain level, comprising:

[0034] Acquire a first video clip to be evaluated; the first video clip to be evaluated is obtained by merging a plurality of facial video clips taken at different camera angles within the same uterine contraction interval;

[0035] The first video clip to be evaluated is input into the trained pain level evaluation model described in the embodiment of the first aspect, and the corresponding pain level is output.

[0036] The embodiment of the third aspect of the present disclosure provides a model training device for a pain level assessment model, comprising:

[0037] A video clip acquisition module, used to acquire multiple first video clips and a pain level label corresponding to each first video clip; each first video clip is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval;

[0038] An image segmentation module, for performing image segmentation on each video frame in any first video segment of the plurality of first video segments based on facial feature points to obtain a second video segment; the second video segment includes a plurality of sample video frames including a human face region; the human face region is composed of a plurality of facial feature points;

[0039] A node heat map generation module, used for generating a node heat map of any sample video frame; the node heat map is used to characterize the position and probability distribution of each facial feature point in the sample video frame;

[0040] a multi-dimensional heat map volume generation module, configured to stack a plurality of node heat maps corresponding one by one to a plurality of sample video frames according to a time dimension of the plurality of sample video frames, to generate a multi-dimensional heat map volume of the second video segment; the multi-dimensional heat map volume is used to characterize the dynamic change of each facial feature point in the time dimension;

[0041] A model training module is used to train a pain level assessment model based on multiple multidimensional heat map volumes and the pain level label corresponding to each multidimensional heat map volume to obtain a trained pain level assessment model; the trained pain level assessment model is used to evaluate the first video clip to be evaluated to obtain the corresponding pain level.

[0042] An embodiment of a fourth aspect of the present disclosure provides a pain level assessment device, comprising:

[0043] A video clip acquisition module is used to acquire a first video clip to be evaluated; the first video clip to be evaluated is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval;

[0044] The pain level assessment module is used to input the first video clip to be assessed into the trained pain level assessment model described in the embodiment of the first aspect, and output the corresponding pain level.

[0045] An embodiment of the fifth aspect of the present disclosure provides an electronic device, which includes a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model training method of the pain level assessment model described in the first aspect or the pain level assessment method described in the second aspect by executing the computer instructions.

[0046] An embodiment of the sixth aspect of the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the model training method of the pain level assessment model described in the first aspect or the pain level assessment method described in the second aspect.

[0047] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will become apparent from the following description or be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present disclosure. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0049] In the attached picture:

[0050] Figure 1 A schematic diagram showing a flow chart of a model training method for a pain level assessment model provided by an embodiment of the present disclosure;

[0051] Figure 2 A schematic diagram of a camera acquisition arrangement provided by an embodiment of the present disclosure is shown;

[0052] Figure 3 A schematic diagram of a process for generating a first video clip provided by an embodiment of the present disclosure is shown;

[0053] Figure 4 A schematic diagram showing a human face region provided by an embodiment of the present disclosure consisting of a plurality of facial feature points;

[0054] Figure 5 A schematic diagram of a process of performing image segmentation on each video frame in a first video segment based on facial feature points to obtain a second video segment according to an embodiment of the present disclosure is shown;

[0055] Figure 6 A schematic diagram showing the structure of a model training device for a pain level assessment model provided by an embodiment of the present disclosure is shown;

[0056] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;

[0057] Figure 8 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present disclosure should have the common meanings understood by those skilled in the art to which the present disclosure belongs.

[0060] The following describes the technical scenarios involved in the embodiments of the present disclosure.

[0061] Existing face acquisition equipment usually uses a single camera to capture static face images or selects only face key frames from multi-view video data, which are then used for algorithm processing. However, this method faces major challenges in the specific scenario of maternal pain assessment. Because women may turn sideways, face sideways, or shake their heads during the delivery process, it is difficult for a single camera system to fully capture information about the front face of the woman from a temporal dimension. This limits the accuracy and real-time nature of pain assessment, because a single perspective may not be able to capture more complete changes in expression, especially when the woman's body position changes frequently. Therefore, there is a need for more advanced, multi-angle face acquisition systems to ensure that comprehensive facial expression information of the woman can be captured at any time, thereby improving the efficiency of pain assessment.

[0062] Existing pain assessment systems are often too generalized, focusing mainly on algorithms for extracting feature points from facial images, while ignoring the need to customize the pain characteristics of different patient groups during data preprocessing. A customized pain assessment system should be able to take into account the physiological parameters of different patient groups, such as incorporating information on maternal contraction monitoring to participate in the screening of effective facial pain expression video clips, thereby improving the individual relevance and applicability of the assessment results.

[0063] Existing pain assessment systems generally predict pain levels by extracting various forms of features from single-frame images. These methods lack temporal information and ignore the dynamic and subtle changes in facial expressions in the temporal dimension. Painful expressions are usually continuous and gradual, and it is difficult for a single-frame image to capture these important temporal features. In addition, the facial features of a person in a single-frame image or multi-frame input video segment may be affected by factors such as instantaneous expression changes, lighting, and occlusion, resulting in incomplete information and a low accuracy rate in maternal pain assessment.

[0064] To this end, the disclosed embodiment adopts multi-view video stream capture technology, which effectively overcomes the defect of incomplete capture of facial features under a single view. This technology ensures that even if the mother moves her head during the delivery process, the front and adjusted side of the face are fully captured at any time. The present disclosure not only focuses on the image feature extraction focused on by traditional technologies, but also screens the original video stream through the mother's physiological data, that is, the interval between contractions, during the data processing stage. This method significantly improves the validity and reliability of the data.

[0065] The present disclosure proposes a method for predicting the pain level of parturients based on a temporal feature extraction model, which combines painful facial expression analysis and spatiotemporal feature extraction technology. Combined with objective facts, the main features of facial expression changes come from facial feature points, and the change in expression is a continuous and small / local process. Facial feature points are understood as heat maps of expression changes, and are stacked along the time dimension to form a multidimensional heat map volume. Each node heat map represents the position and probability distribution of the key point in the facial image, and the multidimensional heat map volume retains the complete spatiotemporal information of facial expression changes. Therefore, the pain level assessment model will more effectively capture the characteristics of local pain expression changes between frames, thereby greatly improving the prediction accuracy of the pain level assessment model for the pain level.

[0066] In addition, the system provides interpretable visualization analysis charts for the classification of pain levels in facial expression images, making the model prediction results transparent and providing medical staff with a certain reference value for decision-making.

[0067] According to an embodiment of the present disclosure, an embodiment of a model training method for a pain level assessment model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0068] In this embodiment, a model training method for a pain level assessment model is provided. Figure 1 is a flow chart of a model training method for a pain level assessment model according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0069] Step S101, obtaining a plurality of first video clips and a pain level label corresponding to each first video clip.

[0070] Each first video segment is obtained by merging a plurality of facial video segments captured at different camera angles within the same contraction interval.

[0071] In some specific embodiments, before the above step S101, the method further includes:

[0072] Step a1, obtaining multiple facial video clips under different camera angles.

[0073] In the disclosed embodiment, multiple cameras arranged around the parturient's face can be used to obtain multiple facial video clips at different camera angles, that is, one camera obtains a facial video clip at one camera angle. Figure 2The number of cameras is not specifically limited in the present embodiment, and may be 3 or more than 3. The location of the cameras may be set according to actual conditions and is not specifically limited here. For example, “multiple facial video clips at different camera angles” Figure 3 "Video clips under angle 1", "Video clips under angle 2" and "Video clips under angle 3" in.

[0074] The disclosed embodiment adopts multi-view video stream capture technology, which effectively overcomes the defect of incomplete capture of facial features under a single view. This technology ensures that even if the mother moves her head during the delivery process, the front and adjusted side of the face are fully captured at any time.

[0075] Step a2: for any facial video segment among the multiple facial video segments, intercept the facial video segment based on the contraction interval to obtain a third video segment corresponding to each contraction interval.

[0076] In the disclosed embodiment, the uterine contraction interval refers to the time between two uterine contractions of the parturient. The uterine contraction interval of the parturient can be determined by medical monitoring equipment, or by the following method: the nurse records the time point of the parturient's self-assessed pain, the corresponding uterine contraction time, the fetal heart rate and other information at the bedside.

[0077] In the disclosed embodiment, after determining the contraction interval of the parturient, the facial video clips can be intercepted according to the contraction interval of the parturient to obtain the third video clip corresponding to each contraction interval. For example, assuming there are two contraction intervals, each facial video clip can be intercepted into two third video clips, namely, the third video clip corresponding to the first contraction interval and the third video clip corresponding to the second contraction interval.

[0078] Step a3: Filter out a plurality of target third video segments under different camera angles within the same contraction interval from the plurality of third video segments corresponding to the plurality of facial video segments.

[0079] Take an example to illustrate the above step a3: assume that there are facial video clips at three angles, namely, facial video clips at angle 1, facial video clips at angle 2, and facial video clips at angle 3; assume that the interval between the contractions of the parturient is 2;

[0080] By intercepting the facial video clips according to the contraction interval, two third video clips corresponding to the facial video clip of angle 1 can be obtained, namely, the third video clip corresponding to the first contraction interval and the third video clip corresponding to the second contraction interval; the facial video clip of angle 2 and the facial video clip of angle 2 are similar. A total of 2*3=6 third video clips can be obtained; and then the 6 third video clips are classified according to the contraction interval, and 3 third video clips corresponding to the first contraction interval and 3 third video clips corresponding to the second contraction interval can be obtained.

[0081] In the embodiment of the present disclosure, for example Figure 3 In the embodiment, the contraction interval and the facial video clips at each angle are subjected to intersection processing, thereby obtaining the target third video clips at different angles within the same contraction interval x.

[0082] Step a4: merge the multiple target third video segments to obtain the first video segment.

[0083] Specifically, multiple target third video clips may be combined into one video clip, and the target third video clips of the same parturient's face at three angles may be played simultaneously.

[0084] Step a5: adding a corresponding pain level label to the first video clip.

[0085] In the disclosed embodiment, the corresponding pain level label can be added to the first video clip in the following manner: the doctor watches each first video clip and gives a corresponding pain level score to it; and adds a corresponding pain level label to each first video clip according to the pain level score.

[0086] Step S102: for any first video segment among the multiple first video segments, perform image segmentation on each video frame in the first video segment based on facial feature points to obtain a second video segment.

[0087] The second video clip includes a plurality of sample video frames containing a face region; the face region is composed of a plurality of facial feature points, such as Figure 4 As shown, the face area can be composed of 68 facial feature points.

[0088] In some specific embodiments, the above step S102 includes steps S1021 to S1023:

[0089] Step S1021: perform facial feature point detection on each video frame in the first video clip to obtain a plurality of facial feature points.

[0090] Specifically, facial feature point detection may be performed on each video frame in the first video clip using the Dlib facial key point detection toolkit, the purpose of which is to obtain the two-dimensional coordinates of 68 facial feature points in the frame where the face is currently recognized.

[0091] Step S1022: perform facial alignment and position adjustment of facial feature points on multiple video frames in the first video clip, so that the position of each facial feature point in the multiple video frames after adjustment is the same.

[0092] Step S1023, performing image segmentation on each adjusted video frame based on multiple facial feature points to obtain a sample video frame containing a face area.

[0093] In the above steps S1021 to S1023, for example Figure 5 As shown: the InsightFace toolkit can be used to detect facial feature points and align faces for each video frame in the first video clip (in order to standardize the direction of the face, including ensuring that the face of the parturient is horizontal in the image, for example, Figure 2 The video frame recorded by the middle camera 3 from the side of the parturient is rotated so that the eyes of the parturient in the video are located above the lips of the parturient instead of on the right side), the positions of the key points of the face are adjusted after rotation, and the face is cropped (448*448) based on the key points to obtain a .avi video clip sample containing only the face area of ​​the parturient.

[0094] In some specific embodiments, in order to process sample video frames containing human face regions at different time segments into a data set format that is convenient for training a pain level assessment model, a custom format is required to process the multi-dimensional heat map volume based on facial feature points. The specific processing steps are as follows: Step S103-Step S104.

[0095] Step S103: for any sample video frame, generate a node heat map of the sample video frame; the node heat map is used to characterize the position and probability distribution of each facial feature point in the sample video frame.

[0096] In the embodiment of the present disclosure, a node heat map of a sample video frame can be generated by a Gaussian function, and each facial feature point in the node heat map has a corresponding triple (x k ,y k ,c k ), where x k ,y k refers to the position of the facial feature point in the video frame, c k It refers to the confidence of whether the facial feature point exists. A confidence of 1 indicates that the facial feature point exists in the video frame, and a confidence of 0 indicates that the facial feature point does not exist in the video frame.

[0097] The Gaussian function above is specifically as follows:

[0098]

[0099] Among them, i, j are the horizontal and vertical coordinates on the heat map, and σ is the parameter that controls the width of the Gaussian distribution. In other words, the confidence value of this point (i, j) on the node heat map decreases as the distance from the node center increases. A motion posture is represented as a heat map of size K×H×W, where K is the number of nodes, and H and W are the height and width of the video frame respectively.

[0100] Step S104, stacking a plurality of node heat maps corresponding to the plurality of sample video frames one by one according to the time dimension of the plurality of sample video frames to generate a multi-dimensional heat map volume of the second video segment; the multi-dimensional heat map volume is used to characterize the dynamic change of each facial feature point in the time dimension;

[0101] In the disclosed embodiment, multiple node heat maps corresponding to multiple sample video frames can be stacked along the time dimension of multiple sample video frames to create a multi-dimensional heat map volume of the second video segment. The volume size of the multi-dimensional heat map volume is K×T×H×W, where T is the time dimension (the number of video frames). The multi-dimensional heat map volume contains the dynamics of each facial feature point changing over time between multiple sample video frames.

[0102] In some specific embodiments, before step S104, the method further includes step b1:

[0103] Step b1, extracting a preset number of sample video frames from the multiple sample video frames.

[0104] In the embodiment of the present disclosure, the preset number of frames can be set according to actual conditions and is not specifically limited here. For example, the preset number of frames can be set to 10.

[0105] In some specific embodiments, the preset number of sample video frames are continuous, each sample video frame contains the multiple facial feature points, and the existence confidence of each facial feature point is a preset confidence.

[0106] In the embodiment of the present disclosure, considering that facial feature points may not be detected in several sample video frames due to the movement of the parturient's head, the continuity of expression changes in the subsequent multi-dimensional heat map volume will be destroyed. To this end, the embodiment of the present disclosure sets the number of frames of the multi-dimensional heat map volume sample video frames used to generate the second video clip to 10, and uses a window function to ensure that the 10 frames in the window are continuous and that all facial feature points, i.e., 68 two-dimensional (x, y) coordinates, can be detected by the Dlib toolkit in each frame, and the confidence of each coordinate is set to 1. The corresponding algorithm is as follows:

[0107] Initialize the cache:

[0108] Cache = []

[0109] For each frame F i (i=1,2,…,N):

[0110] Cache=Cache∪{F i}if FaceDetected(F i )=1

[0111] Cache = [] if FaceDetected = 0

[0112] If the number of elements in the cache = 10:

[0113] If|Cache|=10then SaveFrames(Cache)and Cache=[]

[0114] where F i represents the i-th frame of the video, and Cache=[] represents an initially empty cache. For each frame F i , define a function FaceDeteted(F i ), a value of 1 indicates that facial feature points are detected, and a value of 0 indicates that facial feature points are not detected. SaveFrames(Cache) is a function for saving frames.

[0115] In some specific embodiments, after extracting a preset number of sample video frames from the plurality of sample video frames, the method further includes:

[0116] Step c1, screening out a plurality of target node heat maps corresponding to a preset number of sample video frames from a plurality of node heat maps;

[0117] Step c2, stacking multiple target node heat maps according to the time dimension of sample video frames with a preset number of frames to generate a target multidimensional heat map volume of the second video clip; the target multidimensional heat map volume is used to train the pain level assessment model to obtain a trained pain level assessment model.

[0118] Step S105, based on multiple multidimensional heat map volumes and the pain level label corresponding to each multidimensional heat map volume, the pain level assessment model is trained to obtain a trained pain level assessment model; the trained pain level assessment model is used to evaluate the first video clip to be evaluated to obtain the corresponding pain level.

[0119] In the disclosed embodiment, the pain level assessment model is selected according to the actual situation and is not specifically limited here, such as support vector machine (SVM), random forest, convolutional neural network (CNN) or recurrent neural network (RNN).

[0120] In the disclosed embodiment, during the model training process, the multidimensional heat map volume is used as the input feature and the pain level is used as the label to train the selected model. The performance of the model is evaluated using techniques such as cross-validation, and the model parameters are adjusted to optimize the results. During the model validation process, the accuracy and generalization ability of the model can be verified on an independent test set, and the prediction performance of the model at different pain levels can be analyzed. Finally, the trained model is deployed to a practical application, such as a medical monitoring system or a pain management application.

[0121] In some specific embodiments, in the deployment of the model, the model file trained under the pytorch framework can be converted to the ONNX format, and the deployment reasoning of the deep model can be performed on the Jetson orin nx edge device. The content of the display page includes the real-time shooting content of the camera (including the face detection frame) and the real-time pain monitoring change curve. After discussion with the doctor, it is confirmed that the pain curve within every minute and five minutes is clinically significant. At the same time, in addition to the monitoring function, this system also supports pain detection function. Every minute, the system will automatically extract the 5 most painful pictures of the patient (after standardized facial processing) through an algorithm (size 5) for clinical doctor teaching.

[0122] The technical effects of the embodiments of the present disclosure are as follows:

[0123] Data Collection Methods

[0124] The disclosed embodiment uses a three-view video stream to record the painful expression of the parturient during uterine contractions, and combines the intervention of medical staff and the patient's own physiological data to collect data. This long-term and stable data collection method ensures the frame-level integrity of the parturient's painful expression, as well as the validity and authenticity of the medical data.

[0125] Unique system modeling algorithm design

[0126] This paper applies the advanced model ideas in the field of human skeleton motion recognition to the pain expression recognition system, integrates spatiotemporal information into the training process, and retains the micro-changes of expression at the spatiotemporal level. Because the algorithm focuses on the multi-dimensional heat map volume of the key points of the face, it avoids the influence of noise factors such as the distance between the patient and the camera, angle restrictions, and lighting on the quality of the collected pictures.

[0127] Explainability of deep models after deployment on edge devices

[0128] The deep model involved in the present disclosure is deployed on the edge device, which is specifically manifested in the reasoning of the model, the real-time camera face display, the prompt effect when the face is not detected, the display of the current timestamp, the real-time frame-level automatic prediction of pain values ​​and the real-time display of the pain value curve, and the result visualization analysis report provided to the doctor.

[0129] The present disclosure also provides a method for assessing pain level, including:

[0130] Acquire a first video clip to be evaluated; the first video clip to be evaluated is obtained by merging a plurality of facial video clips taken at different camera angles within the same uterine contraction interval;

[0131] The first video clip to be evaluated is input into the trained pain level evaluation model described in the above embodiment, and the corresponding pain level is output.

[0132] Corresponding to the implementation of the above pain level assessment model training method, the present disclosure also provides a pain level assessment model training device for executing the above Figures 1 to 5 The model training method of the pain level assessment model described in any of the illustrated embodiments. Figure 6 As shown, the model training device of the pain level assessment model includes:

[0133] A video clip acquisition module, used to acquire multiple first video clips and a pain level label corresponding to each first video clip; each first video clip is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval;

[0134] An image segmentation module, for performing image segmentation on each video frame in any first video segment of the plurality of first video segments based on facial feature points to obtain a second video segment; the second video segment includes a plurality of sample video frames including a human face region; the human face region is composed of a plurality of facial feature points;

[0135] A node heat map generation module, used for generating a node heat map of any sample video frame; the node heat map is used to characterize the position and probability distribution of each facial feature point in the sample video frame;

[0136] a multi-dimensional heat map volume generation module, configured to stack a plurality of node heat maps corresponding one by one to a plurality of sample video frames according to a time dimension of the plurality of sample video frames, to generate a multi-dimensional heat map volume of the second video segment; the multi-dimensional heat map volume is used to characterize the dynamic change of each facial feature point in the time dimension;

[0137] A model training module is used to train a pain level assessment model based on multiple multidimensional heat map volumes and the pain level label corresponding to each multidimensional heat map volume to obtain a trained pain level assessment model; the trained pain level assessment model is used to evaluate the first video clip to be evaluated to obtain the corresponding pain level.

[0138] Optionally, the device also includes a video processing module, which is used to obtain multiple facial video clips under different camera angles before obtaining multiple first video clips; for any facial video clip among the multiple facial video clips, intercept the facial video clip based on the contraction interval to obtain a third video clip corresponding to each contraction interval; filter out multiple target third video clips under different camera angles within the same contraction interval from the multiple third video clips corresponding to the multiple facial video clips; merge the multiple target third video clips to obtain the first video clip; and add a corresponding pain level label to the first video clip.

[0139] Optionally, the multiple facial video clips include a first sub-video clip, a second sub-video clip and a third sub-video clip; the video processing module is also used to: obtain the first sub-video clip by using a first camera set at a first position; obtain the second sub-video clip by using a second camera set at a second position; obtain the third sub-video clip by using a third camera set at a third position; wherein the first position, the second position and the first position surround the face of the target user.

[0140] Optionally, the image segmentation module is also used to: perform facial feature point detection on each video frame in the first video clip to obtain multiple facial feature points; perform facial alignment and position adjustment of facial feature points on multiple video frames in the first video clip to make the position of each facial feature point the same in the adjusted multiple video frames; perform image segmentation on each adjusted video frame based on the multiple facial feature points to obtain sample video frames containing face areas.

[0141] Optionally, the device also includes: a sample video frame extraction module, used to extract a preset number of sample video frames from the multiple sample video frames before generating the multidimensional heat map volume of the second video fragment; the preset number of sample video frames are used for the multidimensional heat map volume of the second video fragment; wherein the preset number of sample video frames are continuous, each sample video frame contains the multiple facial feature points, and the existence confidence of each facial feature point is a preset confidence.

[0142] The model training device for the pain level assessment model provided in the above-mentioned embodiment of the present disclosure and the model training method for the pain level assessment model provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0143] Corresponding to the implementation of the above pain level assessment method, the present disclosure also provides a pain level assessment device for executing the pain level assessment method described in the above embodiment. The pain level assessment device includes:

[0144] A video clip acquisition module is used to acquire a first video clip to be evaluated; the first video clip to be evaluated is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval;

[0145] The pain level assessment module is used to input the first video clip to be assessed into the trained pain level assessment model described in the above embodiment, and output the corresponding pain level.

[0146] The present disclosure also provides an electronic device to execute the above-mentioned pain level assessment model training method or pain level assessment method. Figure 7 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 7 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702 and a communication interface 703, and the processor 700, the communication interface 703 and the memory 701 are connected via the bus 702; the memory 701 stores a computer program that can be run on the processor 700, and when the processor 700 runs the computer program, it executes the model training method of the pain level assessment model or the pain level assessment method provided in the aforementioned embodiment of the present disclosure.

[0147] The memory 701 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 703 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0148] The bus 702 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 701 is used to store a program, and the processor 700 executes the program after receiving an execution instruction. The model training method of the pain level assessment model or the pain level assessment method disclosed in the above-mentioned embodiment may be applied to the processor 700, or implemented by the processor 700.

[0149] The processor 700 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 700 or the instruction in the form of software. The above processor 700 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701 and completes the steps of the above method in combination with its hardware.

[0150] The electronic device provided by the embodiments of the present disclosure and the model training method of the pain level assessment model or the pain level assessment method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0151] The present disclosure also provides a computer-readable storage medium corresponding to the model training method or pain level assessment method of the pain level assessment model provided in the above embodiments. Figure 8 The computer-readable storage medium shown is a CD 30 on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the model training method or pain level assessment method of the pain level assessment model provided in any of the aforementioned embodiments.

[0152] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0153] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the model training method of the pain level assessment model or the pain level assessment method provided by the embodiments of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0154] It should be noted that:

[0155] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0156] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed disclosure requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present disclosure.

[0157] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of this disclosure and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0158] The above is only a preferred specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A model training method for a pain level assessment model, characterized in that: The method comprises: Acquire multiple first video clips and a pain level label corresponding to each first video clip; each first video clip is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval; For any first video segment among the plurality of first video segments, image segmentation is performed on each video frame in the first video segment based on facial feature points to obtain a second video segment; the second video segment includes a plurality of sample video frames including a human face region; the human face region is composed of a plurality of facial feature points; For any sample video frame, a node heat map of the sample video frame is generated; the node heat map is used to characterize the position and probability distribution of each facial feature point in the sample video frame; According to the time dimension of the plurality of sample video frames, a plurality of node heat maps corresponding one to one to the plurality of sample video frames are stacked to generate a multi-dimensional heat map volume of the second video segment; the multi-dimensional heat map volume is used to characterize the dynamic change of each facial feature point in the time dimension; Based on multiple multidimensional heat map volumes and the pain level label corresponding to each multidimensional heat map volume, a pain level assessment model is trained to obtain a trained pain level assessment model; the trained pain level assessment model is used to evaluate the first video clip to be evaluated to obtain the corresponding pain level.

2. The method according to claim 1, characterized in that Before acquiring the plurality of first video segments, the method further includes: Obtain multiple facial video clips from different camera angles; For any facial video segment among the multiple facial video segments, intercepting the facial video segment based on the uterine contraction interval to obtain a third video segment corresponding to each uterine contraction interval; Filtering out a plurality of target third video segments under different camera angles within the same uterine contraction interval from a plurality of third video segments corresponding to the plurality of facial video segments; Merging the multiple target third video clips to obtain the first video clip; A corresponding pain level label is added to the first video clip.

3. The method according to claim 2, characterized in that The multiple facial video clips include a first sub-video clip, a second sub-video clip, and a third sub-video clip; obtaining multiple facial video clips at different camera angles includes: Obtaining the first sub-video segment by using a first camera disposed at a first position; obtaining the second sub-video segment by a second camera arranged at a second position; Obtaining the third sub-video segment by a third camera arranged at a third position; The first position, the second position and the first position surround the face of the target user.

4. The method according to claim 1 or 2, characterized in that: Performing image segmentation on each video frame in the first video segment based on facial feature points to obtain a second video segment includes: Performing facial feature point detection on each video frame in the first video clip to obtain a plurality of facial feature points; Performing facial alignment and adjusting the positions of facial feature points on multiple video frames in the first video clip, so that the positions of each facial feature point in the multiple video frames after adjustment are the same; Image segmentation is performed on each adjusted video frame based on multiple facial feature points to obtain a sample video frame containing a face area.

5. The method according to claim 1 or 2, characterized in that: The probability distribution is existence confidence; before generating the multi-dimensional heat map volume of the second video segment, the method further includes: Extracting a preset number of sample video frames from the multiple sample video frames; the preset number of sample video frames are used for the multi-dimensional heat map volume of the second video segment; The preset number of sample video frames are continuous, each sample video frame contains the multiple facial feature points, and the existence confidence of each facial feature point is a preset confidence.

6. The method according to claim 1 or 2, characterized in that: The plurality of facial feature points include facial contour feature points, mouth feature points, nose feature points, eye feature points and eyebrow feature points.

7. A method for assessing pain level, characterized in that: The method comprises: Acquire a first video clip to be evaluated; the first video clip to be evaluated is obtained by merging a plurality of facial video clips taken at different camera angles within the same uterine contraction interval; The first video clip to be evaluated is input into the trained pain level evaluation model according to any one of claims 1 to 6, and the corresponding pain level is output.

8. A model training device for a pain level assessment model, characterized in that: The device comprises: A video clip acquisition module is used to acquire multiple first video clips and a pain level label corresponding to each first video clip; each first video clip is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval; An image segmentation module, configured to perform image segmentation on each video frame in any first video segment of the plurality of first video segments based on facial feature points to obtain a second video segment; the second video segment includes a plurality of sample video frames including a human face region; the human face region is composed of a plurality of facial feature points; A node heat map generation module, used for generating a node heat map of any sample video frame; the node heat map is used to characterize the position and probability distribution of each facial feature point in the sample video frame; a multi-dimensional heat map volume generation module, configured to stack a plurality of node heat maps corresponding one by one to a plurality of sample video frames according to a time dimension of the plurality of sample video frames, to generate a multi-dimensional heat map volume of the second video segment; the multi-dimensional heat map volume is used to characterize the dynamic change of each facial feature point in the time dimension; A model training module is used to train a pain level assessment model based on multiple multidimensional heat map volumes and the pain level label corresponding to each multidimensional heat map volume to obtain a trained pain level assessment model; the trained pain level assessment model is used to evaluate the first video clip to be evaluated to obtain the corresponding pain level.

9. A pain level assessment device, characterized in that: The device comprises: A video clip acquisition module is used to acquire a first video clip to be evaluated; the first video clip to be evaluated is obtained by merging multiple facial video clips taken at different camera angles within the same uterine contraction interval; A pain level assessment module is used to input the first video clip to be assessed into the trained pain level assessment model according to any one of claims 1 to 6, and output a corresponding pain level.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.