Anesthesia postoperative pain assessment method in combination with facial expression of patient

Abstract

The invention relates to the technical field of image analysis, in particular to an anesthesia post-operation pain assessment method in combination with facial expressions of a patient. An optical flow method is adopted to analyze pixel motion of a facial image, optical flow vectors and information points are obtained, and facial expression changes are quantitatively captured. In view of information point validity differences, facial feature points are screened to reduce data redundancy. And in consideration of the facial structure difference of the patient, the traction characteristics of the optical flow vector are analyzed, and the traction direction of the image block is determined. Pain expressions such as eye eyebrow closeness and the like can be represented by the traction direction. By analyzing the distribution confusion degree and difference in the traction direction, the facial pain index is determined, and accurate reference is provided for pain assessment. And finally, screening a data set based on the facial images of multiple patients and pain indexes, training a neural network, and establishing a mapping relation between facial expressions and pain degrees. According to the method, the pain assessment accuracy is improved, the assessment efficiency is also remarkably improved, and automatic pain assessment of the facial image of the patient to be tested is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and particularly to a method for evaluating postoperative pain combined with the facial expressions of patients. Background Art

[0002] In the medical field, the evaluation of postoperative pain is a crucial link, which is directly related to the rehabilitation process and medical effect of patients. Since patients may have difficulty accurately expressing their pain feelings due to the influence of drugs, confusion of consciousness or language expression disorders after anesthesia, in order to improve the accuracy and efficiency of evaluation, pain assessment methods based on facial expressions have gradually emerged.

[0003] Facial expressions are an important way for human emotional expression, and pain, as a strong emotional experience, often leaves obvious traces on the face. Therefore, by analyzing the muscle movements and expression changes of the patient's face, the pain level can be indirectly evaluated. However, when using facial recognition technology for pain recognition in practical applications, facial feature points are often used for recognition. However, due to the differences in facial structures and the distribution of muscle and skin among different people, and some feature points have no relation to pain recognition, when traditional facial pain recognition using feature points is carried out, a large amount of non-pain redundant data may need to be processed simultaneously, which reduces the efficiency and affects the final accuracy. Summary of the Invention

[0004] In order to solve the technical problems that there are differences in facial structures and the distribution of muscle and skin among different people, and some feature points have no relation to pain recognition, so when traditional facial pain recognition using feature points is carried out, a large amount of non-pain redundant data may need to be processed simultaneously, which reduces the efficiency and affects the final accuracy, the purpose of the present invention is to provide a method for evaluating postoperative pain combined with the facial expressions of patients, and the specific technical solutions adopted are as follows: Obtain two temporally adjacent facial images of a patient after anesthesia, and use the previous facial image in time sequence as the current image; Based on the optical flow method, analyze the motion changes of pixel points in the two facial images to obtain optical flow vectors, and use the pixel points corresponding to the optical flow vectors in the current image as information points; divide the current image into blocks to obtain image blocks; analyze the difference between the distribution characteristics of information points in the local neighborhood and in the image blocks, and screen facial feature points from the information points; According to the positional relationship and length difference between optical flow vectors, determine the stretching eigenvalue corresponding to each facial feature point; in each image block, based on the stretching eigenvalue corresponding to the facial feature point, determine the stretching direction of each image block; analyze the distribution chaos and distribution difference of the stretching directions of all image blocks in the current image to determine the facial pain index of the current image; Screen a dataset for neural network training based on the facial pain indicators corresponding to the facial images of multiple patients to obtain a trained neural network; use the trained neural network to perform postoperative pain assessment on the facial images of the patients to be tested.

[0005] Further, the method for obtaining the facial feature points includes: In the current image, for any information point, use the information point as the center of a circle and the length of the optical flow vector corresponding to the information point as the radius to obtain the local neighborhood corresponding to the information point; In the local neighborhood corresponding to the information point, calculate the density of all information points in the local neighborhood as the first density; In the image block to which the information point belongs, calculate the density of all information points in the image block as the second density; Multiply the normalized value of the difference between the first density and the second density corresponding to the information point by the first density, and normalize the obtained product to obtain the information factor corresponding to the information point; Among all information points, use the information points with information factors greater than the preset information threshold as facial feature points.

[0006] Further, the method for obtaining the stretching eigenvalue includes: For the optical flow vector corresponding to any facial feature point, determine the stretching optical flow of the optical flow vector according to the positional relationship between the optical flow vector and other optical flow vectors; Calculate the angle between the optical flow vector and each corresponding stretching optical flow as the angle deviation value, use the difference between the length of the optical flow vector and the length of each corresponding stretching optical flow as the length deviation factor, and use the Euclidean distance between the starting point of the optical flow vector and the starting point of each corresponding stretching optical flow as the distance factor; Arrange the stretching optical flows of the optical flow vector in ascending order according to the distance factor to obtain a sorting sequence. Under the sorting sequence, calculate the Pearson correlation coefficient between the angle deviation values of all stretching optical flows and the distance factor as the first correlation factor, and calculate the Pearson correlation coefficient between the length deviation factors of all stretching optical flows and the distance factor as the second correlation factor; Use the normalized value of the sum of the first correlation factor and the second correlation factor corresponding to all stretching optical flows of the optical flow vector as the stretching eigenvalue of the facial feature point corresponding to the optical flow vector.

[0007] Further, the method for obtaining the stretching optical flow includes: Optionally select the optical flow vector corresponding to a facial feature point as the optical flow to be tested, draw a perpendicular line to the straight line where the optical flow to be tested is located through the starting point of the optical flow to be tested, and use the perpendicular line as the transverse movement line; The optical flow vectors intersecting the lateral translation line are used as the stretched optical flow of the optical flow to be measured.

[0008] Further, the method for obtaining the pulling direction includes: In each image block, the direction of the optical flow vector corresponding to the maximum pulling eigenvalue is used as the pulling direction of each image block.

[0009] Further, the method for obtaining the facial pain index includes: In the current image, obtain the facial symmetry line on the left and right of the face and determine the facial area where each image block is located, where the facial area is divided into a left area and a right area; Taking the facial symmetry line in the current image as the vertical axis and the line perpendicular to the facial symmetry line as the horizontal axis to construct a coordinate system; For any one image block, taking the straight line where the pulling direction is located as the moving line, taking the intersection point of the moving line and the edge line of the facial area where it is located as the first intersection point, and taking the intersection point of the moving line and the facial symmetry line as the second intersection point; Taking the image block with the pulling direction pointing to the first intersection point as the first area, and taking the image block with the pulling direction pointing to the second intersection point as the second area; For all the first areas in the current image, taking the ratio of the information entropy of the ordinate of the first intersection point to the information entropy of the ordinate of the second intersection point as the diffusion index; For all the second areas in the current image, taking the ratio of the information entropy of the ordinate of the first intersection point to the information entropy of the ordinate of the second intersection point as the aggregation index; Taking the sum of the ordinates of the first intersection point and the second intersection point in the second area as the first sum value, taking the sum of the ordinates of the first intersection point and the second intersection point in the first area as the second sum value, and taking the value obtained by normalizing the difference between the first sum value and the second sum value as the height difference; Taking the value obtained by normalizing the product of the diffusion index, the aggregation index, and the height difference corresponding to the current image as the facial pain index corresponding to the current image.

[0010] Further, the method for obtaining the facial symmetry line includes: Arbitrarily take one point on each of the upper boundary and the lower boundary of the current image and connect them, and take the straight line where the connection is located as the dividing line; For any one dividing line, taking the number of information points in the current image that are symmetric about the dividing line as the symmetry index of the dividing line; Taking the dividing line with the maximum symmetry index as the facial symmetry line of the current image.

[0011] Further, the method for obtaining the data set includes: Among the facial pain indicators corresponding to the facial images of all patients, facial images greater than or equal to the pain threshold are used as target images; The optical flow vectors of the facial feature points in the target images are used as pulling vectors, and the pulling vectors corresponding to all target images are used as the dataset.

[0012] Further, the method for evaluating the postoperative pain of a patient to be tested based on the trained neural network includes: Obtain the optical flow vectors of all facial feature points in the facial image to be tested and use them as the input of the trained neural network, and output the pain level of the facial image to be tested.

[0013] Further, the method for obtaining the image blocks includes: Use the superpixel segmentation method to segment the current image to obtain a number of superpixel blocks, and each superpixel block is an image block.

[0014] The present invention has the following beneficial effects: In order to accurately evaluate the postoperative pain of patients, first, the optical flow method is used to analyze the motion changes of pixel points in facial images, obtain the optical flow vectors of the patient's face and get information points, which can objectively and quantitatively capture the subtle changes in facial expressions. Given that not all information points are effective for characterizing postoperative pain, the differences between the distribution characteristics of information points in the local neighborhood and in the image blocks are analyzed, and facial feature points are selected from the information points, thereby reducing data redundancy. Since there are differences in the facial structures and muscle-skin distributions of different patients, but when a patient experiences pain, changes in the muscle directions will occur on the face, that is, facial muscle pulling will occur. Therefore, the pulling characteristics between the optical flow vectors are analyzed to obtain the pulling direction of each image block. When a painful expression appears on the face, features such as the eyes and eyebrows being tightly closed and gathered usually appear, and this feature can be characterized by the pulling direction. Therefore, the chaotic and differential situations of the distribution of the pulling directions of the image blocks are analyzed to determine the facial pain indicator. At this time, the facial pain indicator can provide a more accurate reference for measuring the postoperative pain of the current patient. Finally, the dataset is screened based on the facial pain indicators corresponding to the facial images of multiple patients and the neural network is trained, which can establish the mapping relationship between facial expressions and pain levels, so as to realize the subsequent automatic pain evaluation of the facial images to be tested of patients to be tested. Thus, while improving the evaluation accuracy, the evaluation efficiency is also improved. Description of the Drawings

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0016] Figure 1 The method flow chart of a method for evaluating postoperative pain of anesthesia combined with the facial expression of a patient provided by an embodiment of the present invention; Figure 2 The schematic diagram of a stretching optical flow provided by an embodiment of the present invention; Figure 3 The schematic diagram of the distribution of optical flow vectors when a force acts on the face provided by an embodiment of the present invention; Figure 4 The schematic diagram of a first intersection point and a second intersection point provided by an embodiment of the present invention. Detailed implementation manners

[0017] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following combines the accompanying drawings and preferred embodiments to detail the specific implementation manners, structures, features and effects of a method for evaluating postoperative pain of anesthesia combined with the facial expression of a patient according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0019] The following specifically describes the specific solution of a method for evaluating postoperative pain of anesthesia combined with the facial expression of a patient provided by the present invention with reference to the accompanying drawings.

[0020] Please refer to Figure 1 , which shows the method flow chart of a method for evaluating postoperative pain of anesthesia combined with the facial expression of a patient provided by an embodiment of the present invention. The method includes the following steps: Step S1: Obtain two adjacent facial images of the patient in time sequence after anesthesia, and use the previous facial image in time sequence as the current image.

[0021] After anesthesia, due to the anesthetic effect, the patient's consciousness is unclear, and for patient groups such as children and the elderly who cannot directly express their pain sensations, when doctors evaluate the patient's pain status after anesthesia, they usually conduct pain assessments based on facial expressions. With the rapid development of computer vision and artificial intelligence technologies, in order to improve the evaluation efficiency, pain assessment methods based on the analysis of patients' facial expressions have gradually attracted attention, so as to objectively evaluate the pain of patients after anesthesia and provide more accurate management references for doctors.

[0022] First, it is necessary to obtain the facial images of the patient after anesthesia. Given the differences in facial structures and the distribution of muscles and skin among different people, in order to more accurately identify the pain status represented by facial expressions, in the embodiments of the present invention, obtaining two temporally adjacent facial images of the patient after anesthesia helps to more objectively and quantitatively capture the subtle changes in the face in the subsequent process.

[0023] Specifically, use a high-definition camera or a medical special-purpose camera device to continuously record videos during the recovery period of the patient after anesthesia. During the shooting process, it is necessary to ensure that the position, angle, and focal length of the camera are fixed. Use professional video processing software to extract images from the recorded video at a set time interval or frame rate. Since the embodiments of the present invention mainly focus on the facial state of the patient, in order to avoid interference from other background areas, the extracted images can be operated on based on a trained semantic segmentation network to obtain facial images that only contain the patient's face. Then, in the sequence of extracted facial images, any two temporally adjacent facial images are selected, for example, the nth frame and the (n + 1)th frame are selected. In the two adjacent facial images, the previous frame (the nth frame) is used as the current image.

[0024] It should be noted that the time interval for image extraction can be set to 1 second, and the specific time interval can be adjusted according to the implementation scenario and is not limited here; the training process of the semantic segmentation network is briefly described as follows: (1) Use images containing human faces as training data. Label the pixels in the facial area as 1 and other pixels as 0 to obtain label data. (2) The semantic segmentation network adopts an encoder-decoder structure. After normalizing the training data and label data, they are input into the network. The semantic segmentation encoder is used to extract the features of the input data to obtain a feature map. The semantic segmentation decoder performs sampling transformation on the feature map and outputs the semantic segmentation result. (3) The cross-entropy loss function is used to train the network.

[0025] In the embodiments of the present invention, the acquisition and collection of the patient's facial data are all authorized by relevant users, and the process does not violate relevant laws and regulations and does not violate public order and good customs.

[0026] Step S2: Analyze the motion changes of pixel points in two frames of facial images based on the optical flow method, obtain the optical flow vectors, and use the pixel points corresponding to the optical flow vectors in the current image as information points; divide the current image into blocks to obtain image blocks; analyze the difference between the distribution characteristics of the information points in the local neighborhood and in the image blocks, and screen out facial feature points from the information points.

[0027] Pain is often accompanied by the tension of facial muscles. Therefore, by analyzing the motion changes of pixel points in facial images based on the optical flow method to obtain optical flow vectors and information points, the motion of facial muscles can be captured, which is crucial for recognizing pain expressions. Since not all information points are effective for characterizing the motion of facial muscles, in order to reduce the computational redundancy, facial feature points can be further screened out from all the information points. For the feature points that can characterize the motion of facial muscles, under expressions such as frowning, they will have a relatively obvious displacement. Therefore, in the embodiments of the present invention, the difference between the distribution characteristics of the information points in the local neighborhood and in the image blocks is analyzed, and the information points with relatively obvious displacement are screened out as facial feature points.

[0028] First, take two frames of facial images as the input of the optical flow method, so as to output the optical flow vectors in the current image, which reflect the motion information of pixel points in the current image. The pixel points with optical flow vectors are used as information points, and the information points represent the positions where motion occurs, and these motions are usually closely related to the changes in facial expressions.

[0029] It should be noted that the process of obtaining optical flow vectors based on the optical flow method is a well-known technology, and the specific process will not be elaborated here.

[0030] Then, the current image can be divided into blocks to more carefully analyze the changes in facial expressions in a small range, so as to screen out more representative facial feature points in the subsequent process, that is, the facial feature points that can more significantly reflect the motion of facial muscles.

[0031] Preferably, in an embodiment of the present invention, the method for obtaining image blocks includes: Superpixel segmentation can divide an image into regions (superpixel blocks) with similar attributes, and while reducing the complexity of the image, it can retain the key features of the image, such as edges, textures, and shapes, etc. Therefore, in this embodiment of the present invention, the current image is segmented using the superpixel segmentation method to obtain several superpixel blocks, and each superpixel block is an image block.

[0032] It should be noted that in order to ensure that the area of the image block is not too small, in this embodiment of the present invention, the number of superpixel blocks is set to 16, and the specific number can be adjusted according to the implementation scenario and will not be limited here; the superpixel segmentation method is a well-known technology, and the specific process will not be elaborated here.

[0033] Superpixel segmentation divides the current image into several image patches of texture-structured pixels, which can be regarded as dividing different regions of the patient's face. At this time, the distribution of information points in the image patches can characterize the trend changes of facial muscles within a certain range. To further identify more representative facial feature points, the differences between the distribution characteristics of information points in a smaller local neighborhood and in the image patches can be analyzed to screen facial feature points.

[0034] Preferably, in an embodiment of the present invention, the method for obtaining facial feature points includes: In the current image, for any information point, taking the information point as the center and the length of the optical flow vector corresponding to the information point as the radius, a local neighborhood corresponding to the information point is obtained. By setting the local neighborhood, the relationship between the information point and its surrounding pixels can be more accurately evaluated, as well as whether the information point can represent a significant facial feature.

[0035] In the local neighborhood corresponding to the information point, calculate the density of all information points in the local neighborhood as the first density. Specifically, the ratio of the number of all information points in the local neighborhood to the area of the local neighborhood can be used as the first density. The greater the first density, it indicates that in the local neighborhood corresponding to the information point, the distribution of information points is relatively dense, that is, the distribution of pixel points with motion changes is relatively dense, and then the representativeness of the information point in characterizing the muscle changes during facial pain is higher.

[0036] Then, in the image patch to which the information point belongs, calculate the density of all information points in the image patch as the second density. Similarly, the ratio of the number of all information points in the image patch to the area of the image patch is used as the second density. The greater the second density, it indicates that in the image patch to which the information point belongs, the distribution of information points is relatively dense.

[0037] Next, compare the local neighborhood density (first density) corresponding to the information point and the image patch density (second density) to evaluate whether the information point has significant characteristics in its local neighborhood. If the first density is higher than the second density, then the information point can be regarded as a facial feature point with relatively significant motion characteristics.

[0038] Therefore, calculate the normalized value of the difference between the first density and the second density corresponding to this information point. The larger this value is, the greater the degree to which the local neighborhood density of this information point is higher than the image block density. Then multiply this value by the first density and normalize the obtained product to get the information factor corresponding to this information point. At this time, the larger the information factor is, the more significant the local facial motion feature of this information point is, and the more it can show the motion changes of facial muscles. Among them, normalization is a technical means well-known to those skilled in the art. The choice of the normalization function can be linear normalization or standard normalization, etc. The specific normalization method is not limited here.

[0039] Finally, among all the information points, regard the information points with information factors greater than the preset information threshold as facial feature points.

[0040] It should be noted that in the embodiment of the present invention, the preset information threshold is 0.65, and the specific value can be adjusted according to the implementation scenario and is not limited here.

[0041] Step S3: Determine the pulling feature value corresponding to each facial feature point according to the positional relationship and length difference between the optical flow vectors; in each image block, determine the pulling direction of each image block based on the pulling feature value corresponding to the facial feature point; analyze the distribution confusion and distribution difference of the pulling directions of all image blocks in the current image to determine the facial pain index of the current image.

[0042] In the expression of facial pain, the movement and deformation of muscles are obvious features, and the facial muscles will pull each other. The optical flow vector can capture the movement information of pixels in the image, which is crucial for analyzing the dynamic changes of facial expressions. Therefore, the positional relationship and length difference between the optical flow vectors are analyzed to determine the pulling feature value corresponding to each facial feature point, which is used to reflect the pulling degree at the position of each facial feature point. Then, in each image block, based on the pulling feature value of the facial feature point, the pulling direction of this image block can be determined, which helps to analyze the deformation and muscle movement in different regions of the face and further supports the recognition of pain expressions. Finally, in view of the fact that when there is a painful expression on the face, the eyes, eyebrows, etc. will be closed and gathered, while the mouth, nostrils, etc. will show phenomena such as opening, so by analyzing the distribution confusion and distribution difference of the pulling directions of all image blocks in the current image, the pain degree of the entire face can be comprehensively evaluated. This distribution analysis can capture the subtle changes and overall trends of facial pain and provide a reliable basis for determining the facial pain index.

[0043] First, determine the pulling feature value corresponding to each facial feature point according to the positional relationship and length difference characteristics between the optical flow vectors. Preferably, in an embodiment of the present invention, the method for obtaining the pulling feature value includes: Optionally, select the optical flow vector corresponding to a facial feature point as the optical flow to be measured. Draw a perpendicular line to the line where the optical flow to be measured is located through the starting point of the optical flow to be measured, and use this perpendicular line as the horizontal movement line. The optical flow vectors intersecting with the horizontal movement line are used as the stretching optical flows of the optical flow to be measured. By analyzing the stretching optical flows, it is possible to understand the pulling of the facial muscles by the optical flow to be measured in the direction of the horizontal movement line, which is of great significance for analyzing the deformation of facial features, etc. Please refer to Figure 2 , which shows a schematic diagram of the stretching optical flow in an embodiment of the present invention. Among them, the yellow arrow represents the optical flow to be measured, the yellow line represents the horizontal movement line of the optical flow to be measured, and the blue arrow is the stretching optical flow of the optical flow to be measured.

[0044] In the analysis of facial expressions, when the face is subjected to a force, such as muscle contraction, the area near the point of action of the force is usually subjected to a greater pulling, while the pulling degree of the area far from this point of action gradually decreases. Applying this principle to the analysis of optical flow vectors means that for a certain optical flow vector, if its stretching optical flow is farther away from the optical flow vector, due to the decreasing effect of the pulling degree, the deviation angle and length difference between the two tend to increase accordingly. This deviation and difference provide important information about the facial pulling degree. Please refer to Figure 3 , which shows a schematic diagram of the distribution of optical flow vectors when the face is subjected to a force in an embodiment of the present invention.

[0045] Therefore, for the optical flow vector corresponding to any facial feature point, the angle between the optical flow vector and each corresponding stretching optical flow can be calculated as the angle deviation value. The difference between the length of the optical flow vector and the length of each corresponding stretching optical flow is used as the length deviation factor. The Euclidean distance between the starting point of the optical flow vector and the starting point of each corresponding stretching optical flow is used as the distance factor.

[0046] Then, the stretching optical flows of the optical flow vector are sorted in ascending order according to the distance factor to obtain a sorted sequence. Under the sorted sequence, the Pearson correlation coefficient between the angle deviation values of all stretching optical flows and the distance factor is calculated as the first correlation factor. Based on the foregoing analysis, it can be seen that as the distance factor increases, under the action of traction, the angle deviation value will also increase. Therefore, if the first correlation factor is closer to 1, it indicates a positive correlation between the distance factor and the angle deviation value, and the more it conforms to the change trend under the pulling action, the greater the pulling degree.

[0047] Similarly, calculate the Pearson correlation coefficient between the length deviation factors of all stretching optical flows and the distance factor as the second correlation factor. If the second correlation factor is closer to 1, it indicates a positive correlation between the distance factor and the length deviation factor, which also indicates a greater pulling degree.

[0048] Finally, the value obtained by normalizing the sum of the first correlation factor and the second correlation factor corresponding to all the stretched optical flows of the optical flow vector is used as the pulling feature value of the facial feature point corresponding to the optical flow vector. At this time, the larger the pulling feature value, the more obvious the muscle pulling situation and the greater the pulling degree at the facial feature point. Among them, normalization is a well-known technical means to those skilled in the art. The choice of the normalization function can be linear normalization, standard normalization, etc. The specific normalization method is not limited herein.

[0049] So far, the pulling feature values corresponding to each facial feature point in the current image can be obtained. The pulling feature value reflects the pulling degree of the facial feature point in a specific direction. Therefore, by analyzing the pulling feature values of the facial feature points in the image block, the pulling direction of the image block can be determined, which helps to more accurately understand the change of facial expressions.

[0050] Preferably, in an embodiment of the present invention, the method for obtaining the pulling direction includes: Based on the foregoing analysis, it can be known that when the pulling feature value of a certain facial feature point is larger, it indicates that the muscle pulling situation at the facial feature point is more obvious and the pulling degree is greater. Therefore, in each image block, the optical flow vector corresponding to the maximum pulling feature often represents the most obvious deformation trend in the image block. Therefore, the direction of the optical flow vector corresponding to the maximum pulling feature value is used as the pulling direction of each image block.

[0051] When the patient has pain, a series of painful expressions will appear on the patient's face, which will cause different deformation situations of the facial organs. For example, the eyebrows droop or frown, the eyes are closed or squinted, showing a concentrated state; the nostrils expand, the corners of the mouth droop, showing an outward expansion state. Therefore, there will be differences in the pulling directions of different image blocks. Therefore, in this embodiment of the present invention, by analyzing the distribution chaos and distribution differences of the pulling directions of the image blocks in the current image, the facial pain degree of the current image can be quantified to obtain a facial pain index.

[0052] Preferably, in an embodiment of the present invention, the method for obtaining the facial pain index includes: Since there may be a slight twist of the patient's head during the process of collecting facial images, the position of each image block in the current image can be determined first. Since there is symmetry on the left and right sides of the general patient's face, the facial symmetry line of the left and right sides of the face can be obtained in the current image, so as to determine the facial area where each image block is located. Among them, the facial area can be simply divided into a left area and a right area.

[0053] The method for obtaining the facial symmetry line includes: taking any point on the upper boundary and the lower boundary of the current image respectively and connecting them, and taking the straight line where the connection line is located as the dividing line; for any dividing line, taking the number of information points in the current image that are symmetric about the dividing line as the symmetry index of the dividing line. The larger the symmetry index corresponding to a certain dividing line, the better the symmetry of the left and right faces of the current image when taking this dividing line as the symmetry line. Therefore, the dividing line with the largest symmetry index is taken as the facial symmetry line of the current image. It should be noted that the boundary in this embodiment of the present invention refers to the border of the image.

[0054] Then, taking the facial symmetry line as the vertical axis and the line perpendicular to the facial symmetry line as the horizontal axis, a coordinate system is constructed.

[0055] For any image block, taking the straight line in the pulling direction as the moving line, taking the intersection point of the moving line and the edge line of the facial area where it is located as the first intersection point, and taking the intersection point of the moving line and the facial symmetry line as the second intersection point. Please refer to Figure 4 , which shows a schematic diagram of the first intersection point and the second intersection point in an embodiment of the present invention. Among them, the arrow indicates the pulling direction of the image block, the straight line where the arrow is located represents the moving line, the red dot position represents the first intersection point, and the green dot position represents the second intersection point. It should be noted that the method for obtaining the edge line can be obtained through the Canny operator, and it is a well-known technology, and the specific process will not be elaborated here.

[0056] Taking the image block with the pulling direction pointing to the first intersection point as the first region. Since the first intersection point is the intersection point of the moving line and the edge line of the facial area, if the pulling direction points to the first intersection point, then it can be regarded that the pulling direction of the image block is spreading outward from the face; taking the image block with the pulling direction pointing to the second intersection point as the second region. Since the second intersection point is the intersection point of the moving line and the facial symmetry line, if the pulling direction points to the second intersection point, then it can be regarded that the pulling direction of the image block is gathering inward from the face.

[0057] When expressing pain emotions, the facial muscles often undergo obvious longitudinal deformations. In contrast, the transverse deformations of the face may not be as significant as the longitudinal deformations. Therefore, for all the first regions in the current image, the first region is the image block that spreads outward from the face. Therefore, the degree of disorder of the position of the first intersection point is higher than that of the position of the second intersection point. And the information entropy is highly sensitive to small changes in data. In the evaluation of facial muscle deformations, even if the deformation degree is small, it can be characterized more accurately. Therefore, taking the ratio of the information entropy of the ordinate of the first intersection point to the information entropy of the ordinate of the second intersection point as the diffusion index. At this time, the larger the diffusion index, the greater the degree of outward diffusion of the first region, indicating the greater the deformation of the facial muscles, and the higher the pain degree.

[0058] Similarly, for all the second regions in the current image, where the second region is an image patch with the face gathering inward, the higher the degree of disorder in the position of the first intersection point compared to the degree of disorder in the position of the second intersection point. Therefore, the ratio of the information entropy of the vertical coordinate of the first intersection point to the information entropy of the vertical coordinate of the second intersection point is used as the aggregation index. The larger the aggregation index, the greater the degree of inward aggregation of the second region, indicating a greater deformation of the facial muscles, and thus a higher degree of pain.

[0059] Based on the above analysis, when making a painful expression on the face, the eyebrows, eyes, and nose parts often show muscle deformation of gathering inward, while the mouth and nostrils often show muscle deformation of spreading outward. Therefore, the vertical coordinate of the second region should be larger than the vertical coordinate of the first region. Thus, the sum value of the vertical coordinates of the first intersection point and the second intersection point in the second region is calculated as the first sum value; the sum value of the vertical coordinates of the first intersection point and the second intersection point in the first region is calculated as the second sum value; the value obtained by normalizing the difference between the first sum value and the second sum value is used as the height difference. If the height difference is positive, it indicates that the muscle direction in the current facial expression conforms more to the characteristics of painful expressions. Since the difference between the first sum value and the second sum value may be positive or negative, the normalization here can use a function.

[0060] Finally, the value obtained by normalizing the product of the diffusion index, aggregation index, and height difference corresponding to the current image is used as the facial pain index corresponding to the current image. At this time, the larger the facial pain index, the more the muscle direction in the current image conforms to the trend of painful expressions. The normalization is a well-known technical means to those skilled in the art. The choice of the normalization function can be linear normalization or standard normalization, etc. The specific normalization method is not limited here.

[0061] Step S4: Screen the data set for the training of the neural network based on the facial pain indexes corresponding to the facial images of multiple patients to obtain a trained neural network; perform postoperative pain assessment for the to-be-tested facial image of the to-be-tested patient based on the trained neural network.

[0062] To improve the efficiency of postoperative pain assessment for patients, in the embodiment of the present invention, based on the foregoing steps, the facial images of multiple patients can be analyzed to obtain facial pain indexes, and then the data set can be screened for the training of the neural network to obtain a trained neural network. Thus, in the subsequent process, the to-be-tested facial image of the to-be-tested patient can be directly subjected to postoperative pain assessment based on the trained neural network.

[0063] Preferably, in an embodiment of the present invention, the method for obtaining the data set includes: Among the facial pain indicators corresponding to the facial images of all patients, the facial images greater than or equal to the pain threshold are taken as target images. At this time, the target images can more accurately and representatively express the pain state, thus helping to improve the learning efficiency and accuracy of the neural network.

[0064] The optical flow vector can characterize the muscle changes of the face. Therefore, the optical flow vectors of the facial feature points in the target images are taken as the traction vectors, and all the traction vectors corresponding to the target images are used as the data set. At this time, the data set can more truly and effectively reflect the pain state of the patients, providing rich and accurate information for the training of the neural network.

[0065] It should be noted that the pain threshold here is set to 0.5, and the specific value can be adjusted according to the implementation scenario and is not limited here.

[0066] After obtaining the data set, the neural network can be trained: the pain indicators are classified. For example, [0.5, 0.7) is the first level of pain, [0.7, 0.9) is the second level of pain, and [0.9, 1] is the third level of pain. At this time, each traction vector in the data set corresponds to a pain level. The network structure uses Encoder-FC, and the loss function is the cross-entropy function; the input is the data set, and the output is the pain level. During the training process of the neural network, it is trained by the gradient descent method until the loss function converges to complete the training.

[0067] It should be noted that the training process of the neural network is a well-known technology, which is only briefly described here and the specific process will not be elaborated.

[0068] After obtaining the trained neural network, the anesthetic postoperative pain assessment of the facial image to be measured of the patient to be measured can be carried out based on the trained neural network. Preferably, in an embodiment of the present invention, this process includes: Based on the foregoing steps, the optical flow vectors of all facial feature points in the facial image to be measured are obtained and used as the input of the trained neural network, so as to output the pain level of the facial image to be measured.

[0069] In summary, in order to accurately evaluate the postoperative pain of patients after anesthesia, the optical flow method is first used to analyze the motion changes of pixels in the facial image, obtain the optical flow vector of the patient's face and obtain information points, which can objectively and quantitatively capture the subtle changes in facial expressions. Given that not all information points are effective for characterizing postoperative pain after anesthesia, the difference between the distribution characteristics of information points in the local neighborhood and in the image block is analyzed, and facial feature points are screened in the information points to reduce data redundancy. Due to differences in facial structure and muscle skin distribution among different patients, when patients experience pain, the direction of the muscles on the face will change, that is, the facial muscles will be pulled. Therefore, the pulling characteristics between the optical flow vectors are analyzed to obtain the pulling direction of each image block. When the face shows a painful expression, the eyes and eyebrows are usually closed and gathered, which can be characterized by the pulling direction. Therefore, the distribution disorder and difference of the pulling direction of the image block are analyzed to determine the facial pain index. The facial pain index at this time can provide a more accurate reference for measuring the postoperative pain of the current patient after anesthesia. Finally, by screening the facial pain index data set corresponding to the facial images of multiple patients and training the neural network, a mapping relationship between facial expressions and pain levels can be established, thereby realizing the subsequent automated pain assessment of the facial images of the patients to be tested, thereby improving the assessment accuracy while also improving the assessment efficiency.

[0070] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0071] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. An anesthetic postoperative pain assessment method combined with the facial expressions of patients, characterized in that, The method includes: Obtain two temporally adjacent facial images of a patient after anesthesia, and use the previous facial image in time sequence as the current image; Analyze the motion changes of pixel points in the two facial images based on the optical flow method to obtain optical flow vectors, and use the pixel points corresponding to the optical flow vectors in the current image as information points; divide the current image into blocks to obtain image blocks; analyze the differences between the distribution characteristics of information points in the local neighborhood and in the image blocks, and screen facial feature points from the information points; Determine the traction eigenvalue corresponding to each facial feature point according to the positional relationship and length difference between the optical flow vectors; in each image block, determine the traction direction of each image block based on the traction eigenvalue corresponding to the facial feature point; analyze the distribution chaos and distribution differences of the traction directions of all image blocks in the current image to determine the facial pain index of the current image; Screen a dataset for the training of a neural network based on the facial pain indices corresponding to the facial images of multiple patients to obtain a trained neural network; perform pain assessment after anesthesia on the to-be-tested facial image of the to-be-tested patient based on the trained neural network.

2. The method for evaluating postoperative pain combined with the facial expression of a patient according to claim 1, wherein The method for obtaining the facial feature points includes: In the current image, for any information point, use the information point as the center of a circle and the length of the optical flow vector corresponding to the information point as the radius to obtain the local neighborhood corresponding to the information point; In the local neighborhood corresponding to the information point, calculate the density of all information points in the local neighborhood as the first density; In the image block to which the information point belongs, calculate the density of all information points in the image block as the second density; Multiply the normalized value of the difference between the first density and the second density corresponding to the information point by the first density, and perform normalization processing on the obtained product to obtain the information factor corresponding to the information point; Among all the information points, use the information points with information factors greater than the preset information threshold as facial feature points.

3. The method for evaluating postoperative pain combined with the patient's facial expression according to claim 1, wherein The method for obtaining the traction eigenvalue includes: For the optical flow vector corresponding to any facial feature point, determine the stretching optical flow of the optical flow vector according to the positional relationship between the optical flow vector and other optical flow vectors; Calculate the angle between the optical flow vector and each corresponding stretching optical flow as the angle deviation value, use the difference between the length of the optical flow vector and the length of each corresponding stretching optical flow as the length deviation factor, and use the Euclidean distance between the starting point of the optical flow vector and the starting point of each corresponding stretching optical flow as the distance factor; Arrange the stretching optical flows of the optical flow vector in ascending order according to the distance factor to obtain a sorting sequence. Under the sorting sequence, calculate the Pearson correlation coefficient between the angle deviation values of all stretching optical flows and the distance factor as the first correlation factor, and calculate the Pearson correlation coefficient between the length deviation factors of all stretching optical flows and the distance factor as the second correlation factor; Use the normalized value of the sum of the first correlation factor and the second correlation factor corresponding to all stretching optical flows of the optical flow vector as the traction eigenvalue of the facial feature point corresponding to the optical flow vector.

4. The method for evaluating postoperative pain combined with the facial expressions of patients according to claim 3, wherein The method for obtaining the stretching optical flow includes: Optionally, select the optical flow vector corresponding to a facial feature point as the optical flow to be measured, draw a perpendicular line to the straight line where the optical flow to be measured is located through the starting point of the optical flow to be measured, and use the perpendicular line as the horizontal movement line; Use the optical flow vector intersecting with the horizontal movement line as the stretching optical flow of the optical flow to be measured.

5. A method for evaluating postoperative pain combined with the patient's facial expression according to claim 1, characterized in that, The method for obtaining the pulling direction includes: In each image block, use the direction of the optical flow vector corresponding to the maximum pulling eigenvalue as the pulling direction of each image block.

6. The method for evaluating postoperative pain combined with the facial expression of a patient according to claim 1, wherein, The method for obtaining the facial pain index includes: In the current image, obtain the facial symmetry line on the left and right of the face and determine the facial area where each image block is located, where the facial area is divided into a left area and a right area; Construct a coordinate system with the facial symmetry line in the current image as the vertical axis and the line perpendicular to the facial symmetry line as the horizontal axis; For any image block, use the straight line where the pulling direction is located as the movement line, use the intersection point of the movement line and the edge line of the facial area where it is located as the first intersection point, and use the intersection point of the movement line and the facial symmetry line as the second intersection point; Use the image block with the pulling direction pointing to the first intersection point as the first area, and use the image block with the pulling direction pointing to the second intersection point as the second area; For all the first areas in the current image, use the ratio of the information entropy of the ordinate of the first intersection point to the information entropy of the ordinate of the second intersection point as the diffusion index; For all the second areas in the current image, use the ratio of the information entropy of the ordinate of the first intersection point to the information entropy of the ordinate of the second intersection point as the aggregation index; Use the sum of the ordinate of the first intersection point and the ordinate of the second intersection point in the second area as the first sum value, use the sum of the ordinate of the first intersection point and the ordinate of the second intersection point in the first area as the second sum value, and use the value obtained by normalizing the difference between the first sum value and the second sum value as the height difference; Use the value obtained by normalizing the product of the diffusion index, the aggregation index, and the height difference corresponding to the current image as the facial pain index corresponding to the current image.

7. The method for evaluating postoperative pain combined with the facial expressions of patients according to claim 6, characterized in that The method for obtaining the facial symmetry line includes: Arbitrarily select a point on each of the upper boundary and the lower boundary of the current image and connect them, and use the straight line where the connection is located as the dividing line; For any dividing line, use the number of information points in the current image that are symmetric about the dividing line as the symmetry index of the dividing line; Use the dividing line with the largest symmetry index as the facial symmetry line of the current image.

8. A method for evaluating postoperative pain combined with the patient's facial expression according to claim 1, characterized in that, The method for obtaining the data set includes: Among the facial pain indexes corresponding to the facial images of all patients, use the facial images greater than or equal to the pain threshold as the target images; Use the optical flow vectors of the facial feature points in the target images as the pulling vectors, and use all the pulling vectors corresponding to the target images as the data set.

9. A method for evaluating postoperative pain combined with the patient's facial expression according to claim 1, characterized in that, The method for evaluating the postoperative pain of the patient to be measured based on the trained neural network for the facial image to be measured of the patient to be measured includes: Obtain the optical flow vectors of all facial feature points in the facial image to be measured and use them as the input of the trained neural network, and output the pain level of the facial image to be measured.

10. The method for evaluating postoperative pain combined with the facial expression of a patient according to claim 1, wherein The method for obtaining the image block includes: Use the superpixel segmentation method to segment the current image to obtain a number of superpixel blocks, and each superpixel block is an image block.

Images