A pain grading method and system based on image processing

Abstract

The present invention relates to the field of image processing technology, and in particular to a pain grading method and system based on image processing. When grading pain for an object to be graded, the present invention selects a pain grading sub-model based on changes in the object's heart rate, thereby improving the efficiency of pain prediction. Simultaneously, in a second pain grading sub-model, to reduce the amount of multi-feature fusion computation, a method for constraining the generation of facial feature vectors using heart rate data is employed, based on the characteristic that heart rate can better indicate the state to be assessed, thereby improving the efficiency of pain grading. Furthermore, by adjusting the weight adjustment parameters of each organ, the changing trend of each feature weight can be flexibly controlled. Furthermore, the present invention can more accurately reflect the contribution of each feature to facial expression at different heart rate levels, thereby improving the generalization capability of pain grading.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a pain grading method and system based on image processing. Background Art

[0002] Pain is a common symptom in clinical medicine, and accurately assessing the pain level of a patient is crucial for diagnosis and treatment. Traditional pain assessment methods rely primarily on subjective descriptions from the patient, such as visual analogue scales (VAS) or numerical rating scales (NRS). However, these methods have the following problems: 1. High subjectivity: The patient's pain experience may vary due to individual differences, psychological state, and other factors, resulting in inaccurate assessment results; 2. Unsuitable for patients who cannot express themselves: For example, patients in coma, children, or those with cognitive impairments find it difficult to assess pain through subjective descriptions; 3. Lack of real-time performance: Pain changes cannot be monitored in real time, making it difficult to adjust treatment plans in a timely manner.

[0003] In recent years, with the development of artificial intelligence and image processing technology, research on pain classification based on image processing and artificial intelligence models has gradually become an important research hotspot in existing technologies, especially pain classification research based on multi-feature fusion. However, in the above-mentioned studies, facial image data and heart rate data are generally fused with multiple features. Direct fusion methods and model fusion methods can be used for fusion. Among them, direct fusion has low accuracy. Using support vector machines (SVM) and deep learning models (such as CNN, LSTM, etc.) for feature fusion will undoubtedly greatly increase the computational complexity of the entire pain classification task, resulting in low classification efficiency.

[0004] At the same time, the existing technology generally adopts a pain grading model to achieve pain grading. However, for the objects to be graded, their states are different when performing pain grading, resulting in that in some states, simple calculations can obtain the pain grading results. Therefore, the efficiency of using a single model for pain grading is low; at the same time, the pain grading method in the existing technology also has the problem of insufficient generalization ability. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a pain grading method and system based on image processing, which are used to solve the problems existing in the prior art.

[0006] The present invention provides a pain grading method based on image processing, comprising the following steps:

[0007] S1: Obtain a facial image sequence of the object to be classified through an image acquisition device;

[0008] S2: performing an image preprocessing operation on the facial image sequence to obtain a preprocessed facial image sequence;

[0009] S3: Establishing a pain grading model, wherein the pain grading model includes a first pain grading sub-model and a second pain grading sub-model; wherein the pain grading sub-model is selected based on the heart rate change of the subject to be graded, specifically: if the heart rate change of the subject to be graded is less than a preset heart rate change threshold, then entering the first pain grading sub-model; otherwise, entering the second pain grading sub-model;

[0010] The second pain grading sub-model includes a dynamic feature extraction module, a feature vector determination module and a pain grading module;

[0011] S4: collecting heart rate data of the subject to be graded, and inputting the heart rate data and the pre-processed facial image sequence into the pain grading model to obtain a pain grading result.

[0012] Preferably, the dynamic features extracted by the dynamic feature extraction module include: eye features, eyebrow features and mouth features. The eye features include the frequency of eye opening and closing, and the frequency of pupil diameter change; the eyebrow features may include the frequency of eyebrow raising, and the duration of eyebrow wrinkling; the mouth features may include the frequency of mouth opening and closing, and the number of times the corners of the mouth change.

[0013] Preferably, the feature vector determination module is used to determine the weights of eye features, eyebrow features and mouth features, and weight each feature to obtain a facial feature vector.

[0014] Preferably, the initial weights of the eye features, the initial weights of the eyebrow features and the initial weights of the mouth features are first determined according to the heart rate of the object to be classified; then, a normalization operation is performed based on the initial weights of the eye features, the initial weights of the eyebrow features and the initial weights of the mouth features to obtain the eye feature weights, the eyebrow feature weights and the mouth feature weights; finally, each feature is weighted based on the eye feature weights, the eyebrow feature weights and the mouth feature weights to obtain the facial feature vector.

[0015] Preferably, the initial weight of the eye feature The calculation formula is:

[0016] ;

[0017] In the formula, HR eye is the center heart rate of the eye feature weight, k eye is the eye feature weight adjustment parameter;

[0018] Among them, the initial weight of the eyebrow feature The calculation formula is:

[0019] ;

[0020] In the formula, HR eyebrow is the central heart rate of the eyebrow feature weight, k eyebrow is the eyebrow feature weight adjustment parameter;

[0021] Among them, the initial weight of the mouth feature The calculation formula is:

[0022] ;

[0023] In the formula, HR mouth is the center heart rate of the mouth feature weight, k mouth It is the mouth feature weight adjustment parameter.

[0024] Preferably, in S4, the heart rate of the subject to be classified is the maximum heart rate value within the image sequence acquisition time.

[0025] Preferably, the method for determining the central heart rate of the eye feature weight, the central heart rate of the eyebrow feature weight, and the central heart rate of the mouth feature weight is specifically as follows:

[0026] Sa: collects the heart rate data of the subjects to be classified;

[0027] Sb: Calculate the power spectrum density data of the heart rate variation rate of the heart rate data;

[0028] Sc: Calculating the weighted average frequency of the heart rate data according to the power spectrum density data;

[0029] Sd: Calculate the center heart rate of the eye feature weight, the center heart rate of the eyebrow feature weight, and the center heart rate of the mouth feature weight based on the weighted average frequency.

[0030] Preferably, the weighted average method is used to realize the grayscale conversion of the image, and the specific formula is:

[0031] ;

[0032] Wherein, R, G, and B represent the pixel values ​​of the red, green, and blue channels in the facial image, respectively; Gray is the pixel value after grayscale conversion; the linear normalization method is used to normalize the pixel values ​​of the facial image; and the mean filtering method is used to achieve image noise reduction.

[0033] Preferably, in S3, the first pain grading sub-model adopts a lightweight convolutional neural network architecture, and specifically selects the Mobile Net architecture as the basic model.

[0034] According to another aspect of the present invention, a pain grading system based on image processing is provided. The system adopts the above-mentioned pain grading method based on image processing, and the system includes:

[0035] An image acquisition device for acquiring a facial image sequence of an object to be classified;

[0036] A preprocessing module, configured to perform image preprocessing on the facial image sequence to obtain a preprocessed facial image sequence;

[0037] A model building module, configured to build a pain grading model, wherein the pain grading model includes a first pain grading sub-model and a second pain grading sub-model;

[0038] A heart rate collection module is used to collect heart rate data of the subject to be classified;

[0039] The processor is configured to input the heart rate data and the preprocessed facial image sequence into the pain grading model to obtain a pain grading result.

[0040] The embodiments of the present invention have the following technical effects:

[0041] When grading pain for a subject, the present invention selects a pain grading sub-model based on the subject's heart rate changes. This allows computing power and time to be focused on more complex pain grading scenarios during the pain grading process, thereby improving the efficiency of pain prediction.

[0042] More importantly, in the second pain grading sub-model, to reduce the computational effort of multi-feature fusion, a method was adopted to constrain the generation of facial feature vectors using heart rate data, based on the fact that heart rate can better indicate the state to be evaluated. That is, the weights of facial features are affected by heart rate values, and the feature weights can be dynamically adjusted according to the real-time heart rate to adapt to the pain grading requirements under different heart rate states. This reduces the computational effort of multi-feature fusion and improves the efficiency of pain grading. At the same time, by adjusting the weight adjustment parameters of each organ, the changing trend of each feature weight can be flexibly controlled. It can also more accurately reflect the contribution of each feature to the expression at different heart rate levels, improving the generalization ability of pain grading.

[0043] At the same time, the power spectral density is used to indirectly determine a value reflecting the central trend of the heart rate, so as to more accurately measure the heart rate differences between different individuals and further improve the generalization ability of pain grading. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flowchart of a pain grading method based on image processing provided by an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the Mobile Net architecture provided by an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of a pain grading module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0049] Example 1, attached Figure 1 A flowchart of a pain grading method based on image processing is shown in the attached Figure 1 As shown, a pain grading method based on image processing includes the following steps:

[0050] S1: Obtain a facial image sequence of the object to be classified through an image acquisition device;

[0051] Image acquisition is a key step in the pain grading method based on image processing, and its accuracy directly affects the accuracy of subsequent feature extraction and pain grading. When selecting the image acquisition device, a high-definition camera with a resolution of not less than 1920×1080 is selected to ensure that facial details can be clearly captured. High resolution helps to more accurately extract facial expression features. At the same time, ensure that the camera frame rate is not less than 30fps to avoid image distortion caused by device delay, especially distortion when capturing dynamic expressions.

[0052] When acquiring an image sequence, the acquisition environment has a significant impact on the accuracy of subsequent pain grading. Therefore, during the acquisition process, ensure that the lighting in the acquisition environment is uniform and sufficient, avoiding direct strong light or shadows. In this step, this embodiment uses a soft, scattered light source, such as a diffuse reflector, to provide uniform lighting; ensure that there are no reflective objects, such as mirrors or metal surfaces, in the acquisition area; and select a simple, non-interfering background to reduce background interference with image analysis. For example, use a solid-color background wall to avoid complex patterns or objects appearing in the background.

[0053] At the same time, ensure that the distance between the camera and the object to be classified is moderate. In this step, the distance between the camera and the object to be classified is between 1-2 meters to ensure that the size of the facial image is moderate. The camera should face the face of the object to be classified to avoid facial deformation due to angle problems. In this step, use a tripod or fixed bracket to ensure the stability of the camera and the accuracy of the angle.

[0054] S2: performing an image preprocessing operation on the facial image sequence to obtain a preprocessed facial image sequence;

[0055] Wherein, the preprocessing includes image grayscale, image noise reduction and pixel normalization processing;

[0056] Grayscaling is the process of converting a color facial image into a grayscale facial image. Color facial images usually contain three color channels: red, green, and blue. The pixel value of each channel ranges from 0 to 255. Grayscaling can simplify a color facial image into a single-channel grayscale facial image, reducing the amount of data while retaining the main structure and texture information of the facial image. In this step, the weighted average method is used to achieve image grayscaling. The specific formula is:

[0057] ;

[0058] Wherein, R, G, and B represent the pixel values ​​of the red, green, and blue channels in the facial image, respectively; Gray is the grayscale pixel value.

[0059] Image denoising is the process of removing noise from facial images. Noise can arise from hardware defects in the image acquisition device, poor ambient lighting conditions, interference during signal transmission, and other factors. Image denoising can improve image quality and enhance useful information. In this step, mean filtering is used to achieve image denoising.

[0060] Image normalization is the process of adjusting the range of the facial image pixel values ​​to a specific interval to facilitate subsequent processing. In this step, the facial image pixel values ​​are normalized using a linear normalization method.

[0061] It is worth emphasizing that in this step, in the image preprocessing link, any one or several of the above-mentioned preprocessing operation processes can be adopted, and this embodiment does not specifically limit this.

[0062] Grayscaling, noise reduction, and normalization can effectively improve image quality, reduce noise interference, and enhance useful information, providing a better data foundation for subsequent feature extraction and model training. The proper selection and application of these preprocessing steps can significantly improve the accuracy and reliability of pain grading.

[0063] S3: Establishing a pain grading model, wherein the pain grading model includes a first pain grading sub-model and a second pain grading sub-model;

[0064] In the prior art, when using facial image sequences for pain grading, facial expression features and heart rate features are typically extracted separately, such as frequency-domain heart rate-related features. These features are then fused. However, due to the need to extract multiple features from multiple facial organs and multiple heart rate features, feature fusion is typically performed directly or using a fusion feature model. When direct fusion is inaccurate, model fusion, such as support vector machines (SVMs) or deep learning models (e.g., CNNs and LSTMs), is employed. This undoubtedly increases the computational complexity of the entire pain grading task. This embodiment addresses this issue, considering that heart rate values ​​are highly sensitive to pain grading results during pain grading, and that facial organs react differently to pain under different heart rate states. A method for generating facial feature vectors based on heart rate data constraints is proposed. First, a pain grading sub-model is selected based on the heart rate changes of the subject to be graded. Then, when pain states are difficult to predict, heart rate values ​​are further used to constrain the facial expression feature extraction parameters. This reduces the computational complexity of the pain grading task while ensuring feature extraction accuracy.

[0065] Specifically, the heart rate change is the difference between the maximum heart rate and the minimum heart rate during image acquisition;

[0066] Among them, the pain classification sub-model is selected according to the heart rate changes of the subject to be classified, specifically:

[0067] If the heart rate change of the subject to be graded is less than the preset heart rate change threshold, the first pain grading sub-model is entered; otherwise, the second pain grading sub-model is entered.

[0068] In this embodiment, the preset heart rate change threshold is 10 bpm.

[0069] It is worth emphasizing that other indicators can also be used as indicators for selecting to enter the first pain grading sub-model and the second pain grading sub-model; and similar to the acquisition of the heart rate change indicator, the selection indicator can be obtained by measuring the correlation between the calibrated pain grading results and the indicators that can reflect pain.

[0070] Furthermore, in order to meet the requirements of real-time and efficiency, the first pain classification sub-model adopts a lightweight convolutional neural network (CNN) architecture, specifically the Mobile Net architecture (see Appendix). Figure 2) as the basic model, the Mobile Net uses the Depth-wise Separable Convolution technology to significantly reduce the computational complexity and number of parameters of the model while maintaining high accuracy;

[0071] Specifically, the first pain grading sub-model includes an input layer, a convolutional layer, a fully connected layer, and an output layer;

[0072] The input layer is used to input the pre-processed facial image, and the image size is uniformly adjusted to 224×224 pixels;

[0073] The structure of the convolutional layer includes:

[0074] The first convolutional layer uses a 3×3 convolution kernel, a stride of 2, 32 output channels, and a ReLU activation function to perform preliminary feature extraction on the preprocessed facial image, while reducing the spatial dimension of the feature map and reducing the amount of computation.

[0075] Depthwise Separable Convolution Layer: It consists of multiple depthwise separable convolution modules. Each module includes depthwise convolution and pointwise convolution. The depthwise convolution performs convolution operations on each input channel separately, while the pointwise convolution uses a 1×1 convolution kernel to perform channel fusion on the output of the depthwise convolution. This structure effectively reduces the amount of computation and the number of parameters while ensuring feature extraction capabilities.

[0076] The pooling layer is used to add a maximum pooling layer after the partial depth-separable convolution module. The pooling kernel size is 2×2 and the stride is 2. It is used to further reduce the spatial dimension of the feature map, extract more abstract features, and reduce the amount of computation.

[0077] The fully connected layer is used to flatten the feature map into a one-dimensional vector after multiple layers of convolution and pooling operations, and then pass it through a fully connected layer with an output dimension of 128, using the ReLU activation function to further perform nonlinear transformation and integration on the features;

[0078] The output dimension of the last fully connected layer is determined by the specific classification task. For example, if the task is to recognize four basic facial expressions (smiling, calm eyes, natural eyebrows, and slightly raised mouth corners), the output dimension is 4. Using the softmax activation function, the output values ​​are normalized into a probability distribution, representing the probability of the input image belonging to each facial expression state.

[0079] The output layer is used to provide a pain grading result corresponding to the preprocessed facial image according to the output of the fully connected layer.

[0080] When the subject to be graded is in a state of high heart rate variation (>100bpm), the facial expression generally exhibits the characteristics of "rapid appearance and short duration". Therefore, it is necessary to extract the dynamic characteristics of a large number of facial image sequences of the subject to be graded from multiple organs to improve the accuracy of pain grading.

[0081] In this step, the second pain grading sub-model includes a dynamic feature extraction module, a feature vector determination module and a pain grading module;

[0082] The dynamic features extracted by the dynamic feature extraction module include: eye features, eyebrow features and mouth features;

[0083] As a preferred embodiment, the eye features may include the frequency of eye opening and closing, and the frequency of pupil diameter change; the eyebrow features may include the frequency of eyebrow raising, and the degree and duration of eyebrow wrinkling; the mouth features may include the frequency of mouth opening and closing, and the number of times the corners of the mouth change.

[0084] Among them, the eye opening and closing frequency is calculated by calculating the number of times the eyes open and close in unit time. By detecting the key points of the eye contour, the ratio of the width and height of the eyes is calculated. When the ratio is lower than a certain threshold, the eyes are considered to be closed, and when it is higher than the threshold, the eyes are considered to be open. The number of times the eyes open and close from closed to open in unit time is counted to obtain the eye opening and closing frequency.

[0085] The pupil diameter change frequency is calculated by measuring the change in pupil diameter within a certain period of time. The pupil diameter is calculated by detecting key points of the pupil center and pupil edge, and the change trend of the pupil diameter, such as the frequency of pupil dilation and contraction, is counted.

[0086] The eyebrow raising frequency is calculated by counting the number of times the eyebrows rise within a certain period of time. The key points of the eyebrow contour are detected and the inclination angle of the eyebrows is calculated. When the inclination angle exceeds a certain threshold, the eyebrows are considered to be raised. The number of times the eyebrows rise per unit time is counted to obtain the eyebrow raising frequency.

[0087] The eyebrow frowning degree time is used to calculate the frowning degree of the eyebrows. The eyebrow curvature is calculated by detecting key points of the eyebrow contour. When the curvature exceeds a certain threshold, the eyebrows are considered to be frowned. The duration of the eyebrow frowning in unit time is counted.

[0088] The mouth opening and closing frequency is calculated by calculating the number of times the mouth opens and closes per unit time. The key points of the mouth contour are detected and the vertical distance between the upper and lower lips is calculated. When the distance exceeds a certain threshold, the mouth is considered to be open, and when it is below the threshold, the mouth is considered to be closed. The number of times the mouth switches from closed to open per unit time is counted to obtain the mouth opening and closing frequency.

[0089] The number of times the corners of the mouth change is used to calculate the degree of uplift and droop of the corners of the mouth. The key points of the corners of the mouth are detected and the angle between the corners of the mouth and the horizontal line is calculated. When the angle is positive, the corner of the mouth is considered to be uplifted, and when it is negative, the corner of the mouth is considered to be drooped. The number of times the corners of the mouth rise and droop in unit time is counted to obtain the number of times the corners of the mouth change.

[0090] The feature vector determination module is used to determine the weights of eye features, eyebrow features and mouth features, and weight each feature to obtain a facial feature vector;

[0091] In determining the weights of the eye features, the eyebrow features, and the mouth features, firstly, the initial weights of the eye features, the eyebrow features, and the mouth features are determined according to the heart rate of the subject to be classified;

[0092] The initial weight of the eye feature The calculation formula is:

[0093] ;

[0094] In the formula, HR eye is the center heart rate of the eye feature weight, k eye is the eye feature weight adjustment parameter;

[0095] Among them, the initial weight of the eyebrow feature The calculation formula is:

[0096] ;

[0097] In the formula, HR eyebrow is the central heart rate of the eyebrow feature weight, k eyebrow is the eyebrow feature weight adjustment parameter;

[0098] Among them, the initial weight of the mouth feature The calculation formula is:

[0099] ;

[0100] In the formula, HR mouth is the center heart rate of the mouth feature weight, k mouth It is the mouth feature weight adjustment parameter.

[0101] Then, a normalization operation is performed based on the initial weights of the eye features, the initial weights of the eyebrow features, and the initial weights of the mouth features to obtain eye feature weights, eyebrow feature weights, and mouth feature weights. Finally, each feature is weighted based on the eye feature weights, the eyebrow feature weights, and the mouth feature weights to obtain a facial feature vector.

[0102] Through the above-mentioned weight determination method, the feature weights can be dynamically adjusted according to the real-time heart rate to adapt to the pain grading needs under different heart rate states. At the same time, by adjusting the weight adjustment parameters of each organ, the changing trend of each feature weight can be flexibly controlled; and it can more accurately reflect the contribution of each feature to the expression under different heart rate levels, thereby improving the generalization ability of pain grading.

[0103] Furthermore, the determination of the center heart rate of the eye feature weight, the center heart rate of the eyebrow feature weight, and the center heart rate of the mouth feature weight has a significant impact on the determination of the facial feature vector; therefore, as a preferred embodiment, a method for determining the center heart rate of the eye feature weight, the center heart rate of the eyebrow feature weight, and the center heart rate of the mouth feature weight is further provided, specifically:

[0104] Sa: collects the heart rate data of the subjects to be classified;

[0105] In this step, continuous heart rate data of a certain length of time needs to be collected. The collection time is the same as the image sequence collection time. At the same time, the collection frequency of the heart rate data should be as high as possible to ensure that subtle changes in the heart rate can be captured. For example, the heart rate data can be collected once per second.

[0106] Sb: Calculate the power spectrum density data of the heart rate variation rate of the heart rate data;

[0107] The heart rate variability is used to evaluate the severity of heart rate fluctuations. The power spectrum density of the heart rate variability is calculated using Fourier transform, and the heart rate signal is converted from the time domain to the frequency domain in order to analyze its frequency components.

[0108] Sc: Calculating the weighted average frequency of the heart rate data according to the power spectrum density data;

[0109] Among them, the weighted average frequency f center The calculation formula is:

[0110] ;

[0111] Where, f i is the i-th frequency point, PSD ( f i ) is the power spectrum density corresponding to the i-th frequency point;

[0112] Sd: Calculate the center heart rate of the eye feature weight, the center heart rate of the eyebrow feature weight, and the center heart rate of the mouth feature weight according to the weighted average frequency;

[0113] Among them, the center heart rate of the eye feature weight , the central heart rate of eyebrow feature weight , the center heart rate of the mouth feature weight The calculation formula is:

[0114] .

[0115] Through the above scheme, the power spectral density is used to indirectly determine a value reflecting the central trend of the heart rate, thereby more accurately measuring the heart rate differences between different individuals and further improving the generalization ability of pain grading.

[0116] In addition, as a preferred embodiment, after calculating the weight of each organ, the organ weight is evenly divided for the sub-features included in each organ feature; for example, the weight of the eye is calculated to be 0.56, then the weights for the feature eye opening and closing frequency and pupil diameter change are both 0.28.

[0117] The pain grading module is a convolutional neural network model, and the model architecture includes an input layer, multiple convolutional layers, multiple pooling layers, multiple fully connected layers and an output layer. Figure 3 The specific architecture of the convolutional neural network model is shown, including an input layer, 3 convolutional layers, 3 pooling layers, 2 fully connected layers, and an output layer.

[0118] S4: collecting heart rate data of the subject to be graded, and inputting the heart rate data and the pre-processed facial image sequence into the pain grading model to obtain a pain grading result.

[0119] In this step, in order to avoid the sudden change caused by model switching affecting the stability of the recognition result, the heart rate of the object to be classified is the maximum heart rate value within the image sequence acquisition time.

[0120] In Example 2, the present invention further provides a pain grading system based on image processing, wherein the system adopts the pain grading method based on image processing of Example 1, and the system comprises:

[0121] An image acquisition device for acquiring a facial image sequence of an object to be classified;

[0122] A preprocessing module, configured to perform image preprocessing on the facial image sequence to obtain a preprocessed facial image sequence;

[0123] A model building module, configured to build a pain grading model, wherein the pain grading model includes a first pain grading sub-model and a second pain grading sub-model;

[0124] A heart rate acquisition module, used to acquire the heart rate of the subject to be classified;

[0125] The processor is configured to input the heart rate and the preprocessed facial image sequence into the pain grading model to obtain a pain grading result.

[0126] Specifically, the heart rate change is the difference between the maximum heart rate and the minimum heart rate during image acquisition;

[0127] The pain grading model is divided into two sub-models according to the heart rate changes of the subjects to be graded; specifically:

[0128] If the heart rate change of the subject to be graded is less than the preset heart rate change threshold, the first pain grading sub-model is entered; otherwise, the second pain grading sub-model is entered.

[0129] In this embodiment, the preset heart rate change threshold is 10 bpm.

[0130] The first pain grading sub-model adopts a lightweight convolutional neural network (CNN) architecture, specifically selecting the Mobile Net architecture as the basic model. The Mobile Net uses depth-wise separable convolution technology to significantly reduce the model's computational complexity and number of parameters while maintaining a high accuracy rate.

[0131] The second pain grading sub-model includes a dynamic feature extraction module, a feature vector determination module and a pain grading module;

[0132] The dynamic features extracted by the dynamic feature extraction module include: eye features, eyebrow features and mouth features; the eye features may include the frequency of eye opening and closing, and the frequency of pupil diameter change; the eyebrow features may include the frequency of eyebrow raising, and the duration of eyebrow frowning; the mouth features may include the frequency of mouth opening and closing, and the number of times the corners of the mouth change.

[0133] The feature vector determination module is used to determine the weights of eye features, eyebrow features and mouth features, and weight each feature to obtain a facial feature vector;

[0134] In determining the weights of the eye features, the eyebrow features, and the mouth features, firstly, the initial weights of the eye features, the eyebrow features, and the mouth features are determined according to the heart rate of the subject to be classified;

[0135] Among them, the initial weight of the eye feature The calculation formula is:

[0136] ;

[0137] In the formula, HR eye is the center heart rate of the eye feature weight, k eye is the eye feature weight adjustment parameter;

[0138] Among them, the initial weight of the eyebrow feature The calculation formula is:

[0139] ;

[0140] In the formula, HR eyebrow is the central heart rate of the eyebrow feature weight, k eyebrow is the eyebrow feature weight adjustment parameter;

[0141] Among them, the initial weight of the mouth feature The calculation formula is:

[0142] ;

[0143] In the formula, HR mouth is the center heart rate of the mouth feature weight, k mouth It is the mouth feature weight adjustment parameter.

[0144] Then, a normalization operation is performed based on the initial weights of the eye features, the initial weights of the eyebrow features, and the initial weights of the mouth features to obtain eye feature weights, eyebrow feature weights, and mouth feature weights. Finally, each feature is weighted based on the eye feature weights, the eyebrow feature weights, and the mouth feature weights to obtain a facial feature vector.

[0145] Through the above-mentioned weight determination method, the feature weights can be dynamically adjusted according to the real-time heart rate to adapt to the pain grading needs under different heart rate states. At the same time, by adjusting the central heart rate and weight adjustment parameters of each organ, the changing trend of each feature weight can be flexibly controlled; and it can more accurately reflect the contribution of each feature to the expression at different heart rate levels, thereby improving the generalization ability of pain grading.

[0146] In addition, as a preferred embodiment, after calculating the weight of each organ, the organ weight is evenly divided for the sub-features included in each organ feature; for example, the weight of the eye is calculated to be 0.56, then the weights for the feature eye opening and closing frequency and pupil diameter change are both 0.28.

[0147] The pain grading module is a convolutional neural network model, and the model architecture includes an input layer, multiple convolutional layers, multiple pooling layers, multiple fully connected layers and an output layer.

[0148] Example 3: The present invention also provides an electronic device, including one or more processors and a memory.

[0149] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0150] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the image processing-based pain grading method and / or other desired functions of any embodiment of the present application described above. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A pain grading method based on image processing, characterized in that: The following steps are involved: S1: Obtain a facial image sequence of the object to be classified through an image acquisition device; S2: performing an image preprocessing operation on the facial image sequence to obtain a preprocessed facial image sequence; S3: Establishing a pain grading model, wherein the pain grading model includes a first pain grading sub-model and a second pain grading sub-model; wherein the pain grading sub-model is selected based on the heart rate change of the subject to be graded, specifically: if the heart rate change of the subject to be graded is less than a preset heart rate change threshold, then entering the first pain grading sub-model; otherwise, entering the second pain grading sub-model; The second pain grading sub-model includes a dynamic feature extraction module, a feature vector determination module, and a pain grading module; the feature vector determination module is used to determine the weights of eye features, eyebrow features, and mouth features, and weight each feature to obtain a facial feature vector; first, the initial weights of the eye features, the initial weights of the eyebrow features, and the initial weights of the mouth features are determined based on the heart rate of the subject to be graded; then, a normalization operation is performed based on the initial weights of the eye features, the initial weights of the eyebrow features, and the initial weights of the mouth features to obtain eye feature weights, eyebrow feature weights, and mouth feature weights; finally, each feature is weighted based on the eye feature weights, the eyebrow feature weights, and the mouth feature weights to obtain the facial feature vector; S4: collecting heart rate data of the subject to be graded, and inputting the heart rate data and the pre-processed facial image sequence into the pain grading model to obtain a pain grading result.

2. The pain grading method based on image processing according to claim 1, characterized in that: The dynamic features extracted by the dynamic feature extraction module include: eye features, eyebrow features and mouth features. The eye features include the frequency of eye opening and closing, and the frequency of pupil diameter change; the eyebrow features may include the frequency of eyebrow raising, and the duration of eyebrow frowning; the mouth features may include the frequency of mouth opening and closing, and the number of times the corners of the mouth change.

3. The pain grading method based on image processing according to claim 1, characterized in that: The initial weight of the eye feature The calculation formula is: ； Where HR is the heart rate of the subject to be classified, HR eye is the center heart rate of the eye feature weight, k eye is the eye feature weight adjustment parameter; Among them, the initial weight of the eyebrow feature The calculation formula is: ； In the formula, HR eyebrow is the central heart rate of the eyebrow feature weight, k eyebrow is the eyebrow feature weight adjustment parameter; Among them, the initial weight of the mouth feature The calculation formula is: ； In the formula, HR mouth is the center heart rate of the mouth feature weight, k mouth It is the mouth feature weight adjustment parameter.

4. The pain grading method based on image processing according to claim 1, characterized in that: In S4, the heart rate of the subject to be classified is the maximum heart rate value within the image sequence acquisition time.

5. The pain grading method based on image processing according to claim 3, characterized in that: The method for determining the central heart rate of the eye feature weight, the central heart rate of the eyebrow feature weight, and the central heart rate of the mouth feature weight is specifically as follows: Sa: collects the heart rate data of the subjects to be classified; Sb: Calculate the power spectrum density data of the heart rate variation rate of the heart rate data; Sc: Calculating the weighted average frequency of the heart rate data according to the power spectrum density data; Sd: Calculate the center heart rate of the eye feature weight, the center heart rate of the eyebrow feature weight, and the center heart rate of the mouth feature weight based on the weighted average frequency.

6. The pain grading method based on image processing according to claim 5, characterized in that: The weighted average method is used to realize image grayscale. The specific formula is: ; Where R, G, and B represent the pixel values ​​of the red, green, and blue channels in the facial image, respectively; Gray is the grayscale pixel value; Normalizing the pixel values ​​of the facial image using a linear normalization method; The mean filter method is used to achieve image noise reduction.

7. The pain grading method based on image processing according to claim 1, characterized in that: In S3, the first pain grading sub-model adopts a lightweight convolutional neural network architecture, and selects the Mobile Net architecture as the basic model.

8. A pain grading system based on image processing, characterized in that: The system adopts the image processing-based pain grading method according to any one of claims 1 to 7, and the system comprises: An image acquisition device for acquiring a facial image sequence of an object to be classified; A preprocessing module, configured to perform image preprocessing on the facial image sequence to obtain a preprocessed facial image sequence; A model building module, configured to build a pain grading model, wherein the pain grading model includes a first pain grading sub-model and a second pain grading sub-model; A heart rate collection module is used to collect heart rate data of the subject to be classified; The processor is configured to input the heart rate data and the preprocessed facial image sequence into the pain grading model to obtain a pain grading result.

Images