Data-driven acra hypertrophy patient typing and face portrait drawing method

Abstract

The invention discloses an acra hypertrophy patient typing and face portrait drawing method based on data driving. The method comprises the following steps: preprocessing an acquired facial image of a patient with acromegaly; facial feature point location extraction is carried out on the preprocessed facial image, and feature selection is carried out on facial features after every two feature points are combined; the optimal class cluster number is searched and determined in the acra hypertrophy patient sample through an elbow method, and classification is conducted on the acra hypertrophy patient through a clustering algorithm; and calculating the average coordinate of each feature point of the facial feature points of each class of acra hypertrophy patients in sequence, and drawing facial portraits of different classes of patients according to the average coordinates. In addition, the facial features and clinical features of different types of acra hypertrophy patients are subjected to statistical analysis. The invention provides an effective method which is low in cost and can quickly assist in clinically judging the severity of the acra hypertrophy patient, and is beneficial for doctors to accurately treat the patients with different illness degrees.

Description

Technical Field

[0001] The present invention belongs to the technical field of acromegaly category classification and differential analysis. Specifically, it relates to a method for classifying acromegaly patients and drawing facial portraits based on data-driven Background Art

[0002] Acromegaly is a relatively rare chronic disease. Usually, patients are accompanied by pituitary growth hormone adenomas, resulting in excessive secretion of growth hormone (GH) and insulin-like growth factor 1 (IGF-1). [1] The onset of the disease is insidious, the symptoms are unclear, and the progression is slow. It often delays diagnosis, causes serious complications such as cardiovascular diseases and treatment difficulties, and increases the mortality rate.

[0003] Due to the insidious course of acromegaly, diagnosis is often made only after the appearance of suspected symptoms and signs, which delays for a long time and makes patients have diseases of different severities when diagnosed. Clinically, the severity of acromegaly patients is usually scored by neuroendocrinology experts, and they are scored according to mild, moderate and severe degrees (1-3 points) respectively. However, the method of using manual scoring is usually cumbersome, inefficient and consumes a large amount of manpower and financial resources. Since acromegaly patients usually have typical facial changes, such as wide nose, large tongue, enlarged and protruding mandible, enlarged zygoma, protruding supraorbital ridge, soft tissue swelling and skin thickening, the classification of acromegaly patients can be carried out through facial images. Kong Y et al. [2] Constructed an automatic diagnosis and severity classification model for acromegaly from 2148 facial photo data of different severities, and each photo was given a score (1-3 points) reflecting the severity. However, this method only classifies the severity of patient images, does not analyze the facial feature differences of patients with different degrees, and does not record and statistically analyze the relevant clinical variables of different patients.

[0004] There is little discussion in the existing literature on the method for classifying acromegaly patients. The vast majority of the literature only focuses on the diagnosis of acromegaly and has not reported the relevant work on drawing facial portraits of patients; however, the accurate classification of acromegaly patients can contribute to the accurate diagnosis, evaluation and tracking of the disease.

[0005] [1] Meng T, Guo X, Lian W, et al. Identifying facial features and predicting patients of acromegaly using three-dimensional imaging techniques and machine learning[J]. Frontiers in endocrinology, 2020, 11:492.

[0006] [2] Kong Y, Kong X, He C, et al. Constructing an automatic diagnosis and severity-classification model for acromegaly using facial photographs by deep learning[J]. Journal of Hematology&Oncology, 2020, 13:1-4. Summary of the Invention

[0007] Aiming at the deficiencies of the existing clinical classification techniques for acromegaly patients, the purpose of the present invention is to provide a data-driven classification and facial portrait drawing method for acromegaly patients, which can enable doctors to more accurately diagnose, evaluate and track acromegaly patients with different severities, and alleviate the problems of low efficiency, cumbersome operation and manpower consumption existing in the prior art. The present invention extracts the facial features of patient images, and fuses the physiological and apparent features based on an expert knowledge base. The elbow method is used to find and determine the optimal number of clusters in the acromegaly patient samples, and a clustering algorithm is used to classify the acromegaly patients. The average coordinates of each feature point are calculated for the facial feature points of each type of acromegaly patient in turn, and the facial portraits of patients of different types are drawn according to the average coordinates. At the same time, statistical analysis is performed on the facial features and clinical features of acromegaly patients of different types, which can effectively clarify the feature differences between patients of different types.

[0008] The technical solution of the present invention is specifically introduced as follows.

[0009] The present invention provides a data-driven classification and facial portrait drawing method for acromegaly patients, and the specific steps are as follows:

[0010] 1) Collection and construction of the data set

[0011] Collect the frontal facial images of a number of acromegaly patients to construct a data set;

[0012] 2) Data cleaning

[0013] Preprocess the facial image, including facial alignment, facial cropping, and facial geometric normalization;

[0014] 3) Facial feature point recognition

[0015] For the preprocessed facial image, accurately locate the feature points through the facial image recognition algorithm, including: facial contour, forehead, both eyes, cheekbones, nasolabial area, and mandibular area;

[0016] 4) Facial feature selection and fusion

[0017] Pairwise combine the recognized facial feature points to calculate the corresponding Euclidean distance to construct facial feature information, and then use the L1 regularization method to select the facial feature information to obtain the reduced-dimensional facial features;

[0018] Mark on the face the relevant physiological and epigenetic features of acromegalic patients obtained from the expert knowledge base. The physiological and epigenetic features include: thick lips, broad nose and tongue, prominent supraorbital ridges and cheekbones, and enlarged and protruding mandible;

[0019] Combine the marked physiological and epigenetic features with the reduced-dimensional facial features for feature fusion to obtain the final facial features;

[0020] 5) Determine the optimal number of clusters

[0021] Based on the final facial features, use the elbow method to find the optimal number of clusters in the acromegalic patient samples;

[0022] 6) Data-driven patient category typing and facial portrait drawing

[0023] According to the determined optimal number of clusters, use the clustering algorithm to type the acromegalic patient samples into K categories.

[0024] Calculate the average coordinates of the facial feature points of each category of acromegalic patients in turn, and draw the facial portraits of different categories of patients according to the average coordinates.

[0025] In the present invention, in step 1), the collected facial images include data of the patient at different time periods; at the same time, clinical variables such as gender, age, and BMI that have a greater impact on facial features are recorded.

[0026] In the present invention, in step 2), the method of data cleaning uses the multi-task convolutional neural network MTCNN algorithm to detect, align, and crop the human faces in the facial images, and normalize the facial geometry to obtain a preprocessed facial image with a size of 350×350 pixels.

[0027] In the present invention, in step 3), the facial feature point recognition method adopts the Facemesh algorithm or the Dlib algorithm.

[0028] In the present invention, in step 4), in the feature selection and fusion step, a feature fusion method combining L1 regularization dimensionality reduction with expert knowledge base annotation is adopted, which solves the problem that a single data-driven method ignores clinical prior knowledge, thereby being able to improve the discrimination and clinical relevance of classification.

[0029] In the present invention, in step 5), the optimal number of clusters K is determined by calculating and observing the change trend of the sum of squared errors under different numbers of clusters.

[0030] In the present invention, in step 6), the clustering algorithm is the Kmeans algorithm.

[0031] The present invention also includes step 7): statistical analysis of the clinical characteristics of different categories of patients, specifically including:

[0032] According to the drawn facial portraits, the facial feature differences of different categories of acromegalic patients are compared, and the clinical variables of different categories of patients are statistically analyzed. For clinical characteristics, the variables are expressed as mean ± standard deviation. One-way analysis of variance and LSD post hoc analysis are used to determine the differences between the two groups. And chi-square test or Fisher's exact test is used for analysis. A p-value < 0.05 is considered statistically significant.

[0033] As above, the present invention extracts relevant facial feature information from facial images for the classification of patients with acromegaly in view of the special facial features of acromegalic patients, and can intuitively distinguish acromegalic patients with different degrees of illness severity, which is faster and more efficient compared with the clinical methods for patient classification that are cumbersome, inefficient, and labor-consuming; at the same time, since the course of acromegaly is relatively occult, the present invention allows users to take photos at any time and upload the frontal facial images into the system for classification, thereby helping doctors to discriminate and analyze the degree of illness of patients. Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1) The present invention fully extracts the facial feature information of acromegalic patients, covering features such as facial contour, forehead, both eyes, cheekbones, nasolabial area, and mandible.

[0035] 2) The present invention classifies patients based on data-driven, and can preferably distinguish the subtypes to which the patients belong through their facial features, and show the clinical characteristics possessed by the subtypes.

[0036] 3) The present invention draws corresponding facial portraits for different categories of acromegalic patients, and can intuitively find out the facial feature differences between different patients from the facial portraits according to facial feature points (such as facial contours, facial regions, nose and tongue sizes, zygomatic heights, etc.).

[0037] 4) The present invention uses statistical methods to statistically analyze the clinical characteristics of different categories of acromegalic patients, and can intuitively show the significant differences between different characteristic variables. Brief Description of the Drawings

[0038] Figure 1 is a schematic flow chart of the research on the method for classifying acromegalic patients and drawing facial portraits based on data driving of the present invention.

[0039] Figure 2 is the facial portrait of an acromegalic patient drawn in the embodiment.

[0040] Figure 3 is the schematic diagram showing the visualization change of the heat map in the embodiment. Detailed Embodiment

[0041] The technical solution of the present invention will be introduced in detail below in conjunction with the drawings and embodiments.

[0042] The present invention extracts facial image features based on data driving, uses the elbow method to find the optimal number of clusters in the sample, then classifies acromegalic patients according to the clustering algorithm, and draws the facial portraits of each category of patients. The method proposed in this embodiment uses facial feature information based on data driving to conduct research on the classification of acromegalic patients and draw facial portraits for each category of patients, which is beneficial for doctors to accurately diagnose, evaluate and track the disease.

[0043] As Figure 1 shown, a method for classifying acromegalic patients and drawing facial portraits based on data driving provided in this embodiment specifically includes the following steps:

[0044] 1) Obtain frontal facial image data, including the acromegalic patient dataset. Collect the frontal facial images of the subjects at different times (before surgery, several months after surgery, half a year after surgery, one year after surgery, etc.), and record information such as the gender, age, height, weight, growth hormone, etc. of each subject.

[0045] 2) Clean the facial images. First, use the MTCNN algorithm to detect 5 feature points of the eyes, nose, and mouth in the image, obtain the coordinates of the two eye feature points, calculate the angle between the coordinate line and the horizontal line, obtain the rotation matrix through the angle, and apply the rotation matrix to the entire image to achieve facial alignment. Then, use the face detection model to identify the faces in the image and set the rectangular bounding boxes to crop the face regions. Finally, set the size of each facial image to 350×350 pixels.

[0046] 3) For the preprocessed frontal facial images, identify 468 facial feature points in the facial images through the Facemesh algorithm, including all facial regions such as facial contours, foreheads, both eyes, cheekbones, nasolabial areas, and mandibles, enabling more attention to be paid to more implicit facial feature information.

[0047] Among them, the Facemesh algorithm is a facial feature point detection algorithm that can detect 468 facial feature points. It constructs network branches for specific regions of the eyes and mouth, and uses the Spatial Transformer Network (STN) to transform the feature maps, and predicts key points on the transformed feature maps. The accuracy of this algorithm can be comparable to the cascade method, and the speed is increased by 30%. The feature points detected by this algorithm can cover all facial regions, helping the system to pay more attention to implicit facial feature information.

[0048] 4) Based on data-driven feature selection, pair up and combine the 468 facial feature points recognized by Facemseh, calculate the Euclidean distance between two feature points as facial features, and a total of 109,278 facial feature information is formed. Based on L1 regularization, feature dimensionality reduction is performed to obtain 276 facial features.

[0049] 5) According to the facial manifestations of acromegalic patients described in the expert knowledge base: thick lips, wide nose and tongue, high eyebrow arches and cheekbones, and enlarged and protruding mandibles, the neuroendocrine experts manually calibrate the feature points (the center points of the upper and lower lips, the outermost points of both sides of the nose wings, the points of the nasal root and the tip of the nose, the highest point on the eyebrow arch and the lowest point in the eye socket under the eyebrow arch, the outermost points of the cheekbones and the facial midline, the mental vertex and the mandibular angle points, etc.) to obtain the corresponding physiological and apparent features, and splice the vectors of this feature with the facial features obtained by L1 regularization dimensionality reduction to form 386 finally fused facial features.

[0050] The feature ablation results selected by the present invention are shown in Table 1:

[0051] Table 1 Feature Ablation Results

[0052]

[0053] 6) Use the elbow method to find the optimal number of clusters in the acromegaly patient samples, and determine the optimal number of clusters K = 4 by calculating and observing the change trend of the sum of squared errors of clustering under different numbers of clusters.

[0054] 7) Use the Kmeans clustering algorithm to classify the acromegaly patients. According to the determined optimal number of clusters, the patients are clustered into 4 categories; calculate the average coordinates of the 468 facial feature points of each category of acromegaly patients in turn, and draw the facial portraits of different categories of patients according to the average coordinates, as Figure 2 shown.

[0055] 8) We used heatmaps to visually display the changes ( Figure 3 ). The main differences among the four groups of patients were concentrated in the width and height of the face and the symmetry of facial features. Although relatively small changes were shown in the upper part of the face, more obvious differences were observed in the lower part of the face. The upper eyelids of three groups of patients were dense, and there were obvious asymmetries in the lower two-thirds, especially around the mouth and chin (small face, asymmetry). We observed that the patients in Group 1 showed a well-balanced and symmetric facial structure, with a smaller overall facial dimension and less obvious facial angles. Their lower face, including the chin and jawline, looked more compressed (small, symmetric). The patients in Group 2 showed a slight vertical elongation, resulting in a higher facial structure and a narrower lower wrinkle (longer face, prominent lower wrinkle). In contrast, the patients in Group 4 had the largest overall facial size, especially in the lower third of the face around the cheeks and chin (large and wider lower face).

[0056] 9) To further facilitate the formulation of personalized treatment strategies, we analyzed the relevant clinical variables in the four patient groups (Table 2).

[0057] Table 2 Analysis of clinical variables of four groups of patients

[0058]

[0059] Specifically, we used one-way analysis of variance (ANOVA) and LSD post hoc analysis to compare the differences between each group, and chi-square test or Fisher's exact test was used for analysis. The proportion of female patients in group 4 was higher, and there were no significant differences in age, BMI, disease duration, etc. among the patients. The random growth hormone and the lowest level of growth hormone in group 3 were the lowest, while the IGF-1 index level was comparable to that of other groups. This indicates that the sensitivity of the liver to growth hormone is increased, and even a small change in growth hormone will lead to a similar production of IGF-1. The bone formation marker BGP was significantly higher in cluster 1 than in other groups, while there was no significant difference in the bone turnover index Cros between the two groups. Echocardiography showed that the left atrial diameter and interventricular septum thickness of the patients in group 1 were smaller, but there was no significant difference in cardiac systolic function between the two groups. However, the diastolic function of group 1 was better. Regarding pituitary function, there was no significant difference among the groups.

[0060] The present invention classifies the data of acromegaly patients. According to the elbow method, the optimal number of clusters is 4, and the acromegaly patients are clustered into 4 categories, and the facial portraits of each category of patients are drawn, as Figure 2 shown. By comparing the facial features of the 4 categories of patients obtained by the classification of the present invention, it is found that: the area of the two sides of the face of the first category of patients is significantly different; the nose and mouth of the second category of patients are larger than those of other categories of patients; the facial portrait of the third category of patients seems to be that of a normal person; the face on both sides and the forehead area of the fourth category of patients are larger.

[0061] Different from the traditional facial feature point recognition method, the 468 feature point recognition algorithm of Facemesh adopted by the present invention can detect facial feature point areas such as cheekbones and mandibles that cannot be detected by other methods, and can obtain more implicit facial feature information; at the same time, it conducts data-driven category classification on acromegaly patients, and drawing facial portraits of each category of patients can better compare the facial differences between different categories of patients through facial feature points; performing statistical analysis on different categories of patients can more intuitively show the differences between different categories of patients. The results show that the system can effectively classify acromegaly patients and can better explain the facial and physiological differences of the patients.

Claims

1. A data-driven method for classifying and drawing facial portraits of patients with acromegaly, characterized in that: The specific steps are as follows: 1) Dataset collection and construction Collect frontal facial images of several patients with acromegaly to construct a dataset; 2) Data cleaning Preprocess facial images, including facial alignment, facial cropping, and facial geometry normalization; 3) Facial feature point recognition For the pre-processed facial images, the facial image recognition algorithm is used to accurately locate the facial feature points, including the facial contour, forehead, eyes, cheekbones, nose and lips, and jaw area; 4) Facial feature selection and fusion The identified facial feature points are combined in pairs to calculate the corresponding Euclidean distance to construct facial feature information, and then the L1 regularization method is used to select the facial feature information to obtain the reduced-dimensional facial features; The relevant physiological and epigenetic features of acromegaly patients obtained from the expert knowledge base are annotated on the face, including thick lips, wide nose and tongue, high brow arches and zygomatic bones, and enlarged and protruding mandible; The annotated physiological and appearance features are combined with the reduced-dimensional facial features to perform feature fusion to obtain the final facial features; 5) Determine the optimal number of clusters Based on the final facial features, the elbow method was used to find the optimal number of clusters in the acromegaly patient samples; 6) Data-driven patient classification and facial profiling According to the determined optimal number of clusters, a clustering algorithm was used to classify the samples of acromegaly patients and the patients were clustered into K categories; The average coordinates of the facial feature points of each type of acromegaly patients were calculated in turn, and facial portraits of different types of patients were drawn based on the average coordinates.

2. The data-driven method for classifying acromegaly patients and drawing facial portraits according to claim 1, characterized in that: In step 1), the collected facial images include data of the patient at different time periods; at the same time, the clinical variables of gender, age and BMI that have a greater impact on facial features are recorded.

3. The data-driven method for classifying acromegaly patients and drawing facial portraits according to claim 1, characterized in that: In step 2), the data cleaning method uses a multi-task convolutional neural network MTCNN algorithm to detect, align and crop faces in facial images, and normalize facial geometry to obtain a pre-processed facial image with a size of 350×350 pixels.

4. The data-driven method for classifying acromegaly patients and drawing facial portraits according to claim 1, characterized in that: In step 3), the facial feature point recognition method adopts the Facemesh algorithm or the Dlib algorithm.

5. The data-driven method for classifying acromegaly patients and drawing facial portraits according to claim 1, characterized in that: In step 5), the optimal number of clusters K is determined by calculating and observing the changing trend of the clustering square sum under different numbers of clusters.

6. The data-driven method for classifying acromegaly patients and drawing facial portraits according to claim 1, characterized in that: In step 6), the clustering algorithm is the Kmeans algorithm.

7. The data-driven method for classifying acromegaly patients and drawing facial portraits according to claim 1, characterized in that: Also includes step 7): Statistical analysis of clinical characteristics of different categories of patients: The facial features of patients with different categories of acromegaly were compared based on the drawn facial portraits, and the clinical variables of patients in different categories were statistically analyzed.

Images