A probability prediction system and prediction method based on facial structure recognition

Abstract

The present invention relates to a probability prediction system and prediction method based on facial structure recognition, belonging to the field of intelligent recognition technology. First, data collection, data organization and data processing are used as preliminary work; then the overall deviation of the data is calculated, available samples are screened, and feature engineering is organized and constructed; finally, the original data is encapsulated based on a common deep learning library to divide the training and test sets, and a model is constructed to train, verify and test the deep learning network model with the assistance of regularization, dropout and related loss functions. In order to realize the calculation of key features, artificial intelligence technology including machine learning technology and deep learning technology is used to calculate the features, and the relevant feature information is extracted by identifying the position and structure of the key points of the face, and a fully connected neural network model is established for recognition. Without the assistance of other detection equipment, daily monitoring of AH can be carried out and the initial recognition probability can be improved by analyzing facial image videos through daily mobile phones.

Description

Technical Field

[0001] The invention relates to a probability prediction system and a prediction method based on facial structure recognition, belonging to the technical field of intelligent recognition. Background Art

[0002] Adenoids are lymphoid tissue located on the posterior wall of the nasopharynx and are immune organs in the human body. AH (adenoidal hypertrophy), also known as adenoid hyperplasia, is more common in children. Because adenoids and palatine tonsils are located in the first line of defense of the respiratory tract, they are exposed to various inflammatory stimuli for a long time, resulting in their pathological hyperplasia. AH is mainly manifested by mouth breathing, snoring, runny nose, nasal congestion, and dyspnea during sleep. Long-term mouth breathing will lead to relaxation of the lip muscles and eversion of the lower lip. In severe cases, "adenoid facies" such as narrow upper collar, tilted upper front teeth, and narrowing of the base of the nose will appear; the symptoms lack specificity and are often misdiagnosed by parents as colds, rhinitis, pharyngitis and other diseases. AH can cause attention deficit, hyperactivity disorder, growth retardation, cognitive and behavioral abnormalities in children, etc., which is very harmful to children's physical and mental health.

[0003] Due to the lack of awareness among parents, sleep monitoring in hospitals is complicated and painful, surgery requires general anesthesia, and is expensive (about 10,000 yuan), most children with AH do not receive timely diagnosis and treatment, or the best time for surgery is delayed. Adenoids are lymphoid tissues, which have a certain volume without inflammatory stimulation. However, when they are examined, there is often inflammatory stimulation. At this time, it is normal to find that AH occupies one-third to four-fifths of the space, and swelling is inevitable. There is no consensus in the medical community whether mild to moderate AH should be operated on, but experts agree that if AH has begun to affect sleep, surgical treatment is recommended. Other non-monitoring methods such as questionnaires have a sensitivity and accuracy of only 78% and 72%. Therefore, before making a surgical decision, it is generally based on objective monitoring. However, there is a shortage of monitoring equipment in hospitals, and it is extremely uncomfortable to wear and noisy. In comparison, home monitoring has greater advantages, but current home sleep monitoring products such as smart mattresses generally only consider adults, lack of targeted monitoring of children, and are not suitable for children. Due to various reasons, many patients do not receive timely monitoring and therefore do not receive surgical treatment, resulting in a very low treatment rate for AH in children.

[0004] Traditionally, AH requires professional doctors to refer to the measurement results of professional equipment (such as electronic nasopharyngeal endoscopes, DR, CT, MR, and X-rays) combined with clinical manifestations to determine it, which means that children must go to large hospitals for examination, which results in a lot of time and money costs for the children's families. CT and X-ray examinations cause radiation to the human body, and because the adenoids are located at the top of the nasopharyngeal cavity, nasopharyngeal endoscopy is difficult to accurately understand their hypertrophy. Children are not easy to cooperate with the examination. At the same time, considering the widespread incidence of AH and its frequent misdiagnosis as colds and other diseases. In order to see a doctor as soon as possible and reduce the cost of the children's families, the initial screening of AH at home is particularly important. However, there is a lack of intelligent AH initial screening solutions for home use on the market today.

[0005] The two invention patents of Chinese patent documents CN111904393B-Intelligent children's sleep monitoring system with adenoids hypertrophy probability judgment function and CN114246578A-Adenoids hypertrophy initial screening device, system and terminal equipment use wearable devices or non-wearable monitoring devices to conduct preliminary screening of AH, which is relatively simple and fast, but requires wearing or being equipped with terminal monitoring equipment. Summary of the invention

[0006] In view of the shortcomings of the prior art, in order to solve the above problems and avoid the side effects and inconvenience of the AH identification method, the present invention establishes a model and makes predictions based on AH features, and does not require the assistance of other detection equipment and requires fewer facial information features. It can monitor AH on a daily basis and improve the probability of preliminary recognition by only analyzing facial image videos with a daily mobile phone.

[0007] First, data collection, data organization, and data processing are used as preliminary work; then the overall deviation of the data is calculated, available samples are screened, and feature engineering is organized; finally, the original data is encapsulated based on the common deep learning library to divide the training and test sets, and the model is constructed with regularization, dropout, and related loss functions to train, verify, and test the deep learning network model. After exporting the face video file from the database, combined with the given features that may be related to adenoids hypertrophy, such as the ratio of the upper edge of the upper lip to the upper edge of the lower lip and the lower edge of the upper lip to the lower edge of the lower lip in the calm and natural state of the front face, the ratio of the left eye width to the left eye height, the ratio of the right eye width to the right eye height, the ratio of the horizontal distance of the face at the nose position to the horizontal distance of the face at the mouth position in the side face state, etc. In order to realize the calculation of key features, artificial intelligence technology including machine learning technology and deep learning technology is used to calculate the features. The relevant feature information is extracted by identifying the position and structure of the key points of the face, and logistic regression, support vector machine and neural network models are established. Since logistic regression cannot handle a large number of features or variables well and is prone to overfitting, support vector machines cannot handle large-scale samples well and are sensitive to the selection of parameters and kernel functions. At the same time, combined with the test results of some sample data, a fully connected neural network model is selected for AH recognition.

[0008] Terminology explanation:

[0009] Deep Learning: It is a machine learning method consisting of a multi-layer neural network. It learns the inherent laws and representation levels of sample data. Sample data often includes images, texts, and sounds. Its ultimate goal is to enable machines to analyze and solve problems like humans.

[0010] The technical solution of the present invention is as follows:

[0011] A probability prediction system based on facial structure recognition, comprising a data module, a feature module, and an analysis module;

[0012] The data module is used to obtain the video data of the object under test. The data is shot under a single gray background wall. The pixel of the device used for shooting is not less than 200*200 pixels. The face state of the video data collection needs to include three states: front face, left face, and right face;

[0013] The feature module is used to detect the FacePoint model based on deep learning and including 468 facial key points of 3D coordinate points, calculate the corresponding ratios of related indicators, and convert them into a general array format that can be processed by the program; the FacePoint model can be obtained through the Google official website; the present invention shares multiple groups of feature values ​​including frontal and side face states, and then performs correlation analysis, consistency check and PCA and other statistical methods to reduce the dimension, and obtain a relatively small number of facial ratio features;

[0014] The analysis module is used to input the feature calculation value under the facial state into a pre-established primary screening model for prediction, and output a probability value in the range of 0-1.

[0015] Preferably, the data module obtains data of the face of the subject under test in three states: front face, left face, and right face through a terminal device, and divides the data into a training set and a test set. The data in the training set has been marked with label information of the subject under test, and the label information includes adenoids hypertrophy, tonsil enlargement, mouth breathing, and snoring during sleep.

[0016] Further preferably, the terminal device includes a terminal acquisition device and a terminal display device. The terminal acquisition device is used to acquire and save data in dynamic video situations that meet the requirements, and transmit the data to a cloud server so that users can view the data online. The cloud server is used to store the obtained user data and the constructed database information on the cloud server, perform calculation analysis on the server, and finally return the obtained probability result to the user.

[0017] The terminal display device is used to obtain and display the prediction results of the user, and is integrated into mobile electronic devices such as mobile phones and computers. The main effect of the electronic device is to allow users to upload their facial information data and files more conveniently and obtain the probability prediction results of each recognition at the same time.

[0018] Further preferably, the terminal acquisition equipment includes a tripod, a shooting device, and a rotatable seat. The tripod is used to fix the position and help the measurement personnel to place the movable electronic device more effectively; the shooting device may include a mobile phone or a camera; the rotatable seat is used to turn the face during the shooting process. When the user turns his face to reveal the side face, he cannot have too large a movement range and too fast a speed, so that the side face of the person can be captured more clearly.

[0019] Further preferably, in addition to predicting the probability of suffering from adenoids hypertrophy, the recognition system can also predict the probability of having enlarged tonsils, snoring during sleep and mouth breathing. The sample data needs to be reasonably divided according to the labels. Taking adenoids hypertrophy as an example, the labels that are ambiguous and have empty values ​​are filtered out, and the rest are valid data. The number of data contained in each binary classification prediction problem may also be different due to different empty value records. The four parts of binary classification data training data and test data are divided into 70%:30%, and 20% is reserved from the training set as a validation set.

[0020] Preferably, the model is established using the function interface in the common deep learning library. The fully connected neural network has a better fitting effect on the eigenvalue. When the amount of data is not large, a shallower neural network is generally used. The initial screening model uses a three-layer neural network to construct a three-layer fully connected neural network with 20, 10, and 1 intermediate neurons, except for the last output layer. The weight initialization uses glorot_normal, the activation function uses relu, l2 weight regularization is added, and BatchNormalization batch normalization layer and Dropout layer are added.

[0021] A prediction method using the above-mentioned probability prediction system based on facial structure recognition comprises the following steps:

[0022] Step S10, with the help of mobile electronic devices such as mobile phones and personal computers, the facial video data of the subject in a calm and natural state is conveniently obtained and uploaded to the cloud server. During the video shooting process, the user is required to first maintain a frontal face posture, then turn the head to the left, then turn back to the front, continue to turn the head to the right, and finally turn back to the front. The whole process can be kept at a uniform speed. The video shot in this way should include the front face and the left and right sides of the face.

[0023] The data acquisition module obtains the user's facial video data and label information for adenoids hypertrophy, tonsil enlargement, mouth breathing, and snoring during sleep from the mobile phone and webpage respectively, and stores them in Huawei OBS and the server MySQL database respectively; the video format is MP4, and the image file is stored in jpg format, with a minimum pixel of 400*400 and a maximum pixel of 4096*4096. The short side of the face detected in the video is not less than 200px, and occlusion, blur, improper lighting, and excessive face angles are avoided as much as possible;

[0024] Step S20, generating a series of key detection points related to the human face based on the collected video data, and combining the clinical experience of adenoids hypertrophy related diseases, screening out the key points for calculation and calculating the corresponding length ratios as original features to be input into the initial screening model;

[0025] Preferably, the facial key points are 468 facial 3D coordinate points detected by combining the face and face reference point detection models; the basic architecture of the face reference point detection model is based on MobileNetv1 / v2 and SSD (Single ShotMultiBoxDetector); the 3D face reference point detection model uses the transfer learning model trained by Google to output the three-dimensional face coordinate points, and iterates the coordinate prediction results through bootstrap and refine; the output of the 468 facial key points is as follows Figure 3 shown.

[0026] Further preferably, the feature calculation includes the ratios of key points of the front face part and the side face part, the front face part includes: the first group of ratios: left eye width, left eye height, the second group of ratios: right eye width, right eye height, the third group of ratios: upper edge of upper lip to upper edge of lower lip, lower edge of upper lip to lower edge of lower lip, the fourth group of ratios: forehead face, full facial area, the fifth group of ratios: upper and lower lip distance, columella to lower edge of mandible, the sixth group of ratios: zygomatic plane horizontal distance, outer eye distance; the side face part includes: the seventh group of ratios: horizontal distance of the face at the nose position, horizontal distance of the face at the mouth position, the eighth group of ratios: horizontal distance of the face at the eye position, horizontal distance of the face at the nose position, the ninth group of ratios: nose bridge height, nose tip height.

[0027] Based on the nine sets of feature ratios related to prediction obtained by key point detection, the feature module exports the user's (adenoids hypertrophy, tonsil enlargement, mouth breathing, sleep snoring) label information from the server MySQL database, and after matching with the user ID, adds the label information to the information of a single record and stores it in a json format file, with different label information data stored separately;

[0028] Re-read the json format file to form a numpy.ndarray format training data set. The labels need to be standardized and the records of uncertain specific conditions are excluded. The label is yes, that is, the label state such as adenoids hypertrophy exists, otherwise it does not exist; thus, the label information is converted from an irregular string to digital information data that can be processed by a computer to form user record data. In this way, the four types of diagnostic data are divided.

[0029] Preferably, in order to ensure that the ratio of positive and negative samples in the training, validation and test sets is the same as much as possible, the total number of positive and negative samples for each label state is first calculated, where the positive sample is the data without the label state and the negative sample is the data with the label state. Then, the positive and negative samples are stored separately in the numpy array format at a ratio of 50%:50%. At this point, the data sorting part is completed.

[0030] Preferably, when the data dimension is much larger than the order of magnitude of the user data, the pandas library conversion format is used to calculate the descriptive value of the feature, that is, the standard deviation, such as the standard deviation and quartile; the user records that exceed 3 times the calculated standard deviation are deleted according to the calculated descriptive value. The standard deviation calculation formula is:

[0031]

[0032] Among them, n is the number of data, x i is the current data dimension, μ is the average value of the data dimension;

[0033] Use principal component analysis to reduce the dimensionality of the original data, eliminate the correlation redundancy between the data, and reduce unnecessary computing energy consumption, which is conducive to subsequent model training and prediction. Import the sklearn scientific analysis library to reduce the dimensionality of the original data.

[0034] Step S30, performing forward iterative calculation based on the pre-trained primary screening model, supplemented by relevant activation functions, and finally obtaining the predicted probabilities of adenoids hypertrophy, tonsil enlargement, snoring during sleep, and mouth breathing;

[0035] The model is constructed. The machine learning models for the classification include support vector machine (SVM) and decision tree algorithm, mainly through the sklearn scientific computing library, and a fully connected neural network based on tensorflow is established for prediction. In contrast, the prediction accuracy of the neural network model is better. Because according to the universality criterion of neural networks, to a certain extent, neural networks with nonlinear activation functions can fit any function.

[0036] A three-layer fully connected neural network with 20, 10, and 1 intermediate neurons was constructed. Except for the last output layer, glorot_normal was used for weight initialization, relu was used for activation function, l2 weight regularization was added, and BatchNormalization layer and Dropout layer were added to make the trained model better for prediction. The loss function used was the most commonly used binary cross entropy loss function in binary classification.

[0037]

[0038] The from_logits parameter in the function is set to True to calculate the loss function more effectively. The optimization algorithm uses the Adam gradient descent strategy to update the weight values ​​of neurons in each neural layer.

[0039]

[0040] m t =β1m t-1+(1-β1)g t

[0041]

[0042]

[0043]

[0044]

[0045] Among them, g t is the derivative value of the error function with respect to each weight at time t; w t is the weight value of each neuron at time t; w t+1 is the weight value of each neuron after t+1 update; β1 is the attenuation factor of the first momentum, with a typical value of 0.9; β2 is the attenuation factor of the second momentum, with a typical value of 0.999; ∈ is the smoothing term, which is generally 10 -8 ; η refers to the learning rate; m t is the biased first-order momentum estimate at time t; is the corrected first-order momentum estimate; v t is the biased second-order momentum estimate at time t; is the corrected second-order momentum estimate;

[0046] Set the number of EPOCH training rounds to start the model training iteration. When the prediction accuracy exceeds 75% or the set number of training rounds is reached, the training is stopped (early_stopping). The tensor graph and model weights are saved in the tensorflow ckpt format.

[0047] The video under the new complete process is input, and after being processed by the initial screening model, the final output is the probability of the user in the video suffering from adenoids hypertrophy, swollen tonsils, mouth breathing and snoring during sleep.

[0048] The beneficial effects of the present invention are:

[0049] The present invention proposes a probability prediction system that can be combined with multiple portable electronic devices to obtain video / image data and clear information (including whether the adenoids are enlarged, whether the tonsils are enlarged, whether there is snoring during sleep, and whether the mouth is breathing) taken from the handheld electronic device. The video data information is executed by a deep learning network to execute facial contour key points to further calculate the ratio of certain key lengths of the face; the calculated value is input into a pre-established adenoids prediction model through an analysis module for calculation and analysis to obtain the probability of abnormal facial appearance of the subject. As a three-in-one system, it opens up the full process automation from data to analysis to calculation, and does not require additional related devices. It can be conveniently applied to various scenarios, while effectively improving the accuracy of family initial screening and facilitating the implementation of the Healthy China plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a flow chart of an exemplary embodiment of a probability prediction system of the present invention;

[0051] Figure 2a A schematic diagram of the fifth set of ratios of some required features of the front face of the present invention;

[0052] Figure 2b A sixth group of ratios of some required features of the front face of the present invention is a schematic diagram;

[0053] Figure 3 A drawing of all 468 key points of the face in the present invention;

[0054] Figure 4 It is a practical schematic diagram of adenoids hypertrophy in children according to the present invention;

[0055] Figure 5 Schematic diagram of the functional principle of the prediction system in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0057] Embodiment 1:

[0058] A probability prediction system based on facial structure recognition, comprising a data module, a feature module, and an analysis module;

[0059] The data module is used to obtain the video data of the object under test. The data is shot under a single gray background wall. The pixel of the device used for shooting is not less than 200*200 pixels. The face state of the video data collection needs to include three states: front face, left face, and right face;

[0060] The data module obtains data of the face of the subject under test in three states: the front face, the left side face, and the right side face through the terminal device, and divides the data into a training set and a test set. The data in the training set has been marked with label information of the subject under test, and the label information includes adenoids hypertrophy, tonsil enlargement, mouth breathing, and snoring during sleep.

[0061] In addition to predicting the probability of suffering from adenoids hypertrophy, this recognition system can also predict the probability of swollen tonsils, snoring during sleep, and mouth breathing. The sample data needs to be reasonably divided according to the labels. Taking adenoids hypertrophy as an example, the ambiguous and empty labels are filtered out, and the rest are valid data. The number of data contained in each binary classification prediction problem may also be different due to different empty value records. The four parts of binary classification data training data and test data are divided into 70%:30%, and 20% is reserved from the training set as the validation set.

[0062] The feature module is used to detect the FacePoint model based on deep learning and including 468 facial key points of 3D coordinate points, calculate the corresponding ratios of related indicators, and convert them into a general array format that can be processed by the program; the FacePoint model can be obtained through the Google official website; the present invention shares multiple groups of feature values ​​including frontal and side face states, and then performs correlation analysis, consistency check and PCA and other statistical methods to reduce the dimension, and obtain a relatively small number of facial ratio features;

[0063] The analysis module is used to input the feature calculation value under the facial state into a pre-established primary screening model for prediction, and output a probability value in the range of 0-1.

[0064] The model is established using the function interface in the common deep learning library. The fully connected neural network has a better fitting effect on the eigenvalues. When the amount of data is not large, a shallower neural network is generally used. The initial screening model uses a three-layer neural network to construct a three-layer fully connected neural network with 20, 10, and 1 intermediate neurons. Except for the last output layer, the weight initialization uses glorot_normal, the activation function uses relu, l2 weight regularization is added, and BatchNormalization batch normalization layer and Dropout layer are added.

[0065] Embodiment 2:

[0066] A probability prediction system based on facial structure recognition, the structure of which is as described in Example 1, except that the terminal device includes a terminal acquisition device and a terminal display device, the terminal acquisition device is used to acquire and save data in a dynamic video situation that meets the requirements, and transmit the data to a cloud server so that users can view the data online; the cloud server is used to store the obtained user data and the constructed database information on the cloud server, perform calculation analysis on the server, and finally return the obtained probability result to the user;

[0067] The terminal display device is used to obtain and display the prediction results of the user, and is integrated into mobile electronic devices such as mobile phones and computers. The main effect of the electronic device is to allow users to upload their facial information data and files more conveniently and obtain the probability prediction results of each recognition at the same time.

[0068] The terminal acquisition equipment includes a tripod, a shooting device, and a rotatable seat. The tripod is used to fix the position and help the measurement personnel to place the movable electronic equipment more effectively; the shooting device may include a mobile phone or a camera; the rotatable seat is used to turn the face during the shooting process. When the user turns his face to reveal the side face, he cannot have too large a movement range and too fast a speed, so that the side face of the person can be captured more clearly.

[0069] Embodiment 3:

[0070] A prediction method using the probability prediction system based on facial structure recognition described in Example 2 comprises the following steps:

[0071] Step S10, with the help of mobile electronic devices such as mobile phones and personal computers, conveniently obtain facial video data of the subject in a calm and natural state and upload it to the cloud server. During the video shooting process, the user needs to first maintain a frontal face posture, then turn his head to the left, then turn back to the front, continue to turn his head to the right, and finally turn back to the front. The whole process can be kept stable at a constant speed. The video shot in this way should include the front face and the left and right sides of the face. The data of the left and right sides of the face are used to calculate the ratio of the distances of several facial feature points. These features are used to determine whether a child suffers from adenoids hypertrophy. Some examples of facial feature distances are shown in Figure 2a , Figure 2b Adenoid hypertrophy is common in children aged 2-10 years old and disappears on its own. However, during the growth of children, they are often repeatedly stimulated by certain inflammations, leading to malignant hyperplasia, which causes a chain reaction to surrounding tissues and cells, such as Figure 4 shown.

[0072] The data acquisition module obtains the user's facial video data and label information for adenoids hypertrophy, tonsil enlargement, mouth breathing, and snoring during sleep from the mobile phone and webpage respectively, and stores them in Huawei OBS and the server MySQL database respectively; the video format is MP4, and the image file is stored in jpg format, with a minimum pixel of 400*400 and a maximum pixel of 4096*4096. The short side of the face detected in the video is not less than 200px, and occlusion, blur, improper lighting, and excessive face angles are avoided as much as possible;

[0073] Step S20, generating a series of key detection points related to the human face based on the collected video data, and combining the clinical experience of adenoids hypertrophy related diseases, screening out the key points for calculation and calculating the corresponding length ratios as original features to be input into the initial screening model;

[0074] The facial key points are a combination of face and face reference point detection models to detect a total of 468 facial 3D coordinate points; the basic architecture of the face reference point detection model is based on MobileNetv1 / v2 and SSD (Single ShotMultiBoxDetector); the 3D face reference point detection model uses the transfer learning model trained by Google to output the three-dimensional face coordinate points, and iterates the coordinate prediction results through bootstrap and refine; the output of the 468 facial key points is as follows Figure 3 shown.

[0075] The feature calculation includes the ratios of key points of the front face and the side face. The front face includes: the first group of ratios: left eye width, left eye height, the second group of ratios: right eye width, right eye height, the third group of ratios: upper edge of upper lip to upper edge of lower lip, lower edge of upper lip to lower edge of lower lip, the fourth group of ratios: forehead face, full facial area, the fifth group of ratios: upper and lower lip distance, columella to lower edge of mandible, the sixth group of ratios: zygomatic plane horizontal distance, outer eye distance; the side face includes: the seventh group of ratios: facial horizontal distance at nose position, facial horizontal distance at mouth position, the eighth group of ratios: facial horizontal distance at eye position, facial horizontal distance at nose position, the ninth group of ratios: nose bridge height, nose tip height.

[0076] The key point index table used to calculate features is as follows:

[0077] Table 1 Key point index table

[0078]

[0079]

[0080] According to the above table, nine groups of characteristic ratios related to the identification of adenoids hypertrophy are obtained.

[0081] Based on the nine sets of feature ratios related to prediction obtained by key point detection, the feature module exports the user's (adenoids hypertrophy, tonsil enlargement, mouth breathing, sleep snoring) label information from the server MySQL database, and after matching with the user ID, adds the label information to the information of a single record and stores it in a json format file, with different label information data stored separately;

[0082] Re-read the json format file to form a numpy.ndarray format training data set. The labels need to be standardized and the records of uncertain specific conditions are excluded. The label is yes, that is, the label state such as adenoids hypertrophy exists, otherwise it does not exist; thus, the label information is converted from an irregular string to digital information data that can be processed by a computer to form user record data. In this way, the four types of diagnostic data are divided.

[0083] In order to ensure that the ratio of positive and negative samples in the training, validation and test sets is the same as far as possible, we first calculate the total number of positive and negative samples for each label state. The positive sample is the data without the label state, and the negative sample is the data with the label state. Then, the positive and negative samples are stored separately in numpy array format according to the ratio of 50%:50%. At this point, the data sorting part is completed.

[0084] When the data dimension is much larger than the order of magnitude of the user data, the pandas library is used to convert the format and calculate the descriptive value of the feature, that is, the standard deviation, such as the standard deviation and quartile. The user records that exceed 3 times the calculated standard deviation are deleted based on the calculated descriptive value. The standard deviation calculation formula is:

[0085]

[0086] Among them, n is the number of data, x i is the current data dimension, μ is the average value of the data dimension;

[0087] Use principal component analysis to reduce the dimensionality of the original data, eliminate the correlation redundancy between the data, and reduce unnecessary computing energy consumption, which is conducive to subsequent model training and prediction. Import the sklearn scientific analysis library to reduce the dimensionality of the original data.

[0088] Step S30, performing forward iterative calculation based on the pre-trained primary screening model, supplemented by relevant probability activation functions, and finally obtaining the predicted probabilities of adenoids hypertrophy, tonsil enlargement, sleep snoring and mouth breathing;

[0089] The model is constructed. The machine learning models for the classification include support vector machine (SVM) and decision tree algorithm, mainly through the sklearn scientific computing library, and a fully connected neural network based on tensorflow is established for prediction. In contrast, the prediction accuracy of the neural network model is better. Because according to the universality criterion of neural networks, to a certain extent, neural networks with nonlinear activation functions can fit any function.

[0090] A three-layer fully connected neural network with 20, 10, and 1 intermediate neurons was constructed. Except for the last output layer, glorot_normal was used for weight initialization, relu was used for activation function, l2 weight regularization was added, and BatchNormalization layer and Dropout layer were added to make the trained model better for prediction. The loss function used was the most commonly used binary cross entropy loss function in binary classification.

[0091]

[0092] The from_logits parameter in the function is set to True to calculate the loss function more effectively. The optimization algorithm uses the Adam gradient descent strategy to update the weight values ​​of neurons in each neural layer.

[0093]

[0094] m t =β1m t-1 +(1-β1)g t

[0095]

[0096]

[0097]

[0098]

[0099] Among them, g t is the derivative value of the error function with respect to each weight at time t; w t is the weight value of each neuron at time t; w t+1 is the weight value of each neuron after t+1 update; β1 is the attenuation factor of the first momentum, with a typical value of 0.9; β2 is the attenuation factor of the second momentum, with a typical value of 0.999; ∈ is the smoothing term, which is generally 10- 8 ; η refers to the learning rate; m t is the biased first-order momentum estimate at time t; is the corrected first-order momentum estimate; vt is the biased second-order momentum estimate at time t; is the corrected second-order momentum estimate.

[0100] Set the number of EPOCH training rounds to start the model training iteration. When the prediction accuracy exceeds 75% or the set number of training rounds is reached, the training is stopped (early_stopping). The tensor graph and model weights are saved in the tensorflow ckpt format.

[0101] The video under the new complete process is input, and after being processed by the initial screening model, the final output is the probability of the user in the video suffering from adenoids hypertrophy, swollen tonsils, mouth breathing and snoring during sleep.

Claims

1. A prediction method for a probability prediction system based on facial structure recognition, characterized in that: The probability prediction system based on facial structure recognition includes a data module, a feature module, and an analysis module; The data module is used to obtain the video data of the object under test. The data is shot under a single gray background wall. The pixel of the device used for shooting is not less than 200*200 pixels. The face state of the video data collection needs to include three states: front face, left face, and right face; The feature module is used to use the existing FacePoint model based on deep learning to detect 468 facial key points including 3D coordinate points, calculate the corresponding ratios of relevant indicators, and convert them into a general array format that can be processed by the program; The analysis module is used to input the feature calculation value under the facial state into the pre-established primary screening model for prediction, and output a probability value in the range of 0-1; the prediction method includes the following steps: Step S10, using an electronic device to obtain facial video data of the subject in a calm and natural state and upload it to a cloud server. During the video shooting process, the user is required to first maintain a frontal face posture, then turn his head to the left, then turn back to the front, continue to turn his head to the right, and finally turn back to the front. The whole process is at a constant speed; The data acquisition module obtains the user's facial video data and label information for adenoids hypertrophy, tonsil enlargement, mouth breathing, and snoring during sleep from the mobile phone and webpage respectively, and stores them in Huawei OBS and the server MySQL database respectively; the video format is MP4, the minimum pixel is 400*400, the maximum pixel is 4096*4096, and the short side length of the face detected in the video is not less than 200px; Step S20, generating a series of key detection points related to the face based on the collected video data, screening out the key points available for calculation and calculating the corresponding length ratios as original features to be input into the primary screening model; The facial key points are a total of 468 facial 3D coordinate points detected by combining the face and face reference point detection models; the basic architecture of the face reference point detection model is based on MobileNetv1 / v2 and SSD; the 3D face reference point detection model uses the transfer learning model trained by Google to output the three-dimensional face coordinate points, and iterates the coordinate prediction results through bootstrap and refine. The key points of the face are independent of each other, and the key points form the facial contour, eye, and nose areas; the index list of the facial contour area is: [10,338,297,332,284,251,389,356,454,323,361,288,397,365,379,378,400,377,152,148,176,149,150,136,172,58,132,93,234,127,162,21,54,103,6 7,109,10], lip contour index list: [61,146,91,181,84,17,314,405,321,375,291,61,185,40,39,37,0,267,269,270,409,291,78,95,88,178,87,14,317,402,318,324,308,78,191,80,81,82,13,312,311,310,415,308]; The feature calculation includes the key point ratios of the front face and the side face. The front face includes: the first group of ratios: left eye width, left eye height, the second group of ratios: right eye width, right eye height, the third group of ratios: upper lip edge to lower lip upper edge, upper lip lower edge to lower lip lower edge, the fourth group of ratios: forehead face, full face area, the fifth group of ratios: upper and lower lip distance, columella to lower jaw edge, the sixth group of ratios: zygomatic plane horizontal distance, eye outer distance; the side face includes: the seventh group of ratios: nose position facial horizontal distance, mouth position facial horizontal distance, the eighth group of ratios: eye position facial horizontal distance, nose position facial horizontal distance, the ninth group of ratios: nose bridge height, nose tip height; The feature module obtains nine sets of feature ratios related to prediction based on key point detection, exports the user's tag information from the server MySQL database, matches it with the user ID, adds the tag information to the information of a single record, and stores it in a json format file, with different tag information data stored separately; Reread the json format file to form a numpy.ndarray format training data set. The labels need to be standardized and the records with uncertain specific conditions are excluded. The label is yes, that is, the adenoids hypertrophy label state exists, otherwise it does not exist. In this way, the label information is converted from an irregular string to digital information data that can be processed by a computer to form user record data. Step S30, performing forward iterative calculation based on the pre-trained primary screening model, supplemented by relevant activation functions, and finally obtaining the predicted probabilities of adenoids hypertrophy, tonsil enlargement, snoring during sleep, and mouth breathing; The initial screening model uses a three-layer neural network to construct a three-layer fully connected neural network with 20, 10, and 1 intermediate neurons respectively. Except for the last output layer, the weight initialization uses glorot_normal, the activation function uses relu, l2 weight regularization is added, and BatchNormalization batch normalization layer and Dropout layer are added; the loss function uses the most commonly used binary cross entropy loss function in binary classification. The from_logits parameter in the function is set to True; the optimization algorithm uses the Adam gradient descent strategy to update the weight values ​​of neurons in each neural layer; m t =β1m t-1 +(1-β1)g t Among them, g t is the derivative value of the error function with respect to each weight at time t; w t is the weight value of each neuron at time t; w t+1 is the weight value of each neuron after t+1 update; β1 is the decay factor of the first momentum; β2 is the decay factor of the second momentum; ∈ is the smoothing term; η refers to the learning rate; m t is the biased first-order momentum estimate at time t; is the corrected first-order momentum estimate; v t is the biased second-order momentum estimate at time t; is the corrected second-order momentum estimate; Set the number of EPOCH training rounds to start the model training iteration. When the prediction accuracy exceeds 75% or the set number of training rounds is reached, the training is stopped and the tensor graph and model weights are saved in the tensorflow ckpt format. The video under the new complete process is input, and after being processed by the initial screening model, the final output is the probability of the user in the video suffering from adenoids hypertrophy, swollen tonsils, mouth breathing and snoring during sleep.

2. The prediction method of the probability prediction system based on facial structure recognition according to claim 1, characterized in that: First, calculate the total number of positive and negative samples for each label state. The positive sample is the data without label state, and the negative sample is the data with label state. Then, store the positive and negative samples separately in numpy array format according to the ratio of 50%:50%. At this point, the data sorting part is completed.

3. The prediction method of the probability prediction system based on facial structure recognition according to claim 1, characterized in that: When the data dimension is much larger than the order of magnitude of user data, the pandas library is used to convert the format and calculate the feature description value, i.e., the standard deviation. The calculated description value is used to delete user records that exceed 3 times the calculated standard deviation. The standard deviation calculation formula is: Among them, n is the number of data, x i is the current data dimension, and μ is the average value of the data dimension.

4. The prediction method of the probability prediction system based on facial structure recognition according to claim 1, characterized in that: The data module obtains data of the face of the subject under test in three states: the front face, the left side face, and the right side face through the terminal device, and divides the data into a training set and a test set. The data in the training set has been marked with label information of the subject under test, and the label information includes adenoids hypertrophy, tonsil enlargement, mouth breathing, and snoring during sleep.

5. The prediction method of the probability prediction system based on facial structure recognition according to claim 4, characterized in that: The terminal device includes a terminal acquisition device and a terminal display device. The terminal acquisition device is used to collect and save data in dynamic video situations that meet the requirements, and transmit the data to a cloud server at the same time; the cloud server is used to store the obtained user data and the constructed database information on the cloud server, and perform calculations and analysis on the server, and finally return the obtained probability results to the user; the terminal display device is used to obtain and display the prediction results of the user, and is integrated in a mobile electronic device.

6. The prediction method of the probability prediction system based on facial structure recognition according to claim 5, characterized in that: The terminal acquisition equipment includes a tripod, a shooting device, and a rotatable seat. The tripod is used to fix the position and help the measurement personnel; the shooting device includes a mobile phone or a camera; and the rotatable seat is used to turn the face during the shooting process.

7. The prediction method of the probability prediction system based on facial structure recognition according to claim 4, characterized in that: The sample data is divided according to labels, and unclear and empty labels are filtered out. The remaining data are valid data. The training data and test data are divided into 70%:30%, and 20% is reserved from the training set as the validation set.

Images