A diagnostic method and system for identifying children's mouth breathing using artificial intelligence

Abstract

The present invention discloses a diagnostic method and system for identifying children's mouth breathing using artificial intelligence, belonging to the field of remote intelligent diagnosis technology, including: collecting a real-time facial video stream of a child during sleep based on a set timestamp and sending it to the server side; using a face detector on the server side to perform facial recognition on the real-time facial video stream to obtain the relative relationship α between the mouth opening amplitude and the overall width of the lips, where the relative relationship α is used to represent the proportional relationship between the mouth opening amplitude and the overall width of the lips; obtaining the continuous change of the processed relative relationship α in the video stream through Kalman filtering processing of the relative relationship α; diagnosing the child's mouth breathing according to the continuous change; in the present invention, by presetting an algorithm in the computer, the maxillofacial conditions during sleep are analyzed quickly by doubling, thereby saving a large amount of diagnostic time and improving the diagnostic efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of remote intelligent diagnosis, and more particularly, to a diagnosis method and system for identifying children's mouth breathing using artificial intelligence. Background Art

[0002] Mouth breathing is highly prevalent in children and is an important cause endangering children's developmental health. With the popular science education in recent years, many parents have increasingly realized the harm of mouth breathing and actively seek professional help. According to domestic and foreign literature reports, the incidence rate fluctuates between 4.3% and 45.9%. The significant differences in the reported incidence rates of mouth breathing in different literatures reflect, on the one hand, the high incidence of mouth breathing in children; on the other hand, a review of the literature reveals that there is still a lack of unified and objective diagnostic criteria and methods for mouth breathing.

[0003] According to Moss's "functional matrix theory", the growth and development of the maxillofacial region are adapted to functions, and breathing is an important function of the stomatognathic system. Long-term mouth breathing can change the normal balance of maxillofacial muscle strength and bite force, thereby having an adverse impact on the development of the jawbone, dental arch, alveolar bone, and soft tissues, and thus forming the characteristic "adenoid face appearance". Studies have found that since nasal airflow has a stimulating effect on the normal development of the maxilla, upper airway obstruction may cause abnormal development of the maxilla, and this abnormal development may occur in the sagittal, coronal, and axial planes. Studies have also found that approximately 52.3% of children with mouth breathing are combined with obstructive sleep apnea of varying degrees.

[0004] Currently, there are two common types of methods for diagnosing mouth breathing: questionnaire survey method and simple test method.

[0005] Questionnaire survey method: (1) Simple questionnaire survey: Parents or doctors observe whether children have the habit of opening their mouths. (2) The modified questionnaire survey method is to not only observe children's mouth breathing habits but also understand their sleep conditions, history of upper respiratory diseases, and daily life and learning performances to help improve the accuracy.

[0006] Simple testing methods: (1) Closed-lip experiment: When the child is sleeping soundly and lying on their back, gently close the child's upper and lower lips without gaps, and observe whether the child struggles or wakes up due to a sense of suffocation. (2) Mirror experiment: Place a single-sided mirror at the midpoint of the child's upper lip, with the mirror surface facing the nasal cavity and perpendicular to the plane of the nostrils. Place the mirror at the midpoint between the nostrils and the upper lip to isolate the oral airflow and allow the nasal airflow to spray onto the mirror surface, and observe whether water vapor appears on the mirror surface. (3) Water-containing experiment: Instruct the child to sit still, hold about 15 ml of water for 3 minutes, and observe whether the breathing is significantly affected. However, the existing methods cannot accurately diagnose the situation of children's mouth breathing, and the test process is greatly affected by the cooperation of the child. Therefore, continue to design a non-contact technology that uses artificial intelligence graphic recognition to intelligently diagnose children's open-mouth breathing to meet the technical requirements for diagnosing children's open-mouth breathing. Summary of the Invention

[0007] To solve the above problems, the present invention provides a diagnostic method for identifying children's open-mouth breathing using artificial intelligence, including the following steps:

[0008] Based on a set timestamp, collect the real-time facial video stream of the child during sleep and send it to the server side;

[0009] Through the server side, use a face detector to perform facial recognition on the real-time facial video stream to obtain the relative relationship α between the mouth-opening amplitude and the overall width of the lips, where the relative relationship α is used to represent the proportional relationship between the mouth-opening amplitude and the overall width of the lips;

[0010] Through Kalman filtering processing of the relative relationship α, obtain the continuous change situation of the processed relative relationship α in the video stream;

[0011] Diagnose children's open-mouth breathing according to the continuous change situation.

[0012] Preferably, before the facial recognition process, perform contrast enhancement processing and grayscale processing on the original image of the real-time collected facial video stream, and output an output stream with the same size and frame rate as the input image to the face detector.

[0013] Preferably, during the facial recognition process, send the output stream to a face detector based on the dlib library, and use a deep neural network trained on a self-built children's face database to identify whether the image is a valid face to obtain a face image.

[0014] Preferably, during the process of identifying a valid face, use the deep neural network ResNet-34 to identify whether the image is a valid face.

[0015] Preferably, before obtaining the relative relationship α between the mouth opening amplitude and the overall lip width, for the face image, 68 feature points of the face are generated through the facial landmarks algorithm to locate the eyes, eyebrows, nose, mouth, and jawline in the image.

[0016] Preferably, during the process of obtaining the relative relationship α, the relative relationship α between the mouth opening amplitude and the overall lip width is obtained by obtaining the distance inside the two lips, the distance between the two corners of the mouth, and the distances from the two corners of the mouth to the longitudinal central axis of the lips.

[0017] Preferably, during the process of obtaining the relative relationship α, the relative relationship α is expressed as:

[0018]

[0019] where d h,i is the distance inside the two lips, d w,o is the distance between the two corners of the mouth, and d h,l and d h,r are the distances from the two corners of the mouth to the longitudinal central axis of the lips.

[0020] Preferably, during the process of obtaining the continuous change situation, according to the continuous change situation of the relative relationship α in the video stream, specific values are defined for the size of mouth breathing, expressed as:

[0021]

[0022] Preferably, during the process of diagnosing children's mouth breathing, according to the number of times of children's mouth breathing, the total duration of mouth breathing, the proportion of mouth breathing duration, and the average degree of mouth breathing, the degree of children's mouth breathing is judged.

[0023] The present invention discloses a diagnostic system for identifying children's mouth breathing using artificial intelligence, including:

[0024] A data acquisition module for collecting the real-time facial video stream of children during sleep based on a set time stamp and sending it to the server side;

[0025] A data processing module for performing facial recognition on the real-time facial video stream through the server side using a face detector to obtain the relative relationship α between the mouth opening amplitude and the overall lip width, where the relative relationship α is used to represent the proportional relationship between the mouth opening amplitude and the overall lip width;

[0026] A feature extraction module for obtaining the continuous change situation of the processed relative relationship α in the video stream by performing Kalman filtering processing on the relative relationship α;

[0027] A diagnostic module for diagnosing children's mouth breathing according to the continuous change situation.

[0028] The present invention discloses the following technical effects:

[0029] The present invention can record facial images without affecting or contacting children, regardless of whether they are awake or asleep.

[0030] The analysis of the child's facial image by the present invention is dynamic and can achieve long-term monitoring;

[0031] The present invention can judge the severity of mouth breathing by the ratio of the distance between the upper and lower lips to the distance between the two sides of the mouth corners;

[0032] The recording device used in the present invention only needs to be a high-definition camera;

[0033] The present invention does not require professional equipment and a specific place. Parents of children can analyze the changes in the jaws and faces during sleep at home using common mobile devices such as mobile phones and cameras.

[0034] In the present invention, a corresponding application program will be provided to guide parents to complete the video recording of sleep;

[0035] The present invention uses a preset algorithm in the computer to double the speed of analyzing the maxillofacial condition during sleep, thereby saving a lot of time and improving the diagnosis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a schematic diagram of the distribution definition of feature points for face recognition according to the present invention;

[0038] Figure 2 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are only a part rather than all of the embodiments of this application. Components of the embodiments of this application usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative efforts fall within the scope of protection of this application.

[0040] As Figure 1-2 shown, the present invention provides a diagnostic method for identifying children's mouth breathing using artificial intelligence, including the following steps:

[0041] Based on a set timestamp, collect the real-time facial video stream of a child during sleep and send it to the server side;

[0042] Through the server side, use a face detector to perform facial recognition on the real-time facial video stream, and obtain the relative relationship α between the mouth opening amplitude and the overall width of the lips, where the relative relationship α is used to represent the proportional relationship between the mouth opening amplitude and the overall width of the lips;

[0043] Through Kalman filtering processing of the relative relationship α, obtain the continuous change situation of the processed relative relationship α in the video stream;

[0044] Diagnose children's mouth breathing according to the continuous change situation.

[0045] Further preferably, for the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, before the facial recognition process, perform contrast enhancement processing and grayscale processing on the original image of the real-time collected facial video stream, and output an output stream with the same size and frame rate as the input image to the face detector.

[0046] Further preferably, for the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, during the facial recognition process, the present invention sends the output stream to a face detector based on the dlib library, and uses a deep neural network trained for a self-built children's face database to identify whether the face in the image is a valid face, and obtain a face image.

[0047] Further preferably, for the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, during the process of identifying a valid face, the present invention uses the deep neural network ResNet-34 to identify whether the face in the image is a valid face.

[0048] Further preferably, in the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, before the process of obtaining the relative relationship α between the mouth opening amplitude and the overall width of the lips, the present invention generates 68 feature points of the human face through the faciallandmarks algorithm for the face image, and locates the eyes, eyebrows, nose, mouth, and jawline in the image.

[0049] Further preferably, in the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, during the process of obtaining the relative relationship α, the present invention obtains the relative relationship α between the mouth opening amplitude and the overall width of the lips by obtaining the distance inside the two lips, the distance between the two corners of the mouth, and the distances from the two corners of the mouth to the longitudinal central axis of the lips.

[0050] Further preferably, in the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, during the process of obtaining the relative relationship α, the relative relationship α mentioned in the present invention is expressed as:

[0051]

[0052] where d h,i is the distance inside the two lips, d w,o is the distance between the two corners of the mouth, and d h,l and d h,r are the distances from the two corners of the mouth to the longitudinal central axis of the lips.

[0053] Further preferably, in the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, during the process of obtaining the continuous change situation, the present invention defines specific values for the size of mouth breathing according to the continuous change situation of the relative relationship α in the video stream, which is expressed as:

[0054]

[0055] Further preferably, in the diagnostic method for identifying children's mouth breathing using artificial intelligence provided by the present invention, during the process of diagnosing children's mouth breathing, the present invention judges the degree of mouth breathing of children according to the number of times of children's mouth breathing, the total duration of mouth breathing, the proportion of mouth breathing duration, and the average degree of mouth breathing.

[0056] The present invention discloses a diagnostic system for identifying children's mouth breathing using artificial intelligence, including:

[0057] A data acquisition module, configured to collect the real-time video stream of the face of a child during sleep based on a set timestamp and send it to the server side;

[0058] A data processing module, which is used to perform facial recognition on the real-time facial video stream through the server side by using a face detector, and obtain the relative relationship α between the mouth-opening amplitude and the overall width of the lips, where the relative relationship α is used to represent the proportional relationship between the mouth-opening amplitude and the overall width of the lips;

[0059] A feature extraction module, which is used to obtain the continuous change situation of the processed relative relationship α in the video stream by performing Kalman filtering on the relative relationship α;

[0060] A diagnosis module, which is used to diagnose children's mouth breathing according to the continuous change situation.

[0061] Embodiment 1: The purpose of the present invention is to utilize the recognition effect of artificial intelligence on maxillofacial features to invent a recognition and detection technology, which evaluates whether there is mouth breathing through the positional relationship between the upper and lower lips during children's sleep, and judges the severity of mouth opening based on the ratio of the distance between the upper and lower lips to the bilateral corners of the mouth. In a children's sleep video of a certain duration, objectively calculate and judge whether there is mouth breathing in children and its severity (judge whether there are adenoid facies, dentofacial deformities, and children's obstructive sleep apnea, etc.), so as to objectively evaluate whether there is mouth breathing during children's sleep, the time when mouth breathing occurs, and its severity. Guide children's parents and medical staff to choose appropriate treatment methods, and conduct evaluations and monitoring before and after treatment. Thus, intelligent monitoring and evaluation of mouth breathing are realized, and children's dentofacial deformities caused by mouth breathing are prevented, where mouth breathing refers to breathing air through the mouth due to partial or complete obstruction of the nasal airway; adenoid facies refers to that caused by adenoid hypertrophy, mainly manifested as elongated maxilla, retruded mandible, high-arched palate, irregular dentition, protruding upper incisors, thickened lips, and characteristic facial features lacking facial expressions; dentofacial deformity refers to abnormal growth and development of the jawbones, resulting in abnormal jawbone volume, shape, and the relationship between the upper and lower jawbones and other craniofacial bones, as well as concomitant abnormal occlusion relationship and oral and maxillofacial system functions; children's obstructive sleep apnea refers to clinical manifestations such as growth and development stagnation, abnormal cardiopulmonary function, nerve damage, and behavioral abnormalities caused by hypoxemia during sleep due to partial or complete upper airway obstruction.

[0062] The present invention achieves the prediction of the probability of adenoid hypertrophy in children through the system's learning of the "adenoid facies" characteristics. In actual clinical practice, the causes of children's facial development are not only adenoid hypertrophy. Continuous upper airway obstruction caused by tonsil hypertrophy, allergic rhinitis, nasal septum deviation, nasal polyps, etc. may all cause "adenoid facies". Many children in clinical practice have one or more of the above etiologies. Therefore, upper airway endoscopy or imaging examination is still indispensable in the preoperative preparation of children with mouth breathing.

[0063] The occurrence of dentofacial dysplasia mentioned in the present invention does not occur simultaneously with the presence of adenoid hypertrophy. Research has found that there is a "lag" phenomenon in the impact of mouth breathing caused by upper airway obstruction on maxillofacial development. When many children with dentofacial deformities seek medical treatment, both the adenoids and tonsils have already atrophied and do not meet the diagnostic criteria of "hypertrophy". Therefore, it is not accurate to evaluate the probability of adenoid or tonsil hypertrophy only through the developmental characteristics of the maxillofacial region.

[0064] Research shows that regardless of whether the nasal ventilation volume decreases, as long as the mouth is open, it will affect the descent of the palate. With the appearance of mouth breathing, the pressure in the oral and nasal cavities will change simultaneously, and the overall force on the hard palate will also change accordingly. This also suggests that in clinical practice, not only when the upper airway obstruction affects normal vital activities needs to be treated in a timely manner, but most importantly, children with mouth breathing need to be screened out in a timely manner. Further clarify the cause and promptly relieve the upper airway obstruction. For bad mouth breathing habits such as thumb sucking, tongue thrusting, and finger biting, children need to be further guided to correct them.

[0065] To meet the above technical objectives, the present invention mentions a technology for intelligent diagnosis of children's mouth breathing using artificial intelligence graphics recognition, which specifically includes the following processes:

[0066] (1). Connect the real-time video stream, perform contrast enhancement processing and grayscale processing on the original image, and output a stream with the same size and frame rate as the input image;

[0067] (2). Connect the video stream obtained in (1) to the face detector in the dlib library, and identify whether there is a valid face in the image through a deep neural network (such as ResNet-34) trained for the self-built children's face database;

[0068] (3). If a face is recognized in (2), establish a feature vector and generate 68 feature points (facial landmarks) of the face, locate the eyes, eyebrows, nose, mouth, jawline, etc. in the image, and the specific feature point distribution is as Figure 1 shown;

[0069] (4). Calculate the opening amplitude according to the recognition result in (3). Among them, the outer edge of the upper lip is 49 - 55, the inner edge of the upper lip is 66 - 68, the outer edge of the lower lip is 56 - 60, and the inner edge of the lower lip is 61 - 65. We use the upper and lower vertexes of the lips [63, 67] and the left and right corner vertexes of the mouth [49, 55] as data inputs, and propose the following calculation formula:

[0070]

[0071] where d h,i is the distance between the two inner sides of the lips, d w,o is the distance between the two corners of the mouth, dh,l With d h,r It is the distance between the corners of the mouth on both sides and the longitudinal central axis of the lips. The physical meaning of this formula is to calculate the relative relationship between the opening amplitude of the mouth and the overall width of the lips, which can play a robust role in the distance during shooting and the different sizes of the human face and lips;

[0072] (5), Perform Kalman filtering on the α calculated in (4);

[0073] (6), Based on the above calculation results, record the continuous change of α in the video stream and define specific values for the size of mouth breathing:

[0074]

[0075] (7), According to the detection results in (6), count the following information of the subject in the effective video stream (i.e., the effective detection time period): the number of times of mouth breathing (times), the total duration of mouth breathing (s), the proportion of mouth breathing duration, and the average degree of mouth breathing (none, mild or severe), and conduct a comprehensive evaluation based on the above four indicators.

[0076] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0077] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0078] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A diagnostic method for identifying mouth breathing in children using artificial intelligence, characterized in that: The following steps are involved: Based on the set timestamp, collect the real-time video stream of the child's face while sleeping and send it to the server; By using the server side, using a face detector, performing facial recognition on the real-time facial video stream, and obtaining a relative relationship α between the mouth opening amplitude and the overall lip width, wherein the relative relationship α is used to represent the proportional relationship between the mouth opening amplitude and the overall lip width; By performing Kalman filtering on the relative relationship α, a continuous change of the processed relative relationship α in the video stream is obtained; Diagnosing mouth breathing in children based on the continuous changes; In the process of obtaining the relative relationship α, the relative relationship α between the mouth opening amplitude and the overall lip width is obtained by obtaining the inner distance between the two lips, the distance between the two mouth corners, and the distance between the two sides of the mouth corners and the longitudinal center axis of the lips. The relative relationship α is expressed as: Among them, d h,i is the inner distance between the two lips, d w..o is the distance between the two mouth corners, d h,l With d h,r is the distance between the corners of the mouth and the longitudinal center axis of the lips, <d h,l ,d h,r > is d h,l The corresponding vector and d h,r The minimum positive angle between corresponding vectors; In the process of obtaining the continuous change, according to the continuous change of the relative relationship α in the video stream, a specific value is defined for the size of mouth breathing, which is expressed as: In the process of diagnosing mouth breathing in children, the degree of mouth breathing in children is judged according to the number of mouth breathing times, the total duration of mouth breathing, the proportion of mouth breathing duration and the average degree of mouth breathing, wherein the average degree of mouth breathing includes no mouth breathing, mild mouth breathing or severe mouth breathing.

2. A diagnostic method for identifying mouth breathing in children using artificial intelligence according to claim 1, characterized in that: Before the face recognition process is performed, the original image of the face video stream collected in real time is subjected to contrast enhancement and graying processing, and an output stream with the same size and frame rate as the input image is output to the face detector.

3. A diagnostic method for identifying mouth breathing in children using artificial intelligence according to claim 2, characterized in that: During the facial recognition process, the output stream is sent to a face detector based on the dlib library, and a deep neural network trained on a self-built children's face database is used to identify whether the image is a valid face, thereby obtaining a face image.

4. A diagnostic method for identifying mouth breathing in children using artificial intelligence according to claim 3, characterized in that: In the process of identifying valid faces, the deep neural network ResNet-34 is used to identify whether the image contains a valid face.

5. The diagnostic method for identifying mouth breathing in children using artificial intelligence according to claim 4, characterized in that: Before obtaining the relative relationship α between the mouth opening amplitude and the overall lip width, the facial image is subjected to a facial landmarks algorithm to generate 68 feature points of the face and locate the eyes, eyebrows, nose, mouth and jawline in the image.

6. A diagnostic system for identifying mouth breathing in children using artificial intelligence, characterized in that: include: The data collection module is used to collect the real-time video stream of the child's face during sleep based on the set timestamp and send it to the server; A data processing module, used to perform facial recognition on the real-time facial video stream by using a face detector through the server side, and obtain a relative relationship α between the mouth opening amplitude and the overall lip width, wherein the relative relationship α is used to represent the proportional relationship between the mouth opening amplitude and the overall lip width; A feature extraction module, configured to obtain a continuous change of the relative relationship α in the video stream by performing Kalman filtering on the relative relationship α; A diagnosis module, used for diagnosing mouth breathing of children according to the continuous changes; In the process of obtaining the relative relationship α, the relative relationship α between the mouth opening amplitude and the overall lip width is obtained by obtaining the inner distance between the two lips, the distance between the two mouth corners, and the distance between the two sides of the mouth corners and the longitudinal center axis of the lips. The relative relationship α is expressed as: Among them, d h,i is the inner distance between the two lips, d w..o is the distance between the two mouth corners, d h,l With d h,r is the distance between the corners of the mouth and the longitudinal center axis of the lips, <d h,l ,d h,r > is d h,l The corresponding vector and d h,r The minimum positive angle between corresponding vectors; In the process of diagnosing mouth breathing in children, the degree of mouth breathing in children is judged according to the number of mouth breathing times, the total duration of mouth breathing, the proportion of mouth breathing duration and the average degree of mouth breathing, wherein the average degree of mouth breathing includes no mouth breathing, mild mouth breathing or severe mouth breathing.

Images