Method and system for identifying bone shape based on neural network through face photo

Abstract

The present application relates to a method and system for identifying bone shape based on neural network through face photos, comprising the following steps: S1, constructing a sample set: the sample set is divided into three types of sagittal bone types, I, II, III, each sample includes front, 45-degree and 90-degree side face photos; S2, constructing and training Shuffl e-Attent i on model based on soft tissue face photo to distinguish bone classification: the model adopts lightweight Shuffl eNet as the backbone, specifically including three input layers, three feature extractors, a feature fusion layer for integration, an SE module for attention mechanism and a classifier, inputting three-angle face photos into the trained model to distinguish the bone classification; S3, analyzing the soft tissue features of Shuffl e-Attent i on model to distinguish bone classification, and prompting the sample feature soft tissue performance of three types of bone types. The accuracy of identifying bone type based on soft tissue in the present application is better than that of trained orthodontic doctors by naked eye, and the model can process human difficult clues.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent identification, and particularly relates to a method and system for identifying bone surface types based on facial photos through a neural network. BACKGROUND

[0002] Social progress drives the pursuit of aesthetic profile, and more and more patients seek orthognathic or orthodontic treatment, hoping to improve the overall facial beauty, which highlights the importance of comprehensive understanding of the overall beauty of the face. Various studies have extensively explored the ranking of facial attractiveness, and proposed standards such as the golden ratio and the proportions of facial thirds, symmetry or averageness, etc. However, the above-mentioned measurement methods using spatial relationships are mainly developed based on empirical evidence, introducing considerable subjective bias. In addition, soft tissue features such as facial volume and connecting curves, which are crucial factors in aesthetic perception, are not fully considered by geometric proportions. Therefore, there is an urgent need to develop more comprehensive strategies to evaluate facial beauty.

[0003] From a therapeutic perspective, the main aesthetic issues involved in orthodontic treatment revolve around sagittal skeletal discrepancies. Relative deficiency or excess growth of the maxilla or mandible can disrupt facial balance, leading to disharmony. Although individual perceptions of beauty may vary, relatively harmonious maxillo-mandibular relationships, such as skeletal Class I, are generally considered more pleasing than Class II or III. Therefore, a common goal in treating Class II or III patients is to minimize their differences from Class I or to attenuate the impact of emphasizing their disharmonic features. However, as mentioned above, facial attractiveness depends on the coordinated interaction of all facial features. From this perspective, the most satisfactory outcome of dental treatment is to achieve overall harmony by adjusting the dental arch to accommodate existing features. Therefore, only by considering the patient's expectations and baseline conditions can an appropriate treatment plan be developed

[0004] Although there have been many studies evaluating facial attractiveness, various standards such as the golden ratio, facial symmetry and averageness have been proposed, these standards are mainly based on empirical and quantitative measurements of spatial relationships, are prone to introduce subjective bias, and cannot fully consider the soft tissue features of the face. The geometric proportions in the existing methods cannot fully capture the soft tissue features such as facial volume and connecting curves, which play a key role in aesthetic perception.

[0005] In addition, for different patients, only making a treatment plan based on the condition of hard tissue may not achieve the best overall facial harmony. Therefore, the prior art has limitations in comprehensively considering facial aesthetics and individual differences. At the same time, the existing method often lacks precise quantitative tools to guide the formulation of treatment plans, resulting in unstable effects and difficulty in achieving the expected facial aesthetic improvement effect. Therefore, there is an urgent need for a more comprehensive and more accurate method to evaluate facial aesthetics. SUMMARY

[0006] The present application provides a method and system for identifying bone facial types based on neural networks through facial photos.

[0007] The present application achieves the following technical solutions:

[0008] The method for identifying bone facial types based on neural networks through facial photos provided by the present application includes the following steps:

[0009] S1, constructing a sample set: the sample set is divided into three types of sagittal bone types I, II and III, and each sample includes a front view, a 45-degree side view and a 90-degree side view of a facial photo;

[0010] S2, constructing and training a Shuffle-Attention model:

[0011] The Shuffle-Attention model uses a lightweight ShuffleNet as the backbone, specifically including three input layers, three independent feature extractors, a feature fusion layer for integration, an SE module for attention mechanism, and a classifier. The three input layers are connected to the three independent feature extractors, the three independent feature extractors are connected to the feature fusion layer, the feature fusion layer is connected to the SE module, the SE module is connected to the pooling layer and the full connection layer, and finally the classifier is input.

[0012] The front view, 45-degree side view and 90-degree side view of each sample are input into the three input layers for model training, and a trained Shuffle-Attention model is obtained.

[0013] Optionally, the S1 includes the following steps:

[0014] Collecting data: collecting the front view, 45-degree side view and 90-degree side view of the patient's facial photo, and the cephalometric radiograph image; measuring the ANB value according to the patient's head lateral film; and marking the sagittal bone type of the sample according to the ANB value.

[0015] Optionally, the patient has no history of congenital deformity, infection, trauma and tumor; at the same time, patients with obvious skeletal deformities and in need of surgery are excluded.

[0016] Optionally, when measuring the ANB value, first use the automatic cephalometric marking function to mark the lateral cephalogram image, and then manually check and correct the marking by at least two trained orthodontists; 1°≤ANB≤5° is classified as class I, ANB>5° is classified as class II, and ANB<1° is classified as class III; in case of disagreement in classification, consult another orthodontist, and if three orthodontists still cannot reach a consensus, exclude these confusing images.

[0017] Optionally, S1 further comprises data processing: cropping the three-angle face photo according to the frames of the forehead, ears and chin; using an image enhancement method to expand the data set; dividing the expanded data set into a training set and a test set.

[0018] Optionally, the ratio of the training set and the test set is 8:2.

[0019] Optionally, the method of identifying skeletal pattern from face photo based on neural network further comprises: Grad-CAM reverse analysis of Shuffle-Attention model to determine the soft tissue features of skeletal classification, and prompt the skeletal I, II and III sample feature soft tissue performance.

[0020] The system for identifying skeletal pattern from face photo provided by the application comprises:

[0021] The input module has three input units for inputting the face photos of the front, 45-degree side and 90-degree side of the person to be identified, respectively;

[0022] The recognition module comprises a Shuffle-Attention model, which is trained by the method;

[0023] The output module is used for outputting the recognition result.

[0024] Optionally, the system for identifying skeletal pattern from face photo further comprises a Grad-CAM module: for calculating the weight score of each pixel on the input image and presenting it as a heat map, and superimposing the heat map on the corresponding input image;

[0025] The output module is also used for outputting the heat map prompt.

[0026] The method for identifying bone types based on neural networks through facial photos inputs the front, 45-degree side and 90-degree side of the person to be identified into a Shuffle-Attention model, the Shuffle-Attention model identifies based on three-angle facial photos, and outputs the bone classification of the person to be identified; the front, 45-degree side and 90-degree side of the person to be identified are input into a Grad-CAM module, and the heat map of the person to be identified is output, prompting the soft tissue area of the feature, and used for prompting the clinical aesthetic scheme design.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] 1. The accuracy of identifying bone types based on only soft tissues (facial photos) in the present application is higher than that of trained orthodontists visually distinguishing, the Shuffle-Attention model constructed in the present application can process clues that are difficult for humans to perceive;

[0029] 2. Compared with obtaining bone information according to a cephalogram and cooperating with professional doctors, the present application can obtain information only according to three-angle facial photos, which can effectively reduce the cost of medical equipment and lower the medical technology threshold;

[0030] 3. Compared with the prior art of formulating a treatment plan only according to hard tissues, the present application can identify the characteristic soft tissue area according to a portrait photo through soft tissue analysis of three types of bone types, which can assist doctors in formulating a treatment plan by comprehensively considering the overall soft tissue appearance, and is more comprehensive and more accurate;

[0031] 4. The information of three facial photos from different angles is combined in the present application, and the heat map from different sides is more helpful for comprehensive analysis of the characteristics of each bone type. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which are included to provide a further understanding of the embodiments of the present application and constitute a part of the present application, illustrate embodiments of the present application and are used to explain the principles of the present application.

[0033] Figure 1 is a flowchart of the method and system for identifying bone types based on neural networks through facial photos in the embodiment;

[0034] Figure 2 is a structure diagram of the Shuffle-Attention model in the embodiment;

[0035] Figure 3 In a, the accuracy rates of the training set and the test set in the Sfa model along the learning period are represented by green and orange lines respectively, and the final accuracy rate of the test set reaches 0.9235; b is a confusion matrix diagram of the SfA model; c is a ROC curve diagram of the SfA model.

[0036] Figure 4 a is a representative feature map generated by Grad-CAM; wherein a is a representative feature map of a randomly selected I-class sample; b is a representative feature map of a randomly selected II-class sample; c is a representative feature map of a randomly selected III-class sample;

[0037] Figure 5 a is a representative feature map generated by Grad-CAM; wherein a is a representative feature map of a randomly selected I-class sample; b is a representative feature map of a randomly selected II-class sample; c is a representative feature map of a randomly selected III-class sample;

[0038] a is a 224X224 pixel picture segmented into 9 regions using a picture segmentation strategy, showing the heat map distribution of the 9 region weight scores in bone type I, II and III;

[0039] b is a comparison chart of 1-9 regions of the front photo in the three classifications, and I, II and III are represented by light green, yellow and orange respectively;

[0040] c is a comparison chart of 1-9 regions of the 45-degree photo in the three classifications, and I, II and III are represented by dark gray, pink and dark red respectively;

[0041] d is a comparison chart of 1-9 regions of the side photo in the three classifications, and I, II and III are represented by light gray, blue and navy blue respectively; the significance of the results is indicated by asterisks, P<0.05 (*), P<0.01 (**), P<0.001 (***) and P<0.0001 (****) respectively;

[0042] Figure 6 a is a comparison chart of the face appearance modification of II-class patients based on feature maps and heat maps;

[0043] a is the original photo (a) and the corresponding original heat map prompt;

[0044] b is the photo after modification of the chin and the corresponding heat map prompt, and the modified part is highlighted with a white dashed line frame;

[0045] c is the photo after modification of the nose and the corresponding heat map prompt, and the modified part is highlighted with a white dashed line frame;

[0046] d is the photo after modification of the lips and the corresponding heat map prompt, and the modified part is highlighted with a white dashed line frame;

[0047] e is the photo after modification of the cheeks and the corresponding heat map prompt, and the modified part is highlighted with a white dashed line frame;

[0048] Figure 7Fig. 4 is a comparison chart of face appearance modification of a patient of type III based on feature map and subsequent changes in heat map; wherein a is the original photo and corresponding original heat map prompt; b is the photo after modification of the cheeks and the corresponding heat map prompt, the modified part is highlighted with a white dashed line frame; c is the photo after modification of the chin and the corresponding heat map prompt, the modified part is highlighted with a white dashed line frame. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments.

[0050] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0051] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. It should be noted that each embodiment in the present specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between each embodiment can be referred to each other. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts are referred to the part of the description of the method embodiments.

[0052] All experimental procedures involving photos and human X-ray images in this embodiment have obtained the approval of the ethics committee (WCHSIRB-CT-2022-256), and the research has obtained the informed consent of the patients.

[0053] As shown in Figure 1 The method for identifying bone facial types through face photos based on neural networks disclosed in this embodiment includes the following steps:

[0054] S1, constructing a sample set;

[0055] S1.1 Data collection

[0056] In this embodiment, photos and cephalometric radiographs of patients who visited the hospital for orthodontic treatment from January 1, 2020 to July 1, 2021 were selected for the construction of the data set.

[0057] The frontal, 45-degree right lateral, and 90-degree lateral photographs of each patient were taken in a relaxed and comfortable position. Lateral cephalograms were randomly obtained from two different devices, including PaX-400C (VATECH, Korea) and Morita x550 (Morita, Kyoto, Japan).

[0058] This study mainly included patients aged 18-30 years old, without congenital deformity, infection, trauma, and tumor history. At the same time, patients with obvious skeletal deformities and the need for surgery were excluded to ensure the applicability of the neural network.

[0059] To classify patients by sagittal skeletal type, this embodiment used the automatic cephalometric marker function of Uceph (China) to mark the lateral cephalogram, and then two trained orthodontists manually checked and corrected the markers. Point A-nasal tip-point B (ANB) was used for diagnosis. The sagittal relationship was divided into class I straight group (1°≤ANB≤5°), class II convex group (ANB>5°), and class III concave group (ANB<1°). In the case of disagreement in classification, a senior orthodontist will be consulted. If three experts still cannot reach a consensus, these confusing images will be excluded. Based on the consensus of experts, the samples were divided into three categories of sagittal (I, II, III) skeletal types, and the sample distribution is shown in Table 1.

[0060] Table 1: Sample distribution of different sagittal bone categories

[0061]

[0062] S1.2, Data processing

[0063] To train and test the model, this embodiment first crops the photo according to the frame of the forehead, ears, and chin, removing irrelevant influencing factors. Second, this embodiment uses image enhancement methods such as color jittering, horizontal flipping, scaling, rotation, and cropping to expand the data set and improve the robustness of the network. After enhancement, the number of samples increased from 1044 to 4176. This embodiment uses random sampling method to divide the processed data set into training set and test set, and the ratio of training set and test set sample amount is 8:2.

[0064] S2, Model construction and training

[0065] This embodiment developed a Shuffle-Attention model (hereinafter referred to as SfA model), which uses a lightweight ShuffleNet as the backbone. As shown in FIG. 2, the SfA model consists of a ShuffleNet backbone, a Shuffle-Attention module, and a fully connected layer. Figure 2As shown, overall, the SfA model mainly consists of three independent feature extractors based on ShuffleNet, a connection part for integration, two layers of SE modules for attention mechanism compression and excitation, and a classifier. All weights in the feature extractors except the fully connected layer are initialized as the weights of the ImageNet pre-trained model as transfer learning. The feature maps extracted from the three independent feature extractors are integrated through the connection layer; next, the SE module is used to calculate and optimize the weight of each channel to achieve better classification performance.

[0066] The model framework of ShuffleNet mainly includes the following key components:

[0067] 1. Input layer: Each angle image is input into a separate ShuffleNet for feature extraction.

[0068] 2. Convolutional and pooling layers: The initial layers of ShuffleNet usually include a convolutional layer for extracting low-level features, followed by a pooling layer for reducing the spatial resolution of the image. The role of this layer is to perform preliminary feature extraction and down-sampling on the input image.

[0069] 3. Stage division: ShuffleNet divides the entire network into multiple stages (Stages), each containing a group of ShuffleNet units. The number of channels and resolution in each stage gradually change to adapt to different levels of feature extraction needs. Within each stage, the feature maps are divided into multiple groups, and each group performs independent feature extraction.

[0070] 4. ShuffleNet unit: ShuffleNet unit is the basic building block of ShuffleNet. Each unit includes the following key steps:

[0071] Pointwise Group Convolution: Within each group, pointwise group convolution is used to reduce the exchange of information between channels. This is the core operation of ShuffleNet, used to reduce computational complexity.

[0072] Channel Shuffle: Within each group, the channel shuffle operation is used to increase the flow of information between channels to improve the performance of the model.

[0073] Depthwise Separable Convolution: After pointwise group convolution, ShuffleNet usually uses depthwise separable convolution for feature extraction. This convolution operation reduces the computational load while maintaining the accuracy of the model.

[0074] 5. After performing the three-angle information feature fusion, use the squeeze-and-excitation module to perform attention mechanism calculation to strengthen the channel features.

[0075] 6. Global Average Pooling: The last layer of ShuffleNet is usually a global average pooling layer. The role of this layer is to average pool the feature maps of each channel to generate a fixed-size feature vector for the final classification task.

[0076] 7. Fully Connected Layers and Classifier: Finally, ShuffleNet usually includes one or more fully connected layers to map the output of the global average pooling layer to the probability distribution of class labels. These layers constitute the model's classifier.

[0077] Overall, the model framework structure of ShuffleNet is a deep neural network composed of multiple ShuffleNet units. Each unit includes pointwise group convolution, channel rearrangement, and depthwise separable convolution operations to reduce computational complexity and improve model efficiency. This structure makes ShuffleNet an ideal choice for image classification and object detection tasks in environments with limited computing resources.

[0078] Model training includes the following steps:

[0079] Input the front, 45-degree side, and 90-degree side of each sample in the training set into the three input layers of the model for model training;

[0080] Use the trained model to classify the test set: input the front, 45-degree side, and 90-degree side of the test set samples into the trained Shuffle-Attention model, which recognizes based on three-angle face photos and outputs skeletal classification.

[0081] S3, evaluate model performance

[0082] Use the confusion matrix, accuracy, recall, sensitivity, specificity, and receiver operating characteristic curve to evaluate the model, as shown in Table 2.

[0083] Table 2: Sfa model and professional doctor's accuracy, recall, and F1 score for bone classification based on face photos

[0084]

[0085] As can be seen from Table 2, the accuracy of the Sfa model in identifying the type of bone based only on soft tissue (0.9293) is significantly better than the accuracy of the trained orthodontist in visual identification (0.764), which confirms that the Sfa model of the present embodiment can process clues that are difficult for humans to perceive.

[0086] S4, Grad-CAM reverse analysis

[0087] Grad-CAM is a popular and optimized version of Class Activation Mapping (CAM) technology, and its essential role is to analyze the trained neural network model, prompt the key pixels in the input image that play a key role in the final classification decision, and their importance.

[0088] In the present embodiment, first, the Grad-CAM algorithm is used to calculate the key pixels and their weight scores of the Shufflenet-Attention model for judging bone types I, II and III in the input image, and statistical analysis is performed on all sample results, so as to obtain the computer-determined soft tissue feature points of bone types I, II and III.

[0089] Secondly, a single sample can be input into the model, and the prominent feature area of the sample can be identified by Grad-CAM, so as to assist clinical aesthetic design and design a personalized treatment plan based on the original soft tissue features of the patient.

[0090] As shown in Figure 4 , the heat map superimposed on the original test set image produces interesting results and reveals significant differences between the three classifications.

[0091] Specifically, for type I, as shown in Figure 4 a, Figure 5 a-d, the prominent areas of the frontal and 45-degree photos mainly cover the facial contour, such as the hairline, temple, ear and chin, while in the profile photo, the forehead, nasal area and tip of the nose are highlighted, which indicates that the portrait of type I seems to be classified by the geometric coordination between the contours.

[0092] For type II, as shown in Figure 4 b, Figure 5 a-d, the frontal view is more concerned because the forehead and nasolabial area are significantly highlighted, while the 45-degree photo focuses on the contour and malar bone. Interestingly, the prominent area of type II on the side seems to be composed of two parallel lines from the corner of the eye and the tip of the nose to the tip of the nose. Therefore, it can be inferred that the Sfa model mainly identifies type II through the features of the forehead, malar bone and forehead-nose-chin.

[0093] For type III, as shown in Figure 4 c, Figure 5As shown in a-d, frontal views appear to define the profile, while 45-degree and lateral views both highlight the nasal sides and chin, suggesting that pictures of Class III patients are more easily identified by their characteristic midface and chin.

[0094] From another perspective, Grad-CAM results suggest that the weight of the pictures from the three perspectives of Class I, Class II, and Class III varies, as shown in a. Figure 5 First, it is generally accepted that the lateral view is the most representative picture, which can suggest the relationship between the skull, maxilla, and mandible. The research results of this embodiment support this view, because in all sagittal types, most of the highlighted features are observed on the profile picture. For Class I and Class III, the nasal sides and chin are the most prominent parts. While in Class II, the relative position between the forehead, nose, and chin seems to be more important. The results are highly consistent with existing knowledge about sagittal types, that is, Class III is usually characterized by a prominent chin or a concave maxilla reflected by a concave nasal side, while a relatively retruded mandible is a feature of Class II, as shown in d. Figure 5

[0095] Second, while the frontal view of Class I and Class III does not provide much information in addition to the profile, the forehead and nasolabial area of Class II are highlighted in the frontal view, as shown in b. This can be an interesting finding, suggesting that Class II patients may have more prominent features in the frontal view, while the differences in Class III are more difficult to detect at this angle. Figure 5

[0096] Third, as shown in c, the 45-degree angle pictures of Class II and Class III highlight the zygomatic bone and cheek, which indicates that for most Class II and Class III patients, the imbalance of maxillary and mandibular growth can lead to prominent features such as prominent zygomatic bone and concave midface, which should be fully considered during treatment. Figure 6

[0097] Overall, the portrait of Class I seems to be classified by the geometry of the profile, while Class II is identified by the features between the front face, zygomatic bone, and forehead-nose-chin; the picture of Class III is more easily identified by the characteristic midface and chin.

[0098] S5, verification of clinical significance.

[0099] ​​​To test the clinical significance of the model of the present embodiment, representative class II and class III samples were selected and input into the Sfa model and Grad-CAM network, respectively. Based on the highlighted areas in the Grad-CAM output heat map, the facial features prompted as class II or class III features were modified using Adobe Photoshop CS6 (Adobe Systems Inc., San Jose, CA). For example, for class II samples, facial changes were simulated to reduce prominent features such as a prominent chin, nose, or mouth, or to reduce the fullness of the cheeks. As for class III samples, their features were attenuated by increasing the volume of the relatively concave middle face or by narrowing the chin. Then, the modified photos were re-input into the Sfa model and Grad-CAM network for re-identification and classification to verify whether the classification results and heat map distribution can be changed after modifying the features of specific parts.

[0100] As shown in Figure 7 , the modified facial photos were re-input into the Sfa model and Grad-CAM, and it was found that the classification result changed from the original class II to class I, and the heat map area was consistent with that of class I, indicating that modifying the chin, nose, or cheeks can effectively improve the facial shape.

[0101] As shown in ​ , the modified facial photos were re-input into the Sfa model and Grad-CAM, and it was found that the classification result changed from the original class III to class I, and the heat map area was consistent with that of class I, indicating that modifying the cheeks or chin can effectively improve the facial shape.

[0102] The present embodiment prompts three heat maps through the Grad-CAM network, and combines the information of the three maps to imagine the basis on which the Sfa model predicts the bone type through the appearance of soft tissue. The result is more accurate than that of human experts.

[0103] In particular, it is worth noting that, unlike traditional methods, the Sfa model of the present embodiment adopts a multi-input strategy to combine the information from three photos taken from different angles, and the heat maps from different sides are more helpful for comprehensive analysis of the features of each bone type.

[0104] Based on the above method of identifying bone shape through facial photos based on a neural network, the present embodiment discloses a system for identifying bone shape through facial photos, which comprises:

[0105] An input module having three input units for inputting the facial photos of the subject from the front, 45-degree side, and 90-degree side, respectively;

[0106] The recognition module comprises a Shuffle-Attention model, and the Shuffle-Attention model is trained by using the method;

[0107] The Grad-CAM module is used for calculating a weight score of each pixel on an input image and presenting the weight score as a heat map, and superimposing the heat map on the corresponding input image;

[0108] The output module is used for outputting the recognition result and the heat map prompt;

[0109] The front view, 45-degree side view and 90-degree side view of the person to be recognized are input into the Shuffle-Attention model, the Shuffle-Attention model performs recognition based on the three-angle face photos, and outputs the skeletal classification of the person to be recognized;

[0110] The front view, 45-degree side view and 90-degree side view of the person to be recognized are input into the Grad-CAM module, and the Grad-CAM module outputs the heat map prompt of the person to be recognized.

[0111] The present application can provide valuable help for orthodontists in identifying uncoordinated facial features of patients, and is beneficial to maximizing the aesthetic benefits of orthodontic treatment for each individual patient.

[0112] In several embodiments provided in the present application, it should be understood that the disclosed system and method can also be implemented in other ways. The system embodiments described above are only illustrative, for example, the flowchart and block diagram in the drawings show the possible implementation architecture, function and operation of the system, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0113] In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media. It should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0115] The above detailed description further illustrates the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above is only a specific embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for recognizing skeletal facial features from facial photographs based on neural networks, characterized by: Includes the following steps: S1. Constructing a sample set: The sample set is divided into three types of sagittal skeletons: I, II, and III. Each sample includes a frontal, 45-degree side, and 90-degree side facial photograph. S2. Construct and train the Shuffle-Attention model, which is used to classify bony features based on soft tissue face photos. The Shuffle-Attention model uses the lightweight ShuffleNet as its backbone, specifically consisting of three input layers, three independent feature extractors, a feature fusion layer for integration, an SE module for the attention mechanism, and a classifier. The three input layers are connected to the three independent feature extractors, and the three independent feature extractors are all connected to the feature fusion layer. The feature fusion layer is connected to the SE module, the SE module is connected to the pooling layer and the fully connected layer, and finally, the input is fed into the classifier. The front, 45-degree side, and 90-degree side views of each sample are input into the three input layers respectively for model training, resulting in a trained Shuffle-Attention model.

2. The method for recognizing skeletal facial features from facial photographs based on neural networks according to claim 1, characterized in that: S1 includes the following steps: Data collection: Collect facial photographs of the patient from the front, 45-degree side, and 90-degree side, as well as cephalometric radiographs; ANB values ​​were measured based on the patient's lateral cephalometric radiographs. The sagittal skeleton type of the corresponding sample is labeled according to the ANB value.

3. The method for recognizing skeletal facial features from facial photographs based on neural networks according to claim 2, characterized in that: The patients had no history of congenital malformations, infections, trauma, or tumors; at the same time, patients with significant skeletal deformities requiring surgery were excluded.

4. The method for recognizing skeletal facial features from a face photograph based on a neural network according to claim 2 or 3, characterized in that: When measuring ANB values, the lateral cephalogram image is first marked using the automatic cephalogram measurement and marking function, and then the marking is manually checked and corrected by at least two trained orthodontists. 1°≤ANB≤5° is classified into Class I, ANB>5° is classified into Class II, and ANB<1° is classified into Class III; If there is disagreement on the classification, consult another orthodontist. If the three orthodontists still cannot reach a consensus, the confusing images will be ruled out.

5. The method for recognizing skeletal facial features from a face photograph based on a neural network according to claim 2, characterized in that: S1 also includes data processing: Cropping of a three-dimensional face photo based on the framework of the forehead, ears, and chin; Expand the dataset using image enhancement methods; The expanded dataset is divided into training and testing sets.

6. The method for recognizing skeletal facial features from a face photograph based on a neural network according to claim 5, characterized in that: The ratio of training set to test set is 8:

2.

7. The method for recognizing skeletal facial features from a face photograph based on a neural network according to any one of claims 1-3, 5, and 6, characterized in that: Also includes: Grad-CAM reverse analysis of the Shuffle-Attention model to determine the soft tissue features of bony classification, indicating the soft tissue characteristics of bony type I, II, and III samples.

8. A system for recognizing facial bone structure from a facial photograph, characterized in that: include: The input module has three input units for inputting facial photos of the subject from the front, 45-degree side, and 90-degree side, respectively. The recognition module includes a Shuffle-Attention model, which is trained using the method described in any one of claims 1-7; The output module is used to output the recognition results.

9. The system for recognizing skeletal facial features from a facial photograph according to claim 8, characterized in that: It also includes the Grad-CAM module: used to analyze the soft tissue features of the Shuffle-Attention model to distinguish the bone type, and to indicate the soft tissue performance of bone type I, II and III samples; The output module is also used to output heat map prompts.

10. A method for recognizing skeletal facial features from facial photographs based on neural networks, characterized in that: The system described in claim 8 or 9 includes the following steps: The frontal, 45-degree side, and 90-degree side views of the subject are input into the Shuffle-Attention model. The Shuffle-Attention model recognizes the subject based on the three-angle facial photos and outputs the subject's skeletal classification. The frontal, 45-degree side, and 90-degree side views of the subject are input into the Grad-CAM module, which outputs a heat map of the subject, highlighting the characteristic soft tissue areas to inform the design of clinical aesthetic plans.

Images