Cascaded side face key point positioning and mandibular malformation assessment method and system

Abstract

The invention discloses a cascaded side face key point positioning and mandibular malformation evaluation method and system. The method comprises the following steps: detecting nose, lip and jaw regions in a side face image by using a deep learning method; then, side face contour features are accurately extracted by adopting an adaptive skin color segmentation technology, and needed medical key points are positioned; and finally, constructing a multi-index combined threshold value discrimination model to carry out mandibular malformation evaluation. Compared with a pure deep learning method, the positioning precision of the medical key points of the side face is higher, and the evaluation requirement of mandibular deformity can be met; the cascade architecture is low in computing power requirement and supports immediate evaluation of outpatient service; according to the mandibular malformation assessment method based on the double-triangle geometric model, data support is provided for hundreds of clinical cases, and the judgment threshold value is more robust; the method can be flexibly applied to the fields of oral cavity and mandibular surface malformation analysis, disease diagnosis, surgical planning and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image analysis and computer vision, and particularly relates to a cascaded side face key point positioning and mandibular deformity evaluation method and system. BACKGROUND

[0002] Maxillofacial deformity refers to the structural abnormalities or functional disorders of the skull and facial soft and hard tissues during development, which is a complex condition in oral and maxillofacial surgery with a global prevalence rate of up to 12%. Maxillofacial deformity can have a significant impact on the patient's facial appearance, occlusal function, respiratory system, and language ability, and can even lead to psychological and social problems in severe cases. In order to ensure the success of surgical treatment, accurate preoperative evaluation of the deformity is crucial. By analyzing the medical feature key points of the patient's side face, the patient's mandibular deformity can be accurately evaluated, which not only enables early intervention, but also provides precise guidance for surgical planning.

[0003] Currently, the evaluation of mandibular deformity based on side face medical key points still mainly adopts the manual annotation of key points, which is time-consuming and labor-intensive, and is greatly influenced by human subjective factors. Therefore, there is an urgent need in oral clinics to develop an automatic detection and positioning technology for side face image key points to assist clinicians in judging the patient's mandibular deformity and improve clinical diagnosis efficiency.

[0004] With the good performance of deep learning in image target detection and recognition tasks, there have been precedents for applying deep learning models to the field of oral medical image processing and analysis. For example, combining 3D classification criteria for maxillary and mandibular developmental abnormalities with analysis of facial soft tissue and occlusal relationship provides guidance for orthognathic surgery. For another example, CBCT technology is applied to evaluate complex craniofacial bone deformities, and a 3D surgical simulation method based on CBCT images is proposed, providing theoretical support for personalized surgical planning. And using the corrected active shape model to detect 12 anatomical landmark points, a method for automatically identifying anatomical abnormalities of the skull is proposed.

[0005] Although the above research has achieved certain results in oral medical image processing, the pure deep learning method has low recognition accuracy and requires a large amount of computing power, and false detection may lead to misjudgment of clinical decision-making, and insufficient computing power may delay surgical arrangements. The existing medical image key point detection methods are still not ideal for large-scale data sets and real-time clinical applications, and long training and reasoning time limits the application of these methods in actual clinical environments, and oral clinics urgently need an automatic system that integrates features, supports low-cost image input, achieves sub-millimeter-level positioning, and multi-parameter evaluation. SUMMARY

[0006] The present application aims to at least solve one of the technical problems existing in the related art to some extent.

[0007] One purpose of the present application is to provide a cascaded side face key point positioning and mandibular deformity evaluation method, which detects the nose, lip and chin regions in the side face image by using a deep learning method; then accurately extracts the side face contour features and positions the required medical key points by using an adaptive skin color segmentation technology; finally constructs a multi-index joint threshold discrimination model to evaluate the mandibular deformity, and completes the automatic positioning of the medical key points of the side face image and the evaluation of the mandibular deformity.

[0008] Another purpose of the present application is to provide a cascaded side face key point positioning and mandibular deformity evaluation system.

[0009] In order to achieve the above purpose, the present application provides a cascaded side face key point positioning and mandibular deformity evaluation method, comprising the following steps:

[0010] S1, collect side face images and perform preprocessing; unify the face orientation to the left, label the nose, lip and chin feature regions using a rectangular frame, use the processed side face images as image data, and use the rectangular frame coordinates with different nose, lip and chin labels as labels to construct a data set; use a target detection network based on deep learning to train the data set to obtain a feature region automatic detection model;

[0011] S2, input the side face image to be detected into the feature region automatic detection model to position the nose, lip and chin feature regions; and perform adaptive skin color segmentation processing in the boundary frame of each feature region to distinguish the skin region from the background, extract the connected domain maximum closure contour, and generate the face contour line of each feature region;

[0012] S3, divide the generated face contour line of each feature region into zones, extract the minimum horizontal coordinate point of the contour line in the nose feature region as the nose tip point; vertically divide the lip feature region into two zones, respectively extract the minimum horizontal coordinate points of the contour lines in the two zones as the upper lip point and the lower lip point; for the chin region, draw a vertical reference line through the nose tip point, extract the most forward tangent point of the reference line and the chin feature region contour line as the chin point; output the four-dimensional coordinate set of the nose tip point, the upper lip point, the lower lip point and the chin point;

[0013] S4, based on the four-dimensional coordinates of the nose tip point, the upper lip point, the lower lip point and the chin point, determine the multi-index joint threshold according to the clinical data statistical histogram to evaluate the mandibular deformity.

[0014] In the further preferred technical solution of the present application, the adaptive skin color segmentation processing in the boundary frame of each feature region in step S2 adopts a YCrCb color space dynamic threshold segmentation method, specifically:

[0015] The detected nose, lip and chin feature regions are processed respectively, the side face image in RGB color space is converted to YCrCb color space, Y represents brightness, Cr represents red component in light source, Cb represents blue component in light source, an initial segmentation threshold is set, the background color space at the edge of the image is extracted and the segmentation threshold is dynamically updated to exclude the background color space range, and then a binary skin color mask is generated;

[0016] The mathematical model for generating the binary skin color mask according to the segmentation threshold is:

[0017] ;

[0018] Wherein, represents the intensity value of the channel at the position in the YCrCb color space, represents the maximum and minimum values of the segmentation threshold vector; Then, an elliptical structure element fitting the biological features of the face contour is used for morphological optimization, an open-close operation combination is used to form a morphological filter, and a face contour line of the nose, lip and chin region is generated, each face contour line is composed of a point set.

[0019] As a preferred, the morphological filter formed by the open-close operation combination eliminates noise by open operation and performs hole repair by close operation;

[0020] Wherein, the open operation is divided into an erosion stage and a dilation stage, the erosion stage eliminates noise with an area smaller than A and shrinks the skin region boundary, separates the adhered regions, and is represented as:

[0021]

[0022] ;

[0023] ;

[0024] K represents an elliptical structure element, a and b represent the lengths of the semi-major axis and semi-minor axis of the elliptical structure element, represents the original binary image set, represents the result set obtained after the erosion operation, represents the area of the noise region; The dilation stage restores the original size of the effective region, and the calculation formula is:

[0025]

[0026] ;

[0027] Wherein, represents the result of the open operation,​​​​ This represents the coordinate offset of the structuring element K. Indicates an expansion operation;

[0028] The closing operation is divided into an expansion stage and an erosion stage. The expansion stage fills the holes with a diameter smaller than D and connects the fracture contour line, represented as:

[0029] ;

[0030] ;

[0031] in, This represents the set after dilation of the original binary image set M. This indicates the diameter of the holes in the original image. Represents the minor semi-axis in an elliptical structural element;

[0032] The calculation formula for the raised boundary caused by smooth expansion during the corrosion stage is:

[0033] ;

[0034] in, This represents a compound closed-loop operation that first performs an expansion operation and then an erosion operation. This indicates a corrosion operation.

[0035] Preferably, in step S3, the point with the minimum horizontal coordinate in the contour line within the nasal feature region is extracted as the nasal tip point; specifically:

[0036] Traverse the entire point set of the nose feature region contour line, take the point with the minimum horizontal coordinate x and add it to the candidate point set, and then take the point with the largest y value from the candidate point set as the nose tip point.

[0037] In step S3, the lip feature region is vertically divided into upper and lower equal parts, and the minimum horizontal coordinate point in the contour line of each part is extracted as the upper lip point and lower lip point; specifically:

[0038] Divide the lip feature region into upper and lower lips equally along the vertical midline. Then, take the point with the minimum horizontal coordinate x from all the points of the outline of the upper and lower lip feature regions and add them to the upper lip candidate point set and the lower lip candidate point set respectively. Then, take the point with the largest y value from the upper lip candidate point set and the lower lip candidate point set respectively as the upper lip point and the lower lip point.

[0039] In step S3, for the chin area, a vertical baseline is drawn through the tip of the nose, and the foremost tangent point between this baseline and the contour line of the chin feature area is extracted as the chin point; specifically:

[0040] First, add all points of the contour line of the nose feature area to the point set, draw a straight line through the already determined tip of the nose, and rotate the straight line until it is tangent to the chin contour line, and take the tangent point as the chin point.

[0041] Preferably, the four-dimensional coordinates of the tip of the nose, the upper lip point, the lower lip point and the chin point are used to determine a multi-index combined threshold according to a clinical data histogram, and the jaw deformity condition is evaluated; specifically:

[0042] First, a first feature triangle is formed by the tip of the nose, the upper lip point and the chin point, and an angle at the upper lip point is calculated and denoted as ∠A; a second feature triangle is formed by the upper lip point, the lower lip point and the chin point, and an angle at the lower lip point is calculated and denoted as ∠B;

[0043] The multi-index combined threshold is determined according to a clinical data histogram, and ∠A and ∠B are compared with the determined multi-index combined threshold to evaluate the jaw deformity condition.

[0044] Preferably, the target detection network based on deep learning in step S1 adopts a YOLOv5 model;

[0045] When training with the data set, a multi-task learning strategy is adopted to simultaneously optimize the rectangular frame loss , the confidence loss and the classification loss ; wherein the rectangular frame loss uses a complete intersection ratio loss function, the confidence loss and the classification loss use a cross-entropy loss.

[0046] Another aspect of the present application provides a cascaded side face key point positioning and jaw deformity evaluation system, comprising:

[0047] A data processing module is used to collect side face images and perform preprocessing; the face orientation is unified to the left, the nose, lip and chin feature areas are labeled with rectangular frames, the processed side face images are used as image data, and the rectangular frame coordinates with different nose, lip and chin labels are used as labels to construct a data set.

[0048] A feature area detection module is used to adopt a target detection network based on deep learning, train with the data set to obtain a feature area automatic detection model; input the side face image to be detected into the feature area automatic detection model to locate the nose, lip and chin feature areas

[0049] A face contour line generation module is used to perform adaptive skin color segmentation processing within the boundary frame of each feature area, distinguish the skin area from the background, extract the connected domain maximum closure contour, and generate the face contour line of each feature area.

[0050] A key point positioning module is configured to perform zoning processing on the generated face contour line of each feature region, extract a minimum horizontal coordinate point in the contour line as a nose tip point in a nose feature region, vertically divide a lip feature region into two sub-zones, and extract minimum horizontal coordinate points in the contour lines of the two sub-zones as an upper lip point and a lower lip point, respectively, draw a vertical reference line through the nose tip point for a chin region, and extract a tangent point of the reference line and a contour line of the chin feature region as a chin point, and output a four-dimensional coordinate set of the nose tip point, the upper lip point, the lower lip point and the chin point.

[0051] A mandibular deformity evaluation module is configured to determine a multi-index combined threshold value according to a clinical data statistical histogram based on the four-dimensional coordinates of the nose tip point, the upper lip point, the lower lip point and the chin point, and perform mandibular deformity condition evaluation.

[0052] In another aspect, the application provides a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to perform the cascade side face key point positioning and mandibular deformity evaluation method.

[0053] In yet another aspect, the application provides an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus, and the processor invokes a logic instruction in the memory to perform the cascade side face key point positioning and mandibular deformity evaluation method.

[0054] In still another aspect, the application provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program being executed by a processor to cause a computer to perform the cascade side face key point positioning and mandibular deformity evaluation method.

[0055] Beneficial effects: The cascade side face key point positioning and mandibular deformity evaluation method of the application has higher positioning accuracy of side face medical key points compared to pure deep learning methods by fusing region detection based on deep learning and skin color segmentation methods based on non-deep learning, and can meet the evaluation requirements of mandibular deformity; the cascade architecture has low requirements for computing power and supports outpatient immediate evaluation; the mandibular deformity evaluation method based on a double-triangle geometric model has hundreds of clinical cases as data support, and the decision threshold is more robust; the method can be flexibly applied to the fields of oral mandibular deformity analysis, disease diagnosis and surgical planning, etc. BRIEF DESCRIPTION OF DRAWINGS

[0056] Fig. 1 A flowchart of the cascade side face key point positioning and mandibular deformity evaluation method of the application;

[0057] Fig. 2 A flowchart of the mandibular deformity condition evaluation of the application. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. They should not be understood as limiting the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application. In the description of the present application, it should be understood that the terms used are only for the purpose of description and should not be understood as indicating or implying the relative importance.

[0059] The present application provides a method for cascade positioning of side face key points and evaluation of mandibular deformity. Figs. 1-2 The present application provides a method for cascade positioning of side face key points and evaluation of mandibular deformity.

[0060] Embodiment 1: The present embodiment provides a method for cascade positioning of side face key points and evaluation of mandibular deformity, comprising the following steps:

[0061] Step 1, data preprocessing:

[0062] 300 side face images were collected and flipped to ensure that the faces were uniformly oriented to the left. The image format and resolution size were unified to facilitate subsequent data processing and use of deep learning models. The nose, lip and chin regions were manually labeled using a rectangular frame. The labeling results were saved as the coordinates of the upper left corner and lower right corner of the rectangular frame . The processed side face images were used as image data, and the rectangular frame coordinates with different nose, lip and chin labels were used as labels to construct a data set.

[0063] Step 2, model training and feature region detection:

[0064] The YOLOv5 model was used to train the labeled side face images. After training, the model was saved, and the newly collected side face images to be tested were input into the trained model to automatically detect the nose, lip and chin position region frame and output the coordinates of the upper left corner and lower right corner of the region frame .

[0065] The model training adopts a multi-task learning strategy, simultaneously optimizing the rectangular frame loss , confidence loss and classification loss ; the rectangular frame loss uses the Complete Intersection OverUnion (CIOU) loss function, the confidence loss and the classification loss The loss function expression is as follows using the cross-entropy loss (Binary CrossEntropy, BCE) loss:

[0066] ;

[0067] ;

[0068] ;

[0069] wherein p is the distance between the center points of the real frame and the predicted frame, c is the diagonal length of the minimum enclosing rectangle of the real frame and the predicted frame, v is the aspect ratio similarity of the real frame and the predicted frame, and a is the influence factor of v;

[0070] ;

[0071] wherein N is the number of samples in the small batch, y∈[0,1] represents the input label data, ∈[0,1] is the confidence of the classification of sample j, and the total loss function is the weighted sum of the above three losses to balance the learning of the positioning and classification tasks.

[0072] Step 3, skin color segmentation and face contour line generation:

[0073] Adaptive skin color segmentation processing is performed in the feature region bounding box based on the YCrCb color space, and the detected nose, lip and chin feature regions are processed respectively. The side face image in the RGB color space is converted to the YCrCb color space. Y represents brightness, Cr represents the red component in the light source, and Cb represents the blue component in the light source. An initial segmentation threshold is set, the background color space at the edge of the image is extracted, and the segmentation threshold is dynamically updated to exclude the background color space range, and then a binary skin color mask is generated;

[0074] The mathematical model for generating the binary skin color mask according to the segmentation threshold is:

[0075] ;

[0076] wherein represents the intensity value of the channel at in the YCrCb color space, and represent the maximum and minimum values of the segmentation threshold vector;

[0077] Then, an elliptical structure element that fits the biological features of the face contour is used for morphological optimization, and an open-close operation combination is used to form a morphological filter. The morphological filter formed by the open-close operation combination eliminates noise by open operation and performs hole repair by close operation.

[0078] wherein the opening operation is divided into an erosion stage and a dilation stage, the erosion stage eliminates noise with an area less than A and shrinks the boundary of the skin region, separates the adhered regions, and is expressed as:

[0079] ;

[0080] ;

[0081] K represents an elliptical structural element, a and b represent the lengths of the major and minor semi-axes of the elliptical structural element, M represents a set of original binary images, M represents a set of original binary images, M represents a set of original binary images, A represents the area of the noise region;

[0082] The dilation stage restores the original size of the effective region, and the calculation formula is:

[0083] ;

[0084] wherein, M represents the result of the opening operation, K represents the coordinate offset of the structural element K, M represents the result of the dilation operation;

[0085] The closing operation is divided into a dilation stage and an erosion stage, the dilation stage fills the holes with a diameter less than D and connects the broken contour lines, and is expressed as:

[0086] ;

[0087] ;

[0088] wherein, M represents the set after dilation of the original set of binary images M, D represents the diameter of the hole in the original image, b represents the minor semi-axis in the elliptical structural element;

[0089] The erosion stage smooths the convex boundary caused by dilation, and the calculation formula is:

[0090] ;

[0091] wherein, M represents the composite closing operation process of first performing the dilation operation and then performing the erosion operation, M represents the erosion operation.

[0092] Finally, the face contour mask of the nose, lips, and chin region is generated, and these face contours are composed of point sets, which provide support for positioning the medical key points in the next step.

[0093] Step 4, key point detection:

[0094] The generated facial contour line of each feature region is partitioned. In the nose feature region, the point set of the nose feature region contour line is traversed, the point with the minimum horizontal coordinate x is added to the candidate point set, and the point with the maximum y value in the candidate point set is taken as the nose tip point. The lip feature region is equally divided into upper and lower lips according to the vertical center line, and the points with the minimum horizontal coordinate x in the point sets of the contour lines of the upper and lower lip feature regions are added to the upper and lower lip candidate point sets, respectively. Then, the points with the maximum y value in the upper and lower lip candidate point sets are taken as the upper and lower lip points, respectively. For the chin region, the point set of the nose feature region contour line is added to the point set, a straight line is drawn through the nose tip point, and the tangent point of the chin contour line is taken as the chin point after rotating the straight line until it is tangent to the chin contour line. The four-dimensional coordinate set of the nose tip point, the upper lip point, the lower lip point, and the chin point is output.

[0095] Step 5, joint threshold decision:

[0096] A first feature triangle is formed by the nose tip point, the upper lip point, and the chin point, and the included angle at the upper lip point is calculated and denoted as ∠A. A second feature triangle is formed by the upper lip point, the lower lip point, and the chin point, and the included angle at the lower lip point is calculated and denoted as ∠B. If the upper lip point is located to the right of the line connecting the nose tip point and the chin point, ∠A is positive, otherwise it is negative.

[0097] The multi-index joint threshold is determined according to the hundred-person-level clinical data statistical histogram, which is the first threshold a, the second threshold b, the third threshold c, the fourth threshold d, and the fifth threshold e, respectively.

[0098] If ∠A is positive: judge the size of ∠A and the first threshold a ∠A > a, decide to be temporarily negative, ∠A ≤ a, decide to be temporarily positive, use the second threshold b to judge the temporarily negative sample, if ∠B ≤ b, decide to be Ⅲ type deformity, if ∠B > b, decide to be Ⅰ type negative, use the third threshold c to judge the temporarily positive sample, if ∠B ≤ c, decide to be Ⅲ type deformity, if ∠B > c, decide to be Ⅰ type negative.

[0099] If ∠A is negative, use the fourth threshold d to judge, if ∠A > d, decide to be temporarily negative, if ∠A ≤ d, decide to be Ⅱ type deformity directly, use the fifth threshold e to judge the temporarily negative sample, if ∠B ≤ e, decide to be Ⅱ type deformity, if ∠B > e, decide to be Ⅰ type negative.

[0100] The labeled image can be used for downstream applications such as anatomical analysis, disease diagnosis, or surgical planning.

[0101] In the aspect of oral mandibular facial anatomy: this method can construct part of the patient's mandibular anatomical information by accurately detecting the key points in the patient's side image. It can be used to analyze the spatial position relationship of the soft tissue around the mandible. It provides reliable data support for the pathological structure research of patients with jaw deformity, such as symmetry analysis of dental deformity and bone structure evaluation of joint disorder.

[0102] In the aspect of disease diagnosis: based on key point detection, it can assist doctors in accurately diagnosing three types of mandibular deformities. For common diseases such as class II and class III deformities, it provides intuitive image evaluation and diagnosis basis through key point displacement and anatomical structure changes.

[0103] In the aspect of surgical planning: in jaw surgery (such as orthognathic surgery, implant surgery, and jaw reconstruction surgery), this method can be used for preoperative accurate planning and intraoperative real-time guidance. By detecting the preoperative mandibular facial key point position of the patient, it helps doctors design the best surgical path and cutting scheme to ensure the accuracy and safety of the surgery.

[0104] In addition, this method can also be used for postoperative efficacy evaluation. By comparing the preoperative and postoperative key point position and structure changes, it quantitatively analyzes the surgical results. During the postoperative rehabilitation process, it continuously monitors the key point displacement and timely discovers potential problems such as postoperative complications or abnormal structure recovery.

[0105] Embodiment 2: a cascaded side face key point positioning and mandibular deformity evaluation system, comprising:

[0106] A data processing module is used to collect side face images and pre-process them; unify the face orientation to the left, label the nose, lip, and chin feature areas with a rectangular frame, use the processed side face images as image data, and the rectangular frame coordinates with different nose, lip, and chin labels as labels to construct a data set;

[0107] A feature area detection module is used to adopt a target detection network based on deep learning, train it using the data set, and obtain a feature area automatic detection model; input the side face image to be detected into the feature area automatic detection model to locate the nose, lip, and chin feature areas

[0108] A face contour line generation module is used to perform adaptive skin color segmentation processing within the boundary frame of each feature area, distinguish the skin area from the background, extract the maximum closed contour of the connected domain, and generate the face contour line of each feature area;

[0109] a key point positioning module, configured to perform partition processing on the generated face contour line of each feature region, extract a minimum horizontal coordinate point in the contour line as a tip of the nose in the nose feature region, vertically divide the lip feature region into two equal parts, and extract minimum horizontal coordinate points in the contour lines of the two parts as upper and lower lip points, draw a vertical reference line through the tip of the nose for the chin region, and extract a tangent point of the reference line and the contour line of the chin feature region as a chin point, and output a four-dimensional coordinate set of the tip of the nose, the upper lip point, the lower lip point, and the chin point;

[0110] a lower jaw deformity evaluation module, configured to determine a multi-index joint threshold according to a hundred-person-level clinical data statistical histogram based on the four-dimensional coordinates of the tip of the nose, the upper lip point, the lower lip point, and the chin point, and perform lower jaw deformity condition evaluation.

[0111] Embodiment 3: The embodiment provides a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to execute a cascaded side face key point positioning and lower jaw deformity evaluation method, the method comprising the following steps:

[0112] S1, collect side face images and perform preprocessing, unify the face orientation to the left, use a rectangular frame to label nose, lip, and chin feature regions, use the processed side face images as image data, and use the rectangular frame coordinates with different nose, lip, and chin labels as labels to construct a data set, use a target detection network based on deep learning to train the data set to obtain a feature region automatic detection model;

[0113] S2, input the side face images to be detected into the feature region automatic detection model to locate the nose, lip, and chin feature regions, perform adaptive skin color segmentation processing in the boundary frames of the feature regions to distinguish the skin region and the background, extract the maximum closed contour of the connected domain, and generate face contour lines of the feature regions;

[0114] S3, perform partition processing on the generated face contour lines of each feature region, extract a minimum horizontal coordinate point in the contour line as a tip of the nose in the nose feature region, vertically divide the lip feature region into two equal parts, and extract minimum horizontal coordinate points in the contour lines of the two parts as upper and lower lip points, draw a vertical reference line through the tip of the nose for the chin region, and extract a tangent point of the reference line and the contour line of the chin feature region as a chin point, and output a four-dimensional coordinate set of the tip of the nose, the upper lip point, the lower lip point, and the chin point;

[0115] S4, determine a multi-index joint threshold according to a hundred-person-level clinical data statistical histogram based on the four-dimensional coordinates of the tip of the nose, the upper lip point, the lower lip point, and the chin point, and perform lower jaw deformity condition evaluation.

[0116] Embodiment 4: The electronic device can include a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other through the communications bus. The processor can invoke a logical instruction in the memory to execute a cascaded side face key point positioning and mandibular deformity evaluation method, which includes the following steps:

[0117] S1, collect side face images and perform preprocessing; unify the face orientation to the left, label the nose, lip, and chin feature regions using a rectangular frame, use the processed side face images as image data, and use the rectangular frame coordinates with different nose, lip, and chin labels as labels to construct a data set; use a target detection network based on deep learning to train the data set to obtain a feature region automatic detection model;

[0118] S2, input the side face image to be detected into the feature region automatic detection model to locate the nose, lip, and chin feature regions; and perform adaptive skin color segmentation processing within the boundary frame of each feature region to distinguish the skin region from the background, extract the connected domain maximum closure contour, and generate a face contour line for each feature region;

[0119] S3, perform partitioning processing on the generated face contour line of each feature region, extract the minimum horizontal coordinate point in the contour line as the nose tip point in the nose feature region; vertically divide the lip feature region into two equal parts, and extract the minimum horizontal coordinate points in the contour lines of the two partitions as the upper lip point and the lower lip point; for the chin region, draw a vertical reference line through the nose tip point, and extract the most forward tangent point of the reference line and the chin feature region contour line as the chin point; output the four-dimensional coordinate set of the nose tip point, the upper lip point, the lower lip point, and the chin point;

[0120] S4, based on the four-dimensional coordinates of the nose tip point, the upper lip point, the lower lip point, and the chin point, determine a multi-index joint threshold according to a hundred-person level clinical data statistical histogram to evaluate the mandibular deformity condition.

[0121] In addition, the logic instructions in the memory described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0122] Embodiment 5: The present embodiment provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer-readable storage medium, and the computer program being executable by a processor, so that a computer can execute a cascaded side face key point positioning and mandibular deformity evaluation method, the method comprising the following steps:

[0123] S1, collect side face images and perform preprocessing; unify the face orientation to the left, label the nose, lip and chin feature regions using a rectangular frame, use the processed side face images as image data, and use the rectangular frame coordinates with different nose, lip and chin labels as labels to construct a data set; use a target detection network based on deep learning to train the data set to obtain a feature region automatic detection model;

[0124] S2, input the side face image to be detected into the feature region automatic detection model to locate the nose, lip and chin feature regions; and perform adaptive skin color segmentation processing within the boundary frame of each feature region to distinguish the skin region from the background, extract the connected domain maximum closure contour, and generate the face contour line of each feature region;

[0125] S3, perform partitioning processing on the generated face contour line of each feature region, extract the minimum value point of the horizontal coordinates in the contour line as the nose tip point in the nose feature region; vertically divide the lip feature region into two equal parts, and extract the minimum value points of the horizontal coordinates in the contour lines of the two partitions as the upper lip point and the lower lip point; for the chin region, draw a vertical reference line through the nose tip point, and extract the most forward tangent point of the reference line and the chin feature region contour line as the chin point; output the four-dimensional coordinate set of the nose tip point, the upper lip point, the lower lip point and the chin point;

[0126] S4, based on the four-dimensional coordinates of the nose tip point, upper lip point, lower lip point and chin point, determining a multi-index combined threshold according to a hundred-person level clinical data statistical histogram to perform the lower jaw deformity condition assessment.

[0127] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0128] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0129] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cascaded method for locating key facial features and assessing mandibular deformities, characterized in that, Includes the following steps: S1. Collect side profile images and preprocess them; unify the face orientation to the left, use rectangles to label the nose, lips and chin feature regions, use the processed side profile images as image data, and use the coordinates of the rectangles with different nose, lip and chin labels as labels to construct a dataset; use a deep learning-based object detection network, use the dataset to train it, and obtain an automatic feature region detection model. S2. Input the profile image to be detected into the feature region automatic detection model to locate the nose, lips and chin feature regions; and perform adaptive skin color segmentation within the bounding boxes of each feature region to distinguish the skin region from the background, extract the maximum closure contour of the connected component, and generate the face contour line of each feature region. S3. Divide the generated facial contour lines of each feature region into regions. In the nose feature region, extract the minimum horizontal coordinate point of the contour line as the nose tip point. Divide the lip feature region vertically into upper and lower equal regions, and extract the minimum horizontal coordinate point of the contour lines of the two regions as the upper lip point and lower lip point, respectively. For the chin region, draw a vertical baseline through the nose tip point, and extract the foremost tangent point between the baseline and the chin feature region contour line as the chin point. Output the four-dimensional coordinate set of the nose tip point, upper lip point, lower lip point, and chin point. S4. Based on the four-dimensional coordinates of the nasal tip, upper lip, lower lip and chin points, determine the combined threshold of multiple indicators according to the histogram of clinical data, and evaluate the mandibular deformity.

2. The cascaded method for locating key facial features and assessing mandibular deformities according to claim 1, characterized in that, In step S2, adaptive skin color segmentation is performed within the bounding boxes of each feature region using a dynamic thresholding method in the YCrCb color space. Specifically: The detected nose, lip, and chin feature regions are processed separately. The profile image in RGB color space is converted to YCrCb color space, where Y represents brightness, Cr represents the red component of the light source, and Cb represents the blue component of the light source. An initial segmentation threshold is set, the background color space at the image edge is extracted, and the segmentation threshold is dynamically updated to exclude the background color space range, thereby generating a binary skin color mask. The mathematical model for generating a binary skin color mask based on the segmentation threshold is as follows: ； in, In the YCrCb color space passage The intensity value, and This represents the maximum and minimum values ​​of the segmentation threshold vector; Then, morphological optimization is performed using elliptical structuring elements that fit the biological characteristics of facial contours. Open-closed operations are combined to form morphological filters to generate facial contour lines for the nose, lips, and chin regions. Each facial contour line is composed of a set of points.

3. The cascaded method for locating key facial features and assessing mandibular deformities according to claim 2, characterized in that, The morphological filter formed by the combination of opening and closing operations uses the opening operation to eliminate noise and the closing operation to repair holes. The opening operation is divided into an erosion stage and an expansion stage. The erosion stage eliminates noise with an area smaller than A and shrinks the skin region boundary, separating the adhesion region, as shown below: ； ； K represents the elliptical structural element, and a and b represent the semi-major and semi-minor axis lengths of the elliptical structural element. Represents the original set of binary images. Indicates to The set of results obtained after the etching operation. Indicates the area of ​​the noise region; The expansion phase restores the original size of the effective region, calculated using the following formula: ； in, This represents the result of the opening operation. This represents the coordinate offset of the structuring element K. Indicates an expansion operation; The closing operation is divided into an expansion stage and an erosion stage. The expansion stage fills the holes with a diameter smaller than D and connects the fracture contour line, represented as: ； ； in, This represents the set after dilation of the original binary image set M. This indicates the diameter of the holes in the original image. Represents the minor semi-axis in an elliptical structural element; The calculation formula for the raised boundary caused by smooth expansion during the corrosion stage is: ； in, This represents a compound closed-loop operation that first performs an expansion operation and then an erosion operation. This indicates a corrosion operation.

4. The cascaded method for locating key facial features and assessing mandibular deformities according to claim 2, characterized in that, In step S3, within the nasal feature region, the point with the minimum horizontal coordinate in the contour line is extracted as the nasal tip point; specifically: Traverse the entire point set of the nose feature region contour line, take the point with the minimum horizontal coordinate x and add it to the candidate point set, and then take the point with the largest y value from the candidate point set as the nose tip point. In step S3, the lip feature region is vertically divided into upper and lower equal parts, and the minimum horizontal coordinate point in the contour line of each part is extracted as the upper lip point and lower lip point; specifically: Divide the lip feature region into upper and lower lips equally along the vertical midline. Then, take the point with the minimum horizontal coordinate x from all the points of the outline of the upper and lower lip feature regions and add them to the upper lip candidate point set and the lower lip candidate point set respectively. Then, take the point with the largest y value from the upper lip candidate point set and the lower lip candidate point set respectively as the upper lip point and the lower lip point. In step S3, for the chin area, a vertical baseline is drawn through the tip of the nose, and the foremost tangent point between this baseline and the contour line of the chin feature area is extracted as the chin point; specifically: First, add all the points of the nose feature region outline to the point set. Draw a straight line through the already determined nose tip point, and rotate the straight line until it is tangent to the chin outline. Take the tangent point as the chin point.

5. The cascaded method for locating key facial features and assessing mandibular deformities according to claim 1, characterized in that, Step S4 describes the assessment of mandibular deformity based on the four-dimensional coordinates of the nasal tip, upper lip, lower lip, and chin points, and the determination of a multi-indicator combined threshold according to the histogram of clinical data. Specifically: First, construct the first characteristic triangle using the tip of the nose, the upper lip, and the chin. Calculate the included angle at the upper lip and denote it as ∠A. Then, construct the second characteristic triangle using the upper lip, lower lip, and chin and calculate the included angle at the lower lip and denote it as ∠B. The combined threshold of multiple indicators is determined based on the statistical histogram of clinical data. ∠A and ∠B are compared with the determined combined threshold of multiple indicators to evaluate the mandibular deformity.

6. The cascaded method for locating key facial features and assessing mandibular deformities according to claim 1, characterized in that, The deep learning-based object detection network described in step S1 uses the YOLOv5 model; When training using the dataset, a multi-task learning strategy is employed, while simultaneously optimizing the bounding box loss. Confidence loss and classification loss Among them, the rectangular box loss uses the perfect intersection-union loss function, and the confidence loss... and classification loss Use cross-entropy loss.

7. A cascaded system for locating key facial features and assessing mandibular deformities, characterized in that, include: The data processing module is used to collect profile images and perform preprocessing. All faces are oriented to the left. The nose, lips, and chin feature regions are labeled with rectangles. The processed profile images are used as image data, and the coordinates of the rectangles with different nose, lip, and chin labels are used as labels to construct a dataset. The feature region detection module is used to train a deep learning-based object detection network using the dataset to obtain an automatic feature region detection model. The side profile image to be detected is input into the automatic feature region detection model to locate the nose, lips, and chin feature regions. The face contour generation module is used to perform adaptive skin color segmentation within the bounding boxes of each feature region, distinguish the skin region from the background, extract the maximum closure contour of the connected components, and generate the face contour of each feature region. The key point localization module is used to partition the facial contour lines of each generated feature region. In the nose feature region, the minimum horizontal coordinate point in the contour line is extracted as the nose tip point; the lip feature region is vertically divided into upper and lower equal parts, and the minimum horizontal coordinate point in the contour lines of the two parts is extracted as the upper lip point and lower lip point; for the chin region, a vertical baseline is drawn through the nose tip point, and the foremost tangent point between the baseline and the chin feature region contour line is extracted as the chin point; the four-dimensional coordinate set of the nose tip point, upper lip point, lower lip point, and chin point is output. The mandibular deformity assessment module is used to assess mandibular deformity based on the four-dimensional coordinates of the nasal tip, upper lip, lower lip and chin points, and to determine the combined threshold of multiple indicators according to the histogram of clinical data.

8. A non-transitory computer-readable storage medium, characterized in that, It stores computer instructions that cause the computer to execute the cascaded method for locating key facial features and assessing mandibular deformities as described in any one of claims 1-7.

9. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor calls logical instructions from the memory to execute the cascaded method for locating key facial points and assessing mandibular deformities as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer performs the cascaded method for locating key facial points and assessing mandibular deformities as described in any one of claims 1-7.

Images