Facial data processing method and apparatus, electronic device, and storage medium

Abstract

The application provides a face data processing method and device, electronic equipment and storage medium, and relates to the technical field of face data. The method comprises the following steps: performing spatial configuration based on original coordinates of a plurality of key points obtained from an original 3D image, constructing an original configuration and a mirror configuration corresponding to the original configuration; obtaining a median sagittal plane corresponding to the original 3D image based on a target distance between the original configuration and the mirror configuration; and performing alignment processing on the original 3D image based on the median sagittal plane to obtain an aligned original 3D image, so as to perform face detection on a user's face. Based on the above scheme, the accuracy of the median sagittal plane construction can be improved, so that doctors can diagnose facial deformities through the median sagittal plane with high accuracy, which can improve the accuracy of the doctors' diagnosis of facial deformities, thereby being beneficial to the treatment of facial deformities by the doctors and the evaluation of the postoperative treatment effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of facial data, in particular to a facial data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] When facial deformities are eliminated through facial reconstruction surgery, the diagnosis of facial deformities needs to be made by constructing a median sagittal plane. However, if the diagnosis of facial deformities is made by a median sagittal plane with poor accuracy, it may lead to a decrease in the accuracy of the diagnosis of facial deformities by doctors, thereby being not conducive to the treatment of facial deformities by doctors and the evaluation of postoperative treatment effects.

[0003] Therefore, how to improve the accuracy of the construction of the median sagittal plane is a problem to be solved at present. SUMMARY

[0004] The present application provides a facial data processing method, device, electronic equipment and storage medium, which can improve the accuracy of the construction of the median sagittal plane, so that doctors make a diagnosis of facial deformities by a median sagittal plane with high accuracy, which can improve the accuracy of the diagnosis of facial deformities by doctors, thereby being conducive to the treatment of facial deformities by doctors and the evaluation of postoperative treatment effects.

[0005] In a first aspect, the present application provides a facial data processing method, which comprises:

[0006] Based on the original 3D image, the original coordinates of a plurality of key points are obtained, wherein the original 3D image represents a tomographic image of a user's face, and the plurality of key points represent position points corresponding to craniofacial tissues of the user's face;

[0007] Based on the original coordinates of the plurality of key points, a spatial configuration is made to construct an original configuration;

[0008] Based on the mirror coordinates of the mirror points corresponding to the plurality of key points, a spatial configuration is made to construct a mirror configuration;

[0009] Based on the target distance between the original configuration and the mirror configuration, a median sagittal plane corresponding to the original 3D image is obtained, wherein the degree of coincidence between the original configuration and the mirror configuration is negatively correlated with the target distance;

[0010] Based on the median sagittal plane, the original 3D image is aligned to obtain an aligned original 3D image, wherein the aligned original 3D image is used for facial detection of the user's face.

[0011] In this embodiment, when acquiring the original 3D image representing the user's face using a tomographic scan, multiple key points corresponding to the locations of the craniofacial tissues of the user's face can be obtained from the original 3D image. An original configuration corresponding to the original coordinates is constructed using the original coordinates of these key points, and a mirror configuration corresponding to the mirror coordinates is constructed using the mirror coordinates of the mirror points corresponding to each of these key points. The distance between the constructed original configuration and the mirror configuration is then used to obtain the midsagittal plane corresponding to the original 3D image. The original 3D image is then aligned using this midsagittal plane to obtain the aligned original 3D image, which is used for facial deformity detection of the user's face. Compared to existing methods that directly select multiple sets of symmetrical points on the craniofacial tissues from the original 3D image to construct the midsagittal plane, which may lead to a dependence on local symmetry points in the midsagittal plane construction, and if the accuracy of the selected symmetrical points is insufficient or the number is too small, it will directly reduce the accuracy of the midsagittal plane. This application's embodiment, through the original configuration and mirror configuration, frees the midsagittal plane construction from dependence on local symmetry points. It does not require precisely finding mutually symmetrical points on the craniofacial tissues, but instead determines the original configuration and mirror configuration of the overall space corresponding to the craniofacial tissues through multiple points on the craniofacial tissues. By constructing the midsagittal plane using morphology, both symmetrical and asymmetrical points can be involved in the construction of the original and mirror morphologies. This results in a midsagittal plane that better represents the overall symmetry of the craniofacial tissues, more closely resembling the anatomically functional midsagittal plane rather than a mathematically absolute symmetrical plane. This makes it more realistic and better suited to asymmetrical faces, thus improving the accuracy of midsagittal plane construction. This allows doctors to diagnose facial deformities using a more accurate midsagittal plane, improving the precision of their diagnosis and facilitating the treatment and postoperative evaluation of facial deformities.

[0012] Furthermore, since both symmetrical and asymmetrical points can participate in the construction of the original configuration and the mirror configuration, and are not limited to symmetrical points on the craniofacial tissues, more key points of the craniofacial tissues can be selected. By using more key points of the craniofacial tissues, the error of the selected points can be avoided from being amplified, thereby obtaining a more accurate midsagittal plane.

[0013] Even more remarkably, when a user's face is severely deformed and there are no symmetrical points in the craniofacial tissues, the midsagittal plane can be constructed by using the original and mirror configurations determined by multiple points on the user's craniofacial tissues. This frees the construction of the midsagittal plane from the constraint that it must rely on the symmetrical points of the craniofacial tissues, thus enabling the free construction of the midsagittal plane.

[0014] In the method, the multiple sets of symmetrical points on the craniofacial tissue are directly selected from the original 3D image, and the left and right symmetrical points on the craniofacial tissue must be accurately found. However, the absolutely symmetrical craniofacial tissue is extremely difficult to exist in the real human body, and at most can only be approximately symmetrical. Therefore, the symmetrical points directly selected from the original 3D image will have differences more or less. If the multiple sets of symmetrical points selected from the original 3D image are used to construct the median sagittal plane, the accuracy of the median sagittal plane will be reduced. For example, the left infraorbital point is 0.1 mm lower than the right infraorbital point. The median sagittal plane constructed by using the left infraorbital point and the right infraorbital point will be slightly offset to the right side. In addition, the multiple sets of symmetrical points on the craniofacial tissue directly selected from the original 3D image will also have the problem of less symmetrical points. If the number of symmetrical points is small (for example, 2-3 sets of symmetrical points), the error of the selected points may be magnified due to the small number of symmetrical points, which reduces the accuracy of the median sagittal plane constructed by using the small number of symmetrical points in the original 3D image. Or, when the symmetrical points of the symmetrical craniofacial tissue do not exist in the original 3D image, the median sagittal plane cannot be constructed.

[0015] With reference to the first aspect, in some implementations of the first aspect, the constructing the mirror configuration based on the mirror coordinates of the multiple sets of mirror points comprises:

[0016] determining a mean value of the original coordinates of the multiple key points in a first dimension, wherein the first dimension represents a horizontal dimension of the original 3D image;

[0017] determining an original plane corresponding to the mean value in the first dimension;

[0018] performing horizontal mirror flipping on the original coordinates of the multiple key points with the original plane as a center to obtain mirror coordinates of the multiple mirror points;

[0019] constructing the mirror configuration based on the mirror coordinates of the multiple mirror points.

[0020] In the embodiments of the present application, since the mean value of the original coordinates of the plurality of key points in the horizontal dimension is essentially the center of gravity of the plurality of key points in the horizontal dimension, the specific position information of the plurality of key points in the horizontal dimension can offset the random error of a small number of points, thereby reflecting the symmetry center trend of the plurality of key points in the horizontal dimension as a whole. Therefore, the center plane (i.e., the original plane) determined by the mean value of the plurality of key points in the horizontal dimension is more accurate, so that the mirror coordinates of the plurality of mirror points obtained by horizontally mirroring the original coordinates of the plurality of key points with respect to the center plane are also more accurate. Based on the more accurate mirror coordinates of the plurality of mirror points, the mirror configuration constructed by the mirror coordinates of the plurality of mirror points is also more accurate. Furthermore, a more accurate median sagittal plane can be constructed by the original configuration and the more accurate mirror configuration.

[0021] In addition, when constructing the center plane, it is not necessary to first accurately find the points on the craniofacial structure that are symmetric to each other. Instead, the center plane is determined by the mean value of the plurality of points on the craniofacial structure in the horizontal dimension. This can make the center plane constructed by the symmetric points and the asymmetric points more representative of the overall symmetry trend of the craniofacial structure, more realistic, and more suitable for asymmetric faces, thereby improving the accuracy of the construction of the center plane.

[0022] Furthermore, when the user's face is too deformed and the craniofacial structure does not have symmetric points, the center plane can still be determined by the mean value of the plurality of points on the craniofacial structure in the horizontal dimension. The construction of the center plane is no longer limited by the symmetric points of the craniofacial structure, which realizes the free construction of the center plane and the free construction of the mirror configuration.

[0023] In combination with the first aspect and the above implementation manners, in some implementation manners of the first aspect, the above obtaining the median sagittal plane corresponding to the original 3D image based on the target distance between the original configuration and the mirror configuration comprises:

[0024] normalizing the reciprocal of the target distance to obtain the target weight corresponding to each key point;

[0025] obtaining the median sagittal plane based on the target weight corresponding to each key point.

[0026] In the embodiments of the present application, the key points with small influence degree on facial deformity are given higher weights and the key points with large influence degree on facial deformity are given lower weights by using the reciprocal of the distance between each key point and the mirror point corresponding to the key point, so that the median sagittal plane corresponding to the original 3D image determined by the target weight of each key point is more guided by the key points with small influence degree on facial deformity, and the distortion of the median sagittal plane corresponding to the original 3D image caused by the key points with large influence degree on facial deformity can be avoided, and the problem of affecting the alignment of the original 3D image is avoided, and the accuracy of the median sagittal plane corresponding to the original 3D image is improved. Furthermore, on the basis of the more accurate median sagittal plane corresponding to the original 3D image, more accurate alignment processing can be performed on the original 3D image, so that the original 3D image after alignment can be more accurate, and the accuracy of image alignment is improved.

[0027] In combination with the first aspect and the above implementation manners, in some implementation manners of the first aspect, the obtaining of the median sagittal plane based on the target weight corresponding to each key point comprises:

[0028] obtaining a target spatial transformation matrix based on the original configuration, the mirror configuration and the target weight corresponding to each key point;

[0029] obtaining the median sagittal plane based on the target spatial transformation matrix.

[0030] In combination with the first aspect and the above implementation manners, in some implementation manners of the first aspect, the obtaining of the target spatial transformation matrix based on the original configuration, the mirror configuration and the target weight corresponding to each key point comprises:

[0031] aligning the spatial structures of the original configuration and the mirror configuration based on a preset analysis method and the target weight corresponding to each key point to obtain a minimum weighted function;

[0032] determining the spatial change matrix corresponding to the minimum weighted function as the target spatial transformation matrix.

[0033] In the embodiments of the present application, since the minimum weighted function is obtained by aligning the spatial structures of the original configuration and the mirror configuration based on the preset analysis method and the target weight corresponding to each key point, it can be explained that the spatial structure difference between the original configuration and the mirror configuration is smallest at this time, and the original configuration is almost the same as the mirror configuration after geometric transformation, so the accuracy of the median sagittal plane obtained by the spatial change matrix corresponding to the minimum weighted function is higher, and on the basis of the higher accuracy of the median sagittal plane, more accurate alignment processing can be performed on the original 3D image, so that the original 3D image after alignment can be more accurate, and the accuracy of image alignment is improved.

[0034] With reference to the first aspect and the foregoing implementation manners, in some implementations of the first aspect, the original 3D image is processed based on the median sagittal plane to obtain a processed original 3D image, including:

[0035] A horizontal plane corresponding to the original 3D image is obtained based on the median sagittal plane, the coordinates of the infraorbital point and the coordinates of the ear point in the plurality of key points;

[0036] A coronal plane corresponding to the original 3D image is obtained based on the median sagittal plane and the horizontal plane;

[0037] A rotation matrix is obtained based on the median sagittal plane, the horizontal plane and the coronal plane;

[0038] The original 3D image is processed based on the rotation matrix to obtain a processed original 3D image.

[0039] In the embodiments of the present application, on the basis of the higher accuracy of the median sagittal plane, the accuracy of the horizontal plane corresponding to the original 3D image constructed by the median sagittal plane and the infraorbital point and the ear point is also improved, and the accuracy of the coronal plane corresponding to the original 3D image constructed by the higher accuracy of the median sagittal plane and the horizontal plane is also improved, so that the rotation matrix with higher accuracy can be obtained by the higher accuracy of the median sagittal plane, the horizontal plane and the coronal plane, and the original 3D image can be more accurately processed by the more accurate rotation matrix, so that the processed original 3D image can be more accurate, and the accuracy of image placement is improved; and when the doctor diagnoses the facial deformity by using the 3D image with higher accuracy, the accuracy of the doctor's diagnosis of the facial deformity can be improved, thereby facilitating the doctor's treatment of the facial deformity and the evaluation of the postoperative treatment effect.

[0040] With reference to the first aspect and the foregoing implementation manners, in some implementations of the first aspect, the original coordinates of the plurality of key points are obtained based on the original 3D image, including:

[0041] The original 3D image is processed to obtain a processed original 3D image;

[0042] The first coordinates of the plurality of key points are extracted in the processed original 3D image;

[0043] The original coordinates of the plurality of key points are obtained by performing secondary extraction in the processed original 3D image based on the first coordinates of the plurality of key points;

[0044] The image processing includes at least one of data resampling and data preprocessing.

[0045] In the embodiment of the present application, rough extraction is first performed in the original 3D image after image processing to quickly obtain relatively rough key point coordinates, and then secondary extraction is performed in the original 3D image after image processing through the relatively rough key point coordinates extracted to obtain relatively accurate key point coordinates in a more detailed manner, so that the efficiency of key point coordinate selection is improved, and the accuracy of key point coordinate selection is also improved. Further, on the basis of better efficiency and accuracy of key point coordinate selection, the efficiency and accuracy of the original 3D image alignment can be improved.

[0046] In a second aspect, the present application provides a face data processing apparatus, which comprises:

[0047] An acquisition module is configured to obtain original coordinates of a plurality of key points based on an original 3D image, wherein the original 3D image represents a tomographic image of a user's face, and the plurality of key points represent position points corresponding to craniofacial tissues of the user's face.

[0048] A processing module is configured to perform spatial configuration based on the original coordinates of the plurality of key points to construct an original configuration, perform spatial configuration based on mirror coordinates of mirror points corresponding to the plurality of key points respectively to construct a mirror configuration, obtain a median sagittal plane corresponding to the original 3D image based on a target distance between the original configuration and the mirror configuration, wherein the coincidence degree between the original configuration and the mirror configuration is negatively correlated with the target distance, and perform alignment processing on the original 3D image based on the median sagittal plane to obtain an aligned original 3D image, wherein the aligned original 3D image is used for face detection on the user's face.

[0049] In a third aspect, the present application provides an electronic device comprising a memory and a processor. The memory is configured to store executable program code, and the processor is configured to call and run the executable program code from the memory, so that the electronic device executes the method in the first aspect or any one of the possible implementation manners of the first aspect.

[0050] In a fourth aspect, the present application provides a computer program product, which comprises computer program code. When the computer program code runs on a computer, the computer program code causes the computer to execute the method in the first aspect or any one of the possible implementation manners of the first aspect.

[0051] In a fifth aspect, the present application provides a computer readable storage medium, which stores computer program code. When the computer program code runs on a computer, the computer program code causes the computer to execute the method in the first aspect or any one of the possible implementation manners of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1is a scene schematic diagram of a face data processing method provided by an embodiment of the present application;

[0053] Figure 2 is a flow schematic diagram of a first face data processing method provided by an embodiment of the present application;

[0054] Figure 3 is an architecture schematic diagram of a face model provided by an embodiment of the present application;

[0055] Figure 4 is an architecture schematic diagram of a first face training model provided by an embodiment of the present application;

[0056] Figure 5 is an architecture schematic diagram of a second face training model provided by an embodiment of the present application;

[0057] Figure 6 is a flow schematic diagram of a second face data processing method provided by an embodiment of the present application;

[0058] Figure 7 is a schematic diagram of a face key point provided by an embodiment of the present application;

[0059] Figure 8 is a schematic diagram of a face plane provided by an embodiment of the present application;

[0060] Figure 9 is a structure schematic diagram of a face data processing apparatus provided by an embodiment of the present application;

[0061] Figure 10 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the present application will be described clearly and exhaustively in combination with the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B: "and / or" in the text is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, in addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0063] Hereinafter, the terms "first", "second" are only for description purposes, and cannot be understood as implying or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features.

[0064] The facial attractiveness of a human face has a significant impact on the personal development of an individual, and even influences the social status, economic condition and health level of the individual. The facial symmetry of a human face can be regarded as a key element for measuring the facial attractiveness, because the facial symmetry is a sign of the stability and health of the body development of the individual, i.e. the more stable and healthy the body development is, the more likely the individual has a face with high facial symmetry, and the more attractive the face is. The facial symmetry is positively correlated with the facial attractiveness, and a higher facial symmetry indicates that the face is aesthetically pleasing, attractive or in line with the aesthetic standards, i.e. the so-called "aesthetic face". However, a 100% facial symmetry is more an idealized conception, and it is difficult to exist in a real face, and there will be more or less asymmetric faces (also referred to as "facial deformities"). If the facial symmetry is too low (i.e. the facial deformity is too high), it can cause physical and psychological discomfort, and thus cause great harm to the physical and mental health of the individual. The facial deformity can be measured by a facial deformity index, and the measurement index corresponding to different facial positions can be different. For example, the measurement index corresponding to the craniofacial can be the jaw relationship, the jaw length, etc., and the measurement index corresponding to the face can be the eye distance, the mouth corner height, the facial skewness, the facial angle, etc.

[0065] In order to avoid the harm of facial deformity to the physical and mental health of the individual, the individual can seek facial reconstruction surgery to eliminate the facial deformity, improve the facial symmetry, and restore the harmonious beauty of the face. Since the surgical effect of the facial reconstruction surgery depends largely on the accuracy of the diagnosis of the face in need of facial reconstruction (referred to as "surgical face") by the doctor (e.g. the doctor 101 in Figure 1 , the accurate and comprehensive assessment of the surgical face (e.g. the face 102 in Figure 1 ) is an indispensable and must be strictly followed step before the surgical plan of the facial reconstruction surgery is made. With the rapid development of the three-dimensional imaging technology (3D Imaging Technology), great help is provided for the clinical diagnosis field of the facial reconstruction surgery, and more abundant and intuitive diagnosis means can be provided for the doctor. By means of the 3D image of the surgical face, the 3D image can be rotated from any angle to view the surgical face from all directions, so that the doctor can accurately and comprehensively master the specific situation of the asymmetry of the surgical face, and quantify the facial deformity of the surgical face.

[0066] Currently, there are various methods for quantifying facial deformities. A common method is to calculate the asymmetry of the face to be operated in the 3D space by measuring the linear distance between multiple facial key points in the face to be operated and the symmetry plane, the angle difference and the surface distance, etc. to quantify the facial deformity. In the quantification of facial deformities, the median sagittal plane (also referred to as the "symmetry plane") is the core reference for the analysis of quantifying facial deformities, and there is a high demand for the construction of its own accuracy to improve the accuracy of quantifying facial deformities. There are mainly two ways to construct the median sagittal plane: one is to manually select reference key points that are not affected by asymmetry, i.e. to select symmetrical points as reference key points; the other is to use the best fitting superposition method of Procrustes Analysis to mathematically deduce the reference key points. However, these two ways of constructing the median sagittal plane may lead to unreliable and false manifestations of the true asymmetry characteristics of the face, which may result in poor accuracy of the constructed median sagittal plane. By manually selecting key points, the median sagittal plane is directly constructed by manually selecting reference key points on a three-dimensional (3D) digital model of the face, which includes midline anatomical key points and bilateral anatomical key points of the face. The manual selection of reference key points may result in different median sagittal planes constructed from different facial parts and different numbers of key points, i.e. the median sagittal planes constructed from different facial parts and different numbers of key points may be different, which limits the construction of the median sagittal plane to the selection of reference key points. Moreover, the number of manually selected reference key points is limited, and the median sagittal plane constructed from limited reference key points can only represent part of the facial structure characteristics of the face to be operated, and cannot represent all the facial structure characteristics of the face to be operated, thereby ignoring the rich features of the 3D craniofacial structure, making it difficult to form an ideal median sagittal plane, and resulting in poor accuracy of the constructed median sagittal plane. Furthermore, the process of manually selecting reference key points is time-consuming, error-prone, dependent on experience and has low repeatability, which results in low efficiency of the construction of the median sagittal plane based on the poor accuracy of the constructed median sagittal plane. Furthermore, if the face to be operated is diagnosed for facial deformity by using the median sagittal plane with poor accuracy, the accuracy of the doctor's diagnosis of facial deformity may be reduced, which is not conducive to the treatment of facial deformity and the evaluation of postoperative treatment effect.

[0067] Therefore, the present application provides a face data processing method and device, an electronic device and a storage medium. In the embodiments of the present application, when a tomographic image of a user's face is obtained, a plurality of key points corresponding to positions of craniofacial tissues of the user's face can be obtained, and then the original coordinates of the plurality of key points are spatially configured to construct an original configuration, and the mirror coordinates of the mirror points corresponding to the plurality of key points are spatially configured to construct a mirror configuration. Then, the median sagittal plane corresponding to the original 3D image is obtained according to the distances between each key point in the original configuration and each mirror point in the mirror configuration, and the original 3D image is aligned according to the median sagittal plane to obtain an aligned original 3D image. The aligned original 3D image is used to automatically detect facial deformities of the user's face, which can improve the accuracy of the constructed median sagittal plane, so that the doctor can diagnose facial deformities according to the median sagittal plane with high accuracy, which can improve the accuracy of the doctor's diagnosis of facial deformities, thereby facilitating the doctor's treatment of facial deformities and evaluation of postoperative treatment effect. In addition, the reference key points do not need to be manually selected by artificial manual selection, but are automatically detected by cascade 3D-Unet, which can significantly reduce the time cost and labor cost of selecting key points, greatly improve the efficiency of selecting key points, and further improve the diagnosis efficiency of the doctor's diagnosis of facial deformities on the basis of high efficiency of selecting key points.

[0068] The embodiments of the present application will be described below in conjunction with Figures 2 to 8 The face data processing method provided by the embodiments of the present application will be described in detail.

[0069] Figure 2 is a flowchart of a first face data processing method provided by the embodiments of the present application. The method can be executed by an electronic device with face data processing capability.

[0070] For example, as Figure 2 shown, the first face data processing method 200 includes the following implementation process:

[0071] S210, obtaining original coordinates of a plurality of key points based on the original 3D image.

[0072] The original 3D image represents a tomographic image of a user's face, and the plurality of key points (also referred to as "marker points") represent position points corresponding to craniofacial tissues of the user's face. It should be understood that tomography is a special X-ray imaging technology, and the essence of the tomographic image obtained by scanning is a 3D image.

[0073] Exemplarily, when it is needed to measure the index of facial deformity of a user, a tomographic image of the face of the user (which can be referred to as an "original 3D image") can be obtained by scanning. When the original 3D image is obtained, the original 3D image can be input into the following face model 300 to obtain a plurality of key points (i.e., the following second key points) output by the face model 300 and coordinates (which can be referred to as "original coordinates") of the plurality of key points on the original 3D image, i.e., the following second key point coordinates.

[0074] The tomographic image can be a CBCT image obtained by a Cone Beam Computed Tomography (CBCT) or a CT image obtained by a Computed Tomography Equipment (CT).

[0075] When the original coordinates of the plurality of key points are obtained from the original 3D image, the original 3D image can be first image-processed by the first resampling module 111, the second resampling module 112, the first preprocessing module 113 and the second preprocessing module 114 in the face model 300 to obtain an image-processed original 3D image. The image-processed original 3D image can be input into the first key point solving network 115 to extract the first coordinates (i.e., the following first key point coordinates) of the plurality of key points.

[0076] Further, when the first coordinates of the plurality of key points are obtained, the first coordinates of the plurality of key points can be used to perform secondary extraction in the image-processed original 3D image in the second key point solving network 117 to obtain the original coordinates of the plurality of key points.

[0077] The image processing includes at least one of data resampling and data preprocessing.

[0078] In the embodiments of the present application, rough extraction is first performed in the image-processed original 3D image to quickly obtain relatively rough key point coordinates, and then the relatively rough key point coordinates are used to perform secondary extraction in the image-processed original 3D image to more carefully obtain relatively accurate key point coordinates, so that the efficiency of selecting the coordinates of the key points is improved, and the accuracy of selecting the coordinates of the key points is also improved. Further, on the basis of the improved efficiency and accuracy of selecting the coordinates of the key points, the efficiency and accuracy of the original 3D image alignment can be improved.

[0079] S220, based on the original coordinates of the plurality of key points, performing spatial configuration to construct an original configuration.

[0080] Exemplarily, when the original coordinates of the plurality of key points are obtained, a spatial configuration can be performed on the original coordinates of the plurality of key points to construct a craniofacial tissue configuration corresponding to the original coordinates of the plurality of key points (which can be referred to as an "original configuration").

[0081] The spatial configuration performed on the original coordinates of the plurality of key points is mainly to organize the discrete key point coordinates through point-to-point geometric association to form a structured model reflecting the spatial morphological characteristics of the craniofacial tissue, that is, to use the original coordinates of the plurality of key points to piece together the spatial shape framework of the craniofacial tissue, so that the originally independent coordinates have a morphology. The geometric association can include at least one of the distance, angle, topological relationship, etc. between points.

[0082] S230, performing a spatial configuration based on the mirror coordinates of the mirror points corresponding to the plurality of key points respectively to construct a mirror configuration.

[0083] Exemplarily, when the original coordinates of the plurality of key points are obtained, the mirror coordinates of the mirror points corresponding to the plurality of key points respectively can be obtained. When the mirror coordinates of the plurality of mirror points are obtained, a spatial configuration can be performed on the mirror coordinates of the plurality of mirror points to construct a craniofacial tissue configuration corresponding to the mirror coordinates of the plurality of mirror points (which can be referred to as a "mirror configuration"), that is, to use the mirror coordinates of the plurality of mirror points to piece together a spatial shape framework of the craniofacial tissue. Each key point in the plurality of key points has a mirror point corresponding to itself in the plurality of mirror points, that is, each key point has a corresponding mirror point.

[0084] It should be understood that S220 and S230 can be executed simultaneously or sequentially, and the embodiments of the present application do not limit this.

[0085] When the spatial configuration is performed on the mirror coordinates of the plurality of mirror points to obtain the mirror configuration, the mean value of the original coordinates of the plurality of key points in the first dimension can be calculated first. When the mean value of the plurality of key points in the first dimension is obtained, a plane corresponding to the mean value in the first dimension (which can be referred to as an "original plane") can be determined.

[0086] Further, when the original plane is obtained, the original coordinates of the plurality of key points can be horizontally mirrored and flipped around the original plane to obtain the mirror points corresponding to the plurality of key points and the mirror coordinates of the mirror points. Then, a spatial configuration is performed on the mirror coordinates of the plurality of mirror points to construct a mirror configuration corresponding to the mirror coordinates of the plurality of mirror points.

[0087] The first dimension represents the horizontal dimension of the original 3D image.

[0088] In the embodiments of the present application, since the mean value of the original coordinates of the plurality of key points in the horizontal dimension is essentially the center of gravity of the plurality of key points in the horizontal dimension, the specific position information of the plurality of key points in the horizontal dimension can offset the random error of a small number of points, thereby reflecting the symmetry center trend of the plurality of key points in the horizontal dimension as a whole. Therefore, the center plane (i.e., the original plane) determined by the mean value of the plurality of key points in the horizontal dimension is more accurate, so that the mirror image coordinates of the plurality of mirror image points obtained by horizontally mirroring the original coordinates of the plurality of key points with respect to the center plane are also more accurate. On the basis of the more accurate mirror image coordinates of the plurality of mirror image points, the mirror image configuration constructed by the mirror image coordinates of the plurality of mirror image points is also more accurate. Furthermore, a more accurate median sagittal plane can be constructed by the original configuration and the more accurate mirror image configuration. Moreover, when constructing the center plane, it is not necessary to first accurately find the points on the craniofacial structure that are mutually symmetrical left and right, but the center plane is determined by the mean value of the plurality of points on the craniofacial structure in the horizontal dimension. This can make the center plane constructed by the symmetrical points and the asymmetrical points more represent the overall symmetry trend of the craniofacial structure, be more real, and be more suitable for asymmetric faces, thereby improving the accuracy of the construction of the center plane. Furthermore, when the user's face is too deformed and the craniofacial structure does not have symmetrical points, the center plane can also be determined by the mean value of the plurality of points on the craniofacial structure in the horizontal dimension, rather than necessarily relying on the symmetrical points of the craniofacial structure to construct the center plane. The construction of the center plane is no longer limited by the requirement of relying on the symmetrical points of the craniofacial structure, and the center plane is constructed freely, thereby realizing the free construction of the mirror image configuration.

[0089] S240, obtaining a median sagittal plane corresponding to the original 3D image based on the target distance between the original configuration and the mirror image configuration.

[0090] The coincidence degree between the original configuration and the mirror image configuration is negatively correlated with the target distance. The target distance represents the distance between each key point in the original configuration and the corresponding mirror image point of each key point in the mirror image configuration. Moreover, the distance between each key point and the corresponding mirror image point of each key point is negatively correlated with each key point and the corresponding mirror image point of each key point. The smaller the target distance between each key point and the corresponding mirror image point of each key point, the more coincident they are. For example, the mirror image point corresponding to key point A in the mirror image configuration is A'. The smaller the target distance between key point A and mirror image point A', the more coincident they are, and the greater the target distance between key point A and mirror image point A', the less coincident they are.

[0091] For example, when the original configuration and the mirror configuration are obtained, target distances between each key point in the original configuration and each mirror point corresponding to the key point in the mirror configuration can be determined first, and then a median sagittal plane equation corresponding to the original 3D image is determined according to the target distances, so as to obtain a median sagittal plane (MSP) corresponding to the original 3D image.

[0092] Optionally, the target distance between each key point in the original configuration and each mirror point corresponding to the key point in the mirror configuration can be the Euclidean distance between each key point and each mirror point corresponding to the key point.

[0093] When the median sagittal plane is obtained according to the target distances between the original configuration and the mirror configuration, the reciprocal of the target distance (for example, the Euclidean distance) between each key point and the mirror point corresponding to the key point can be calculated first, and the reciprocal of each distance is normalized to obtain a normalized distance (which can be referred to as a target weight). The target distance represents the distance between each key point and each mirror point corresponding to the key point.

[0094] Further, when the target weight of each key point is obtained, the median sagittal plane corresponding to the original 3D image can be determined according to the target weight of each key point.

[0095] In the embodiment of the present application, the reciprocal of the distance between each key point and each mirror point corresponding to the key point is used to give a key point with a small degree of influence on facial deformity a higher weight, and give a key point with a large degree of influence on facial deformity a lower weight, so that the median sagittal plane corresponding to the original 3D image determined according to the target weight of each key point is more guided by the key point with a small degree of influence on facial deformity, which can avoid the distortion of the median sagittal plane corresponding to the original 3D image caused by the key point with a large degree of influence on facial deformity, and finally affect the alignment of the original 3D image, and can improve the accuracy of the median sagittal plane corresponding to the original 3D image. Further, on the basis of the more accurate median sagittal plane corresponding to the original 3D image, the original 3D image can be more accurately aligned, so that the original 3D image after alignment can be more accurate, and the accuracy of image alignment is improved.

[0096] When the median sagittal plane corresponding to the original 3D image is determined according to the target weight of each key point, a target space transformation matrix (i.e., the optimal space transformation matrix described below) can be obtained according to the original configuration, the mirror configuration and the target weight of each key point first, and then the median sagittal plane is obtained according to the target space transformation matrix.

[0097] Wherein, when the mirror configuration is geometrically transformed by the target space transformation matrix, the difference between the original configuration and the mirror configuration after geometric transformation can be minimized, i.e., original configuration ≈ mirror configuration after geometric transformation.

[0098] When the target space transformation matrix is obtained by the original configuration, the mirror configuration, and the target weight corresponding to each key point, the spatial structure of the original configuration and the mirror configuration can be aligned by the preset analysis method and the target weight corresponding to each key point to obtain the minimum weighted function.

[0099] Further, when the minimum weighted function is obtained, the spatial change matrix corresponding to the minimum weighted function can be determined as the target space transformation matrix.

[0100] Wherein, the preset analysis method can be any one of ordinary Procrustes analysis (Procrustes), inverse distance weighting interpolation (Inverse Distance Weighting, IDW), etc., and the Procrustes analysis is exemplarily described in the embodiments of the present application.

[0101] In the embodiments of the present application, since the minimum weighted function is obtained by aligning the spatial structure of the original configuration and the mirror configuration by the preset analysis method and the target weight corresponding to each key point, it can be explained that the spatial structure difference between the original configuration and the mirror configuration is minimized at this time, and the original configuration and the mirror configuration after geometric transformation are nearly the same. Therefore, the accuracy of the median sagittal plane obtained by the spatial change matrix corresponding to the minimum weighted function is higher, so that the original 3D image can be more accurately aligned on the basis of the higher accuracy of the median sagittal plane, so that the original 3D image after alignment can be more accurate, and the accuracy of image alignment is improved.

[0102] S250, based on the median sagittal plane, the original 3D image is aligned to obtain the original 3D image after alignment.

[0103] Wherein, the original 3D image after alignment is used for face detection of the user's face.

[0104] Exemplarily, when the median sagittal plane is obtained, the original 3D image can be aligned by the median sagittal plane to obtain the original 3D image after alignment, so as to facilitate the doctor to use the original 3D image after alignment to detect the user's face.

[0105] In the case of Figure 2In the first facial data processing method 200 shown, when the original 3D image representing the tomographic image of the user's face is collected, a plurality of key points corresponding to the positions of the craniofacial tissues of the user's face can be obtained from the original 3D image, and a raw configuration corresponding to the original coordinates of the plurality of key points can be constructed by the original coordinates of the plurality of key points, and a mirror configuration corresponding to the mirror coordinates of the plurality of key points can be constructed by the mirror coordinates of the mirror points corresponding to the plurality of key points; then a median sagittal plane corresponding to the original 3D image can be obtained by the distance between the constructed raw configuration and mirror configuration, and the original 3D image can be aligned by the median sagittal plane to obtain an aligned original 3D image, so as to detect facial deformities of the user's face. Compared with the existing scheme of directly selecting a plurality of symmetric points on the craniofacial tissues from the original 3D image to construct the median sagittal plane, the construction of the median sagittal plane may rely on local symmetric points, and if the accuracy of the selected symmetric points is insufficient or the number of the selected symmetric points is small, the accuracy of the median sagittal plane will be directly reduced. The embodiments of the present application make the construction of the median sagittal plane free from the dependence on the local symmetric points by the raw configuration and the mirror configuration, and do not need to accurately find the points symmetric to each other on the craniofacial tissues, but determine the raw configuration and the mirror configuration of the overall space corresponding to the craniofacial tissues by a plurality of points on the craniofacial tissues to construct the median sagittal plane, that is, the symmetric points and the asymmetric points can both participate in the construction of the raw configuration and the mirror configuration, so that the constructed median sagittal plane can better represent the overall symmetry trend of the craniofacial tissues, and is closer to the median sagittal plane with functional significance in anatomy rather than the mathematically absolute symmetric plane, and is more true and more suitable for asymmetric faces, thereby improving the accuracy of the construction of the median sagittal plane, and enabling the doctor to diagnose facial deformities by the median sagittal plane with high accuracy, so as to improve the accuracy of the doctor's diagnosis of facial deformities, thereby being beneficial to the treatment of facial deformities by the doctor and the evaluation of the postoperative treatment effect. Furthermore, since the symmetric points and the asymmetric points can both participate in the construction of the raw configuration and the mirror configuration, and are not limited to the symmetric points on the craniofacial tissues, more key points of the craniofacial tissues can be selected, and the errors of the selected points can be avoided from being magnified by more key points of the craniofacial tissues, so that a more accurate median sagittal plane can be further obtained. Moreover, when the user's face is too deformed (for example, severe facial deviation, hemifacial microsomia, and severe asymmetry of the jawbone), there is no symmetric point on the craniofacial tissues, and the raw configuration and the mirror configuration determined by a plurality of points on the craniofacial tissues of the user's face can be used to construct the median sagittal plane, so that the construction of the median sagittal plane is free from the limitation of the symmetric points on the craniofacial tissues, and the median sagittal plane is constructed freely, thereby enabling the median sagittal plane to overcome the interference of facial deformities, and obtaining a stable, reliable and truly reflecting individual anatomical features of the user's face.

[0106] In the method, the multiple sets of symmetrical points on the craniofacial tissue are directly selected from the original 3D image, and the left and right symmetrical points on the craniofacial tissue must be accurately found. However, the absolutely symmetrical craniofacial tissue is extremely difficult to exist in the real human body, and at most can only be approximately symmetrical. Therefore, the symmetrical points directly selected from the original 3D image will have differences to some extent. If the multiple sets of symmetrical points selected from the original 3D image are used to construct the median sagittal plane, the accuracy of the median sagittal plane will be reduced. For example, the left infraorbital point is 0.1 mm lower than the right infraorbital point. The median sagittal plane constructed by using the left infraorbital point and the right infraorbital point will be slightly offset to the right side. In addition, the multiple sets of symmetrical points on the craniofacial tissue directly selected from the original 3D image will also have the problem of less number of symmetrical points. If the number of symmetrical points is small (for example, 2-3 sets of symmetrical points), the error of the selected points may be magnified due to the small number of symmetrical points, which reduces the accuracy of the median sagittal plane constructed by using the small number of symmetrical points in the original 3D image. Or, when the symmetrical points of the craniofacial tissue do not exist in the original 3D image, the median sagittal plane cannot be constructed, because the severe deformation of the craniofacial tissue distorts the conventional anatomical landmarks and symmetry assumption.

[0107] In the process of aligning the original 3D image by the median sagittal plane to obtain the aligned original 3D image, the horizontal plane equation corresponding to the original 3D image can be obtained by the coordinates of the infraorbital points (i.e. the coordinates of the left and right infraorbital points described below) and the coordinates of the ear points (i.e. the coordinates of the left and right ear points described below) in the multiple key points and the determined median sagittal plane, so as to obtain the horizontal plane corresponding to the original 3D image.

[0108] Further, the coronal plane equation corresponding to the original 3D image can be determined by the median sagittal plane equation corresponding to the median sagittal plane and the horizontal plane equation corresponding to the horizontal plane. Then, the rotation matrix for aligning the original 3D image is obtained by the median sagittal plane equation corresponding to the median sagittal plane, the horizontal plane equation corresponding to the horizontal plane, and the coronal plane equation corresponding to the coronal plane. The original 3D image is aligned by the rotation matrix, so as to obtain the aligned original 3D image.

[0109] In the embodiments of the present application, on the basis of higher accuracy of the median sagittal plane, the accuracy of the horizontal plane corresponding to the original 3D image constructed by the median sagittal plane and the infraorbital point and the ear point will also be improved, and then the accuracy of the coronal plane corresponding to the original 3D image constructed by the median sagittal plane and the horizontal plane with higher accuracy will also be improved, so that a more accurate rotation matrix can be obtained through the median sagittal plane, the horizontal plane and the coronal plane with higher accuracy, and the original 3D image can be more accurately aligned through the more accurate rotation matrix, so that the original 3D image after alignment can be more accurate, and the accuracy of image alignment is improved. Then, when the doctor diagnoses facial deformity through the 3D image with higher accuracy, the accuracy of the doctor's diagnosis of facial deformity can be improved, thereby facilitating the doctor's treatment of facial deformity and the evaluation of postoperative treatment effect.

[0110] It should be noted that, Figure 2 All steps in the embodiments are described in detail in the following Figures 3 to 7 , and will not be repeated here.

[0111] Figure 3 is a schematic diagram of a face model architecture provided by the embodiments of the present application. The face model can be configured in an electronic device that has a face data processing requirement.

[0112] For example, as shown in Figure 3 , the face model 300 can include a first resampling module 111, a second resampling module 112, a first preprocessing module 113, a second preprocessing module 114, a first key point solving network 115, a cropping window 116, and a second key point solving network 117. The first resampling module 111, the second resampling module 112, and the cropping window 116 do not require learning parameters and can be used directly. The first preprocessing module 113 and the second preprocessing module 114 represent the preprocessing rules set according to the unique format and attributes of the data. The first key point solving network 115 and the second key point solving network 117 require learning parameters, which can be trained continuously to learn the parameters, and then used after learning is completed. Among them, the collected tomographic image (i.e. the original 3D image described above) is used as input, and the second key point coordinates are used as output.

[0113] The original 3D image generally has 3 dimensions, 2 dimensions of which have 1024*1024 voxels, and the remaining one dimension has about 1000 voxels; and the voxel spacing of the original 3D image can be about [0.22*0.22*0.3125], the voxel spacing can represent the actual length of a single voxel, and the voxel spacing can determine the detail resolution of the original 3D image, the smaller the spacing, the finer the corresponding voxel, and the more detailed facial structure can be displayed. For example, the original 3D image has 1024 voxels in the horizontal direction (i.e., left and right), 1024 voxels in the vertical direction (i.e., front and back), and 1000 voxels in the up-down direction (i.e., height), and through the 1024*1024*1000 voxel matrix and the [0.22*0.22*0.3125] voxel spacing, the image data of the entire craniofacial tissue can be ensured to be obtained, and the part of the facial deformity area missed by the scanning due to insufficient field of view can be avoided.

[0114] The first resampling module 111 and the second resampling module 112 can select part of the data that meets the voxel spacing requirement in the original 3D image, unify the voxel spacing, ensure the spatial consistency of the selected key points, and avoid the dimensional confusion caused by the difference of the scanning device. The voxel spacing corresponding to the first resampling module 111 and the second resampling module 112 can be different to adapt to different diagnostic requirements; for example, the voxel spacing of the first resampling module 111 is [1.4*1.4*1.4], and the voxel spacing of the second resampling module 112 is [0.22*0.22*0.3125]. Through the larger voxel spacing, the selection of the key points (i.e., the first key point coordinates) can be completed faster, but the accuracy of the selection is poor. In order to improve the accuracy of the selection of the key points, the selection of the key points can be more detailed through the smaller voxel spacing, so that more accurate key points (i.e., the second key point coordinates) can be selected.

[0115] The first preprocessing module 113 and the second preprocessing module 114 can perform at least one of voxel thresholding, voxel normalization, and the like on the resampled data. The voxel thresholding can set voxels exceeding a set threshold range to a voxel threshold value, that is, replace voxels greater than an upper boundary of the set threshold range with an upper boundary of the set threshold value, and replace voxels less than a lower boundary of the set threshold range with a lower boundary of the set threshold value. This can retain as many voxels as possible in the 3D image that are related to the craniofacial region, making the outline of the craniofacial tissue clearer, avoiding interference of air, fat, and the like with the craniofacial tissue, and thus avoiding interference of voxels that are too large or too small with the selection of key points, and improving the accuracy of the selection of key points. The voxel normalization can compress all voxel values after resampling to a fixed range (for example, [0, 1] or [-1, 1]) through mathematical transformation, eliminating differences between voxels, allowing voxels to be compared with and compatible with each other, and being conducive to the accuracy of the selection of key points. In addition, the first preprocessing module 113 and the second preprocessing module 114 can further include at least one of spatial transformation, intensity transformation, random inversion, and the like. The image data after transformation can be represented as , , I can represent the original 3D image after image processing by at least one of spatial transformation, intensity transformation, random inversion, and the like.

[0116] The first key point solving network 115 can quickly locate the voxels related to the craniofacial tissue and roughly select the key points (i.e., the first key point coordinates). The output of the first key point solving network 115 is a heat map corresponding to the first key point coordinates. The second key point solving network 117 can accurately locate the voxels related to the craniofacial tissue and accurately select the key points (i.e., the second key point coordinates). The output of the second key point solving network 117 is a heat map corresponding to the second key point coordinates. The structure of the first key point solving network 115 and the second key point solving network 117 can be the same. Specifically, the structure of the first key point solving network 115 and the second key point solving network 117 can be a 3D U-Net neural network module, which can include 6 layers of encoders and 6 layers of decoders. Each layer of the encoder can include two 3x3 convolution layers, a rectified linear unit (RELU) activation function, and a 2x2 max-pooling layer to form a down-sampling module. Each layer of the decoder can include an up-sampling convolution layer, a feature concatenation layer, two 3x3 convolution layers, and a RELU activation function. The local features in the three-dimensional space are captured through the convolution layers, and the feature perception and expression capabilities of the neural network module are increased through multiple convolution layers in the encoder. The RELU activation function is used in the encoder because the features of the craniofacial tissue are generally nonlinearly distributed. The RELU activation function can enable the neural network module to better learn the nonlinear relationship of the craniofacial features rather than simple linear fitting, for example, the nonlinear relationship between teeth and jaw bones with similar density. The pooling layer can retain the most critical features of the craniofacial tissue, such as the strongest boundary signal, and expand the feature perception capability of the neural network module to enable the neural network module to perceive the global structure of the craniofacial tissue in a larger range, such as from a "single tooth" to a "full jaw shape". The size of the voxel is restored through the up-sampling convolution layer to obtain deeper global features, so that the fusion features obtained through the feature concatenation layer after the feature concatenation combine the local detail features retained by the decoder and the deep global features obtained by the up-sampling convolution layer, which can accurately locate the boundary of the craniofacial tissue. The multiple convolution layers in the decoder can remove the redundant information in the fusion features and retain the fusion features of the local detail features and the deep global features. The RELU activation function is used in the decoder to enable the neural network module to better learn the nonlinear fusion features.

[0117] The cropping window 116 (may also be referred to as "size normalization processing") can remove redundant areas irrelevant to the craniofacial tissue by cropping, and / or fill in insufficient areas related to the craniofacial tissue, unify the three-dimensional matrix of all the resampled data to a fixed dimension, for example, [192x192x192], to ensure the spatial consistency of the preprocessed data and exclude interference factors. For example, the original dimension of voxel A is 180x180x200, 8 layers can be cropped along the y-axis (i.e. the above-mentioned up-down direction), 192 layers including the craniofacial tissue in the middle are retained, cropping to the voxels related to the craniofacial tissue should be avoided, and 6 layers are symmetrically filled in front and back, left and right on the x-axis (i.e. the above-mentioned horizontal direction) and the z-axis (i.e. the above-mentioned vertical direction), respectively, to finally obtain a voxel with a dimension of 192x192x192.

[0118] After the first key point solving network 115 and the second key point solving network 117 output the heat maps, the corresponding key point coordinates of each heat map cannot be directly obtained, but the heat maps need to be processed by the post-processing module to obtain the key point coordinates corresponding to the maximum confidence in the heat maps. Specifically, the output result of the heat map is , represents a set of n heat maps, and n represents the total number of heat maps. When is obtained, each heat map in is traversed to find the key point coordinates (may also be referred to as "extreme point positions") corresponding to the maximum confidence in each heat map. The key point coordinates corresponding to the maximum confidence of n heat maps are reconstructed to obtain the final inference result , represents a set of key point coordinates corresponding to the maximum confidence of n heat maps. All key point coordinates in output by the first key point solving network 115 are determined as the first key point coordinates, and all key point coordinates in output by the second key point solving network 117 are determined as the second key point coordinates.

[0119] The facial model 300 provides two branches: an upper branch and a lower branch. The upper branch consists of: a first resampling module 111, a first preprocessing module 113, a first keypoint solving network 115, and first keypoint coordinates. The lower branch consists of: first keypoint coordinates, a second resampling module 112, a second preprocessing module 114, a cropping window 116, a second keypoint solving network 117, and second keypoint coordinates. The upper branch allows for rapid selection of relevant keypoints of the craniofacial tissues (i.e., the aforementioned first keypoint coordinates) from the original 3D image, improving the efficiency of keypoint selection. Furthermore, the lower branch allows for more detailed selection of the relevant voxels of the craniofacial tissues located by the upper branch, obtaining the second keypoint coordinates and improving the accuracy of keypoint selection.

[0120] For example, when acquiring an original 3D image with a data dimension of approximately [1024×1024×1000] and a voxel spacing of approximately [0.22×0.22×0.3125], the original 3D image can be resampled once by the first resampling module 111 to resample the original 3D image into image data with a voxel spacing of [1.4×1.4×1.4]; and the original 3D image can also be resampled once by the second resampling module 112 to resample the original 3D image into image data with a voxel spacing of [0.22×0.22×0.3125]. The image data with a voxel spacing of [1.4×1.4×1.4] is then preprocessed by the first preprocessing module 113. Values ​​greater than 1024 in the voxel spacing of [1.4×1.4×1.4] are set to 1024, and values ​​less than -1024 are set to -1024. These values ​​are then divided by the mean and normalized to 0-1 to obtain the first preprocessed image data. This first preprocessed image data is then input into the first keypoint solving network 115 to obtain the output value of the first keypoint solving network 115. The corresponding heatmap (also known as a "Gaussian heatmap") will finally be... The coordinates of all key points in the corresponding heatmap are determined as the coordinates of the first key point.

[0121] Furthermore, the image data with a voxel spacing of [0.22×0.22×0.3125] is preprocessed by the second preprocessing module 114. Values ​​with voxel spacing greater than 1024 are set to 1024, and values ​​less than -1024 are set to -1024. Then, the values ​​are divided by the mean and normalized to 0-1 to obtain the second preprocessed image data. When obtaining the second preprocessed image data, the first preprocessed image data and the second preprocessed image data can be cropped to a fixed data dimension, for example... , to obtain cropped image data. Then, the cropped image data is input into the second key point solving network 117 to obtain an output value of the second key point solving network 117 all key point coordinates in the corresponding heat map are determined as second key point coordinates, and the second key point coordinates are gradually refined according to the heat map regression in the patch near the first key point coordinates. all key point coordinates in the corresponding heat map are determined as second key point coordinates, and the second key point coordinates are gradually refined according to the heat map regression in the patch near the first key point coordinates.

[0122] Figure 4 is a schematic diagram of a first face training model provided by an embodiment of the present application. The first face model can be configured in an electronic device that has a face data processing requirement.

[0123] For example, as shown in Figure 4 The first resampling module 111, the first preprocessing module 113, the first key point solving network 115, and the first loss module 118 can be included in the first face training model 400. The first resampling module 111 and the first loss module 118 do not need to learn parameters and can be directly used. The first preprocessing module 113 represents a preprocessing rule set according to unique formats and attributes of data. The first key point solving network 115 needs to learn parameters and can learn parameters through continuous training. After learning is completed, the first key point solving network 115 is used. That is, when the first face training model 400 is trained, the training is mainly performed on the first key point solving network 115. The collected tomographic image (that is, the original 3D image described above) is used as the input of the first face training model 400, and the heat map corresponding to the sample space coordinate converted sample voxel coordinate is used as the true value of the first face training model 400, that is, the true value heat map.

[0124] It should be noted that the first resampling module 111, the first preprocessing module 113, and the first key point solving network 115 in the first face training model 400 can refer to the related descriptions in Figure 3 and will not be described herein again.

[0125] The first loss module 118 can calculate a loss function between the heat map predicted by the first key point solving network 115 and the true value heat map when the first face training model 400 is trained. The first face training model 400 is iteratively trained through the calculated loss function until the calculated loss function is small and meets the training requirement, so that the trained first face training model 400 is obtained.

[0126] Figure 5 is a schematic diagram of a second face training model provided by an embodiment of the present application. The second face model can be configured in an electronic device that has a face data processing requirement.

[0127] For example, as shown inFigure 5 As shown, the second face training model 500 can include a second resampling module 112, a second preprocessing module 114, a cropping window 116, a second key point solving network 117, and a second loss module 119. The second resampling module 112, the cropping window 116, and the second loss module 119 do not need learning parameters and can be directly used. The second preprocessing module 114 represents a preprocessing rule set according to the unique format and attributes of the data. The second key point solving network 117 needs learning parameters, which can be learned through continuous training, and is used after learning is completed. The collected tomographic image (i.e., the original 3D image described above) is used as the input of the second face training model 500, and the heat map corresponding to the sample voxel coordinate after the sample space coordinate is converted is used as the true value of the second face training model 500, i.e., the true value heat map.

[0128] It should be noted that the second resampling module 112, the second preprocessing module 114, the cropping window 116, and the second key point solving network 117 in the second face training model 500 can refer to the related descriptions in the first face training model 400, which will not be described here. Figure 3

[0129] The second loss module 119 can calculate the loss function between the heat map predicted by the second key point solving network 117 and the true value heat map when training the second face training model 500, so as to iteratively train the second face training model 500 through the calculated loss function until the calculated loss function is small and meets the training requirements, thereby obtaining the trained second face training model 500.

[0130] For example, the loss function calculated by the first loss module 118 or the second loss module 119 can be divided into two parts: an adaptive wing loss (denoted as “wing loss”) and a mean square error loss (denoted as “L2 loss”). The wing loss can adaptively adjust the loss weight of the loss function, focus on the boundary area of the heat map and important features, thereby enhancing the accuracy and stability of the trained face training model (e.g., the trained first face training model 400 or the trained second face training model 500), while also providing smooth gradients. When processing relatively complex heat map data, especially heat map data with noise and outliers, it can help the face training model (e.g., the first face training model 400 or the second face training model 500) to stably converge during the training process. The L2 loss can ensure the regression stability of the face training model during the training process, avoid the heat map predicted by the face training model from deviating too much from the true distribution, and improve the convergence speed of the face training model during the early training.

[0131] ​Wherein, each key point has its own corresponding ground truth heat map and predicted heat map, after calculating the loss function (i.e. wing loss and L2 loss) between the ground truth heat map and the predicted heat map, the learnable parameter gradient in the first key point solving network 115 and the second key point solving network 117 can be calculated by using back propagation (i.e. iterative training), and the learnable parameters are updated by using the stochastic gradient descent algorithm.

[0132] For example, after calculating the loss function between the predicted heat map of the first facial training model 400 or the predicted heat map of the second facial training model 500 and the ground truth heat map each time, the learnable parameter gradient in the first key point solving network 115 and the second key point solving network 117 can be calculated by using back propagation, and the learnable parameters in the first key point solving network 115 and the second key point solving network 117 are updated by using an optimizer (e.g. Adam with Weight Decay (AdamW)) until the calculated loss function converges.

[0133] It should be noted that the predicted heat map of the first facial training model 400 (i.e. the output value of the first key point solving network 115 corresponding heat map) or the predicted heat map of the second facial training model 500 (i.e. the output value of the second key point solving network 117 corresponding heat map) can refer to the related description in Figure 3 , which will not be repeated here.

[0134] Figure 6 is a flowchart of a second facial data processing method provided by the embodiments of the present application. The method can be executed by an electronic device with facial data processing capability.

[0135] For example, as shown in Figure 6 , the second facial data processing method 600 includes the following implementation process:

[0136] S1, obtaining the soft and hard tissue key point coordinates in the original 3D image.

[0137] For example, when the tomography is collected, the collected tomography can be used as the original 3D image for facial deformity diagnosis. It is judged whether the original 3D image is righted. When the original 3D image is righted, the facial deformity index of the original 3D image can be directly measured.

[0138] When the original 3D image is not righted, in order to avoid the influence of the original 3D image not being righted on the facial deformity index measurement result, the original 3D image can be input into the face model 300 in Figure 3 , and the Figure 3The facial model 300 obtains the coordinates of the soft and hard tissue key points corresponding to the original 3D image (i.e., the coordinates of the second key points mentioned above), so as to correct the misaligned original 3D image using the coordinates of the soft and hard tissue key points corresponding to the original 3D image. The soft and hard tissue key points can more comprehensively reflect the 3D structure and interrelationships of facial deformities.

[0139] The number of key points in both soft and hard tissues can be multiple, for example, 72. Furthermore, the number of soft tissue key points and hard tissue key points can be equally distributed or unevenly distributed; for example, both soft and hard tissue key points can be 36; or, there can be 31 soft tissue key points and 41 hard tissue key points. Additionally, the hard tissue in the term "soft and hard tissues" (i.e., the craniofacial tissues mentioned above) can refer to the craniofacial skeletal tissues, while the soft tissue can refer to the skin, muscles, fascia, and other tissues covering the surface of the craniofacial bones. Similarly, the hard tissue key points can represent key points of the craniofacial bones, and the soft tissue key points can represent the corresponding key points of the skin, muscles, fascia, etc., covering the surface of the craniofacial bones.

[0140] S2, based on the coordinates of key points in both soft and hard tissues, constructs the original configuration.

[0141] For example, when multiple soft and hard tissue key point coordinates are obtained, spatial configuration can be performed on the multiple soft and hard tissue key point coordinates to obtain the original configuration corresponding to the multiple soft and hard tissue key point coordinates, so as to obtain the real spatial structure of the craniofacial region.

[0142] S3, construct the mirror configuration corresponding to the original configuration.

[0143] For example, when obtaining the coordinates of multiple (e.g., 72) soft and hard tissue key points, the mean of the first dimension among the 72 soft and hard tissue key point coordinates can be calculated first to obtain the center position corresponding to the 72 soft and hard tissue key points. Here, the first dimension can represent the horizontal dimension corresponding to the x-axis, because the x-axis can serve as a reference axis for left-right symmetry of the craniofacial region.

[0144]

[0145] In formula (1), Let represent the mean of the first dimension, N represent the total number of multiple soft and hard tissue keypoints, and i represent the nth soft and hard tissue keypoint. This represents the coordinate value of the i-th soft and hard tissue keypoint in the first dimension.

[0146] In obtaining At that time, it can be determined that The original plane at its position on the x-axis. And the x-coordinates of the original plane. As the new x-axis origin, i.e. the original plane = 0, through the original plane = 0 to correct the plurality of soft and hard tissue key points, forming a reference plane (i.e. the original plane) with the center of the plurality of soft and hard tissue key points as the reference. This can avoid false positive misalignment using the original x-axis coordinate values of the plurality of soft and hard tissue key points, for example, if the original 3D image is left, it will cause the x-axis coordinate values of the plurality of soft and hard tissue key points to be smaller as a whole, thereby causing false positive misalignment. By correcting the plurality of soft and hard tissue key points through the original plane = 0, the center position of the plurality of soft and hard tissue key points can be taken as the new reference origin, so that even if the original 3D image is left, the deviation of the center position corresponding to each soft and hard tissue key point can be focused on, and the adverse effects of the original 3D image offset on facial symmetry are eliminated.

[0147] Specifically, the original x-axis coordinate value of each of the plurality of soft and hard tissue key points can be subtracted by to obtain the new x-axis coordinate value of each of the plurality of soft and hard tissue key points.

[0148]

[0149] In formula (2), represents the original x-axis coordinate value of the i-th soft and hard tissue key point coordinate, represents the new x-axis coordinate value of the i-th soft and hard tissue key point coordinate.

[0150] Optionally, when obtaining the plurality of soft and hard tissue key point coordinates, the original plane (which can be referred to as "MSP plane") can also be calculated by the nasion point (which can be referred to as "N"), the right medial canthal point (which can be referred to as "Al'R"), the left medial canthal point (which can be referred to as "Al'L"), the right ala point (which can be referred to as "ZfR"), and the left ala point (which can be referred to as "ZfL").

[0151] For example, the center point of the right medial canthal point and the left medial canthal point is determined, and the center point of the right ala point and the left ala point is determined. The plane formed by the three points of the center point of the right medial canthal point and the left medial canthal point, the center point of the right ala point and the left ala point, and the nasion point is taken as the original plane.

[0152] Further, when the original plane is obtained, the plurality of soft and hard tissue coordinate points can be horizontally mirrored and flipped with the original plane as the center to obtain a plurality of flipped soft and hard tissue coordinate points. When the plurality of flipped soft and hard tissue coordinate points are obtained, the plurality of flipped soft and hard tissue coordinate points can be spatially configured to obtain a mirror configuration corresponding to the plurality of flipped soft and hard tissue coordinate points to obtain a spatial structure after the craniofacial mirror.

[0153] wherein the x-axis coordinate value of the plurality of flipped soft and hard tissue coordinate points is the opposite of the x-axis coordinate value of the plurality of unflipped soft and hard tissue coordinate points, and the y-axis and z-axis coordinate values of the plurality of flipped soft and hard tissue coordinate points are the same as the y-axis and z-axis coordinate values of the plurality of unflipped soft and hard tissue coordinate points. That is, the coordinate value of the unflipped soft and hard tissue coordinate point (i.e., the key point) is (x, y, z), and the coordinate value of the flipped soft and hard tissue coordinate point (i.e., the mirror point) is (-x, y, z). For example, the coordinate value of the unflipped soft and hard tissue coordinate point A is (2, 6, 10), and the coordinate value of the corresponding flipped soft and hard tissue coordinate point A is (-2, 6, 10).

[0154] It should be understood that each key point has its own corresponding mirror point.

[0155] S4, aligning the original configuration and the mirror configuration by a preset analysis method to obtain a spatial transformation matrix.

[0156] For example, since the influence of each point in the plurality of soft and hard tissue coordinate points on the facial deformity can be different, each soft and hard tissue coordinate point can be assigned a weight. Moreover, since the soft and hard tissue coordinate points with high symmetry have a small influence on the facial deformity, and the soft and hard tissue coordinate points with low symmetry have a large influence on the facial deformity, in order to ensure the accuracy of the median sagittal plane, the soft and hard tissue coordinate points with high symmetry (e.g., the anatomical landmark points with high symmetry) can be given a higher weight, and the soft and hard tissue coordinate points with low symmetry (e.g., the anatomical landmark points with low symmetry) can be given a lower weight.

[0157] Specifically, when assigning weights to each soft and hard tissue coordinate point, the distance (e.g., the Euclidean distance) between each key point and its corresponding mirror point can be calculated first. Then, the weight of each key point is calculated based on the calculated distance.

[0158]

[0159] In formula (3), i represents the i-th key point in the plurality of key points, i' represents the mirror point corresponding to the i-th key point, represents the x-axis coordinate value of the i-th key point, represents the x-axis coordinate value of the mirror point corresponding to the i-th key point, This represents the y-axis coordinate value corresponding to the i-th key point. This represents the y-axis coordinate of the mirror image point corresponding to the i-th keypoint. This represents the z-axis coordinate value corresponding to the i-th key point. Let represent the z-axis coordinate of the mirror point corresponding to the i-th keypoint, and D represent the Euclidean distance between the i-th keypoint and its mirror point.

[0160] When the Euclidean distance between the i-th keypoint and its corresponding mirror point is extremely close to 0 (i.e., D≈0), it indicates that the i-th keypoint and its corresponding mirror point almost completely overlap, exhibiting a high degree of symmetry. Therefore, the i-th keypoint and its corresponding mirror point can be considered a perfectly symmetrical set of keypoints, having a minimal impact on facial deformities. Thus, the reciprocal of D can be used to assign a larger weight to the i-th keypoint.

[0161] When the Euclidean distance between the i-th keypoint and its corresponding mirror point deviates significantly from 0, it indicates a large discrepancy between the i-th keypoint and its mirror point, violating the principle of symmetry and significantly impacting facial deformity. Therefore, the reciprocal of D can be used to assign a smaller weight to the i-th keypoint.

[0162] When calculating the Euclidean distance between a keypoint and its corresponding mirror image, the reciprocal of this Euclidean distance can be taken. This reciprocal can then be normalized to obtain a normalized reciprocal, which is then used as the weight corresponding to the keypoint. This normalized weighting avoids the interference of uneven weight distribution on the results, resulting in more reasonable weights and thus improving the accuracy of the results.

[0163]

[0164] In formula (4), It represents the reciprocal of the Euclidean distance between the i-th keypoint and its corresponding mirror point.

[0165] In calculation At that time, it is possible to Perform normalization to obtain the normalized result. (can be written as " ), that is, the aforementioned target weights, can be used to... The weight (also called "weight factor") of the i-th key point is determined.

[0166] Optionally, when the original configuration and the mirror configuration are obtained, the weight corresponding to each key point (which can be denoted as "w") can be determined by the difference between each key point and the corresponding mirror point of each key point in the mirror configuration, and the attenuation coefficient determined by D. This is because, when D is close to 0, it will lead to infinity, and poor stability, and when D is close to a particularly large value, it will lead to tend to 0, and the change is relatively large, which is easy to introduce noise data. The weight corresponding to each key point determined by the smooth calculation of the difference between each key point and the corresponding mirror point of each key point can not lead to infinity when D is close to 0, but tends to 1, or, when D is close to a particularly large value, reduces the introduction of noise data, so that the weight corresponding to each key point determined has good smoothness and stability.

[0167] For example, first determine the difference between each key point and the corresponding mirror point of each key point and the attenuation coefficient, and then perform smooth calculation by the difference and the attenuation coefficient to obtain the weight corresponding to each key point.

[0168]

[0169] In formula (5), w represents the weight corresponding to each key point determined by the difference between each key point and the corresponding mirror point of each key point and the attenuation coefficient, k represents the attenuation coefficient, represents the 3D coordinate value corresponding to the i-th key point, represents the 3D coordinate value corresponding to the mirror point corresponding to the i-th key point. Wherein, k can be obtained by pre-calibration, which can be 0.05, 0.06 or 0.055, and the embodiments of the present application are not limited thereto. By adjusting k, the attenuation speed of the weight corresponding to each key point with D can be flexibly controlled, and k is negatively correlated with D.

[0170] When the original configuration, the mirror configuration, and the 3D coordinate values corresponding to each of the plurality of soft and hard tissue coordinate points are obtained, or , the spatial structure of the original configuration and the mirror configuration can be aligned by a preset analysis method to obtain a spatial transformation matrix. For ease of illustration, the following embodiments are exemplified by .

[0171] For example, when using Procrustes to analyze the spatial structure of the original configuration and the mirror configuration, the optimal spatial transformation matrix can be found to perform a geometric transformation on the mirror configuration to obtain the minimum weighting function, so that the difference between the original configuration and the geometrically transformed mirror configuration is minimized, and the original configuration and the geometrically transformed mirror configuration are almost identical, that is, the original configuration ≈ the geometrically transformed mirror configuration. In other words, the i-th key point and the mirror point corresponding to the i-th key point in the geometrically transformed mirror configuration almost completely coincide.

[0172]

[0173] In formula (6), Indicates the original configuration. Indicates a mirror configuration. This represents the spatial transformation matrix corresponding to the geometric transformation. This represents the weighting function corresponding to the Procrustes analysis. The geometric transformation can be at least one of rotation, scaling, or translation.

[0174] S5, the equation of the median sagittal plane is obtained through the spatial transformation matrix.

[0175] For example, when the optimal spatial transformation matrix (which can be denoted as "Q′") is obtained, the mirror configuration can be geometrically transformed using this optimal spatial transformation matrix to obtain a mirror configuration with the smallest difference from the original configuration. Principal component analysis is then performed on all mirror points corresponding to the geometrically transformed mirror configuration and all key points corresponding to the original configuration to analyze and derive the median sagittal plane equation corresponding to the original 3D image. The plane corresponding to the median sagittal plane equation in the original 3D image is called the median sagittal plane (e.g., ...). Figure 8 The first page of the middle (801).

[0176] Optionally, when obtaining the equation of the median sagittal plane through Q′, the normal vector of the plane corresponding to Q′ can be calculated first, and then a point (e.g., point P) through which the plane corresponding to Q′ passes in the original plane can be selected. The equation of the median sagittal plane can be determined by the coordinates of point P and the normal vector of the plane corresponding to Q′.

[0177] For example, the normal vector of the plane corresponding to Q′ is n = ( , , The coordinates of point P are ( If ), then the equation of the median sagittal plane can be: (x- )+ (y- )+ (z- ) = 0, (x- )+ (y- )+ (z- Expanding ) = 0, we get the expanded equation: + And the expanded equation + The equation for the median sagittal plane has been determined. Represents a constant term. It is the normal vector of the midsagittal plane.

[0178] S6 minimizes the constraints on the coordinates of the left and right infraorbital points and the coordinates of the left and right ear points to obtain the equation of the horizontal plane.

[0179] For example, when obtaining the midsagittal plane equation, the coordinates of the left infraorbital point can be determined first among multiple key points in soft and hard tissues (such as...). Figure 7 The coordinates of point E in the image), and the coordinates of the right infraorbital point (e.g.) Figure 7 The coordinates of point F in the diagram), and the coordinates of the left ear point (e.g.) Figure 7 (The coordinates of point G in the diagram). The infraorbital point represents the lowest point of the bony lower margin of the orbit. The auricular point represents the highest point of the bony upper margin of the external auditory canal.

[0180] Once the coordinates of the left infraorbital point, right infraorbital point, left ear point, and right ear point are determined, the plane (i.e., the horizontal plane) with the smallest sum of distances to these coordinates can be identified. For example... Figure 8 The second side, 802.

[0181] Suppose that the normal vector corresponding to the horizontal plane (also called the "mid-level plane") is m = ( , , Then the equation of the horizontal plane is: + . Represents a constant term. This is the normal vector to the horizontal plane.

[0182] The distances from the coordinates of the left and right infraorbital points, the left ear point, and the right ear point to the horizontal plane are constrained by the constraint equation shown in formula (6), so that the sum of the distances from these coordinates to the horizontal plane is minimized. Furthermore, the normal vector m corresponding to the horizontal plane and the normal vector n corresponding to the median sagittal plane equation should be perpendicular to each other, that is, the product of the normal vector corresponding to the horizontal plane and the normal vector corresponding to the median sagittal plane equation is 0. + =0, thus solving for , The corresponding values ​​are used to obtain the equation of the horizontal plane.

[0183]

[0184] In formula (7), j represents the number of point coordinates among the left infraorbital point, right infraorbital point, left ear point, and right ear point, and d represents the sum of the distances between the four point coordinates corresponding to the left infraorbital point, right infraorbital point, left ear point, and right ear point and the horizontal plane.

[0185] S7. By combining the midsagittal plane equation and the horizontal plane equation, the coronal plane equation is obtained.

[0186] For example, when determining the equations of the median sagittal plane and the horizontal plane, a plane (i.e., the coronal plane) perpendicular to both the median sagittal plane and the horizontal plane can be determined using these equations. Figure 8 The third page, 803.

[0187] Specifically, by cross-multiplying the normal vector *m* corresponding to the horizontal plane and the normal vector *n* corresponding to the midsagittal plane equation, a normal vector *u* perpendicular to both normal vectors *m* and *n* is calculated. Normal vector *u* is the normal vector corresponding to the coronal plane. Furthermore, the intersection line between the midsagittal plane and the horizontal plane (which can be called the "craniofacial vertical axis") is determined. A point on this intersection line (e.g., point L) is selected, and the coronal plane equation is determined using the coordinates of point L and the normal vector *u*.

[0188] Suppose that the normal vector u = ( , , The coordinates of point L are ( If the coronal plane equation is: (x- )+ (y- )+ (z- ) = 0, (x- )+ (y- )+ (z- Expanding ) = 0, we get the expanded equation: + And the expanded equation + The equation is determined to be the coronal plane equation.

[0189] S8, obtain the rotation matrix through the median sagittal plane equation, the horizontal plane equation and the coronal plane equation.

[0190] Illustratively, when obtaining the median sagittal plane equation, the horizontal plane equation and the coronal plane equation, the normal vector corresponding to each of the median sagittal plane equation, the horizontal plane equation and the coronal plane equation can be determined first, and then the rotation matrix can be determined through the normal vector corresponding to each of the median sagittal plane equation, the horizontal plane equation and the coronal plane equation. Then the rotation angle can be determined through the rotation matrix.

[0191] S9, correct the original 3D image through the rotation matrix to obtain the aligned original 3D image.

[0192] Illustratively, when obtaining the rotation matrix, the rotation angle determined through the rotation matrix can be used to correct the original 3D image to align the original 3D image, and obtain the aligned original 3D image.

[0193] S10, perform facial deformity index measurement through the aligned original 3D image.

[0194] Illustratively, when obtaining the aligned original 3D image, facial deformity index measurement can be performed through the aligned original 3D image to realize highly automated, objective and quantitative measurement of facial deformity, and significantly reduce facial deformity diagnosis errors caused by human judgment.

[0195] Specifically, according to the aligned 3D image, based on the 2D cephalometric theory, the distance or angle between the points or lines projected onto the 2D plane, and the Euclidean distance between two symmetrical key points in the 3D space are evaluated to comprehensively evaluate the degree of facial deformity of the patient.

[0196] In summary, when measuring facial deformities, determining the key coordinates (also known as "3D anatomical key points") on the original 3D image data corresponding to the tomographic scan is crucial for facial deformity assessment and quantification of anatomical abnormalities. A two-stage detection method can be primarily used to detect key coordinates on the original 3D image. First, coarse initial key points are located on the downsampled tomographic image using heatmap regression. Then, patches near the initial key points are cropped at high resolution, and heatmap regression is used again to gradually refine the landmarks, obtaining the final key point coordinates. Based on the automatically detected key point coordinates, the original and mirror configurations are then analyzed. Procrustes analysis is used to align the original and mirror configurations to obtain the optimal spatial transformation matrix, which in turn yields the median sagittal plane equation. Furthermore, the horizontal plane equation is obtained by combining the median sagittal plane equation with the coordinates of the left and right infraorbital points and the left and right auricular points. Finally, the coronal plane equation is obtained by combining the horizontal plane equation with the median sagittal plane equation. Subsequently, a rotation matrix is ​​obtained using the midsagittal plane equation, horizontal plane equation, and coronal plane equation to correct the unerected original 3D image. This rotation matrix is ​​then used to correct the unerected original 3D image, resulting in a corrected original 3D image. Finally, facial deformity indicators are measured using the corrected original 3D image. This improves the accuracy of midsagittal plane construction, allowing doctors to diagnose facial deformities using a more accurate midsagittal plane. This enhances the precision of facial deformity diagnosis, which in turn benefits doctors in treating facial deformities and evaluating postoperative treatment outcomes. Furthermore, instead of manually selecting reference key points, multiple key points corresponding to craniofacial tissues are automatically detected using a cascaded 3D-Unet. This significantly reduces the time and labor costs associated with key point selection, greatly improving the efficiency of key point selection. This high efficiency in key point selection further enhances the diagnostic efficiency of doctors in diagnosing facial deformities. Furthermore, using facial deformity index measurements to diagnose facial deformities can reduce diagnostic errors caused by insufficient doctor experience or subjective factors, ensuring the consistency and repeatability of the assessment results for facial deformity diagnosis.

[0197] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios exemplified. Those skilled in the art can obviously make various equivalent modifications or variations based on the above examples, and such modifications or variations also fall within the scope of the embodiments of this application.

[0198] The above text combined Figures 1 to 8 The facial data processing method provided in the embodiments of this application has been described in detail; the following will be combined with Figure 9 and Figure 10The apparatus embodiments of this application are described in detail below. It should be understood that the apparatus in the embodiments of this application can perform the various methods described in the foregoing embodiments of this application, that is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0199] Figure 9 This is a schematic diagram of the facial data processing device provided in the embodiments of this application.

[0200] For example, such as Figure 9 As shown, the device 900 includes:

[0201] The acquisition module 910 is used to obtain the original coordinates of multiple key points based on the original 3D image, wherein the original 3D image represents a tomographic scan image of the user's face, and the multiple key points represent the location points corresponding to the craniofacial tissues of the user's face.

[0202] The processing module 920 is used to construct an original configuration by performing spatial configuration based on the original coordinates of multiple key points; to construct a mirror configuration by performing spatial configuration based on the mirror coordinates of the mirror points corresponding to each of the multiple key points; to obtain the median sagittal plane corresponding to the original 3D image based on the target distance between the original configuration and the mirror configuration, wherein the overlap between the original configuration and the mirror configuration is negatively correlated with the target distance; and to perform a straightening process on the original 3D image based on the median sagittal plane to obtain a straightened original 3D image, wherein the straightened original 3D image is used for face detection of the user's face.

[0203] In one possible implementation, the processing module 920 is specifically used for:

[0204] Determine the mean of the original coordinates of multiple key points in the first dimension, where the first dimension represents the horizontal dimension of the original 3D image;

[0205] Determine the original plane corresponding to the mean in the first dimension;

[0206] Using the original plane as the center, the original coordinates of multiple key points are horizontally mirrored to obtain the mirror coordinates of multiple mirror points;

[0207] Spatial configuration is constructed based on the mirror coordinates of multiple mirror points to create a mirror configuration.

[0208] In one possible implementation, the processing module 920 is specifically used for:

[0209] The reciprocal of the target distance is normalized to obtain the target weights corresponding to each key point;

[0210] Based on the target weights corresponding to each key point, the midsagittal plane is obtained.

[0211] In a possible implementation, the processing module 920 is specifically configured to:

[0212] obtain a target spatial transformation matrix based on the original configuration, the mirror configuration, and the target weight corresponding to each key point;

[0213] obtain the median sagittal plane based on the target spatial transformation matrix.

[0214] In a possible implementation, the processing module 920 is specifically configured to:

[0215] align the spatial structures of the original configuration and the mirror configuration based on the preset analysis method and the target weight corresponding to each key point, to obtain a minimum weighted function;

[0216] determine the spatial change matrix corresponding to the minimum weighted function as the target spatial transformation matrix.

[0217] In a possible implementation, the processing module 920 is specifically configured to:

[0218] perform image processing on the original 3D image to obtain an image-processed original 3D image;

[0219] extract first coordinates of a plurality of key points in the image-processed original 3D image;

[0220] perform secondary extraction in the image-processed original 3D image based on the first coordinates of the plurality of key points, to obtain original coordinates of the plurality of key points;

[0221] The image processing includes at least one of data resampling and data preprocessing.

[0222] In a possible implementation, the processing module 920 is specifically configured to:

[0223] obtain a horizontal plane corresponding to the original 3D image based on the median sagittal plane, a coordinate of a suborbicular point, and a coordinate of an auricular point in the plurality of key points;

[0224] obtain a coronal plane corresponding to the original 3D image based on the median sagittal plane and the horizontal plane;

[0225] obtain a rotation matrix based on the median sagittal plane, the horizontal plane, and the coronal plane;

[0226] perform alignment processing on the original 3D image based on the rotation matrix, to obtain an aligned original 3D image.

[0227] It should be noted that the apparatus 900 is embodied in the form of functional modules. The term “module” herein can be implemented by software and / or hardware, and is not limited in this regard.

[0228] For example, the "module" can be a software program, hardware circuit, or a combination of both, which implements the above-described functions. The hardware circuit can include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor (for example, a shared processor, a dedicated processor, or a combination processor, etc.) and a memory for executing one or more software or firmware programs, and a combination logic circuit, and / or other suitable components that support the described functions.

[0229] Therefore, the modules of each example described in the embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0230] Figure 10 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0231] For example, as shown in FIG. 1, the electronic device 1000 includes a memory 1010 and a processor 1020, wherein the memory 1010 stores executable program code 1011, and the processor 1020 is configured to invoke and execute the executable program code 1011 to perform a face data processing method. Figure 10

[0232] It should be noted that the electronic device can be a smart device capable of face data processing, including but not limited to: a personal computer, a tablet computer, a handheld device, a vehicle-mounted device, a wearable device, a computing device, or other processing devices connected to a wireless modem, etc. In different networks, the electronic device can be called by different names, such as: user equipment, access electronic device, user unit, user station, mobile station, mobile station, remote station, remote electronic device, mobile device, user electronic device, electronic device, wireless communication device, user agent or user equipment, cellular phone, cordless phone, electronic device in 5G network or future evolution network, etc. The present application does not make any limitation.

[0233] The present application can divide the functional modules of the electronic device according to the above-mentioned method examples, for example, each functional module can be divided, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware. It should be noted that the division of modules in the present embodiment is illustrative, and is only a logical function division. When actually implemented, another division method can be used.

[0234] ​In the case of adopting the respective functional modules corresponding to the respective functions, the electronic device can include an acquisition module, a processing module, and the like. It should be noted that all relevant content of the respective steps involved in the above method embodiments can be cited to the functional description of the corresponding functional modules, which will not be described here.

[0235] The electronic device provided in the present application is used to execute the above face data processing method, and thus can achieve the same effect as the above implementation method.

[0236] In the case of adopting the integrated unit, the electronic device can include a processing module and a storage module. The processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute relevant program codes and data.

[0237] The processing module can be a processor or a controller, which can realize or execute various exemplary logical blocks, modules and circuits shown in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as including one or more microprocessor combinations, a combination of digital signal processing (Digital Signal Processing, DSP) and microprocessor, etc., and the storage module can be a memory.

[0238] The present application also provides a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the steps of the method of any of the preceding embodiments. The computer readable storage medium can include, but is not limited to, any type of disk, including floppy disks, optical disks, DVD (Digital Video Disc), CD-ROM (Compact Disc Read-Only Memory), micro-drives, and magneto-optical disks, ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), DRAM (Dynamic Random Access Memory), VRAM (Video Random Access Memory), flash memory device, magnetic or optical cards, nanosystem (including molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0239] The application further provides a computer program product, which, when running on a computer, causes the computer to perform the above related steps to implement the face data processing method in the above embodiments.

[0240] In addition, the electronic device provided by the embodiments of the application can be a chip, a component or a module, and the electronic device can include a connected processor and a memory; the memory is used to store instructions, and the processor can invoke and execute the instructions when the electronic device is running, so that the chip executes the face data processing method in the above embodiments.

[0241] The electronic device, the computer readable storage medium, the computer program product or the chip provided by the application are all used to execute the corresponding method provided above, and thus the beneficial effects achieved thereby can refer to the beneficial effects of the corresponding method provided above, which will not be described here again.

[0242] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0243] In the embodiments provided by the application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, and for the convenience of description, the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0244] The above is only a specific implementation of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered by the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A facial data processing method, characterized in that, The method includes: Based on the original 3D image, the original coordinates of multiple key points are obtained, wherein the original 3D image represents a tomographic scan image of the user's face, and the multiple key points represent the location points corresponding to the craniofacial tissues of the user's face. Based on the original coordinates of the multiple key points, a spatial configuration is performed to construct the original configuration. Based on the mirror coordinates of the mirror points corresponding to the multiple key points, a spatial configuration is constructed to create a mirror configuration. Based on the target distance between the original configuration and the mirror configuration, the midsagittal plane corresponding to the original 3D image is obtained, wherein the degree of overlap between the original configuration and the mirror configuration is negatively correlated with the target distance; Based on the median sagittal plane, the original 3D image is straightened to obtain a straightened original 3D image, wherein the straightened original 3D image is used for face detection of the user's face; The step of obtaining the median sagittal plane corresponding to the original 3D image based on the target distance between the original configuration and the mirror configuration includes: The reciprocal of the target distance is normalized to obtain the target weights corresponding to each key point; Based on the original configuration, the mirror configuration, and the target weights corresponding to each key point, the target space transformation matrix is ​​obtained; Based on the target space transformation matrix, the median sagittal plane is obtained.

2. The method according to claim 1, characterized in that, The step of constructing a mirror configuration based on the mirror coordinates of the mirror points corresponding to the multiple key points includes: The mean of the original coordinates of the plurality of key points is determined in the first dimension, wherein the first dimension represents the horizontal dimension of the original 3D image; Determine the original plane corresponding to the mean in the first dimension; Centered on the original plane, the original coordinates of the multiple key points are horizontally mirrored to obtain the mirror coordinates of multiple mirror points; The mirror configuration is constructed by performing spatial configuration based on the mirror coordinates of the multiple mirror points.

3. The method according to claim 1, characterized in that, The process of obtaining the target space transformation matrix based on the original configuration, the mirror configuration, and the target weights corresponding to each key point includes: Based on the preset analysis method and the target weights corresponding to each key point, the spatial structure of the original configuration and the mirror configuration is aligned to obtain the minimum weighting function; The spatial transformation matrix corresponding to the minimum weighting function is determined as the target spatial transformation matrix.

4. The method according to any one of claims 1 to 3, characterized in that, The process of obtaining the original coordinates of multiple key points based on the original 3D image includes: The original 3D image is processed to obtain the processed original 3D image; Extract the first coordinates of the multiple key points from the original 3D image after image processing; Based on the first coordinates of the multiple key points, a secondary extraction is performed on the original 3D image after image processing to obtain the original coordinates of the multiple key points. The image processing includes at least one of data resampling and data preprocessing.

5. The method according to any one of claims 1 to 3, characterized in that, The step of straightening the original 3D image based on the median sagittal plane to obtain a straightened original 3D image includes: Based on the median sagittal plane, the coordinates of the infraorbital point and the auricular point among the multiple key points, the horizontal plane corresponding to the original 3D image is obtained; Based on the median sagittal plane and the horizontal plane, the coronal plane corresponding to the original 3D image is obtained; Based on the median sagittal plane, the horizontal plane, and the coronal plane, a rotation matrix is ​​obtained; Based on the rotation matrix, the original 3D image is straightened to obtain the straightened original 3D image.

6. A facial data processing device, characterized in that, The device includes: The acquisition module is used to obtain the original coordinates of multiple key points based on the original 3D image, wherein the original 3D image represents a tomographic scan image of the user's face, and the multiple key points represent the location points corresponding to the craniofacial tissues of the user's face. The processing module is used to construct an original configuration by performing spatial configuration based on the original coordinates of the multiple key points; construct a mirror configuration by performing spatial configuration based on the mirror coordinates of the mirror points corresponding to each of the multiple key points; obtain the median sagittal plane corresponding to the original 3D image based on the target distance between the original configuration and the mirror configuration, wherein the overlap between the original configuration and the mirror configuration is negatively correlated with the target distance; and perform alignment processing on the original 3D image based on the median sagittal plane to obtain an aligned original 3D image, wherein the aligned original 3D image is used for face detection of the user's face. Specifically, the processing module is used to: normalize the reciprocal of the target distance to obtain the target weights corresponding to each key point; obtain the target space transformation matrix based on the original configuration, the mirror configuration, and the target weights corresponding to each key point; and obtain the median sagittal plane based on the target space transformation matrix.

7. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 5.

Images