Method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, and computer-readable recording medium having recorded thereon a program for executing the method on a computer

Abstract

A method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning includes the steps of extracting scan features from the 3D facial scan data using a convolutional neural network, extracting volume features from the 3D volumetric medical image data using the convolutional neural network, and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volume features.

Description

[Technical field]

[0001] The present invention relates to a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, and a computer-readable recording medium having a program recorded thereon for executing the same on a computer. More specifically, the present invention relates to a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, which does not require user input or conversion of data structure, and a computer-readable recording medium having a program recorded thereon for executing the same on a computer. [Background technology]

[0002] 3D volumetric medical image data refers to data that contains hundreds of 2D slide images from Computed Tomography (CT), Cone-beam CT (CBCT), Magnetic Resonance Imaging (MRI), etc. Various tissues and structures of the body can be obtained from 3D image data, so it is widely used in many medical fields, including internal medicine and surgery, for diagnosis, treatment, and surgery planning. In dentistry, oral and maxillofacial surgery, cosmetic surgery, etc., 3D volumetric medical image data is obtained targeting the head and neck region and used to diagnose and treat the patient's maxillofacial and oral cavity.

[0003] The head and neck region is an important body region where not only functional aspects such as breathing, chewing, and speech are emphasized, but also aesthetics. Therefore, maxillofacial and oral procedures and surgical plans reflect both functional considerations and the patient's aesthetic preferences. However, 3D volumetric medical image data only contains single-channel intensity information and does not contain information about the hues of various tissues and structures of the body.

[0004] In addition, to analyze the shape of a specific area in 3D, it is necessary to segment the area or reconstruct the segmented data into a 3D model, which is a time-consuming task. In addition, 3D volumetric medical image data is expensive, and in the case of CT or CBCT, there is a risk of radiation exposure, making it difficult to obtain multiple images. Therefore, it is difficult to compare before and after surgery or procedures, or to use surgery or surgery simulations in various facial expression states.

[0005] To overcome these limitations of 3D volumetric medical image data, 3D face scan mesh model data is also being used. This data includes 3D facial contour information as well as color information, texture, and is inexpensive and does not involve any radiation exposure during the acquisition process. Therefore, it is easy to check the appearance changes before and after surgery or treatment, or with various facial expressions.

[0006] In addition, when used together with 3D volumetric medical image data, it is useful for establishing treatment and surgical plans by considering aesthetic shape and color along with anatomical information within the head and neck. However, since 3D volumetric medical image data and 3D face scan mesh model data have different 3D space coordinate systems, a matching process must be performed to align and overlap these two data. However, manual work in 3D space requires skill and is a time-consuming process even for an expert, so a method to automate this process is needed.

[0007] For this reason, a method for matching 3D face scan mesh model data and 3D volumetric medical image data has been proposed. Since the 3D face scan mesh model data and the 3D volumetric medical image data have different data structures, the conventional method has been to unify the different data structures into one. This can be roughly divided into two methods. The first method is to segment the skin area in the 3D volumetric medical image data, reconstruct it in 3D, unify it into 3D mesh model data, and then perform matching based on the 3D mesh. The second method is to voxelize the 3D face scan mesh data into 3D volumetric medical image data and perform matching based on the image. However, such segmentation and 3D reconstruction are time-consuming tasks. Also, when the image quality of the 3D volumetric medical image data is low, such as CBCT, or when there is a lot of metal noise due to dental prostheses or orthodontic appliances, segmentation itself is difficult, making it difficult to apply the method. In the case of a method in which 3D face scan mesh data is converted into voxels and then matched, the shape information contained in the mesh data is lost, resulting in a decrease in matching accuracy.

[0008] Other methods include a method of manually inputting corresponding points for each of the two data and performing matching based on the matching corresponding points, and a method of attaching a matching marker to the patient, acquiring each data, and then performing matching based on the extracted marker. However, the method using manual corresponding points has limitations in that it is difficult to select accurate corresponding points and consistent results are not obtained depending on the user or the same user's repeated work. In addition, when using markers, data must be acquired with multiple markers attached, which is inconvenient for the patient and requires a separate process of extracting markers or inputting marker positions by the user. Summary of the Invention [Problem to be solved by the invention]

[0009] The objective of the present invention is to provide a method for automatic matching of 3D facial scan data and 3D volumetric medical image data using deep learning, which can quickly and automatically match 3D facial scan data and 3D volumetric medical image data using the 3D facial scan data and 3D volumetric medical image data themselves, without user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0010] Another object of the present invention is to provide a computer-readable recording medium having recorded thereon a program for causing a computer to execute the method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning. [Means for solving the problem]

[0011] To achieve the above objective, one embodiment of a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning includes the steps of: extracting scan features from the 3D facial scan data using a convolutional neural network; extracting volume features from the 3D volumetric medical image data using the convolutional neural network; and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volume features.

[0012] The step of extracting the scan features includes generating a first two-dimensional depth image indicating a distance from a first reference plane positioned outside the 3D facial scan data to a facial surface within the 3D facial scan data.

[0013] An input of the convolutional neural network is the first two-dimensional depth image, and an output of the convolutional neural network is a first two-dimensional coordinate corresponding to the scan feature point.

[0014] The step of extracting the scan features further includes a step of back-projecting the first two-dimensional coordinates into first three-dimensional coordinates based on the transformation method used in the step of generating the first two-dimensional depth image.

[0015] An input of the convolutional neural network is the first 2D depth image, and an output of the convolutional neural network is a first 3D coordinate corresponding to the scan feature point.

[0016] The step of extracting the volumetric features includes generating a second two-dimensional depth image indicating a distance from a second reference plane located outside the three-dimensional volumetric medical image data to a facial surface within the three-dimensional volumetric medical image data.

[0017] The step of generating the second two-dimensional depth image involves proceeding from a starting point of the three-dimensional volumetric medical image data into the three-dimensional volumetric medical image data, and generating a depth value of the second two-dimensional depth image from the position where the change in brightness value is greater than a threshold value for the first time.

[0018] The step of generating the second two-dimensional depth image involves proceeding from a starting point of the three-dimensional volumetric medical image data into the three-dimensional volumetric medical image data, and generating a depth value of the second two-dimensional depth image based on the position where the change in brightness value is greater than a threshold and the brightness value is within a predetermined range of skin brightness values.

[0019] The input of the convolutional neural network is the second 2D depth image, and the output of the convolutional neural network is second 2D coordinates corresponding to the volumetric feature points.

[0020] The step of extracting the volumetric feature points further includes a step of back-projecting the second 2D coordinates onto second 3D coordinates based on the transformation method used in the step of generating the second 2D depth image.

[0021] The input of the convolutional neural network is the second 2D depth image and the output of the convolutional neural network is second 3D coordinates corresponding to the volumetric feature points.

[0022] Extracting the scan features includes imaging the 3D facial scan to generate a first 2D image. An input of the convolutional neural network is the first two-dimensional captured image, and an output of the convolutional neural network is a first three-dimensional coordinate corresponding to the scan feature point.

[0023] Extracting the volumetric features includes imaging the three-dimensional volumetric medical image data to generate a second two-dimensional image. An input of the convolutional neural network is the second two-dimensional captured image, and an output of the convolutional neural network is second three-dimensional coordinates corresponding to the volumetric feature points.

[0024] The number of scan feature points extracted in the step of extracting the scan feature points is three or more, the number of volume feature points extracted in the step of extracting the volume feature points is three or more, and the three or more scan feature points and the three or more volume feature points correspond to each other.

[0025] The scan feature points extracted in the step of extracting the scan feature points and the volume feature points extracted in the step of extracting the volume feature points each include a soft tissue nasion, a right exocanthion, and a left exocanthion.

[0026] The scan feature points extracted in the step of extracting the scan feature points and the scan feature points extracted in the step of extracting the volume feature points are Soft tissue gabella, Soft tissue nasion, Supratip, Pronasale, Columella, Subnasale, Soft tissue A-point, Upper lip anterior point, Stomiom Superious, Lower point of gap between maxillary central incisors, Upper point of gap between mandibular central incisors, Stomiom Inferious, Lower lip anterior point, Soft tissue B-point, Soft tissue pogonion, Soft tissue gnathion, Soft tissue menton, Trichion, Right Orbitale superius, Left Orbitale superius, Right Palpebrale superius, Right Palpebrale inferious, Right Exocanthion, Right Endocanthion, Left Palpebrale superius, Left Palpebrale inferious, Left Endocanthion, Left Exocanthion, Right Includes at least three of the following: Zygion, Left Zygion, Right Cheilion, Left Cheilion, Right Tragus, Left Tragus, Right Soft tissue gonion, Left Soft tissue gonion, Right Alare, Left Alare, Right Alar curvature, and Left Alar curvature.

[0027] The 3D face scan data is mesh data including 3D points (Vertex) and triangular or rectangular surfaces (Triangle) generated by connecting the points.

[0028] The three-dimensional volumetric medical image data is a medical image in which a patient's anatomical structure is acquired in three-dimensional volumetric image data, and the three-dimensional volumetric medical image data is one of a Computed Tomography (CT) image, a Cone-beam CT (CBCT) image, a Magnetic Resonance Imaging (MRI) image, and a Positron Emission Tomography (PET) image.

[0029] After the step of initially matching the 3D facial scan data and the 3D volumetric medical image data, the method further includes a fine matching step of adjusting the distance between a plurality of sampling points of the 3D facial scan data and a plurality of sampling points of the 3D volumetric medical image data to be less than a threshold distance.

[0030] To achieve the above-mentioned object, one embodiment of a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning includes the steps of: extracting scan features from the 3D facial scan data using a first convolutional neural network; extracting volume features from the 3D volumetric medical image data using a second convolutional neural network different from the first convolutional neural network; and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volume features.

[0031] In one embodiment, a method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning to achieve the above object includes the steps of: extracting first scan features from first 3D face scan data using a convolutional neural network; extracting second scan features from second 3D face scan data using the convolutional neural network; extracting volumetric features from the 3D volumetric medical image data using the convolutional neural network; initially matching the first 3D face scan data, the second 3D face scan data, and the 3D volumetric medical image data using the first scan features, the second scan features, and the volumetric features; and calculating a surface distance between the first 3D face scan data and the second 3D face scan data.

[0032] The first 3D face scan data and the second 3D face scan data differ in at least one of the following: imaging device, imaging time, facial expression, shape of a specific part of the face, and coordinate axis.

[0033] A program for causing a computer to execute the method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning is recorded on a computer-readable recording medium. Effect of the Invention

[0034] According to the method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to the present invention, it is possible to quickly and automatically match the 3D facial scan data and 3D volumetric medical image data themselves, without user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0035] Furthermore, since no user input or conversion of data structure is required, the accuracy of matching can be improved. In addition, 3D facial scan data acquired with different expressions and at different times can be automatically aligned with 3D volumetric medical image data, making it useful for patient maxillofacial analysis, surgery, and pre- and post-operative analysis.

[0036] In addition, when the feature points of the 3D face scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced. [Brief description of the drawings]

[0037] [Figure 1] FIG. 1 is a flow chart illustrating a method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning in accordance with one embodiment of the present invention. [Diagram 2] FIG. 2 is a conceptual diagram illustrating an example of detecting scan features from 3D face scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Diagram 3] FIG. 3 is a conceptual diagram illustrating an example of detecting scan features from 3D face scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Figure 4] FIG. 4 is a conceptual diagram illustrating a method for generating a first two-dimensional depth image from three-dimensional face scan data. [Diagram 5] FIG. 5 is a conceptual diagram illustrating a method for generating a second two-dimensional depth image from three-dimensional volumetric medical image data. [Figure 6] FIG. 6 is a conceptual diagram illustrating a method for generating a second two-dimensional depth image from three-dimensional volumetric medical image data. [Figure 7] FIG. 7 is a diagram showing an example of scan feature points and volume feature points. [Figure 8] FIG. 8 is a diagram showing an example of scan feature points and volume feature points. [Figure 9] FIG. 9 is a conceptual diagram showing the initial matching stage of FIG. [Figure 10] FIG. 10 is a conceptual diagram illustrating the detail matching step of FIG. [Figure 11] FIG. 11 is a diagram showing the detail matching result of the detail matching stage of FIG. [Figure 12] FIG. 12 is a conceptual diagram illustrating an example of detecting scan features from 3D face scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Figure 13] FIG. 13 is a conceptual diagram illustrating an example of detecting scan features from 3D face scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Figure 14] FIG. 14 is a diagram showing a first captured image generated by capturing the three-dimensional face scan data. [Figure 15] FIG. 15 is a diagram showing a second captured image generated by capturing three-dimensional volumetric medical image data. [Figure 16] FIG. 16 illustrates an example of automatic alignment of first 3D facial scan data, second 3D facial scan data, and 3D volumetric medical image data. [Figure 17] FIG. 17 illustrates an example of automatic alignment of first 3D facial scan data, second 3D facial scan data, and 3D volumetric medical image data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] With respect to the embodiments of the present invention set forth herein, specific structural or functional descriptions are merely exemplary for purposes of describing the embodiments of the present invention, and the example embodiments of the present invention may be embodied in various forms and should not be construed as being limited to the embodiments set forth herein.

[0039] The present invention can be modified in various ways and can have various forms, and specific embodiments are shown by way of example in the drawings and will be described in detail herein, but it is to be understood that this is not intended to limit the invention to the specific disclosed forms, but rather to include all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0040] Terms such as "first" and "second" are used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another. For example, a first component can be referred to as a second component, and similarly, a second component can be referred to as a first component, without departing from the scope of the present invention.

[0041] When an element is said to be "coupled" or "connected" to another element, it should be understood that it may be directly coupled or connected to the other element, but there may also be other elements in between. On the other hand, when an element is said to be "directly coupled" or "directly connected" to another element, it should be understood that there are no other elements in between. Other expressions describing the relationship between elements, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.

[0042] The terms used in this application are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this application, the terms "include" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0043] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and not as being idealized or overly formal, unless expressly defined in this application.

[0044] However, when an embodiment can be implemented differently, the functions or acts specified in a particular block may occur differently from the sequence specified in the flowchart. For example, two consecutive blocks may actually be performed substantially simultaneously, or the blocks may be performed in reverse depending on the functions or acts involved.

[0045] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same components in the drawings are designated by the same reference numerals, and duplicated descriptions of the same components will be omitted.

[0046] FIG. 1 is a flow chart illustrating a method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning in accordance with one embodiment of the present invention. As shown in Fig. 1, the method for automatic matching of 3D facial scan data and 3D volumetric medical image data includes the steps of: extracting scan features from the 3D facial scan data using a convolutional neural network (S100), extracting volumetric features from the 3D volumetric medical image data using the convolutional neural network (S200), and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features (S300). The automatic matching method further includes, after the step of initially matching the 3D facial scan data and the 3D volumetric medical image data (S300), a fine matching step of adjusting distances between a plurality of sampling points of the 3D facial scan data and a plurality of sampling points of the 3D volumetric medical image data so as to be smaller than a threshold distance (S400).

[0047] The 3D face scan data is mesh data including 3D points (Vertex) and triangles or rectangles generated by connecting the points. The 3D face scan data is image data captured by a 3D scanner. The 3D face scan data is also a 3D face model generated by reconstructing a 2D face photo and a 3D face template model that is matched using feature points of the 2D face photo. The file extension of the 3D face scan data is not limited, and may be, for example, one of ply, obj, and stl.

[0048] The three-dimensional volumetric medical image data is a medical image in which a patient's anatomical structure is acquired as three-dimensional volumetric image data. The three-dimensional volumetric medical image data is one of a Computed Tomography (CT) image, a Cone-beam CT (CBCT) image, a Magnetic Resonance Imaging (MRI) image, and a Positron Emission Tomography (PET) image. In this embodiment, the three-dimensional volumetric medical image data is described by taking a Computed Tomography (CT) image as an example.

[0049] For example, the number of scan feature points extracted in the step of extracting scan feature points is three or more. For example, the number of volume feature points extracted in the step of extracting volume feature points is three or more. The three or more scan feature points and the three or more volume feature points correspond to each other.

[0050] The method for automatically matching 3D facial scan data and 3D volumetric medical image data of the present invention is performed by a computing device. Figure 2 is a conceptual diagram showing an example of detecting scan features from the 3D facial scan data and detecting volume features from the 3D volumetric medical image data in Figure 1. Figure 3 is a conceptual diagram showing an example of detecting scan features from the 3D facial scan data and detecting volume features from the 3D volumetric medical image data in Figure 1. Figure 4 is a conceptual diagram showing a method of generating a first 2D depth image from the 3D facial scan data. Figures 5 and 6 are conceptual diagrams showing a method of generating a second 2D depth image from the 3D volumetric medical image data.

[0051] 1-6, the step of extracting scan features (S100) generates a first 2D depth image showing the distance between a first reference plane located outside the 3D face scan data and a face surface within the 3D face scan data. The first 2D depth image is shown in the upper left of FIG.

[0052] FIG. 2 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is two-dimensional coordinates. For example, the input of the convolutional neural network is the first 2D depth image, and the output of the convolutional neural network is a first 2D coordinate corresponding to the scan feature point.

[0053] Here, the step of extracting the scan feature points (S100) further includes back-projecting the first 2D coordinates onto first 3D coordinates based on a transformation method used in generating the first 2D depth image, the first 3D coordinates representing feature points of the 3D face scan data.

[0054] Similarly, the step of extracting volumetric features (S200) generates a second two-dimensional depth image showing the distance to a second reference plane located outside the three-dimensional volumetric medical image data and to a face surface within the three-dimensional volumetric medical image data, the second two-dimensional depth image being shown in the lower left of FIG.

[0055] For example, the input of the convolutional neural network is the second 2D depth image, and the output of the convolutional neural network is second 2D coordinates corresponding to the volumetric feature points.

[0056] Here, the step of extracting volumetric feature points (S200) further includes back-projecting the second 2D coordinates onto second 3D coordinates based on the transformation method used in generating the second 2D depth image, the second 3D coordinates representing feature points of the 3D volumetric medical image data.

[0057] In Figure 2, x i , y i denotes the first two-dimensional coordinate of the i-th feature point detected from the scan data, and x′ i , y′ i denotes the second two-dimensional coordinates of the i-th feature point detected from the volumetric medical image data.

[0058] The first two-dimensional coordinates in FIG. 2 are back projected onto the 3D face scan data to obtain the scan feature points, and the second two-dimensional coordinates in FIG. 2 are back projected onto the 3D volumetric medical image data to obtain the volumetric feature points.

[0059] FIG. 3 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is a three-dimensional coordinate. For example, an input of the convolutional neural network is the first 2D depth image for the 3D face scan data, and an output of the convolutional neural network is first 3D coordinates corresponding to the scan features.

[0060] Additionally, an input of the convolutional neural network is the second 2D depth image for the 3D volumetric medical image data, and an output of the convolutional neural network is second 3D coordinates corresponding to the volumetric feature points.

[0061] In Figure 3, x i , y i , z i denotes the i-th feature point detected from the face scan data, and x′ i , y′ i , z′ i denotes the i-th feature point detected from the volumetric medical image data.

[0062] In this embodiment, the convolutional neural network for extracting the scan features is the same as the convolutional neural network for detecting the volume features, i.e., the convolutional neural network receives the first 2D depth image for the 3D face scan data and the second 2D depth image for the 3D volumetric medical image data, and outputs the first 2D coordinates or the first 3D coordinates corresponding to the scan features and the second 2D coordinates or the second 3D coordinates corresponding to the volume features.

[0063] Alternatively, the convolutional neural network for extracting the scan features is the same as the convolutional neural network for detecting the volumetric features, i.e., a first convolutional neural network receives the first 2D depth image for the 3D face scan data and outputs the first 2D coordinates or the first 3D coordinates corresponding to the scan features, and a second convolutional neural network different from the first convolutional neural network receives the second 2D depth image for the 3D volumetric medical image data and outputs the second 2D coordinates or the second 3D coordinates corresponding to the volumetric features.

[0064] Figure 4 shows the process of generating a depth image from the 3D face scan data. The minimum distance from the surface of the 3D face scan data to a virtual plane (first reference plane) that exists at a certain distance from the face scan data can be expressed as a brightness value (Intensity) to generate a depth image. In Figure 4, light colors indicate closer distances and dark colors indicate farther distances. Alternatively, the brightness of a depth image can be inverted and expressed in the opposite way.

[0065] Figures 5 and 6 show a process of obtaining a skin region from the 3D volumetric medical image data and generating a depth image using the skin region. Figure 5 shows a process of extracting the skin surface position where the intensity value changes significantly as the air and skin region intersect by moving a virtual ray from the beginning to the end of the volume data in 3D coordinates. When the ray is irradiated at the beginning of the volume data, the position where the intensity value changes significantly for the first time and whose Hounsfield Unit (HU) value is between -100 and 300, which corresponds to skin and fat, can be considered as the skin region.

[0066] For example, the step of generating the second two-dimensional depth image may involve proceeding from a starting point of the three-dimensional volumetric medical image data into the three-dimensional volumetric medical image data, and generating a depth value of the second two-dimensional depth image from the position where the change in brightness value is greater than a threshold value for the first time.

[0067] For example, the step of generating the second two-dimensional depth image may include generating a depth value of the second two-dimensional depth image at a position where a change in brightness value is greater than a threshold value and the brightness value is within a predetermined range of skin brightness values ​​from a start point of the three-dimensional volumetric medical image data into the three-dimensional volumetric medical image data, for example, the predetermined range of skin brightness values ​​is a Hounsfield Unit (HU) value between -100 and 300.

[0068] A depth image can be generated by expressing the minimum distance from the 3D coordinates of the extracted skin surface to a virtual plane (second reference plane) that exists at a certain distance outside the volume data as a brightness value (Intensity). In Figure 6, bright colors indicate closer distances and dark colors indicate farther distances. Alternatively, the brightness of a depth image can be inverted and expressed in the opposite way.

[0069] 7 and 8 are diagrams showing examples of scan feature points and volume feature points. 1 to 8, the scan feature points extracted in the step of extracting the scan feature points (S100) and the scan feature points extracted in the step of extracting the volume feature points (S200) are Soft tissue gabella (1), Soft tissue nasion (2), Supratip (3), Pronasale (4), Columella (5), Subnasale (6), Soft tissue A-point (7), Upper lip anterior point (8), Stomiom Superious (9), Lower point of gap between maxillary central incisors (10), Upper point of gap between mandibular central incisors (11), Stomiom Inferious (12), Lower lip anterior point (13), Soft tissue B-point (14), Soft tissue pogonion (15), Soft tissue gnathion (16), Soft tissue menton (17), Trichion (18), Right Orbitale (19), and Soft tissue nasion (20). superius(19), Left Orbitale superius(20), Right Palpebrale superius(21), Right Palpebrale inferious(22), Right Exocanthion(23), Right Endocanthion(24), Left Palpebrale superius(25), Left Palpebrale inferious(26), Left Endocanthion(27), Left Exocanthion(28), Right Zygion(29), Left Zygion(30), Right Cheilion(31), Left Cheilion(32), Right Tragus(33), Left Tragus(34), Right Soft tissue gonion(35), Left Soft tissue gonion(36), Right Alare(37), LeftContains at least three of the following: Alare (38), Right Alar curvature (39), and Left Alar curvature (40).

[0070] For example, the scan feature points extracted in the step of extracting scan feature points (S100) and the scan feature points extracted in the step of extracting volume feature points (S200) include skin surface feature points and anterior tooth surface feature points.

[0071] Fig. 9 is a conceptual diagram showing the initial matching step (S300) of Fig. 1, Fig. 10 is a conceptual diagram showing the detailed matching step (S400) of Fig. 1, and Fig. 11 is a diagram showing a detailed matching result of the detailed matching step (S400) of Fig. 1.

[0072] 1 to 11, in the initial matching step (S300) based on corresponding feature points, a transformation matrix M can be calculated by landmark transform using a minimum of three pairs of corresponding feature points detected in steps S100 and S200, and an error can be calculated by calculating the difference in distance between the corresponding feature points.

[0073] The transformation matrix M is a transformation that minimizes the difference in distance between corresponding feature points and moves the scan feature points of the 3D face scan data into the domain of the 3D volumetric medical image data. Figure 9 shows an initial matching result using the corresponding feature points.

[0074] For example, the scan feature points extracted in the step of extracting scan feature points (S100) and the volume feature points extracted in the step of extracting volume feature points (S200) include soft tissue nasion, right exocanthion, and left exocanthion, respectively.

[0075] FIG. 10 illustrates the alignment of the 3D facial scan data and the 3D volumetric medical image data using the soft tissue nasion, right exocanthion, and left exocanthion.

[0076] In the minutiae matching step (S400), after the initial matching step (S300), a refined matching is performed to further match the 3D volumetric medical image data and the 3D facial scan data. Figures 10 and 11 show the minutiae matching results.

[0077] From FIG. 10, it can be seen that the fine-grained matching result is in comparison with the initial matching result, and that the 3D volumetric medical image data and the 3D facial scan data are more consistent.

[0078] The detailed matching result is visualized in a multi-view slide image of the 3D volumetric medical image data in Figure 11. The solid outline in Figure 11 corresponds to the skin surface of the 3D face scan data, and it can be seen in Figure 11 that the solid outline showing the skin surface of the 3D face scan data is precisely matched to the 3D volumetric medical image data.

[0079] According to this embodiment, the 3D facial scan data and the 3D volumetric medical image data themselves can be used to quickly and automatically match the 3D facial scan data and the 3D volumetric medical image data, without the need for user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0080] Furthermore, since no user input or conversion of data structure is required, the accuracy of matching can be improved. In addition, when the feature points of the 3D face scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced.

[0081] Fig. 12 is a conceptual diagram showing an example of the steps of detecting scan features from the three-dimensional face scan data and detecting volume features from the three-dimensional volumetric medical image data in Fig. 1. Fig. 13 is a conceptual diagram showing an example of the steps of detecting scan features from the three-dimensional face scan data and detecting volume features from the three-dimensional volumetric medical image data in Fig. 1. Fig. 14 is a diagram showing a first captured image generated by capturing the three-dimensional face scan data. Fig. 15 is a diagram showing a second captured image generated by capturing the three-dimensional volumetric medical image data.

[0082] The automatic matching method according to the present embodiment is similar to the automatic matching method of FIGS. 1 to 11 except for the input of the convolutional neural network, so the same or similar components are denoted by the same reference numerals and duplicated descriptions are omitted.

[0083] 1, 7 to 15, a method for automatically matching 3D facial scan data and 3D volumetric medical image data includes the steps of: extracting scan features from the 3D facial scan data using a convolutional neural network (S100); extracting volumetric features from the 3D volumetric medical image data using the convolutional neural network (S200); and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features (S300). The automatic matching method further includes, after the step of initially matching the 3D facial scan data and the 3D volumetric medical image data (S300), a detailed matching step of adjusting distances between a plurality of sampling points of the 3D facial scan data and a plurality of sampling points of the 3D volumetric medical image data so that they are less than a threshold distance (S400).

[0084] The step of extracting scan features (S100) involves capturing the 3D facial scan data to generate a first 2D captured image, which is shown in the upper left of FIG.

[0085] FIG. 12 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is two-dimensional coordinates. For example, an input of the convolutional neural network is the first two-dimensional captured image, and an output of the convolutional neural network is a first two-dimensional coordinate corresponding to the scan feature point.

[0086] Here, the step of extracting the scan feature points (S100) further includes back-projecting the first 2D coordinates into first 3D coordinates, the first 3D coordinates representing feature points of the 3D face scan data.

[0087] Similarly, the step of extracting volumetric features (S200) involves imaging the three-dimensional volumetric medical image data to generate a second two-dimensional image, as shown in the lower left of FIG. 12 and in FIG. 15.

[0088] For example, an input of the convolutional neural network is the second two-dimensional captured image, and an output of the convolutional neural network is second two-dimensional coordinates corresponding to the volumetric feature points.

[0089] Here, the step of extracting volumetric features (S200) further includes back-projecting the second 2D coordinates onto second 3D coordinates, the second 3D coordinates representing feature points of the 3D face scan data.

[0090] FIG. 13 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is a three-dimensional coordinate. For example, an input of the convolutional neural network is the first 2D captured image of the 3D face scan data, and an output of the convolutional neural network is a first 3D coordinate corresponding to the scan feature.

[0091] Additionally, an input of the convolutional neural network is the second 2D captured image of the 3D volumetric medical image data, and an output of the convolutional neural network is second 3D coordinates corresponding to the volumetric feature points.

[0092] According to this embodiment, the 3D facial scan data and the 3D volumetric medical image data themselves can be used to quickly and automatically match the 3D facial scan data and the 3D volumetric medical image data, without the need for user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0093] Furthermore, since no user input or conversion of data structure is required, the accuracy of matching can be improved. In addition, when the feature points of the 3D face scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced.

[0094] Figure 16 shows an example of automatic alignment of the first 3D face scan data (S1), the second 3D face scan data (S2) and the 3D volumetric medical image data (V1). Figure 17 shows an example of automatic alignment of the first 3D face scan data (S1), the second 3D face scan data (S2) and the 3D volumetric medical image data (V1).

[0095] The automatic matching method of this embodiment is similar to the automatic matching method of Figures 1 to 11 except that two 3D facial scan data are matched into one 3D volumetric medical image data. Therefore, the same or similar components are given the same reference numerals and redundant descriptions are omitted.

[0096] 1 to 17, a method for automatically matching 3D facial scan data and 3D volumetric medical image data includes the steps of: extracting first scan features from a first 3D facial scan data (S1) using a convolutional neural network; extracting second scan features from a second 3D facial scan data (S2) using the convolutional neural network; extracting volumetric features from the 3D volumetric medical image data (V1) using the convolutional neural network; initially matching the first 3D facial scan data (S1), the second 3D facial scan data (S2), and the 3D volumetric medical image data (V1) using the first scan features, the second scan features, and the volumetric features; and calculating a surface distance between the first 3D facial scan data (S1) and the second 3D facial scan data (S2).

[0097] For example, the first scan features, the second scan features, and the volumetric features are extracted using the same convolutional neural network. Alternatively, the first scan feature points and the second scan feature points can be extracted using a first convolutional neural network, and the volumetric feature points can be extracted using a second convolutional neural network.

[0098] For example, the first 3D face scan data (S1) and the second 3D face scan data (S2) differ in at least one of the following: photographing equipment, photographing time, facial expression, shape of specific parts of the face, and coordinate axes.

[0099] FIG. 16 illustrates the alignment of two 3D facial scans (S1, S2) of the same patient, with different imaging modalities, facial expressions and coordinate axes, into one 3D volumetric medical image data set (V1).

[0100] FIG. 17 shows two 3D facial scan data (S1, S2) of the same patient, captured using the same imaging equipment but with different facial expressions and coordinate axes, aligned into a single 3D volumetric medical image data set (V1).

[0101] FIG. 17 shows that the surface distances of the first 3D face scan data (S1) and the second 3D face scan data (S2) are calculated, and the difference between the surface distances is analyzed. As shown in Figures 16 and 17, the 3D facial scan data (S1, S2) with different coordinate axes are aligned to the same 3D volumetric medical image data (V1) and the coordinate axes are aligned with each other, so that the first 3D facial scan data (S1) and the second 3D facial scan data (S2) can be compared and analyzed.

[0102] Fig. 16 shows that 3D facial scan data of the same patient taken from different devices, at different times, and with different facial expressions are aligned into one 3D volumetric medical image data, and the aligned data are compared and analyzed at various angles. Fig. 17 shows that 3D facial scan data of the same patient taken with different facial expressions are aligned into one 3D volumetric medical image data, and the facial change area is analyzed.

[0103] 3D volumetric medical image data is expensive, and in the case of CT or CBCT, it is difficult to obtain multiple times due to the risk of radiation exposure. Therefore, it is difficult to compare before and after surgery or treatment, or to use it for treatment or surgery simulation with various facial expressions. In this embodiment, 3D facial scan data obtained with various expressions and at various times can be automatically matched to 3D volumetric medical image data, so that it can be used for maxillofacial analysis, surgery, and pre- and post-treatment analysis of patients.

[0104] According to this embodiment, the 3D facial scan data and the 3D volumetric medical image data themselves can be used to quickly and automatically match the 3D facial scan data and the 3D volumetric medical image data, without the need for user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0105] Furthermore, since no user input or conversion of data structure is required, the accuracy of matching can be improved. In addition, when the feature points of the 3D face scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced.

[0106] According to an embodiment of the present invention, there is provided a computer readable recording medium having a program recorded thereon for executing the method for automatically matching 3D facial scan data and 3D volumetric medical image data according to the embodiment. The method can be written as a computer executable program and can be embodied in a general purpose digital computer that runs the program using the computer readable medium. Also, data structures used in the method can be recorded on the computer readable medium by various means. The computer readable medium can include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium can be those specifically designed and constructed for the present invention, or those that are known and usable by those of ordinary skill in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. The program instructions include not only machine language codes, such as those created by a compiler, but also high-level language codes executed by a computer using an interpreter, and the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of this invention.

[0107] The method for automatically matching 3D facial scan data and 3D volumetric medical image data may also be embodied in the form of a computer program or application that is recorded on a recording medium and executed by a computer.

[0108] [Industrial Applicability] The present invention relates to a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, and a computer-readable recording medium having a program for executing the method recorded on a computer, which can reduce the effort required for automatic matching and improve accuracy. By automatically matching 3D facial scan data obtained with various expressions and at various times to 3D volumetric medical image data, it can be used for maxillofacial analysis of patients, surgery, and pre- and post-operative analysis.

[0109] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will appreciate that various modifications and variations of the present invention may be made without departing from the spirit and scope of the present invention as set forth in the following claims.

Claims

1. extracting scan features from the 3D face scan data using a convolutional neural network; extracting volumetric features from the three-dimensional volumetric medical image data using the convolutional neural network; performing an initial registration of the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features; A method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning, comprising:

2. The step of extracting scan feature points comprises: generating a first two-dimensional depth image indicative of a distance between a first reference plane located outside the 3D face scan and a facial surface within the 3D face scan.

2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

3. an input of the convolutional neural network is the first two-dimensional depth image; an output of the convolutional neural network is a first two-dimensional coordinate corresponding to the scan feature point; 3. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 2.

4. The step of extracting the scan feature points further includes a step of back-projecting the first two-dimensional coordinates into first three-dimensional coordinates based on a transformation method used in the step of generating the first two-dimensional depth image.

4. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 3.

5. an input of the convolutional neural network is the first two-dimensional depth image; an output of the convolutional neural network is a first three-dimensional coordinate corresponding to the scan feature point; 3. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 2.

6. wherein the step of extracting the volumetric features includes the step of generating a second two-dimensional depth image indicating a distance between a second reference plane located outside the three-dimensional volumetric medical image data and a face surface within the three-dimensional volumetric medical image data.

2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

7. The step of generating the second two-dimensional depth image includes generating a depth value of the second two-dimensional depth image based on a position where a change in brightness value is greater than a threshold value by proceeding from a start point of the three-dimensional volumetric medical image data into the three-dimensional volumetric medical image data.

7. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 6.

8. The step of generating the second two-dimensional depth image is characterized in that the position of a location where a change in brightness value is greater than a threshold value and the brightness value is within a predetermined range of skin brightness values ​​from a start point of the three-dimensional volumetric medical image data to the inside of the three-dimensional volumetric medical image data is generated as a depth value of the second two-dimensional depth image.

7. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 6.

9. an input of the convolutional neural network is the second two-dimensional depth image; an output of the convolutional neural network is a second two-dimensional coordinate corresponding to the volume feature point.

7. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 6.

10. The step of extracting the volumetric feature points further includes a step of back-projecting the second two-dimensional coordinates onto second three-dimensional coordinates based on a transformation method used in the step of generating the second two-dimensional depth image.

10. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 9.

11. an input of the convolutional neural network is the second two-dimensional depth image; an output of the convolutional neural network is a second three-dimensional coordinate corresponding to the volume feature point.

7. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 6.

12. The step of extracting the scan features includes capturing the 3D facial scan data to generate a first captured 2D image.

2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

13. an input of the convolutional neural network is the first two-dimensional captured image; an output of the convolutional neural network is a first three-dimensional coordinate corresponding to the scan feature point; 13. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 12.

14. The step of extracting volumetric features includes imaging the three-dimensional volumetric medical image data to generate a second two-dimensional image.

13. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 12.

15. an input of the convolutional neural network is the second two-dimensional captured image; an output of the convolutional neural network is a second three-dimensional coordinate corresponding to the volume feature point.

15. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 14.

16. The number of the scan feature points extracted in the step of extracting the scan feature points is three or more; The number of the volume feature points extracted in the step of extracting the volume feature points is three or more. the three or more scan feature points and the three or more volume feature points correspond to each other, 2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

17. The scan feature points extracted in the step of extracting the scan feature points and the volume feature points extracted in the step of extracting the volume feature points each include a soft tissue nasion, a right exocanthion, and a left exocanthion.

17. The method for automatic matching of 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 16.

18. The scan feature points extracted in the step of extracting the scan feature points and the scan feature points extracted in the step of extracting the volume feature points are Soft tissue gabella, Soft tissue nasion, Supratip, Pronasale, Columella, Subnasale, Soft tissue A-point, Upper lip anterior point, Stomiom Superious, Lower point of gap between maxillary central incisors, Upper point of gap between mandibular central incisors, Stomiom Inferious, Lower lip anterior point, Soft tissue B-point, Soft tissue pogonion, Soft tissue gnathion, Soft tissue menton, Trichion, Right Orbitale superius, Left Orbitale superius, Right Palpebrale superius, Right Palpebrale inferious, Right Exocanthion, Right Endocanthion, Left Palpebrale superius, Left Palpebrale inferious, Left Endocanthion, Left Exocanthion, Right The present invention is characterized in that it includes at least three of Zygion, Left Zygion, Right Cheilion, Left Cheilion, Right Tragus, Left Tragus, Right Soft tissue gonion, Left Soft tissue gonion, Right Alare, Left Alare, Right Alar curvature, and Left Alar curvature.

2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

19. The 3D face scan data is mesh data including 3D points (Vertex) and triangular or quadrangular surfaces (Triangle) generated by connecting the points.

2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

20. the three-dimensional volumetric medical image data being medical images of a patient's anatomy captured in three-dimensional volumetric image data; The three-dimensional volume medical image data is one of a Computed Tomography (CT) image, a Cone-beam CT (CBCT) image, a Magnetic Resonance Imaging (MRI) image, and a Positron Emission Tomography (PET) image.

2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

21. and after the step of initial alignment of the 3D facial scan data and the 3D volumetric medical image data, the method further comprises a fine alignment step of adjusting a distance between a plurality of sampling points of the 3D facial scan data and a plurality of sampling points of the 3D volumetric medical image data to be less than a threshold distance.

2. The method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning as claimed in claim 1.

22. extracting scan features from the 3D face scan data using a first convolutional neural network; extracting volumetric features from the three-dimensional volumetric medical image data using a second convolutional neural network different from the first convolutional neural network; performing an initial registration of the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features; A method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning, comprising:

23. extracting first scan features from the first 3D face scan using a convolutional neural network; extracting second scan features from a second 3D face scan using the convolutional neural network; and extracting volumetric features from the three-dimensional volumetric medical image data using the convolutional neural network; performing an initial alignment of the first 3D facial scan data, the second 3D facial scan data, and the 3D volumetric medical image data using the first scan features, the second scan features, and the volumetric features; calculating a surface distance between the first 3D face scan data and the second 3D face scan data; A method for automatically matching 3D face scan data and 3D volumetric medical image data using deep learning, comprising:

24. The first 3D face scan data and the second 3D face scan data are different in at least one of the following: a photographing device, a photographing time, a facial expression, a shape of a specific part of a face, and a coordinate axis.

24. A method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning as claimed in claim 23.

25. A computer-readable recording medium having a program recorded thereon for causing a computer to execute the method according to any one of claims 1 to 24.

Images