A method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, and a computer-readable recording medium having recorded thereon a program for executing the method on a computer

Abstract

A method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning includes the steps of extracting scan features from the 3D facial scan data using a convolutional neural network, extracting volume features from the 3D volumetric medical image data using the convolutional neural network, and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volume features.

Description

[Technical Field]

[0001] The present invention relates to a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, and a computer-readable recording medium having a program recorded thereon for executing the method on a computer. More specifically, the present invention relates to a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, which does not require user input or data structure conversion, and a computer-readable recording medium having a program recorded thereon for executing the method on a computer. [Background technology]

[0002] 3D volumetric medical image data refers to data that contains hundreds of 2D slide images, such as those from Computed Tomography (CT), Cone-beam CT (CBCT), and Magnetic Resonance Imaging (MRI). Because various tissues and structures of the body can be obtained from 3D image data, it is widely used in many medical fields, including internal medicine and surgery, for diagnosis, treatment, and surgical planning. In dentistry, orthodontic surgery, and cosmetic surgery, 3D volumetric medical image data is acquired targeting the head and neck region and used to diagnose and treat patients' maxillofacial and oral regions.

[0003] The head and neck region is an important body region where aesthetics are also emphasized, as well as functional aspects such as breathing, chewing, and speech. Therefore, maxillofacial and oral surgery and surgical planning reflect both functional considerations and the patient's aesthetic preferences. However, 3D volumetric medical image data only contains single-channel intensity information and does not include information about the color of various tissues and structures in the body.

[0004] Furthermore, analyzing the shape of a specific region in 3D requires segmenting the region or reconstructing the segmented data into a 3D model, which is a time-consuming process. Furthermore, 3D volumetric medical image data is expensive, and in the case of CT or CBCT, there is a risk of radiation exposure, making it difficult to acquire multiple images. Therefore, it has been difficult to compare before and after surgery or procedures, or to use surgery or surgical simulations under various facial expression conditions.

[0005] To overcome these limitations of 3D volumetric medical image data, 3D facial scan mesh model data is also being utilized. This data includes 3D facial contour information as well as color information (texture), is inexpensive, and does not involve any radiation exposure during acquisition. This makes it easy to check changes in appearance before and after surgery or treatment, or with various facial expressions.

[0006] Furthermore, when used with 3D volumetric medical image data, it is useful for establishing treatment and surgical plans by considering aesthetic shape and color along with anatomical information within the head and neck. However, because 3D volumetric medical image data and 3D facial scan mesh model data have different 3D spatial coordinate systems, a matching process must be performed to align and superimpose these two data. However, manual work in 3D space requires skill and is a time-consuming process even for experts, so a method to automate this process is needed.

[0007] For this reason, methods for aligning 3D facial scan mesh model data and 3D volumetric medical image data have been proposed. Because 3D facial scan mesh model data and 3D volumetric medical image data have different data structures, conventional methods have focused on unifying the different data structures into one. These methods can be broadly divided into two. The first involves segmenting and 3D reconstruction of the skin region in the 3D volumetric medical image data, unifying it into 3D mesh model data, and then performing 3D mesh-based alignment. The second involves voxelizing the 3D facial scan mesh data into 3D volumetric medical image data and performing image-based alignment. However, such segmentation and 3D reconstruction are time-consuming tasks. Furthermore, when the image quality of the 3D volumetric medical image data is low, such as with CBCT, or when there is significant metallic noise from dental prostheses or orthodontic appliances, segmentation itself is difficult, making this method difficult to apply. In the case of a method in which 3D face scan mesh data is converted into voxels and then matched, the shape information contained in the mesh data is lost, resulting in a decrease in matching accuracy.

[0008] Other methods include manually inputting corresponding points for each of the two sets of data and performing matching based on the matching corresponding points, or attaching matching markers to the patient, acquiring each set of data, and then performing matching based on the extracted markers. However, the method using manual corresponding points has limitations in that it is difficult to select accurate corresponding points and consistent results cannot be obtained depending on the user or the same user's repeated work. Furthermore, when using markers, data must be acquired with multiple markers attached, which is inconvenient for the patient and requires the user to go through a separate process of extracting markers or inputting marker positions. Summary of the Invention [Problem to be solved by the invention]

[0009] The objective of the present invention is to provide a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, which can quickly and automatically match 3D facial scan data and 3D volumetric medical image data using the 3D facial scan data and 3D volumetric medical image data themselves, without requiring user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0010] Another object of the present invention is to provide a computer-readable recording medium having recorded thereon a program for causing a computer to execute the method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning. [Means for solving the problem]

[0011] To achieve the above object, one embodiment of a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning includes the steps of: extracting scan features from the 3D facial scan data using a convolutional neural network; extracting volume features from the 3D volumetric medical image data using the convolutional neural network; and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volume features.

[0012] Extracting the scan features includes generating a first two-dimensional depth image indicating the distance from a first reference plane located outside the three-dimensional facial scan data to a facial surface within the three-dimensional facial scan data.

[0013] The input of the convolutional neural network is the first two-dimensional depth image, and the output of the convolutional neural network is first two-dimensional coordinates corresponding to the scanned feature points.

[0014] The step of extracting the scan feature points further includes a step of back-projecting the first two-dimensional coordinates into first three-dimensional coordinates based on the transformation method used in the step of generating the first two-dimensional depth image.

[0015] The input of the convolutional neural network is the first two-dimensional depth image, and the output of the convolutional neural network is first three-dimensional coordinates corresponding to the scan feature points.

[0016] The step of extracting the volumetric features includes generating a second two-dimensional depth image indicating the distance from a second reference plane positioned outside the three-dimensional volumetric medical image data to the facial surface within the three-dimensional volumetric medical image data.

[0017] The step of generating the second two-dimensional depth image involves proceeding from the starting point of the three-dimensional volume medical image data into the interior of the three-dimensional volume medical image data, and generating the depth value of the second two-dimensional depth image from the position where the change in brightness value is greater than a threshold value for the first time.

[0018] The step of generating the second two-dimensional depth image involves proceeding from the starting point of the three-dimensional volume medical image data into the interior of the three-dimensional volume medical image data, and generating the depth value of the second two-dimensional depth image at the position where the change in brightness value is greater than a threshold and the brightness value is within a predetermined range of skin brightness values.

[0019] The input of the convolutional neural network is the second two-dimensional depth image, and the output of the convolutional neural network is second two-dimensional coordinates corresponding to the volumetric feature points.

[0020] The step of extracting the volumetric feature points further includes a step of back-projecting the second two-dimensional coordinates onto second three-dimensional coordinates based on the transformation method used in the step of generating the second two-dimensional depth image.

[0021] The input of the convolutional neural network is the second 2D depth image, and the output of the convolutional neural network is second 3D coordinates corresponding to the volumetric feature points.

[0022] Extracting the scan features includes imaging the three-dimensional facial scan to generate a first two-dimensional image. An input of the convolutional neural network is the first two-dimensional captured image, and an output of the convolutional neural network is a first three-dimensional coordinate corresponding to the scanned feature point.

[0023] Extracting the volumetric features includes imaging the three-dimensional volumetric medical image data to generate a second two-dimensional imaged image. An input of the convolutional neural network is the second two-dimensional captured image, and an output of the convolutional neural network is second three-dimensional coordinates corresponding to the volumetric feature points.

[0024] The number of scan feature points extracted in the step of extracting scan feature points is three or more, the number of volume feature points extracted in the step of extracting volume feature points is three or more, and the three or more scan feature points and the three or more volume feature points correspond to each other.

[0025] The scan feature points extracted in the step of extracting the scan feature points and the volume feature points extracted in the step of extracting the volume feature points include soft tissue nasion, right exocanthion, and left exocanthion, respectively.

[0026] The scan feature points extracted in the step of extracting the scan feature points and the scan feature points extracted in the step of extracting the volume feature points are Soft tissue gabella, Soft tissue nasion, Supratip, Pronasale, Columnella, Subnasale, Soft tissue A-point, Upper lip anterior point, Stomiom superius, Lower point of gap between maxillary central incisors, Upper point of gap between mandibular central incisors, Stomiom inferious, Lower lip anterior point, Soft tissue B-point, Soft tissue pogonion, Soft tissue gnathion, Soft tissue menton, Trichion, Right Orbitale superius, Left Orbitale superius, Right Palpebrale superius, Right Palpebrale inferious, Right Exocanthion, Right Endocanthion, Left Palpebrale superius, Left Palpebrale inferious, Left Endocanthion, Left Exocanthion, and Right Includes at least three of Zygion, Left Zygion, Right Cheilion, Left Cheilion, Right Tragus, Left Tragus, Right Soft tissue gonion, Left Soft tissue gonion, Right Alare, Left Alare, Right Alar curvature and Left Alar curvature.

[0027] The 3D face scan data is mesh data including 3D points (vertices) and triangular or quadrangular surfaces (triangles) generated by connecting the points.

[0028] The three-dimensional volumetric medical image data is a medical image in which a patient's anatomical structure is acquired in three-dimensional volumetric image data, and the three-dimensional volumetric medical image data is one of a Computed Tomography (CT) image, a Cone-beam CT (CBCT) image, a Magnetic Resonance Imaging (MRI) image, and a Positron Emission Tomography (PET) image.

[0029] After the step of initially matching the 3D facial scan data and the 3D volumetric medical image data, the method further includes a fine matching step of adjusting the distance between the plurality of sampling points of the 3D facial scan data and the plurality of sampling points of the 3D volumetric medical image data so that it is less than a threshold distance.

[0030] To achieve the above object, one embodiment of a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning includes the steps of: extracting scan features from the 3D facial scan data using a first convolutional neural network; extracting volume features from the 3D volumetric medical image data using a second convolutional neural network different from the first convolutional neural network; and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volume features.

[0031] To achieve the above object, one embodiment of a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning includes the steps of: extracting first scan features from first 3D facial scan data using a convolutional neural network; extracting second scan features from second 3D facial scan data using the convolutional neural network; extracting volumetric features from the 3D volumetric medical image data using the convolutional neural network; initially matching the first 3D facial scan data, the second 3D facial scan data, and the 3D volumetric medical image data using the first scan features, the second scan features, and the volumetric features; and calculating a surface distance between the first 3D facial scan data and the second 3D facial scan data.

[0032] The first 3D face scan data and the second 3D face scan data differ in at least one of the following: imaging equipment, imaging time, facial expression, shape of a specific part of the face, and coordinate axis.

[0033] A program for causing a computer to execute the method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning is recorded on a computer-readable recording medium. [Effects of the Invention]

[0034] The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to the present invention enables fast automatic matching using the 3D facial scan data and 3D volumetric medical image data themselves, without requiring user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0035] Furthermore, since no user input or data structure conversion is required, the accuracy of matching can be improved. In addition, 3D facial scan data acquired at various times and with various facial expressions can be automatically aligned with 3D volumetric medical image data, making it useful for maxillofacial analysis, surgery, and pre- and post-operative analysis of patients.

[0036] Furthermore, if the feature points of the 3D facial scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced. [Brief explanation of the drawings]

[0037] [Figure 1] FIG. 1 is a flowchart illustrating a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to one embodiment of the present invention. [Figure 2] FIG. 2 is a conceptual diagram illustrating an example of detecting scan features from 3D facial scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Figure 3] FIG. 3 is a conceptual diagram illustrating an example of detecting scan features from 3D facial scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Figure 4] FIG. 4 is a conceptual diagram illustrating a method for generating a first two-dimensional depth image from three-dimensional facial scan data. [Figure 5] FIG. 5 is a conceptual diagram illustrating a method for generating a second two-dimensional depth image from three-dimensional volumetric medical image data. [Figure 6] FIG. 6 is a conceptual diagram illustrating a method for generating a second two-dimensional depth image from three-dimensional volumetric medical image data. [Figure 7] FIG. 7 is a diagram showing examples of scan feature points and volume feature points. [Figure 8] FIG. 8 is a diagram showing examples of scan feature points and volume feature points. [Figure 9] FIG. 9 is a conceptual diagram illustrating the initial matching stage of FIG. [Figure 10] FIG. 10 is a conceptual diagram illustrating the detail matching step of FIG. [Figure 11] FIG. 11 is a diagram showing the detail matching result of the detail matching stage of FIG. [Figure 12] FIG. 12 is a conceptual diagram illustrating an example of detecting scan features from 3D facial scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Figure 13] FIG. 13 is a conceptual diagram illustrating an example of detecting scan features from 3D facial scan data and detecting volumetric features from 3D volumetric medical image data in FIG. [Figure 14] FIG. 14 is a diagram showing a first captured image generated by capturing three-dimensional face scan data. [Figure 15] FIG. 15 is a diagram showing a second captured image generated by capturing three-dimensional volume medical image data. [Figure 16] FIG. 16 illustrates an example of automatic alignment of first 3D facial scan data, second 3D facial scan data, and 3D volumetric medical image data. [Figure 17] FIG. 17 illustrates an example of automatic alignment of first 3D facial scan data, second 3D facial scan data, and 3D volumetric medical image data. DETAILED DESCRIPTION OF THE INVENTION

[0038] Specific structural or functional descriptions of the embodiments of the present invention set forth herein are merely exemplary for purposes of describing the embodiments of the present invention, and the example embodiments of the present invention may be embodied in various forms and should not be construed as being limited to the embodiments set forth herein.

[0039] The present invention can be modified in various ways and can have various forms, and specific embodiments are shown by way of example in the drawings and will be described in detail herein, but it should be understood that this is not to limit the invention to the particular forms disclosed, but rather to include all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

[0040] Terms such as "first" and "second" are used to describe various components, but the components should not be limited by these terms. These terms are used only to distinguish one component from another. For example, a first component can be referred to as a second component, and similarly, a second component can be referred to as a first component, without departing from the scope of the present invention.

[0041] When a component is said to be "coupled" or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, but there may also be other components in between. On the other hand, when a component is said to be "directly coupled" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationship between components, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.

[0042] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or possible addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0043] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an idealized or overly formal meaning unless expressly defined in this application.

[0044] However, when an embodiment can be implemented differently, the functions or operations specified in a particular block may occur differently from the sequence specified in the flowchart. For example, two consecutive blocks may actually occur substantially simultaneously, or the blocks may be reversed depending on the functions or operations involved.

[0045] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same components in the drawings are designated by the same reference numerals, and duplicated descriptions of the same components will be omitted.

[0046] FIG. 1 is a flowchart illustrating a method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to one embodiment of the present invention. 1, a method for automatically matching 3D facial scan data and 3D volumetric medical image data includes the steps of: extracting scan features from the 3D facial scan data using a convolutional neural network (S100); extracting volumetric features from the 3D volumetric medical image data using the convolutional neural network (S200); and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features (S300). The automatic matching method further includes, after the step of initially matching the 3D facial scan data and the 3D volumetric medical image data (S300), a fine-tuning step of adjusting distances between a plurality of sampling points in the 3D facial scan data and a plurality of sampling points in the 3D volumetric medical image data so that they are less than a threshold distance (S400).

[0047] The 3D face scan data is mesh data including 3D points (vertices) and triangles or rectangles generated by connecting the points. The 3D face scan data is image data captured by a 3D scanner. The 3D face scan data is also a 3D face model generated by reconstructing a 2D face photograph and a 3D face template model that is matched using feature points from the 2D face photograph. The file extension of the 3D face scan data is not limited, and may be one of ply, obj, or stl.

[0048] The three-dimensional volumetric medical image data is a medical image in which a patient's anatomical structure is acquired as three-dimensional volumetric image data. The three-dimensional volumetric medical image data is one of a computed tomography (CT) image, a cone-beam CT (CBCT) image, a magnetic resonance imaging (MRI) image, and a positron emission tomography (PET) image. In this embodiment, the three-dimensional volumetric medical image data will be described taking a computed tomography (CT) image as an example.

[0049] For example, the number of scan feature points extracted in the step of extracting scan feature points is three or more. For example, the number of volume feature points extracted in the step of extracting volume feature points is three or more. The three or more scan feature points and the three or more volume feature points correspond to each other.

[0050] The method for automatically matching 3D facial scan data and 3D volumetric medical image data of the present invention is performed by a computing device. Figure 2 is a conceptual diagram illustrating an example of detecting scan features from 3D facial scan data and detecting volume features from 3D volumetric medical image data in Figure 1. Figure 3 is a conceptual diagram illustrating an example of detecting scan features from 3D facial scan data and detecting volume features from 3D volumetric medical image data in Figure 1. Figure 4 is a conceptual diagram illustrating a method for generating a first 2D depth image from 3D facial scan data. Figures 5 and 6 are conceptual diagrams illustrating a method for generating a second 2D depth image from 3D volumetric medical image data.

[0051] 1-6, the step of extracting scan features (S100) generates a first 2D depth image showing the distance between a first reference plane located outside the 3D facial scan data and the facial surface within the 3D facial scan data. The first 2D depth image is shown in the upper left of FIG.

[0052] FIG. 2 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is two-dimensional coordinates. For example, the input of the convolutional neural network is the first two-dimensional depth image, and the output of the convolutional neural network is first two-dimensional coordinates corresponding to the scanned feature points.

[0053] Here, the step of extracting the scan feature points (S100) further includes back-projecting the first 2D coordinates onto first 3D coordinates based on the transformation method used in the step of generating the first 2D depth image, where the first 3D coordinates represent the feature points of the 3D face scan data.

[0054] Similarly, the step of extracting volumetric features (S200) generates a second two-dimensional depth image showing the distance to a second reference plane located outside the three-dimensional volumetric medical image data and the facial surface within the three-dimensional volumetric medical image data, as shown in the lower left of Figure 2.

[0055] For example, the input of the convolutional neural network is the second 2D depth image, and the output of the convolutional neural network is second 2D coordinates corresponding to the volumetric feature points.

[0056] Here, the step of extracting volumetric feature points (S200) further includes a step of back-projecting the second two-dimensional coordinates onto second three-dimensional coordinates based on the transformation method used in the step of generating the second two-dimensional depth image, where the second three-dimensional coordinates represent feature points of the three-dimensional volumetric medical image data.

[0057] In Figure 2, x i , y i denotes the first two-dimensional coordinate of the i-th feature point detected from the scan data, and x′ i , y′ i denotes the second two-dimensional coordinates of the i-th feature point detected from the volumetric medical image data.

[0058] The first two-dimensional coordinates in FIG. 2 are back-projected onto the three-dimensional face scan data to obtain the scan feature points, and the second two-dimensional coordinates in FIG. 2 are back-projected onto the three-dimensional volumetric medical image data to obtain the volumetric feature points.

[0059] FIG. 3 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is a three-dimensional coordinate. For example, the input of the convolutional neural network is the first 2D depth image for the 3D face scan data, and the output of the convolutional neural network is first 3D coordinates corresponding to the scan feature points.

[0060] Additionally, the input of the convolutional neural network is the second two-dimensional depth image for the three-dimensional volumetric medical image data, and the output of the convolutional neural network is second three-dimensional coordinates corresponding to the volumetric feature points.

[0061] In Figure 3, x i , y i , z i denotes the i-th feature point detected from the face scan data, and x′ i , y′ i , z′ i denotes the i-th feature point detected from the volumetric medical image data.

[0062] In this embodiment, the convolutional neural network for extracting the scan features is the same as the convolutional neural network for detecting the volume features, i.e., the convolutional neural network receives the first 2D depth image for the 3D face scan data and the second 2D depth image for the 3D volumetric medical image data, and outputs the first 2D coordinates or the first 3D coordinates corresponding to the scan features and the second 2D coordinates or the second 3D coordinates corresponding to the volume features.

[0063] Alternatively, the convolutional neural network for extracting the scan feature points is the same as the convolutional neural network for detecting the volume feature points. That is, a first convolutional neural network receives the first two-dimensional depth image of the 3D face scan data and outputs the first two-dimensional coordinates or the first three-dimensional coordinates corresponding to the scan feature points. A second convolutional neural network, different from the first convolutional neural network, receives the second two-dimensional depth image of the 3D volumetric medical image data and outputs the second two-dimensional coordinates or the second three-dimensional coordinates corresponding to the volume feature points.

[0064] Figure 4 illustrates the process of generating a depth image from the 3D facial scan data. The minimum distance from the surface of the 3D facial scan data to a virtual plane (first reference plane) that exists at a certain distance from the facial scan data is expressed as a brightness value (intensity), and a depth image can be generated. In Figure 4, bright colors indicate closer distances and dark colors indicate farther distances. Alternatively, the brightness of a depth image can be inverted and expressed in the opposite way.

[0065] 5 and 6 show the process of extracting a skin region from the 3D volumetric medical image data and generating a depth image using the skin region. Figure 5 shows the process of extracting, in 3D coordinates, the location of the skin surface where the intensity value changes significantly as the air and skin region intersect when a virtual ray moves from the beginning to the end of the volume data. When the ray is irradiated at the beginning of the volume data, the location where the intensity value changes significantly first and whose Hounsfield Unit (HU) value is between -100 and 300, which corresponds to skin and fat, can be considered the skin region.

[0066] For example, the step of generating the second two-dimensional depth image involves proceeding from the starting point of the three-dimensional volume medical image data into the interior of the three-dimensional volume medical image data, and generating the depth value of the second two-dimensional depth image from the position where the change in brightness value is greater than a threshold value for the first time.

[0067] For example, the step of generating the second two-dimensional depth image may involve proceeding from a starting point of the three-dimensional volumetric medical image data to the interior of the three-dimensional volumetric medical image data until a change in brightness value is greater than a threshold and the brightness value is within a predetermined skin brightness value range, where the predetermined skin brightness value range is, for example, a Hounsfield Unit (HU) value between -100 and 300.

[0068] A depth image can be generated by expressing the minimum distance from the 3D coordinates of the extracted skin surface to a virtual plane (second reference plane) that exists at a certain distance outside the volume data as a brightness value (intensity). In Figure 6, bright colors indicate closer distances and dark colors indicate farther distances. Alternatively, the brightness of a depth image can be inverted and expressed in the opposite way.

[0069] 7 and 8 are diagrams showing examples of scan feature points and volume feature points. 1 to 8, the scan feature points extracted in the step of extracting scan feature points (S100) and the scan feature points extracted in the step of extracting volume feature points (S200) are Soft tissue gabella (1), Soft tissue nasion (2), Supratip (3), Pronasale (4), Columella (5), Subnasale (6), Soft tissue A-point (7), Upper lip anterior point (8), Stomiom Superius (9), Lower point of gap between maxillary central incisors (10), Upper point of gap between mandibular central incisors (11), Stomiom Inferior (12), Lower lip anterior point (13), Soft tissue B-point (14), Soft tissue pogonion (15), Soft tissue gnathion (16), Soft tissue menton (17), Trichion (18), Right Orbital superius(19), Left Orbitale superius(20), Right Palpebrale superius(21), Right Palpebrale inferious(22), Right Exocanthion(23), Right Endocanthion(24), Left Palpebrale superius(25), Left Palpebrale inferious(26), Left Endocanthion(27), Left Exocanthion(28), Right Zygion(29), Left Zygion(30), Right Cheilion(31), Left Cheilion(32), Right Tragus(33), Left Tragus(34), Right Soft tissue gonion(35), Left Soft tissue gonion(36), Right Alare(37), LeftContains at least three of the following: Alare (38), Right Alert curvature (39), and Left Alert curvature (40).

[0070] For example, the scan feature points extracted in the step of extracting scan feature points (S100) and the scan feature points extracted in the step of extracting volume feature points (S200) include skin surface feature points and anterior tooth surface feature points.

[0071] Fig. 9 is a conceptual diagram showing the initial matching step (S300) of Fig. 1, Fig. 10 is a conceptual diagram showing the detailed matching step (S400) of Fig. 1. Fig. 11 is a diagram showing the detailed matching result of the detailed matching step (S400) of Fig. 1.

[0072] Referring to FIGS. 1 to 11, in the initial matching step (S300) based on corresponding feature points, a transformation matrix M can be calculated using landmark transform using at least three pairs of corresponding feature points detected in steps S100 and S200, and an error can be calculated by calculating the difference in distance between the corresponding feature points.

[0073] The transformation matrix M is a transformation that minimizes the difference in distance between corresponding feature points, and moves the scanned feature points of the 3D facial scan data into the domain of the 3D volumetric medical image data. Figure 9 shows the initial matching result using the corresponding feature points.

[0074] For example, the scan feature points extracted in the step of extracting scan feature points (S100) and the volume feature points extracted in the step of extracting volume feature points (S200) include soft tissue nasion, right exocanthion, and left exocanthion, respectively.

[0075] FIG. 10 illustrates the alignment of the 3D facial scan data and the 3D volumetric medical image data using the Soft Tissue Indentation, Right Exocanthion, and Left Exocanthion.

[0076] In the detailed matching step (S400), after the initial matching step (S300), a refined matching is performed to further match the 3D volumetric medical image data with the 3D facial scan data. Figures 10 and 11 show the detailed matching results.

[0077] From FIG. 10, it can be seen that the detailed matching result is in comparison with the initial matching result, and the 3D volumetric medical image data and the 3D facial scan data are more consistent.

[0078] Figure 11 visualizes the detailed matching results using a multi-view slide image of the 3D volumetric medical image data. The solid outline in Figure 11 corresponds to the skin surface of the 3D facial scan data, and it can be seen in Figure 11 that the solid outline representing the skin surface of the 3D facial scan data is precisely matched to the 3D volumetric medical image data.

[0079] According to this embodiment, 3D facial scan data and 3D volumetric medical image data can be quickly and automatically matched using the data itself, without requiring user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0080] Furthermore, since no user input or conversion of data structure is required, the accuracy of matching can be improved. Furthermore, if the feature points of the 3D facial scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced.

[0081] Figure 12 is a conceptual diagram illustrating an example of the steps of detecting scan features from the 3D facial scan data and detecting volume features from the 3D volumetric medical image data in Figure 1. Figure 13 is a conceptual diagram illustrating an example of the steps of detecting scan features from the 3D facial scan data and detecting volume features from the 3D volumetric medical image data in Figure 1. Figure 14 is a diagram illustrating a first captured image generated by capturing the 3D facial scan data. Figure 15 is a diagram illustrating a second captured image generated by capturing the 3D volumetric medical image data.

[0082] The automatic matching method according to this embodiment is similar to the automatic matching method of FIGS. 1 to 11 except for the input of the convolutional neural network. Therefore, the same or similar components are denoted by the same reference numerals and redundant description will be omitted.

[0083] 1 and 7 to 15, a method for automatically matching 3D facial scan data and 3D volumetric medical image data includes the steps of: extracting scan features from the 3D facial scan data using a convolutional neural network (S100); extracting volumetric features from the 3D volumetric medical image data using the convolutional neural network (S200); and initially matching the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features (S300). The automatic matching method further includes, after the step of initially matching the 3D facial scan data and the 3D volumetric medical image data (S300), a fine-tuning step (S400) of adjusting distances between a plurality of sampling points of the 3D facial scan data and a plurality of sampling points of the 3D volumetric medical image data so that they are less than a threshold distance.

[0084] The step of extracting scan features (S100) captures the 3D facial scan data to generate a first 2D captured image, which is shown in the upper left of FIG. 12 and in FIG. 14.

[0085] FIG. 12 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is two-dimensional coordinates. For example, the input of the convolutional neural network is the first two-dimensional captured image, and the output of the convolutional neural network is first two-dimensional coordinates corresponding to the scanned feature points.

[0086] Here, the step of extracting the scan feature points (S100) further includes back-projecting the first two-dimensional coordinates onto first three-dimensional coordinates, which represent feature points of the 3D face scan data.

[0087] Similarly, the step of extracting volumetric features (S200) involves imaging the three-dimensional volumetric medical image data to generate a second two-dimensional image, as shown in the lower left of Figure 12 and in Figure 15.

[0088] For example, the input of the convolutional neural network is the second two-dimensional captured image, and the output of the convolutional neural network is second two-dimensional coordinates corresponding to the volumetric feature points.

[0089] Here, the step of extracting the volumetric feature points (S200) further includes back-projecting the second 2D coordinates onto second 3D coordinates, which represent the feature points of the 3D face scan data.

[0090] FIG. 13 shows a case where the input of the convolutional neural network is a two-dimensional image and the output is a three-dimensional coordinate. For example, the input of the convolutional neural network is the first 2D captured image of the 3D face scan data, and the output of the convolutional neural network is first 3D coordinates corresponding to the scan feature points.

[0091] The input of the convolutional neural network is the second two-dimensional captured image of the three-dimensional volumetric medical image data, and the output of the convolutional neural network is second three-dimensional coordinates corresponding to the volumetric feature points.

[0092] According to this embodiment, 3D facial scan data and 3D volumetric medical image data can be quickly and automatically matched using the data itself, without requiring user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0093] Furthermore, since no user input or conversion of data structure is required, the accuracy of matching can be improved. Furthermore, if the feature points of the 3D facial scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced.

[0094] Figure 16 shows an example of automatic alignment of a first 3D facial scan (S1), a second 3D facial scan (S2), and a 3D volumetric medical image data (V1). Figure 17 shows an example of automatic alignment of a first 3D facial scan (S1), a second 3D facial scan (S2), and a 3D volumetric medical image data (V1).

[0095] The automatic matching method according to this embodiment is similar to the automatic matching method of FIGS. 1 to 11 except that two 3D facial scan data are matched into one 3D volumetric medical image data. Therefore, the same or similar components are designated by the same reference numerals and redundant descriptions are omitted.

[0096] 1 to 17, a method for automatically matching 3D facial scan data and 3D volumetric medical image data includes the steps of: extracting first scan features from first 3D facial scan data (S1) using a convolutional neural network; extracting second scan features from second 3D facial scan data (S2) using the convolutional neural network; extracting volumetric features from 3D volumetric medical image data (V1) using the convolutional neural network; initially matching the first 3D facial scan data (S1), the second 3D facial scan data (S2), and the 3D volumetric medical image data (V1) using the first scan features, the second scan features, and the volumetric features; and calculating a surface distance between the first 3D facial scan data (S1) and the second 3D facial scan data (S2).

[0097] For example, the first scan feature points, the second scan feature points, and the volume feature points are extracted using the same convolutional neural network. Alternatively, the first scan feature points and the second scan feature points may be extracted using a first convolutional neural network, and the volumetric feature points may be extracted using a second convolutional neural network.

[0098] For example, the first 3D face scan data (S1) and the second 3D face scan data (S2) differ in at least one of the following: photographing equipment, photographing time, facial expression, shape of specific parts of the face, and coordinate axes.

[0099] FIG. 16 shows two 3D facial scans (S1, S2) of the same patient, with different imaging devices, facial expressions, and coordinate axes, aligned into one 3D volumetric medical image data set (V1).

[0100] Figure 17 shows two 3D facial scan data (S1, S2) of the same patient, taken using the same imaging equipment but with different facial expressions and coordinate axes, aligned into one 3D volumetric medical image data (V1).

[0101] FIG. 17 shows that the surface distances of the first 3D face scan data (S1) and the second 3D face scan data (S2) are calculated and the difference between the surface distances is analyzed. As shown in Figures 16 and 17, the 3D facial scan data (S1, S2) with different coordinate axes are aligned to the same 3D volumetric medical image data (V1), and the coordinate axes are aligned with each other, allowing the first 3D facial scan data (S1) and the second 3D facial scan data (S2) to be compared and analyzed.

[0102] Figure 16 shows how 3D facial scan data of the same patient taken from different devices, at different times, and with different facial expressions are aligned into a single 3D volumetric medical image data set, and the aligned data are compared and analyzed from various angles. Figure 17 shows how 3D facial scan data of the same patient taken with different facial expressions are aligned into a single 3D volumetric medical image data set, and the facial change areas are analyzed.

[0103] 3D volumetric medical image data is expensive, and in the case of CT or CBCT, there is a risk of radiation exposure, making it difficult to acquire multiple times. Therefore, it is difficult to use it for comparing pre- and post-surgical images or for surgical simulations of various facial expressions. In this embodiment, 3D facial scan data acquired at various times and with various facial expressions can be automatically aligned with 3D volumetric medical image data, enabling it to be used for maxillofacial analysis, surgery, and pre- and post-surgical analysis of patients.

[0104] According to this embodiment, 3D facial scan data and 3D volumetric medical image data can be quickly and automatically matched using the data itself, without requiring user input or data structure conversion such as a separate 3D model extraction process or voxelization process.

[0105] Furthermore, since no user input or conversion of data structure is required, the accuracy of matching can be improved. Furthermore, if the feature points of the 3D facial scan data and the feature points of the 3D volumetric medical image data are extracted using the same convolutional neural network, the network structure can be simplified and the computer load for feature point extraction can be reduced.

[0106] According to one embodiment of the present invention, a computer-readable recording medium is provided having a program recorded thereon for executing a method for automatically matching 3D facial scan data and 3D volumetric medical image data according to the embodiment. The method can be written as a computer-executable program and implemented on a general-purpose digital computer that runs the program using a computer-readable medium. Furthermore, data structures used in the method can be recorded on the computer-readable medium by various means. The computer-readable medium may include program instructions, data files, data structures, and the like, singly or in combination. The program instructions recorded on the medium may be those specifically designed and constructed for the present invention, or those well known and available to those of ordinary skill in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of this invention.

[0107] The method for automatically matching 3D facial scan data and 3D volumetric medical image data can also be implemented in the form of a computer program or application that is recorded on a recording medium and executed by a computer.

[0108] [Industrial Applicability] The present invention relates to a method for automatically aligning 3D facial scan data and 3D volumetric medical image data using deep learning, and a computer-readable recording medium having a program recorded thereon for executing the method, which can reduce the effort required for automatic alignment and improve accuracy. Automatic alignment of 3D facial scan data acquired with various expressions and at various times with 3D volumetric medical image data can be used for maxillofacial analysis, surgery, and pre- and post-operative analysis of patients.

[0109] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will appreciate that various modifications and variations of the present invention may be made without departing from the spirit and scope of the present invention as set forth in the following claims.

Claims

1. extracting scan features from the 3D face scan data using a convolutional neural network; extracting volumetric features from the three-dimensional volumetric medical image data using the convolutional neural network; initial alignment of the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features; A method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, comprising:

2. The step of extracting scan feature points includes: generating a first two-dimensional depth image indicating a distance from a first reference plane located outside the three-dimensional facial scan to a facial surface within the three-dimensional facial scan. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

3. an input of the convolutional neural network is the first two-dimensional depth image; an output of the convolutional neural network is a first two-dimensional coordinate corresponding to the scan feature point; The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 2.

4. The step of extracting the scan feature points further includes a step of back-projecting the first two-dimensional coordinates onto first three-dimensional coordinates based on a transformation method used in the step of generating the first two-dimensional depth image. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 3.

5. an input of the convolutional neural network is the first two-dimensional depth image; an output of the convolutional neural network is a first three-dimensional coordinate corresponding to the scan feature point; The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 2.

6. The step of extracting the volumetric feature points includes generating a second two-dimensional depth image indicating a distance from a second reference plane located outside the three-dimensional volumetric medical image data to a face surface within the three-dimensional volumetric medical image data. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

7. The step of generating the second two-dimensional depth image includes proceeding from a starting point of the three-dimensional volumetric medical image data to an interior of the three-dimensional volumetric medical image data, and generating a depth value of the second two-dimensional depth image at a position where a change in brightness value is greater than a threshold value for the first time. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 6.

8. The step of generating the second two-dimensional depth image is characterized in that the depth value of the second two-dimensional depth image is generated by proceeding from a starting point of the three-dimensional volumetric medical image data to the inside of the three-dimensional volumetric medical image data, and then generating a position of a point where a change in brightness value is greater than a threshold and the brightness value is within a predetermined range of skin brightness value. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 6.

9. an input of the convolutional neural network is the second two-dimensional depth image; an output of the convolutional neural network is a second two-dimensional coordinate corresponding to the volume feature point; The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 6.

10. The step of extracting the volumetric feature points further includes a step of back-projecting the second two-dimensional coordinates onto second three-dimensional coordinates based on a transformation method used in the step of generating the second two-dimensional depth image. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 9.

11. an input of the convolutional neural network is the second two-dimensional depth image; an output of the convolutional neural network is a second three-dimensional coordinate corresponding to the volume feature point; The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 6.

12. The step of extracting the scan feature points includes capturing the 3D facial scan data to generate a first 2D captured image. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

13. an input of the convolutional neural network is the first two-dimensional captured image; an output of the convolutional neural network is a first three-dimensional coordinate corresponding to the scan feature point; The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 12.

14. The step of extracting volumetric features includes imaging the three-dimensional volumetric medical image data to generate a second two-dimensional image. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 12.

15. an input of the convolutional neural network is the second two-dimensional captured image; an output of the convolutional neural network is a second three-dimensional coordinate corresponding to the volume feature point; 15. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 14.

16. The number of the scan feature points extracted in the step of extracting the scan feature points is three or more, the number of volume feature points extracted in the extracting of the volume feature points is three or more; the three or more scan feature points and the three or more volume feature points correspond to each other. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

17. The scan feature points extracted in the step of extracting the scan feature points and the volume feature points extracted in the step of extracting the volume feature points include soft tissue nasion, right exocanthion, and left exocanthion, respectively.

17. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 16.

18. The scan feature points extracted in the step of extracting the scan feature points and the scan feature points extracted in the step of extracting the volume feature points are Soft tissue gabella, Soft tissue nasion, Supratip, Pronasale, Columnella, Subnasale, Soft tissue A-point, Upper lip anterior point, Stomiom superius, Lower point of gap between maxillary central incisors, Upper point of gap between mandibular central incisors, Stomiom inferious, Lower lip anterior point, Soft tissue B-point, Soft tissue pogonion, Soft tissue gnathion, Soft tissue menton, Trichion, Right Orbitale superius, Left Orbitale superius, Right Palpebrale superius, Right Palpebrale inferious, Right Exocanthion, Right Endocanthion, Left Palpebrale superius, Left Palpebrale inferious, Left Endocanthion, Left Exocanthion, and Right Zygion, Left Zygion, Right Cheilion, Left Cheilion, Right Tragus, Left Tragus, Right Soft tissue gonion, Left Soft tissue gonion, Right Alare, Left Alare, Right Alar curvature and Left Alar curvature, The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

19. The 3D face scan data is mesh data including 3D points (vertices) and triangular or quadrangular surfaces (triangles) generated by connecting the 3D points. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

20. the three-dimensional volumetric medical image data is a medical image of a patient's anatomy captured in three-dimensional volumetric image data; The three-dimensional volume medical image data is one of a Computed Tomography (CT) image, a Cone-beam CT (CBCT) image, a Magnetic Resonance Imaging (MRI) image, and a Positron Emission Tomography (PET) image. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

21. The method further comprises, after the step of initially aligning the 3D facial scan data and the 3D volumetric medical image data, a detailed alignment step of adjusting the distance between the plurality of sampling points of the 3D facial scan data and the plurality of sampling points of the 3D volumetric medical image data so that the distance is less than a threshold distance. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning according to claim 1.

22. extracting scan features from the 3D face scan data using a first convolutional neural network; extracting volumetric features from the three-dimensional volumetric medical image data using a second convolutional neural network different from the first convolutional neural network; initial alignment of the 3D facial scan data and the 3D volumetric medical image data using the scan features and the volumetric features; A method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, comprising:

23. extracting first scan features from the first 3D facial scan using a convolutional neural network; extracting second scan features from a second 3D facial scan using the convolutional neural network; extracting volumetric features from the three-dimensional volumetric medical image data using the convolutional neural network; initial alignment of the first 3D facial scan data, the second 3D facial scan data, and the 3D volumetric medical image data using the first scan features, the second scan features, and the volumetric features; calculating a surface distance between the first 3D facial scan and the second 3D facial scan; A method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning, comprising:

24. The first 3D face scan data and the second 3D face scan data are different in at least one of the following: photographing equipment, photographing time, facial expression, shape of a specific part of the face, and coordinate axis.

24. The method for automatically matching 3D facial scan data and 3D volumetric medical image data using deep learning as claimed in claim 23.

25. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method of any one of claims 1 to 24.

Images