Method for automatically matching three-dimensional face scan data to three-dimensional face volume medical image data, and computer-readable recording medium having recorded thereon program for executing same on computer

Abstract

A method for automatically matching three-dimensional face scan data to three-dimensional face volume medical image data comprises the steps of: obtaining a first nose end point from three-dimensional face scan data; obtaining, from the three-dimensional face scan data, a first vertical center line passing through the first nose end point; obtaining a second nose end point from three-dimensional face volume medical image data; and primarily matching the three-dimensional face scan data to the three-dimensional face volume medical image data to match the first nose end point to the second nose end point and to match the first vertical center line to the axis of the three-dimensional face volume medical image data.

Description

A method for automatically matching 3D facial scan data and 3D facial volume medical image data, and a computer-readable recording medium having recorded thereon a program for executing the method on a computer.

[0001] The present invention relates to a method for automatically matching three-dimensional facial scan data and three-dimensional facial volume medical image data, and to a computer-readable recording medium having recorded thereon a program for executing the same on a computer, and more particularly, to a method for automatically matching three-dimensional facial scan data and three-dimensional facial volume medical image data, which is performed automatically without manual input from a user, and to a computer-readable recording medium having recorded thereon a program for executing the same on a computer.

[0002] 3D volumetric medical imaging data refers to data accumulated from hundreds of 2D slide images, such as Computed Tomography (CT), Cone-beam CT (CBCT), and Magnetic Resonance Imaging (MRI). Because 3D image data can capture various tissues and structures of the body, it is widely used for diagnosis, treatment, and surgical planning across most medical fields, including internal medicine and surgery. Dentistry, oral and maxillofacial surgery, and plastic surgery target the head and neck region, acquiring 3D volumetric medical imaging data to diagnose and treat patients' maxillofacial and oral structures.

[0003] The head and neck region is a crucial area of ​​the body, requiring significant aesthetic considerations in addition to functional aspects such as breathing, mastication, and pronunciation. Therefore, maxillofacial and oral surgery planning incorporates both functional considerations and the patient's aesthetic preferences. However, 3D volumetric medical imaging data only contains single-channel intensity information, lacking information on the color of the body's various tissues and structures.

[0004] Furthermore, 3D analysis of a specific area requires segmenting the area or reconstructing the segmented data into a 3D model, which are time-consuming tasks. Furthermore, 3D volumetric medical imaging data is expensive to acquire, and the risk of radiation exposure from CT or CBCT makes it difficult to acquire multiple scans. Therefore, it is difficult to compare pre- and post-surgical images or to simulate procedures and surgeries under various facial expressions.

[0005] To address these limitations of 3D volumetric medical imaging data, 3D facial scan mesh model data is being utilized. This data can include texture (color information) in addition to 3D facial shape information. It is inexpensive to acquire and requires no radiation exposure during acquisition. Therefore, it is convenient for assessing changes in appearance before and after surgery and procedures, or with various facial expressions.

[0006] Furthermore, when used in conjunction with 3D volumetric medical imaging data, it can be helpful in establishing treatment and surgical plans by considering not only anatomical information within the head and neck region but also aesthetic shape and color. However, because 3D volumetric medical imaging data and 3D facial scan mesh model data have different 3D spatial coordinate systems, a registration process that aligns and superimposes the two data must be performed first. However, manual work in 3D space requires skill and is a time-consuming process even for experienced professionals, so a method to automate this process is needed.

[0007] Registration is the process of aligning multiple 3D models based on a single model. Registration is primarily used to reconstruct a single, complete 3D model by aligning 3D models acquired from multiple perspectives of an object. It is also used to align 3D models acquired from different paths of an object to their closest locations. The process by which a medical 3D scanner digitally models an object is an example of the former, while the process of unifying 3D medical data acquired through different modalities into a common coordinate system is an example of the latter.

[0008] The purpose of the present invention is to provide a method for automatically matching 3D facial scan data and 3D facial volume medical image data without manual input from a user.

[0009] Another object of the present invention is to provide a computer-readable recording medium having recorded thereon a program for executing a method for automatically matching 3D facial scan data and 3D facial volume medical image data on a computer.

[0010] In order to achieve the above object of the present invention, a method for automatically matching 3D facial scan data and 3D facial volume medical image data according to one embodiment includes the steps of obtaining a first nose tip point from 3D facial scan data, obtaining a first vertical center line passing through the first nose tip point from the 3D facial scan data, obtaining a second nose tip point from the 3D facial volume medical image data, and a first matching step of the 3D facial scan data and the 3D facial volume medical image data, which matches the first nose tip point and the second nose tip point and matches the first vertical center line and an axis of the 3D facial volume medical image data.

[0011] In one embodiment of the present invention, the step of obtaining the first nose tip point may include a step of generating a first depth map by projecting the three-dimensional facial scan data onto a reference plane.

[0012] In one embodiment of the present invention, the step of obtaining the first nose tip point may further include the step of converting the first depth map into a second depth map using a first weight function having a maximum weight at the center coordinate and a weight that decreases as the distance from the center coordinate increases.

[0013] In one embodiment of the present invention, the first depth map and the second depth map may be two-dimensional depth maps having depth values ​​for two-dimensional coordinates. The first weight function may have weight values ​​for two-dimensional coordinates.

[0014] In one embodiment of the present invention, the first weight function may be a first Gaussian function corresponding to a two-dimensional space or a function that transforms the first Gaussian function.

[0015] In one embodiment of the present invention, the step of obtaining the first vertical centerline may determine the first vertical centerline along a direction having a minimum gradient from the first nose tip.

[0016] In one embodiment of the present invention, the step of obtaining the second nose tip point may include the step of generating a first depth line by projecting first cross-sectional data of the three-dimensional facial volume medical image data onto a reference line.

[0017] In one embodiment of the present invention, the step of obtaining the second nose tip point may further include the step of converting the first depth line into a second depth line using a second weighting function having a maximum weight at the center coordinate and a weight that decreases as the distance from the center coordinate increases.

[0018] In one embodiment of the present invention, the first depth line and the second depth line may be one-dimensional depth lines having depth values ​​for one-dimensional coordinates. The second weight function may have weight values ​​for one-dimensional coordinates.

[0019] In one embodiment of the present invention, the second weighting function may be a second Gaussian function corresponding to a one-dimensional space or a function transformed from the second Gaussian function.

[0020] In one embodiment of the present invention, the step of obtaining the first nose tip point may include the step of generating a first depth map by projecting the 3D facial scan data onto a reference plane, and the step of converting the first depth map into a second depth map using a first weighting function having a maximum weight at a center coordinate of a 2D space and a weight that decreases as the distance from the center coordinate of the 2D space increases. The step of obtaining the second nose tip point may include the step of generating a first depth line by projecting first cross-sectional data of the 3D facial volume medical image data onto a reference line, and the step of converting the first depth line into a second depth line using a second weighting function having a maximum weight at a center coordinate of a 1D space and a weight that decreases as the distance from the center coordinate of the 1D space increases.

[0021] In one embodiment of the present invention, after the first matching step, the method may further include a second matching step of the 3D facial scan data and the 3D facial volume medical image data, which moves at least one of the 3D facial scan data and the 3D facial volume medical image data so that the distance from a plurality of sampling points extracted from the skin surface of the 3D facial scan data to the skin surface of the 3D facial volume medical image data becomes minimal.

[0022] In order to achieve the above object of the present invention, a method for automatically matching 3D facial scan data and 3D facial volume medical image data according to one embodiment includes the steps of: obtaining a first nose tip point from 3D facial scan data; obtaining a first vertical center line passing through the first nose tip point from the 3D facial scan data; obtaining second 3D facial scan data from the 3D facial volume medical image data; obtaining a second nose tip point from the second 3D facial scan data; obtaining a second vertical center line passing through the second nose tip point from the second 3D facial scan data; and a first matching step of the 3D facial scan data and the 3D facial volume medical image data, which matches the first nose tip point and the second nose tip point and matches the first vertical center line and the second vertical center line.

[0023] In one embodiment of the present invention, the step of obtaining the first nose tip point may include the step of projecting the three-dimensional face scan data onto a reference plane to generate a first depth map, and the step of converting the first depth map into a second depth map using a first weighting function that has a maximum weight at a center coordinate of a two-dimensional space and a weight that decreases as the distance from the center coordinate of the two-dimensional space increases. The step of obtaining the second nose tip point may include the step of projecting the second three-dimensional face scan data onto a second reference plane to generate a third depth map, and the step of converting the third depth map into a fourth depth map using a second weighting function that has a second maximum weight at a second center coordinate of a second two-dimensional space and a weight that decreases as the distance from the second center coordinate of the second two-dimensional space increases.

[0024] In order to achieve the above object of the present invention, a method for automatically matching 3D facial scan data and 3D facial volume medical image data according to one embodiment includes the steps of: acquiring a first nose tip point from 3D facial scan data; acquiring a second nose tip point from 3D facial volume medical image data; and a first matching step of the 3D facial scan data and the 3D facial volume medical image data, wherein the first nose tip point and the second nose tip point are aligned, and an axis of the 3D facial scan data and an axis of the 3D facial volume medical image data are aligned. The step of acquiring the first nose tip point includes the step of generating a first depth map by projecting the 3D facial scan data onto a reference plane. The step of acquiring the second nose tip point includes the step of generating a first depth line by projecting first cross-sectional data of the 3D facial volume medical image data onto a reference line.

[0025] In one embodiment of the present invention, the step of obtaining the first nose tip point may further include a step of converting the first depth map into a second depth map using a first weighting function that has a maximum weight at a center coordinate of a two-dimensional space and a weight that decreases as the distance from the center coordinate of the two-dimensional space increases. The step of obtaining the second nose tip point may further include a step of converting the first depth line into a second depth line using a second weighting function that has a maximum weight at a center coordinate of a one-dimensional space and a weight that decreases as the distance from the center coordinate of the one-dimensional space increases.

[0026] In one embodiment of the present invention, a program for executing a method for automatically matching the 3D facial scan data and the 3D facial volume medical image data on a computer can be recorded on a computer-readable recording medium.

[0027] According to the automatic alignment method of 3D facial scan data and 3D facial volume medical image data according to the present invention, the 3D facial scan data and the 3D facial volume medical image data can be automatically aligned using a first nose tip point of the 3D facial scan data, a vertical center line passing through the first nose tip point, a second nose tip point of the 3D facial volume medical image data, and vertical axis information of the 3D facial volume medical image data. Since alignment is performed automatically without manual input from the user, the speed and accuracy of alignment can be improved.

[0028] In addition, in order to further improve the accuracy of the first nose tip, the first depth map of the 3D facial scan data may be converted into a second depth map using a first weighting function. In addition, in order to further improve the accuracy of the second nose tip, the first depth line of the 3D facial volume medical image data may be converted into a second depth line using a second weighting function. Therefore, the accuracy of automatic alignment of the 3D facial scan data and the 3D facial volume medical image data may be further improved.

[0029] FIG. 1 is a diagram showing automatic alignment of 3D facial scan data and 3D facial volume medical image data according to one embodiment of the present invention.

[0030] Figure 2 is a diagram showing manual alignment of 3D facial scan data and 3D facial volume medical image data according to a comparative example.

[0031] Figure 3 is a diagram showing manual alignment of 3D facial scan data and 3D facial volume medical image data according to a comparative example.

[0032] FIG. 4 is a drawing showing a first depth map of the three-dimensional facial scan data of FIG. 1.

[0033] FIG. 5 and FIG. 6 are diagrams showing a second depth map, a first weighting function, and a first nose tip point of the three-dimensional facial scan data of FIG. 1.

[0034] Fig. 7 is a drawing showing the first nose tip and the first vertical center line of the three-dimensional facial scan data of Fig. 1.

[0035] FIG. 8 is a diagram showing a first depth line, a second depth line, a second weight function, and a second nose tip of the three-dimensional facial volume medical image data of FIG. 1.

[0036] FIG. 9 is a drawing showing a first nose tip and a first vertical center line of the three-dimensional facial scan data of FIG. 1 and a second nose tip and a vertical axis of the three-dimensional facial volume medical image data of FIG. 1.

[0037] FIG. 10 is a diagram showing a first alignment step of the 3D facial scan data of FIG. 1 and the 3D facial volume medical image data of FIG. 1.

[0038] FIG. 11 and FIG. 12 are diagrams showing a second alignment step of the 3D facial scan data of FIG. 1 and the 3D facial volume medical image data of FIG. 1.

[0039] FIG. 13 is a diagram illustrating three-dimensional facial volume medical image data according to one embodiment of the present invention.

[0040] FIG. 14 is a diagram showing second 3D facial scan data acquired based on the 3D facial volume medical image data of FIG. 13.

[0041] FIG. 15 is a drawing showing the second nose tip and the second vertical center line of the second three-dimensional facial scan data of FIG. 14.

[0042] With respect to the embodiments of the present invention disclosed in the text, specific structural and functional descriptions are merely illustrative for the purpose of explaining the embodiments of the present invention, and the embodiments of the present invention may be implemented in various forms and should not be construed as being limited to the embodiments described in the text.

[0043] The present invention is susceptible to various modifications and takes various forms. Specific embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the present invention to specific disclosed forms, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0044] While terms like "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0045] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0046] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0047] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be construed in an idealized or overly formal sense unless explicitly defined herein.

[0048] Meanwhile, if a particular embodiment can be implemented differently, the functions or operations specified within a particular block may occur in a different order than specified in the flowchart. For example, two consecutive blocks may actually be executed substantially simultaneously, or, depending on the related functions or operations, the blocks may be executed in reverse order.

[0049] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. Identical components in the drawings are designated by the same reference numerals, and redundant descriptions of identical components are omitted.

[0050] FIG. 1 is a diagram showing automatic alignment of 3D facial scan data (DT1) and 3D facial volume medical image data (DT2) according to one embodiment of the present invention.

[0051] Referring to FIG. 1, a method for automatically matching 3D facial scan data (DT1) and 3D facial volume medical image data (DT2) may include a step of obtaining a first nose tip point from the 3D facial scan data (DT1), a step of obtaining a first vertical center line passing through the first nose tip point from the 3D facial scan data (DT1), a step of obtaining a second nose tip point from the 3D facial volume medical image data (DT2), and a first matching step of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) to match the first nose tip point and the second nose tip point and to match the first vertical center line and an axis of the 3D facial volume medical image data (DT2). The first matching step may be referred to as initial matching. The first matching step may also be referred to as initial positioning.

[0052] The automatic registration method of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) may further include, after the first registration step, a second registration step of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) of moving at least one of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) so as to minimize the distance between the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2). The second registration step may be referred to as fine registration.

[0053] The above 3D face scan data (DT1) may be mesh data including 3D points (Vertex) and triangular or rectangular faces (Rectangle) created by connecting the points. The above 3D face scan data (DT1) may be image data captured by a 3D scanner. There is no limitation on the file extension of the 3D face scan data (DT1), and may be, for example, one of ply, obj, and stl.

[0054] The above 3D facial volume medical image data (DT2) may be a medical image in which the anatomical structure of the patient's face is acquired as 3D volume image data. The 3D facial volume medical image data may be one of a Computed Tomography (CT) image, a Cone-beam CT (CBCT) image, a Magnetic Resonance Imaging (MRI) image, and a Positron Emission Tomography (PET) image. In the present embodiment, the 3D facial volume medical image data is described by exemplifying a case in which the Computed Tomography (CT) image is used.

[0055] The above 3D facial volume medical image data (DT2) generally includes axis information. In Fig. 1, the X-axis, Y-axis, and Z-axis are illustrated, and these X-axis, Y-axis, and Z-axis may refer to axis information of the 3D facial volume medical image data (DT2). The axis information of the 3D facial volume medical image data (DT2) may be considered to exactly match the horizontal direction, front-back direction, and vertical direction of the patient's face due to the characteristics of the medical device.

[0056] The automatic alignment method of the 3D facial scan data and the 3D facial volume medical image data of the present embodiment can be performed by a computing device.

[0057] Registration is the process of aligning multiple 3D models based on a single model. In digital dentistry, patients' teeth and faces are frequently acquired through two methods: scanners and CT. There is a strong need to align the 3D models obtained from scanners and CT scans. Therefore, registration is crucial in digital dentistry. Facial scan data (e.g., DT1) and facial CT medical image data (e.g., DT2) aligned in a unified coordinate system can be widely utilized, such as for establishing surgical strategies, such as determining the insertion direction of surgical tools, or for providing patients with post-surgical predicted images.

[0058] Fig. 2 is a diagram showing manual alignment of 3D facial scan data (DT1) and 3D facial volume medical image data (DT2) according to a comparative example. Fig. 3 is a diagram showing manual alignment of 3D facial scan data (DT1) and 3D facial volume medical image data (DT2) according to a comparative example.

[0059] The initial alignment described above may refer to the process of moving the source model to a roughly similar position to the fixed target model. The fine alignment may refer to the process of repeatedly moving the initially aligned source model to bring it closer to the target model. Since the quality of the fine alignment results is significantly dependent on the initial alignment results, the accuracy of the initial alignment is of paramount importance.

[0060] In the past, the initial matching was performed by receiving manual input of feature points from the user. For example, in a comparative example, three feature points (P11, P12, P13) are input on the 3D facial scan data (DT1), and three feature points (P21, P22, P23) are input on the 3D facial volume medical image data (DT2), and then the positions of the corresponding feature points are matched to perform the initial matching.

[0061] However, as shown in Fig. 2, if the user's manual input of the feature points changes slightly, the matching result may vary greatly.

[0062] Figure 3 illustrates a case where the left and right of P12 and P13 captured on the 3D facial scan data (DT1) and the left and right of P22 and P23 captured on the 3D facial volume medical image data (DT2) are swapped. In this case, the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) may not be aligned so that they face each other, but may be aligned so that they face each other.

[0063] That is, when a user manually inputs the above-described feature points (P11, P12, P13, P21, P22, P23), not only must the input locations of the corresponding feature points (P11-P21, P12-P22, P13-P23) be consistent, but also the input order of the corresponding feature points (P11-P21, P12-P22, P13-P23) must be consistent, so there is a problem that the work time is long and the accuracy of the work is reduced.

[0064] On the other hand, in the case of the present embodiment, the initial alignment can be performed quickly and accurately by matching the first nose tip obtained from the 3D facial scan data (DT1) and the second nose tip obtained from the 3D facial volume medical image data (DT2) with each other without manual input from the user.

[0065] Fig. 4 is a diagram showing a first depth map of the 3D facial scan data of Fig. 1. Figs. 5 and 6 are diagrams showing a second depth map, a first weighting function, and a first nose tip of the 3D facial scan data of Fig. 1. Fig. 7 is a diagram showing a first nose tip and a first vertical center line of the 3D facial scan data of Fig. 1.

[0066] Referring to FIGS. 1 to 7, the step of obtaining the first nose tip point (NS1) may include a step of projecting the three-dimensional face scan data (DT1) onto a reference plane (PL) to generate a first depth map (DM1).

[0067] When the direction of the face is aligned so that it faces forward in the above 3D face scan data (DT1), the reference plane (PL) can be formed in a direction parallel to the front of the face.

[0068] The step of obtaining the first nose tip point (NS1) may further include a step of converting the first depth map (DM1) into the second depth map (DM2) using a first weight function (WF1) having a maximum weight at the center coordinate and a weight that decreases as it moves away from the center coordinate.

[0069] In the above 3D facial scan data (DT1), the location of the patient's nose is likely to be close to the center coordinates of the reference plane (PL). Therefore, by converting the first depth map (DM1) into the second depth map (DM2) using a first weight function (WF1) having a maximum weight at the center coordinates and a weight that decreases as the distance from the center coordinates increases, the location of the patient's nose (NS1 in FIGS. 5 and 6) can be more accurately obtained.

[0070] In the first depth map (DM1) of FIG. 4, the position of the patient's nose is not emphasized, and the first depth map (DM1) includes surface information of the three-dimensional facial scan data (DT1) as is.

[0071] On the other hand, in the second depth map (DM2) of FIGS. 5 and 6, the position of the patient's nose is emphasized and more protruded than in FIG. 4, and therefore, by using the second depth map (DM2), the position of the patient's nose (NS1 of FIGS. 5 and 6) can be more accurately obtained.

[0072] Even if the position of the patient's nose is not located exactly at the center coordinates of the reference plane (PL), the possibility that the position of the patient's nose is located at the edge of the reference plane (PL) is extremely low, so by converting the first depth map (DM1) into the second depth map (DM2) using the first weighting function (WF1), the accuracy of obtaining the position of the patient's nose can be increased.

[0073] In the present embodiment, the first depth map (DM1) and the second depth map (DM2) may be two-dimensional depth maps having depth values ​​for two-dimensional coordinates. In addition, the first weight function (WF1) may have weight values ​​for two-dimensional coordinates.

[0074] For example, the first weight function (WF1) may be a first Gaussian function corresponding to a two-dimensional space or a function that transforms the first Gaussian function.

[0075] The coordinates of the tip of the nose obtained from the second depth map (DM2) can be moved to coordinates within the first depth map (DM1) through inverse transformation of the first weight function (WF1), and the coordinates within the first depth map (DM1) can be moved to coordinates within the three-dimensional face scan data (DT1) through inverse transformation of the projection that generates the first depth map (DM1).

[0076] Referring to Fig. 7, the step of obtaining the first vertical center line may determine the first vertical center line (CL1) along a direction with the minimum gradient from the first nose tip point (NS1). The direction with the lowest gradient from the first nose tip point (NS1) may be the direction of the nose bridge. Accordingly, by drawing a line from the first nose tip point (NS1) in the direction with the minimum gradient, the direction of the patient's nose bridge is determined, and the direction of the patient's nose bridge can be determined as the first vertical center line (CL1) of the 3D facial scan data (DT1).

[0077] For example, the first vertical center line (CL1) can be determined along a direction having a minimum gradient from the first nose tip point (NS1) within the second depth map (DM2).

[0078] Alternatively, the first vertical center line (CL1) may be determined along a direction having a minimum gradient from the first nose tip point (NS1) within the first depth map (DM1).

[0079] In this embodiment, the first vertical center line (CL1) is exemplified as being determined using the minimum gradient direction from the first nose tip point (NS1), but the first vertical center line (CL1) may also be obtained using axis information of the three-dimensional face scan data (DT1).

[0080] In one embodiment of the present invention, a method for automatically matching 3D facial scan data (DT1) and 3D facial volume medical image data (DT2) may include a step of obtaining a first nose tip point (NS1) from the 3D facial scan data (DT1), a step of obtaining a second nose tip point (NS2) from the 3D facial volume medical image data (DT2), and a first matching step of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) by matching the first nose tip point (NS1) and the second nose tip point (NS2) and matching an axis of the 3D facial scan data (DT1) and an axis of the 3D facial volume medical image data (DT2).

[0081] FIG. 8 is a diagram showing a first depth line (DL1), a second depth line (DL2), a second weight function (WF2), and a second nose tip point (NS2) of the three-dimensional facial volume medical image data (DT2) of FIG. 1.

[0082] Referring to FIGS. 1 to 8, the step of acquiring the second nose tip point (NS2) may include a step of projecting first cross-sectional data of the three-dimensional facial volume medical image data (DT2) onto a reference line (LN) to generate a first depth line (DL1). Here, the first cross-sectional data of the three-dimensional facial volume medical image data (DT2) may be cross-sectional data including the tip of the patient's nose.

[0083] When the face is aligned in the direction facing forward in the above 3D facial volume medical image data (DT2), the reference line (LN) can be formed in a direction parallel to the front of the face.

[0084] The step of obtaining the second nose tip point (NS2) may further include a step of converting the first depth line (DL1) into the second depth line (DL2) using a second weight function (WF2) having a maximum weight at the center coordinate and a weight that decreases as it moves away from the center coordinate.

[0085] In the cross-sectional data including the patient's nose in the above 3D facial volume medical image data (DT2), the location of the patient's nose is likely to be close to the center coordinate of the reference line (LN). Therefore, by converting the first depth line (DL1) into the second depth line (DL2) using a second weighting function (WF2) having a maximum weight at the center coordinate and a weight that decreases as it moves away from the center coordinate, the location of the patient's nose (NS2 in FIG. 8) can be more accurately obtained.

[0086] In the first depth line (DL1) of FIG. 8, the position of the patient's nose is not emphasized, and the first depth line (DL1) includes surface information of cross-sectional data including the patient's nose of the three-dimensional facial volume medical image data (DT2) as it is.

[0087] On the other hand, in the second depth line (DL2) of FIG. 8, the position of the patient's nose is emphasized and protrudes more than in the first depth line (DL1), and therefore, the position of the patient's nose (NS2 of FIG. 8) can be more accurately obtained by using the second depth line (DL2).

[0088] Even if the position of the patient's nose is not exactly located at the center coordinates of the reference line (LN), the possibility that the position of the patient's nose is located at the edge of the reference line (LN) is extremely low, so by converting the first depth line (DL1) into the second depth line (DM2) using the second weighting function (WF2), the accuracy of obtaining the position of the patient's nose can be increased.

[0089] In the present embodiment, the first depth line (DL1) and the second depth line (DL2) may be one-dimensional depth lines having depth values ​​for one-dimensional coordinates. In addition, the second weight function (WF2) may have weight values ​​for one-dimensional coordinates.

[0090] For example, the second weight function (WF2) may be a second Gaussian function corresponding to a one-dimensional space or a function that transforms the second Gaussian function.

[0091] The coordinates of the tip of the nose obtained from the second depth line (DL2) can be moved to coordinates within the first depth line (DL1) through inverse transformation of the second weight function (WF2), and the coordinates within the first depth line (DL1) can be moved to coordinates within the three-dimensional facial volume medical image data (DT2) through inverse transformation of the projection that generates the first depth line (DL1).

[0092] Fig. 9 is a diagram showing a first nose tip point (NS1) and a first vertical center line (CL1) of the three-dimensional facial scan data (DT1) of Fig. 1 and a second nose tip point (NS2) and a vertical axis (AX) of the three-dimensional facial volume medical image data (DT2) of Fig. 1. Fig. 10 is a diagram showing a first matching step of the three-dimensional facial scan data (DT1) of Fig. 1 and the three-dimensional facial volume medical image data (DT2) of Fig. 1.

[0093] Referring to FIGS. 1 to 10, in the first alignment step, the first nose tip point (NS1) and the second nose tip point (NS2) can be aligned, and the first vertical center line (CL1) and the axis (AX) of the three-dimensional facial volume medical image data (DT2) can be aligned.

[0094] In Fig. 10, for convenience of explanation, the first nose tip point (NS1) and the second nose tip point (NS2) are first aligned, and then the first vertical center line (CL1) and the axis (AX) of the three-dimensional facial volume medical image data (DT2) are aligned. However, the present invention is not limited thereto.

[0095] After matching the first vertical center line (CL1) and the axis (AX) of the three-dimensional facial volume medical image data (DT2), the first nose tip point (NS1) and the second nose tip point (NS2) may be matched first, or the first nose tip point (NS1) and the second nose tip point (NS2), the first vertical center line (CL1), and the axis (AX) of the three-dimensional facial volume medical image data (DT2) may be matched simultaneously.

[0096] FIG. 11 and FIG. 12 are diagrams showing a second alignment step of the 3D facial scan data (DT1) of FIG. 1 and the 3D facial volume medical image data (DT2) of FIG. 1.

[0097] Referring to FIGS. 1 to 12, the automatic alignment method of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) may further include, after the first alignment step, a second alignment step of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) of moving at least one of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) to minimize the distance between the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2).

[0098] In the second alignment step, at least one of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) may be moved so that the distance from a plurality of sampling points extracted from the skin surface of the 3D facial scan data (DT1) to the skin surface of the 3D facial volume medical image data (DT2) is minimized.

[0099] For example, since the 3D facial volume medical image data (DT2) has a relatively accurate axis, the 3D facial scan data (DT1) can be moved so that the distance from a plurality of sampling points extracted from the skin surface of the 3D facial scan data (DT1) to the skin surface of the 3D facial volume medical image data (DT2) becomes minimal.

[0100] The above multiple sampling points are exemplified as being extracted from the skin surface of the 3D facial scan data (DT1), but alternatively, the multiple sampling points may be extracted from the skin surface of the 3D facial volume medical image data (DT2).

[0101] According to the present embodiment, the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) can be automatically aligned using the first nose tip point (NS1) of the 3D facial scan data (DT1), the vertical center line (CL1) passing through the first nose tip point (NS1), the second nose tip point (NS2) of the 3D facial volume medical image data (DT2), and the vertical axis information (AX) of the 3D facial volume medical image data (DT2). Since the alignment is performed automatically without manual input from the user, the speed and accuracy of the alignment can be improved.

[0102] In addition, in order to further improve the accuracy of the first nose tip point (NS1), the first depth map (DM1) of the 3D facial scan data (DT1) can be converted into a second depth map (DM2) using a first weighting function (WF1). In addition, in order to further improve the accuracy of the second nose tip point (NS2), the first depth line (DL1) of the 3D facial volume medical image data (DT2) can be converted into a second depth line (DL2) using a second weighting function (WF2). Therefore, the accuracy of automatic alignment of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) can be further improved.

[0103] Fig. 13 is a diagram illustrating three-dimensional facial volume medical image data (DT2) according to one embodiment of the present invention. Fig. 14 is a diagram illustrating second three-dimensional facial scan data (DT3) acquired based on the three-dimensional facial volume medical image data (DT2) of Fig. 13. Fig. 15 is a diagram illustrating a second nose tip point (NS2) and a second vertical center line (CL2) of the second three-dimensional facial scan data (DT3) of Fig. 14.

[0104] The automatic alignment method according to the present embodiment is substantially the same as the automatic alignment method of FIGS. 1 to 12, except that it uses second 3D facial scan data (DT3) acquired based on 3D facial volume medical image data (DT2) instead of the 3D facial volume medical image data (DT2), and therefore the same reference numbers are used for the same or similar components, and redundant descriptions are omitted.

[0105] Referring to FIGS. 1, 4 to 7 and 9 to 15, a method for automatically matching 3D facial scan data (DT1) and 3D facial volume medical image data (DT2) includes the steps of: obtaining a first nose tip point (NS1) from the 3D facial scan data (DT1); obtaining a first vertical center line (CL1) passing through the first nose tip point (NS1) from the 3D facial scan data (DT1); obtaining a second 3D facial scan data (DT3) from the 3D facial volume medical image data (DT2); obtaining a second nose tip point (NS2) from the second 3D facial scan data (DT3); obtaining a second vertical center line (CL2) passing through the second nose tip point (NS2) from the second 3D facial scan data (DT3); and matching the first nose tip point (NS1) and the second nose tip point (NS2), and the first It includes a first alignment step of the three-dimensional facial scan data (DT1) and the three-dimensional facial volume medical image data (DT2) to align the vertical center line (CL1) and the second vertical center line (CL2).

[0106] Here, for example, the second 3D facial scan data (DT3) may be generated by extracting a portion corresponding to skin from the 3D volume medical image data (DT2). The second 3D facial scan data (DT3) may be 3D data composed of points, lines, and surfaces generated by extracting a portion corresponding to skin from the 3D volume medical image data (DT2). However, the method for generating the second 3D facial scan data (DT3) in the present invention is not limited thereto.

[0107] Here, the second nose tip point (NS2) and the second vertical center line (CL2) may refer to coordinates of the second 3D facial scan data (DT3) or may refer to coordinates of the 3D facial volume medical image data (DT2).

[0108] Although the above first alignment step is described as being performed between the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2), the present invention is not limited thereto, and the above first alignment step may also be performed between the 3D facial scan data (DT1) and the second 3D facial scan data (DT2).

[0109] The automatic alignment method of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) may further include, after the first alignment step, a second alignment step of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) of moving at least one of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) so as to minimize the distance between the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2).

[0110] Although the above second alignment step is described as being performed between the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2), the present invention is not limited thereto, and the above second alignment step may be performed between the 3D facial scan data (DT1) and the second 3D facial scan data (DT2).

[0111] As described in FIGS. 4 to 6, the step of obtaining the first nose tip point (NS1) may include a step of projecting the three-dimensional face scan data (DT1) onto a reference plane to generate a first depth map (DM1), and a step of converting the first depth map (DM1) into a second depth map (DM2) using a first weight function (WF1) having a maximum weight at a center coordinate of a two-dimensional space and a weight that decreases as it moves away from the center coordinate of the two-dimensional space.

[0112] Additionally, the first vertical center line (CL1) of the three-dimensional face scan data (DT1) can be obtained in the same manner as described in FIG. 7.

[0113] Since the above second 3D face scan data (DT2) is scan data having the same properties as the above 3D face scan data (DT1), the step of obtaining the second nose tip point (NS2) may be substantially the same as the step of obtaining the first nose tip point (NS1).

[0114] For example, the step of obtaining the second nose tip point (NS2) may include the step of projecting the second three-dimensional face scan data (DT3) onto a second reference plane to generate a third depth map, and the step of converting the third depth map into a fourth depth map using a second weight function having a second maximum weight at a second center coordinate of a second two-dimensional space and having a weight that decreases as it gets farther from the second center coordinate of the second two-dimensional space.

[0115] In addition, since the second 3D face scan data (DT2) is scan data having the same properties as the 3D face scan data (DT1), the step of obtaining the second vertical center line (CL2) may be substantially the same as the step of obtaining the first vertical center line (CL1).

[0116] According to the present embodiment, the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) (or the second 3D facial scan data (DT2)) can be automatically aligned using the first nose tip point (NS1) of the 3D facial scan data (DT1), the first vertical center line (CL1) passing through the first nose tip point (NS1), and the second nose tip point (NS2) of the second 3D facial scan data (DT3), the second vertical center line (CL2) passing through the second nose tip point (NS2). Since the alignment is performed automatically without manual input from the user, the speed and accuracy of the alignment can be improved.

[0117] In addition, in order to further improve the accuracy of the first nose tip point (NS1), the first depth map (DM1) of the 3D facial scan data (DT1) can be converted into a second depth map (DM2) using a first weighting function (WF1). In addition, in order to further improve the accuracy of the second nose tip point (NS2), the third depth map of the second 3D facial scan data (DT3) can be converted into a fourth depth map using a second weighting function. Therefore, the accuracy of automatic alignment of the 3D facial scan data (DT1) and the 3D facial volume medical image data (DT2) can be further improved.

[0118] According to one embodiment of the present invention, a computer-readable recording medium having recorded thereon a program for executing a method for automatically matching 3D facial scan data and 3D facial volume medical image data according to the above embodiments on a computer may be provided. The above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that executes the program using the computer-readable medium. In addition, the structure of data used in the above-described method can be recorded on the computer-readable medium through various means. The computer-readable medium may include program commands, data files, data structures, etc., singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention, or may be known and usable by those skilled in the art in the field of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes, such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention.

[0119] Additionally, the method for automatically matching the aforementioned 3D facial scan data and 3D facial volume medical image data can also be implemented in the form of a computer program or application executed by a computer and stored in a recording medium.

[0120] The present invention relates to a method for automatically matching 3D facial scan data and 3D facial volume medical image data, and to a computer-readable recording medium having recorded thereon a program for executing the method on a computer, whereby the matching is automatically performed without manual input from a user, thereby improving the speed and accuracy of the matching.

[0121] Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims

1. A step of obtaining a first nose end point from 3D facial scan data; A step of obtaining a first vertical center line passing through the first nose tip from the three-dimensional facial scan data; A step of obtaining a second nose endpoint from 3D facial volume medical image data; and A method for automatically matching 3D facial scan data and 3D facial volume medical image data, comprising a first matching step of matching the 3D facial scan data and the 3D facial volume medical image data by matching the first nose tip point and the second nose tip point and matching the first vertical center line and the axis of the 3D facial volume medical image data.

2. In the first paragraph, the step of obtaining the first nose tip point is A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it comprises a step of generating a first depth map by projecting the 3D facial scan data onto a reference plane.

3. In the second paragraph, the step of obtaining the first nose tip point is A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it further comprises a step of converting the first depth map into a second depth map using a first weighting function having a maximum weight at the center coordinate and a weight that decreases as the distance from the center coordinate increases.

4. In the third paragraph, the first depth map and the second depth map are two-dimensional depth maps having depth values for two-dimensional coordinates, A method for automatically matching 3D facial scan data and 3D facial volume medical image data, wherein the first weighting function has weight values for 2D coordinates.

5. A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that in the third paragraph, the first weighting function is a first Gaussian function corresponding to a two-dimensional space or a function transformed from the first Gaussian function.

6. In the third paragraph, the step of obtaining the first vertical center line is A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that the first vertical center line is determined along a direction having a minimum gradient from the first nose tip.

7. In the first paragraph, the step of obtaining the second nose tip point is A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it comprises a step of generating a first depth line by projecting first cross-sectional data of the 3D facial volume medical image data onto a reference line.

8. In the 7th paragraph, the step of obtaining the second nose tip point is A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it further comprises a step of converting the first depth line into a second depth line using a second weighting function having a maximum weight at the center coordinate and a weight that decreases as the distance from the center coordinate increases.

9. In the 8th paragraph, the first depth line and the second depth line are one-dimensional depth lines having depth values for one-dimensional coordinates, A method for automatically matching 3D facial scan data and 3D facial volume medical image data, wherein the second weighting function has weight values for 1D coordinates.

10. A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that in paragraph 8, the second weighting function is a second Gaussian function corresponding to a one-dimensional space or a function transformed from the second Gaussian function.

11. In the first paragraph, the step of obtaining the first nose tip point is A step of generating a first depth map by projecting the above 3D facial scan data onto a reference plane; and A step of converting the first depth map into a second depth map using a first weight function having a maximum weight at the center coordinates of the two-dimensional space and a weight that decreases as the distance from the center coordinates of the two-dimensional space increases, The step of obtaining the second nose tip is A step of generating a first depth line by projecting the first cross-sectional data of the above 3D facial volume medical image data onto a reference line; and A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it comprises a step of converting the first depth line into a second depth line using a second weighting function having a maximum weight at the center coordinates of the 1-dimensional space and a weight that decreases as the distance from the center coordinates of the 1-dimensional space increases.

12. In the first paragraph, after the first matching step, A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it further includes a second matching step of moving at least one of the 3D facial scan data and the 3D facial volume medical image data so that the distance from a plurality of sampling points extracted from the skin surface of the 3D facial scan data to the skin surface of the 3D facial volume medical image data is minimized. Step of obtaining the first nose end point from 13.3D facial scan data; A step of obtaining a first vertical center line passing through the first nose tip from the three-dimensional facial scan data; A step of acquiring second 3D facial scan data from 3D facial volume medical image data; A step of obtaining a second nose tip point from the second three-dimensional facial scan data; A step of obtaining a second vertical center line passing through the second nose tip from the second 3D facial scan data; and A method for automatically matching 3D facial scan data and 3D facial volume medical image data, comprising a first matching step of matching the 3D facial scan data and the 3D facial volume medical image data by matching the first nose tip point and the second nose tip point and the first vertical center line and the second vertical center line.

14. In the 13th paragraph, the step of obtaining the first nose tip point is A step of generating a first depth map by projecting the above 3D facial scan data onto a reference plane; and A step of converting the first depth map into a second depth map using a first weight function having a maximum weight at the center coordinates of the two-dimensional space and a weight that decreases as the distance from the center coordinates of the two-dimensional space increases, The step of obtaining the second nose tip is A step of generating a third depth map by projecting the second three-dimensional face scan data onto a second reference plane; and A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it comprises a step of converting the third depth map into a fourth depth map using a second weighting function having a second maximum weight at a second center coordinate of a second 2-dimensional space and a weight that decreases as the distance from the second center coordinate of the second 2-dimensional space increases.

15. A step of obtaining a first nose end point from 3D facial scan data; A step of obtaining a second nose endpoint from 3D facial volume medical image data; and Including a first matching step of the 3D facial scan data and the 3D facial volume medical image data, which matches the first nose end point and the second nose end point and matches the axis of the 3D facial scan data and the axis of the 3D facial volume medical image data, The step of obtaining the first nose tip is A step of generating a first depth map by projecting the above 3D facial scan data onto a reference plane, The step of obtaining the second nose tip is A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it comprises a step of generating a first depth line by projecting first cross-sectional data of the 3D facial volume medical image data onto a reference line.

16. In the 15th paragraph, the step of obtaining the first nose tip point is Further comprising a step of converting the first depth map into a second depth map using a first weight function having a maximum weight at the center coordinates of the two-dimensional space and a weight that decreases as the distance from the center coordinates of the two-dimensional space increases, The step of obtaining the second nose tip is A method for automatically matching 3D facial scan data and 3D facial volume medical image data, characterized in that it further comprises a step of converting the first depth line into a second depth line using a second weighting function having a maximum weight at the center coordinate of the 1-dimensional space and a weight that decreases as the distance from the center coordinate of the 1-dimensional space increases.

17. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 1 to 16 on a computer.

Images