How to generate 3D facial scan data

Abstract

The present invention relates to a method for generating 3D face scan data. [Solution] A method for generating 3D face scan data according to the present invention includes the steps of obtaining a face image by photographing a patient's face, generating face 3D data from the face image, obtaining a dental region image by photographing the patient's dental region, generating dental region 3D data from the dental region image, extracting a dental region contour for the face image or the face 3D data, isolating and obtaining dental region contour 3D data corresponding to the dental region contour from the dental region 3D data, and aligning and inserting the dental region contour 3D data into the face 3D data.

Description

[Technical Field]

[0001] The present invention relates generally to techniques for generating 3D facial scans of a person.

[0002] In particular, the present invention relates to a technology for generating 3D face scan data, which involves photographing the face and teeth with a structured light pattern projected onto them, acquiring 3D data for the face and teeth separately, extracting a tooth region contour using face landmarks, and then aligning and inserting the acquired tooth region contour 3D data, separated from the tooth region 3D data, into the face 3D data. [Background technology]

[0003] In the past, for cosmetic dental treatments, etc., it was necessary to obtain 3D facial scan data by scanning the patient's face in three dimensions. For example, when performing orthodontic treatment, 3D facial scan data of the patient's face is required to simulate in advance how facial expressions will change during the orthodontic treatment process.

[0004] For this purpose, specialized equipment such as Bellus3D was used, which could take a 3D image of the entire patient's face at once. This specialized equipment is expensive and is not suitable for use in the dental field because it is difficult to obtain precise 3D data for the teeth by taking a 3D image of the entire face at once.

[0005] On the other hand, it is possible to try equipment that scans the patient's face, such as Face Hunter, but this equipment also has the problem that it is difficult to obtain accurate data on the dental area because it only takes a single scan of the face. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent Publication No. 10-2020-0014645 (2020.02.11) "Method for manufacturing dentures for dental implants using intraoral scanning" [Patent Document 2] Republic of Korea Patent Publication No. 10-2022-0023914 (March 3, 2022) "Dental Scan Body" [Patent Document 3] Korean Patent No. 10-1176770 (2012.08.17) "Dental 3D scanner and scanning method using the same" [Patent Document 4] Korean Patent No. 10-2334519 (November 30, 2021) "Simplified automatic face landmark detection method for dental 3D scan data" [Patent Document 5] Korean Patent No. 10-1744079 (May 31, 2017) "Method for generating a facial model for dental surgery simulation" Summary of the Invention [Problem to be solved by the invention]

[0007] SUMMARY OF THE INVENTION It is an object of the present invention to provide a technique for generating 3D facial scans of a person in general.

[0008] In particular, an object of the present invention is to provide a technology for generating 3D face scan data, which involves photographing the face and teeth with a structured light pattern projected onto them, acquiring 3D data for the face and teeth separately, extracting a tooth region contour using facial landmarks, and then separating and aligning the acquired 3D tooth region contour data with the 3D face data and inserting it into the 3D face data.

[0009] However, the problems to be solved by the present invention are not limited to these matters, and other problems to be solved should be understood from the description of this specification. [Means for solving the problem]

[0010] To achieve the above object, a method for generating 3D face scan data according to the present invention includes the steps of: acquiring a face image by photographing a patient's face; generating face 3D data from the face image; acquiring a dental image by photographing a dental region of the patient; generating dental 3D data from the dental image; extracting a dental region contour for the face image or the face 3D data; separating and obtaining dental region contour 3D data corresponding to the dental region contour from the dental region 3D data; and aligning and inserting the dental region contour 3D data into the face 3D data.

[0011] In the above-described method for generating 3D face scan data, the step of acquiring dental region images by photographing the patient's dental region includes a step of photographing the dental region from a plurality of directions to acquire a plurality of partial dental region images, and the step of generating 3D dental region data from the partial dental region images includes a step of generating a plurality of partial dental region data from the plurality of partial dental region images, and a step of aligning the plurality of partial dental region data to generate 3D dental region data for the entire dental region.

[0012] In the method for generating 3D face scan data described above, the step of aligning and inserting the tooth region contour 3D data into the face 3D data includes the steps of removing data of the tooth region contour portion from the face 3D data, and aligning and inserting the tooth region contour 3D data into the tooth region contour portion of the face 3D data.

[0013] In the method for generating 3D face scan data described above, the method further includes the steps of generating auxiliary region 3D data for a predetermined region of the face, and aligning and inserting the auxiliary region 3D data into the face 3D data.

[0014] In the above-described method for generating 3D facial scan data, the facial image, the facial 3D data, the dental region image, and the dental region 3D data are acquired by projecting a structured light pattern onto the patient's face or dental region.

[0015] In the above-mentioned method for generating 3D face scan data, the step of extracting a tooth region contour for the face image or the face 3D data further includes the steps of recognizing facial landmarks from the face image or the face 3D data, and extracting a tooth region contour for the face image or the face 3D data based on the facial landmarks.

[0016] Meanwhile, a computer program according to the present invention is stored in a non-transitory computer-readable storage medium for causing a computer to execute the method for generating 3D face scan data as described above. [Effects of the Invention]

[0017] According to the present invention, 3D facial scan data including high-resolution dental data can be generated using relatively inexpensive equipment, which has the advantage of being able to efficiently assist in various fields (e.g., cosmetic dental treatment). [Brief explanation of the drawings]

[0018] [Figure 1] 1 illustrates a 3D face scanning system according to the present invention; [Figure 2] 1 is a flowchart illustrating a method for generating 3D face scan data according to the present invention. [Figure 3] 1A and 1B are diagrams showing examples of images captured by a camera after projecting a structured light pattern onto a face and teeth area according to the present invention. [Figure 4] FIG. 1 is a diagram conceptually showing 3D data of a face and teeth region according to the present invention. [Figure 5] FIG. 1 illustrates the concept of aligning 3D data of face and dental regions according to the present invention. [Figure 6]1A and 1B are diagrams illustrating the concept of extracting facial landmarks and tooth region contours from a facial image according to the present invention. [Figure 7] 1 is a diagram showing a concept of separating and acquiring tooth region contour 3D data from tooth portion 3D data according to the present invention. FIG. [Figure 8] FIG. 10 is a diagram showing the concept of removing tooth region contour portions from face 3D data and aligning and inserting tooth region contour 3D data according to the present invention. [Figure 9] FIG. 2 illustrates another embodiment of the 3D face scanning system according to the present invention. [Figure 10] 10 is a flowchart illustrating another embodiment of a method for generating 3D face scan data according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] The invention will now be explained in more detail with reference to the drawings.

[0020] FIG. 1 is a diagram showing a 3D Face Scan System according to the present invention.

[0021] 1, the 3D face scanning system comprises a structured light projector 101, a face camera 102, teeth cameras 103 and 104, and a data processing device 200. These components may be embodied as separate devices, or all or part of them may be embodied as an integrated device.

[0022] First, the structured light projector 101 is a device for generating a structured light pattern and projecting the structured light pattern onto the patient's face (A) as shown in Fig. 3. However, instead of structured light, other methods such as using a stereo camera or time of flight can also be used.

[0023] The face photographing camera 102 is a device for photographing a patient's face (A) to obtain a face image such as that shown in Figure 3(a). At this time, photographing of the patient's face (A) is performed while a structured light pattern is projected onto the patient's face (A). Furthermore, photographing of the patient's face (A) may be performed from at least one of the front, left and right sides, bottom, and top of the patient's face, but is not limited thereto.

[0024] The dental imaging cameras 103 and 104 are devices for photographing the dental region of the patient's face (A) and acquiring dental region images such as those shown in (b) and (c) of Figure 3. At this time, the dental region of the patient's face can be photographed while a structured light pattern is projected onto the patient's face (A).

[0025] In the present invention, a face camera 102 for photographing the face and teeth cameras 103, 104 for focusing on photographing the teeth are configured separately. Although only one tooth camera 103, 104 may be provided, it is preferable to provide multiple cameras so that the patient's teeth can be photographed from various directions to obtain multiple images of the teeth. FIG. 1 shows an embodiment in which two teeth cameras 103, 104 are provided. In this case, there are areas visible from all of the multiple cameras 103, 104 and areas visible only from some of the cameras (103 or 104), and by aligning and synthesizing these areas, the accuracy of the 3D data can be improved.

[0026] The data processing device 200 is a device that receives the face image captured by the face camera 102 and the dental region images captured by the dental cameras 103 and 104, and generates 3D face scan data through the data processing process shown in Figure 2 or Figure 10.

[0027] FIG. 2 is a flow chart illustrating a method for generating 3D facial scan data according to the present invention.

[0028] Steps S110 and S120: First, while a predetermined structured light pattern is projected onto the patient's face (A) by the structured light projector 101, a facial image of the patient's face is captured using the facial camera 102, as shown in FIG. 3(a).

[0029] Then, 3D face data such as that shown in Figure 4(a) is generated from the facial image on which the structured light pattern is formed. The structured light pattern allows the three-dimensional appearance of the face to be grasped, allowing the generation of 3D data. The technology for acquiring 3D data from an image of an object onto which a structured light pattern is projected is well known, and therefore a detailed description thereof will be omitted.

[0030] Steps S130 and S140: In addition, while a predetermined structured light pattern is projected onto the patient's face (A) by the structured light projector 101, dental imaging cameras 103 and 104 are used to capture dental images of the patient's teeth, as shown in Figures 3(b) and 3(c). At this time, the structured light pattern for the face image and the structured light pattern for the dental image may be the same or different.

[0031] Next, 3D data of the teeth region, as shown in Figures 4(b) and (c), is generated from the image of the teeth region on which the structured light pattern is formed. Comparing the images of the teeth region, the images of the teeth region in Figures 3(b) and (c) have higher resolution than the teeth region included in the face image in Figure 3(a). As a natural consequence, the 3D data of the teeth region in Figures 4(b) and (c) is more precise than the 3D data of the teeth region included in the 3D face data in Figure 4(a).

[0032] As shown in Fig. 1, it is preferable that a plurality of tooth photographing cameras 103 and 104 are provided, and a plurality of tooth region images are acquired by photographing the tooth region from a plurality of directions in S130. In this case, S140 may be configured to generate a plurality of tooth region 3D data pieces as shown in Fig. 4(b) and 4(c) from a plurality of tooth region images on which structured light patterns are formed as shown in Fig. 3(b) and 3(c), and align these plurality of tooth region 3D data pieces to generate tooth region 3D data for the entire patient's tooth region.

[0033] There are parts that are commonly visible from multiple tooth imaging cameras 103, 104 and parts that are only visible from individual tooth imaging cameras (103 or 104), so by aligning and combining these, accurate and precise high-resolution 3D data can be obtained for the patient's tooth area.

[0034] Fig. 5 is a diagram showing the concept of aligning 3D data of face and teeth regions according to the present invention. Fig. 5(a) shows an example of generating 3D data of teeth regions by aligning two 3D data of tooth regions shown in Fig. 4(b) and (c), and Fig. 5(b) shows an example of aligning such 3D data of teeth regions with 3D data of face.

[0035] Steps S150 and S160: Next, face landmarks are recognized for the face, and a teeth zone contour is extracted for the face based on the face landmarks. This process may be performed on a face image such as that shown in Fig. 3(a) or on 3D face data such as that shown in Fig. 4(a).

[0036] FIG. 6 is a diagram illustrating the concept of extracting facial landmarks and tooth region contours from a facial image such as that shown in FIG. 3(a). The concept of facial landmarks and techniques for extracting facial landmarks from a facial image are well-known, and therefore detailed description thereof will be omitted. For example, entering the keyword "face landmark detection" in a Google search will provide various well-known techniques. The tooth region contour corresponds to the boundary line of the tooth region on the face, and in FIG. 6, the lip boundary line is used as the tooth region contour.

[0037] On the other hand, using facial landmarks to extract tooth region contours is just one example, and other methods can also be used, such as using object recognition algorithms based on artificial intelligence, but this is not limited to these.

[0038] Step S170: From the tooth region 3D data generated in S140 (e.g., tooth region 3D data in (a) of FIG. 5), tooth region contour 3D data corresponding to the tooth region contour extracted in S160 is separated and obtained. Fig. 7 is a diagram showing the concept of separating and obtaining tooth region contour 3D data from tooth region 3D data along the tooth region contour according to the present invention.

[0039] Step S180: Subsequently, the 3D data of the tooth region contour separated and acquired in S170 is aligned and inserted into the 3D data of the face acquired in S120 to obtain 3D face scan data. Figure 8 is a diagram showing the concept of aligning and inserting the 3D data of the tooth region contour into the 3D data of the face according to the present invention. As shown in (b) of Figure 8, the 3D data of the tooth region contour related to the tooth region contour is 3D data with relatively higher resolution than the 3D data of the face dealing with the general area of ​​the patient's face.

[0040] In this case, the tooth region contour 3D data can be directly augmented to the face 3D data, or it can be realized by first removing the data of the tooth region contour part from the face 3D data as shown in Figure 8(a), and then aligning and inserting the tooth region contour 3D data into the tooth region contour part of the face 3D data as shown in Figure 8(b).

[0041] FIG. 9 is a diagram showing another embodiment of a 3D face scanning system according to the present invention, and FIG. 10 is a flowchart showing another embodiment of a method for generating 3D face scan data according to the present invention.

[0042] Compared to Fig. 1, a second embodiment of the present invention shown in Fig. 9 further includes an auxiliary 3D camera 105. When the face capturing camera 102 is used to capture an image of the front of the face, the auxiliary 3D camera 105 is configured to obtain 3D data of other parts of the face, such as the sides of the face or the jaw area, other than the front of the face. In this case, it is preferable that the subject to be captured by the auxiliary 3D camera 105 can be set by an administrator (e.g., a doctor).

[0043] Therefore, compared to FIG. 2, the second embodiment of the present invention shown in FIG. 10 differs in that in step S250, auxiliary region 3D data for one or more predetermined regions of the patient's face (e.g., side of the face, jaw region) is generated by the auxiliary 3D camera 105, and in step S280, the tooth region contour 3D data and auxiliary region 3D data are aligned and inserted into the face 3D data.

[0044] Meanwhile, the present invention may be embodied in the form of computer-readable code stored on a computer-readable non-transitory recording medium, for example, a non-volatile recording medium. Such non-volatile recording media include various types of storage devices, such as hard disks, SSDs, CD-ROMs, NASs, magnetic tapes, web disks, and cloud disks. The present invention may also be embodied in the form of a computer program stored on a medium to execute specific procedures in combination with hardware.

Claims

1. acquiring a facial image of a patient's face; generating 3D facial data from the facial image; acquiring an image of the patient's dental site; generating 3D tooth site data from the tooth site image; extracting a tooth region contour for the facial image or the facial 3D data; Separating and obtaining tooth region contour 3D data corresponding to the tooth region contour from the tooth portion 3D data; and inserting the tooth region contour 3D data into the face 3D data in an aligned manner.

2. The step of acquiring dental site images of the patient's dental site includes: Photographing a tooth region from a plurality of directions to obtain a plurality of partial images of the tooth region; The step of generating 3D data of a tooth region from the tooth region image includes: generating a plurality of 3D data of tooth region portions from the plurality of tooth region portion images; and aligning the plurality of partial 3D data of teeth to generate 3D data of the entire teeth.

3. The step of aligning and inserting the tooth region contour 3D data into the face 3D data includes: removing data of the tooth region contour portion from the facial 3D data; and inserting the tooth region contour 3D data into the tooth region contour portion of the face 3D data in an aligned manner.

4. generating auxiliary region 3D data for a predetermined region of the face; The method of claim 3 further comprising the step of aligning and inserting the auxiliary feature 3D data into the face 3D data.

5. 2. The method of claim 1, wherein the facial image, the facial 3D data, the dental region image, and the dental region 3D data are acquired by projecting a structured light pattern onto the patient's face or dental region.

6. The step of extracting a tooth region contour for the facial image or the facial 3D data comprises: Recognizing facial landmarks from the facial image or the facial 3D data; The method of claim 1 , further comprising: extracting a tooth region contour for the facial image or the facial 3D data based on the facial landmarks.

7. A computer program stored on a storage medium for causing a computer to execute the method for generating 3D face scan data according to any one of claims 1 to 6.

Images