Automatic three-dimensional face scanning matching device with automatic interfeity, driving method

Abstract

An automatic three-dimensional face scan matching device to which artificial intelligence is applied, a method for driving the device, and a computer program stored in a medium are provided. According to the concept of the present invention, the automatic three-dimensional face scan matching apparatus to which artificial intelligence is applied comprises: a communication interface unit that receives first image data obtained by capturing a central portion of a face during a computer tomography (CT) scan of the face of a user, and second image data obtained by capturing a central portion of the face during the CT scan of the face of the user; the second image data is obtained by scanning the face of the user through a three-dimensional scanning method; and a control unit that visualizes the face of the user into three dimensions by analyzing the first image data and the second image data by applying artificial intelligence (AI) to find a detection point of a commonly detected keypoint, converting a two-dimensional coordinate value of the detection point found on the second image data into a three-dimensional coordinate value, and automatically matching the second image data to the computed tomography scan image data based on the converted three-dimensional coordinate values.

Description

Technical Field

[0001] The present invention concept relates to an automatic three-dimensional facial scan matching device with artificial intelligence applied and a method for driving the device, and more specifically, to the following automatic three-dimensional facial scan matching device with artificial intelligence applied, a method for driving the device and a computer program stored in a medium: the device is capable of observing the CT of a user's smiling face by visualizing the computer tomography (CT) image into three dimensions (3D), for example during dental treatment or cosmetic treatment, thereby improving the accuracy of diagnosis. Background Art

[0002] Implants originally refer to substitutes for repairing missing human tissues. However, in the field of dentistry; implantation refers to a series of surgical procedures for implanting artificial teeth. Implantation is a series of surgical procedures to restore tooth function by placing a fixture on the alveolar bone of the missing teeth and then fixing the artificial teeth on the fixture. The fixture is a tooth root made of titanium, etc. that is used to replace the missing tooth root (root) and is not rejected by the human body. For general prostheses or dentures, the surrounding teeth and bones will be damaged over time, while implants will not damage the surrounding tooth tissues, have the same function and shape as natural teeth, and will not cause tooth decay, and therefore, advantageously, implants can be used semi-permanently.

[0003] In artificial tooth surgery (or implantation or implant surgery), a drill is generally used to make a screw hole in the alveolar bone at the location where the fixture is to be placed, the fixture is placed in the screw hole to make it osseointegrated with the bone to form an artificial tooth root, the abutment is coupled to the fixture, and then the final prosthesis is used to completely cover the abutment. Dental implants will not only repair single missing teeth, but also improve the function of dentures for partially edentulous patients and completely edentulous patients, and will improve the aesthetic aspects of dental prosthetic restorations. In addition, the implant will disperse the excessive stress applied to the surrounding supporting bone tissue and also help stabilize the dentition.

[0004] Meanwhile, in order to improve the accuracy of the implantation surgery during the implantation surgery, simulation surgery and surgical planning are involved. For the simulation surgery and surgical planning, accurate data of the oral region of a subject undergoing surgery is essential. Generally, in order to obtain data of the oral region of a subject undergoing surgery, a method of obtaining three-dimensional (3D) image data by capturing an image of the oral region of a subject undergoing surgery using a computed tomography (CT) device is used.

[0005] Although computer tomography has the advantage of being able to observe the actual proportions of a user (i.e., the patient's face) in 3D, it is generally recommended to close the eyes and mouth when performing a scan due to radiation exposure. However, for actual dental or cosmetic treatment, it is important to reflect the user's smiling face, and therefore, a method for matching the position of the user (e.g., teeth in the mouth, etc.) on the smiling face is urgently needed. Summary of the invention

[0006] Technical issues

[0007] An object of the inventive concept is to provide an automatic three-dimensional facial scan matching device to which artificial intelligence (AI) is applied, a method for driving the device, and a computer program stored in a medium, the device being capable of observing the CT of a user's smiling face by visualizing a computer tomography image into 3D, for example, during dental treatment or cosmetic treatment, thereby improving the accuracy of diagnosis.

[0008] The beneficial effects of disclosure

[0009] According to an embodiment of the inventive concept, during dental treatment or cosmetic treatment, a CT image may be observed by being visualized in 3D by reflecting a smiling face of a user, and thus, the accuracy of diagnosis may be improved.

[0010] In addition, according to an embodiment of the present inventive concept, when image data obtained by CT scanning and scan data obtained by scanning a user's smiling face (which are data with heterogeneity) are combined with each other, an artificial intelligence program is applied thereto, and therefore, the accuracy and speed of data processing can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram illustrating an example of a detailed structure of an AI-based automatic 3D face scan matching apparatus according to an embodiment of the inventive concept.

[0012] Figure 2 A view configured for interpreting data for reconnaissance and face scan images.

[0013] Figure 3 To show Figure 1 A block diagram showing an example of the detailed structure of an AI-based automatic 3D facial scan matching unit.

[0014] Figure 4 A diagram for explaining a comparison between a manual method and an automatic method to which artificial intelligence is applied when marking (or detecting) detection points.

[0015] Figure 5 For interpretation Figure 3 A diagram showing the detailed operation of the face and keypoint detection unit.

[0016] Figure 6 For interpretation Figure 5 A diagram showing operations S500 to S520.

[0017] Figure 7 For interpretation Figure 5 A view showing operation S540 is shown.

[0018] Figure 8 For interpretation Figure 3 A view showing the detailed operation of the 3D point detection unit.

[0019] Fig. 9 For interpretation Figure 8 A diagram showing operations S800 to S850 .

[0020] Fig.10 To show Figure 1 The flowchart of the driving process of the AI-based automatic 3D facial scanning matching device is shown.

[0021] Main component markings

[0022] 100: Automatic 3D facial scanning matching device (software) based on artificial intelligence (AI)

[0023] 110: Communication interface unit

[0024] 120: Control unit

[0025] 130: AI-based automatic 3D facial scanning matching unit

[0026] 140: Storage unit

[0027] 300: Data receiving unit

[0028] 310: AI-based detection point marking unit

[0029] 311: Data input unit

[0030] 312: Face and key point detection unit

[0031] 313: 3D detection point (feature point) detection unit

[0032] 314: Result output unit

[0033] 320: Detection point matching unit

[0034] S500, S510, S520, S530, S540, S550, S800, S810, S820, S830, S840, S850, S1000, S1010: Operation DETAILED DESCRIPTION

[0035] According to one aspect of the inventive concept, there is provided an automatic three-dimensional face scan matching device with artificial intelligence applied, the device comprising: a communication interface unit, receiving first image data and second image data, the first image data being obtained by capturing a central portion of a face during a computer tomography (CT) scan of a user's face, and the second image data being obtained by scanning the user's face with a 3D scanning method; and a control unit, visualizing the user's face into three dimensions by the following steps: analyzing the first image data and the second image data by applying artificial intelligence (AI) to find detection points of commonly detected key points (landmarks), converting two-dimensional coordinate values ​​of the detection points found on the second image data into three-dimensional coordinate values, and automatically matching the second image data to the CT scan image data based on the converted three-dimensional coordinate values.

[0036] The control unit can analyze the first image data and the second image data by applying artificial intelligence to extract multiple detection points related to the feature points from the first image data and the second image data respectively, and extract multiple key point detection points common to each other from the extracted multiple detection points.

[0037] The control unit may extract detection points where feature points are not changed in the facial part during the CT scan and the facial part during the 3D scan as key point detection points.

[0038] The control unit may preset reference information for detecting the feature points and the plurality of key point detection points, and detect the feature points and the key point detection points by applying artificial intelligence based on the preset reference information.

[0039] The control unit may determine a shape formed using the feature points as reference information, and finally determine the key point detection point based on a result of the determination.

[0040] The control unit can convert the second image data into three-dimensional UV mapping data, use the converted UV mapping data and the three-dimensional data generated during the facial scanning to convert the two-dimensional coordinate value of the detection point into a three-dimensional coordinate value, by determining the shape formed by the three coordinates during the conversion of the three-dimensional coordinate value and the color information of each coordinate, and perform the conversion of the three-dimensional coordinate value based on the determined result.

[0041] In addition, according to another aspect of the inventive concept, a method for driving a three-dimensional facial scanning automatic matching device applied with artificial intelligence is provided, the method comprising: receiving first image data and second image data by a communication interface unit, the first image data being obtained by capturing a central part of a face during a computer tomography scan of a user's face, and the second image data being obtained by scanning the user's face with a 3D scanning method; and visualizing the user's face into 3D by a control unit through the following steps: analyzing the first image data and the second image data by applying artificial intelligence to find detection points of key points detected in common, converting two-dimensional coordinate values ​​of the detection points found on the second image data into three-dimensional coordinate values, and automatically matching the second image data to the CT scan image data based on the converted three-dimensional coordinate values.

[0042] Visualization may include: analyzing the first image data and the second image data by applying artificial intelligence, thereby extracting a plurality of detection points related to the feature points from the first image data and the second image data, respectively; and extracting a plurality of key point detection points that are common to each other from the extracted plurality of detection points.

[0043] Extracting the plurality of key point detection points may include extracting detection points where feature points do not change in the facial portion during the CT scan and the facial portion during the 3D scan as key point detection points.

[0044] The visualization may include: presetting reference information for detecting feature points and the plurality of key point detection points, and detecting the feature points and the key point detection points by applying artificial intelligence based on the pre-set reference information,

[0045] Detecting the key point detection point may include determining a shape formed using the feature points as reference information, and finally determining the key point detection point based on a result of the determination.

[0046] Visualization may include: converting the second image data into three-dimensional UV mapping data; converting the two-dimensional coordinate value of the detection point into a three-dimensional coordinate value using the converted UV mapping data and the three-dimensional data generated during the facial scan; and determining a shape formed by the three coordinate values ​​and color information of each coordinate during the conversion of the three-dimensional coordinate value, and performing the conversion of the three-dimensional coordinate value based on the determined result.

[0047] Furthermore, according to another aspect of the inventive concept, there is provided a computer program stored in a medium, the computer program being used to implement a three-dimensional face scanning automatic matching method with artificial intelligence applied, wherein the three-dimensional face scanning automatic matching method with artificial intelligence applied may include: receiving first image data and second image data, the first image data being obtained by capturing a central part of a face during a computer tomography scan of a user's face, and the second image data being obtained by scanning the user's face in a 3D scanning method; and visualizing the user's face into 3D by the following steps: analyzing the first image data and the second image data by applying artificial intelligence to find detection points of key points detected in common, converting two-dimensional coordinate values ​​of the detection points found on the second image data into three-dimensional coordinate values, and automatically matching the second image data to the CT scan image data based on the converted three-dimensional coordinate values. .

[0048] Public method

[0049] The embodiments are provided to further fully explain the present disclosure to those skilled in the art. However, the present disclosure is not limited thereto, and it should be understood that various changes may be made to the form and details without departing from the spirit and scope of the appended claims. That is, the description of a specific structure or function may be presented only to illustrate the embodiments of the present disclosure.

[0050] Various modifications may be applied to the disclosed embodiments, and specific embodiments will be illustrated in the drawings and described in the detailed description. However, the various embodiments of this document and the terms used therein are not intended to limit the disclosure to a specific practice, and it should be recognized that the disclosure encompasses various modifications, equivalent forms and / or alternative forms that do not depart from the spirit and technical scope of the disclosure.

[0051] For example, the terms "first" and "second" are only used in this article to explain various constituent elements, but these constituent elements are not limited by these terms. Such terms are only used to distinguish one constituent element from another constituent element. For example, without departing from the scope of the present disclosure, a first constituent element may be referred to as a second constituent element, and vice versa.

[0052] When a constituent element is “connected” or “connected” to another constituent element, the constituent element contacts or is connected to the other constituent element directly or through at least one of the other constituent elements. Conversely, when a constituent element is described as “directly connected” or “directly connected” to another constituent element, the constituent element should be interpreted as being directly connected to the other constituent element without any other constituent element between the constituent element and the other constituent element. Other expressions such as “located between” and “directly located between” used to describe the relationship between constituent elements may be interpreted in the same manner.

[0053] The terms used in this disclosure are used to illustrate specific embodiments and are not used to limit this disclosure. Therefore, unless otherwise clearly stated in the context, expressions used in the singular form in this disclosure also include their plural forms. In addition, terms such as "include" or "comprise" may be interpreted as indicating a certain characteristic, number, step, operation, constituent element or combination thereof, but may not be interpreted as excluding the existence or possibility of adding one or more other characteristics, numbers, steps, operations, constituent elements or combinations thereof.

[0054] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meanings as those commonly understood by those skilled in the art in the art to which the present disclosure may belong. Terms defined in commonly used dictionaries are interpreted as having meanings matching their meanings in the context of the relevant technology, and unless otherwise clearly defined, they should not be interpreted as having idealized or overly formal meanings.

[0055] In the drawings, the thickness of each layer and region is exaggerated for clarity. It should also be understood that when a layer is referred to as being "on" another layer or substrate, the layer may be directly on the other layer or substrate, or there may be intervening layers. The same reference numerals in the drawings represent the same elements and therefore will not be described in detail.

[0056] Terms such as upper end, lower end, upper surface, lower surface, front surface, rear surface, upper part, or lower part are used to distinguish the relative positions of components. For example, for convenience, when the upper side of the figure is referred to as the upper part and the lower side of the figure is referred to as the lower part, in fact, without departing from the scope of the present disclosure, the upper part may be referred to as the lower part, and the lower part may be referred to as the upper part. In addition, in order to help understand the present disclosure, the drawings are not shown in actual proportion, but are shown exaggerated in the size of some components.

[0057] Hereinafter, embodiments of the inventive concept will be explained in detail with reference to the accompanying drawings.

[0058] Figure 1is a block diagram illustrating an example of a detailed structure of an AI-based automatic 3D face scan matching apparatus according to an embodiment of the inventive concept. Figure 2 This is a view for explaining the data configuration of the SCOUT image and the face scan image.

[0059] like Figure 1 As shown in the figure, an automatic 3D facial scan matching device (and software) based on (or applying) artificial intelligence (AI) (hereinafter referred to as a device) 100 according to an embodiment of the inventive concept includes some or all of the components in a communication interface unit 110, a control unit 120, an AI-based automatic 3D facial scan matching unit 130 and a storage unit 140.

[0060] Before describing in detail, the AI-based automatic 3D facial scanning matching device 100 according to an embodiment of the present inventive concept may include various devices used for dentistry, etc., such as desktop computers, laptop computers, tablet personal computers (PCs), smart phones, etc., but may also include a CT scanning device for performing CT scanning and a 3D scanning device for performing a 3D scan of the face of a user (i.e., a patient), and the embodiments of the present inventive concept may not be particularly limited to any one device. In addition, the AI-based automatic 3D facial scanning matching device 100 may include a computer-readable recording medium included in the corresponding device or a computer program (or software) stored in the medium. For example, since the scanning is performed simultaneously from cameras located in three directions, the 3D scanning device can obtain distortion-free three-dimensional scanning data. This is Figure 2 As shown in (b) of FIG. 1 , since it is a 3D scanning device, Figure 2 As shown in (b), it is obvious that two-dimensional texture image data and three-dimensional data of a black and white three-dimensional image are obtained.

[0061] In addition, a cone beam CT (CBCT) can be used as a CT scanning device according to an embodiment of the present inventive concept. CBCT is mainly used in dentistry. CBCT is smaller than a medical CT used for scanning in a general hospital, can perform vertical scanning, and takes about 15 to 30 seconds, which is shorter than the time of a medical CT. Tomographic imaging is performed by a sensor (i.e., a detector) that receives an X-ray signal. When an X-ray is irradiated, signals reflected by various tissues, organs, etc. in the face are scanned by the detector, and these signals are used as depth information, thereby generating three-dimensional data. Therefore, the oral cavity or the head is visually observed using the CBCT used in dentistry. A captured image of the central part of the patient's face is obtained during a CT scan or by a separate camera, etc., and the captured image may be referred to as a scout image (or image). Before a main examination of a patient is performed, a scan is performed to find the location of a lesion or an area of ​​interest. In this state, a scout image is used to set a reference for determining the inspection range. In other words, a scout image is an image used for scanning, which is used to specify a scan portion by checking whether a patient has a disease, etc., before a direct examination. For example, when the scanning operator specifies an area to be scanned while observing a scout image, images are continuously captured at a certain thickness within the specified range. Therefore, when a line is generated on the scout image by identifying the position of the scanned slice image while browsing an image such as a CT image and then the scout image and the slice image are linked to each other, the position of the image can be checked, and when the generated line is clicked, the corresponding slice image is output, so that a function of quickly accessing a desired image can be assigned.

[0062] Images such as CT images are captured by at least tens of images or at most hundreds of images at a time depending on the patient's condition or as needed. In general, doctors diagnose the patient's condition while observing the slice images of continuous scanning, and therefore, it takes too long to find abnormal areas or necessary parts from dozens of images. Therefore, by generating a line on the scout image to directly click on the line, the slice image of the corresponding part is directly output, so that the image of the desired part can be observed in a short time. When generating a scout line, the most important thing is to accurately match the position of the captured slice image with the position of the line to be generated. Therefore, the scout line is generated at an accurate position by distinguishing the correlation between the scout image and each slice image.

[0063] The AI-based automatic 3D facial scan matching device 100 according to an embodiment of the present inventive concept can implement R2 STUDIO Q scanning. In this state, the generated data may include CBCT images (i.e., Figure 2 The reconnaissance image in (a) and Figure 2Here, "R2 Studio Q Scan" may refer to a scan performed by the AI-based automatic 3D facial scan matching device 100 (i.e., a device separately manufactured to perform operations according to an embodiment of the inventive concept). Figure 2 It is observed that the scout image or 3D facial image is three-dimensional data including a texture image file (bmp, png, jpg, etc.), a three-dimensional image file (.obj), and a setting file (.mtl).

[0064] A texture (a texture or texture) image file is a file obtained by scanning a patient and is a file having a color (i.e., color information), which may be skin data as the outer surface of the face. A texture image file is an image for an OBJ file and is used in association with a three-dimensional image. In addition, a three-dimensional image file refers to data formed by white (i.e., black and white). A three-dimensional image file is used in the OBJ file format and includes coordinates (values) for expressing in a three-dimensional manner and data that can assign a hue or color to a facial surface. A three-dimensional image may include a V factor and a VT factor. V records the coordinate values ​​of three-dimensional coordinates (X, Y, Z) and generates a surface in groups of three. Therefore, the three coordinates have a polygonal structure. In other words, the polygonal structure is formed by three coordinates. VT records the texture image coordinates of the color to be assigned to the surface of the three-dimensional image (i.e., the facial skin surface). For example, by obtaining the texture image file color, the color is applied to the surface. The MTL file includes the system path of the texture image file to be used by the three-dimensional image and the setting value of the effect.

[0065] Based on the description presented above, Figure 1 The communication interface unit 110 shown may operate as a data receiving unit. The communication interface unit 110 receives each of the scout data of the scout image and the scan data of the face scan image through CBCT scanning or 3D face scanning. For example, the data of the scout image may be received through a tomographic imaging device such as CBCT, and the data of the face scan image may be received through a 3D face scanning device. Figure 2 The facial scan image data in (b) are data having, for example, different (i.e., heterogeneous) characteristics for the patient's face. The scout data are data of scan images captured during a CT scan when the patient must close his eyes and mouth to avoid radiation, while the facial scan image data are data about facial images of the patient smiling in daily life (because there is no radiation exposure as in a CT scan).

[0066] The control unit 120 is responsible for Figure 1The overall control operation of the communication interface unit 110, the AI-based automatic 3D facial scan matching unit 130, and the storage unit 140 is shown. When the scout image data and the 3D facial scan image data are received through the communication interface unit 110, the control unit 120 temporarily stores the received data in the storage unit 140, and then retrieves and provides the data to the AI-based automatic 3D facial scan matching unit 130.

[0067] When the integration (or overlap) of the CBCT image data and the facial scan image is achieved by the AI-based automatic 3D facial scan matching unit 130, the control unit 120 can display the integrated data according to the dentist's command on the screen by visualizing the data in 3D. The screen may be a monitor of a computer, and may also be a separate display unit.

[0068] The AI-based automatic 3D facial scan matching unit 130 performs an automatic matching operation on the scout image data obtained by capturing the central part of the patient's face during, for example, a CT scan, and the facial scan image data, and by applying the AI ​​program during the process, the accuracy of the matching operation can be improved. It can be said that such an operation is to match the position of the patient through the patient's smiling face during an actual dental or cosmetic treatment, and during a CT scan, the scanning is performed with the eyes or mouth closed to prevent the patient from being exposed to radiation. In other words, although in the CT scan image, the scanning is performed while the patient closes his eyes and mouth, the AI-based automatic 3D facial scan matching unit 130 only uses the facial scan image data to replace the image data of the facial surface. However, in this case, the bone structure, tooth position, etc. of the patient when smiling are different from the bone structure, tooth position, etc. observed during the CT scan, and therefore, it can be said that the difference is corrected based on the facial scan image data to achieve 3D visualization. In this state, by converting the CT image data based on the detection points of the key points detected jointly from the two-dimensional scout image data and the facial scan image data, the accuracy of data processing can be improved.

[0069] In more detail, the AI-based automatic 3D facial scan matching unit 130 uses two-dimensional and three-dimensional data to detect n three-dimensional coordinates with clinical significance. The matching process of the scout data and the facial scan data is automatically performed by applying artificial intelligence (i.e., an AI program). For example, feature points of two different faces can be detected from the scout image data and the facial scan image data. In an embodiment of the inventive concept, the face is identified by detecting, for example, 51 feature points. Then, n key points of the common part (i.e., the part that does not change when the eyes are closed or the smile) are detected among the detected feature points. The detection points of the key points can be 4 to 6, and the maximum is 51. In an embodiment of the inventive concept, in this process, the reference point (or reference information) is set at a consistent position by artificial intelligence to detect the feature point while also accurately detecting the key point. A plurality of various information (e.g., pixel information or pixel boundary information on the image or the shape of the pixel information, etc.) can be used as reference information. In an embodiment of the inventive concept, a part of the nose tip or the nostril tip, etc. corresponds to the key point. Therefore, when setting the reference point, image analysis is performed by artificial intelligence and a learning operation is performed without doubt, so that a part of the nose tip and the nostril tip, etc. can be accurately marked. More precisely, it may be preferable to identify the corresponding detection point based on reference information rather than marking. In some cases, this can be expressed as extracting the detection point. Therefore, when observing the 3D visualization data, by matching the coordinate values ​​(or first coordinate values) of the reconnaissance image data corresponding to the same part with the coordinate values ​​(or second coordinate values) of the 3D scan image data, the CT image can be observed through the smiley face image. For example, since a conversion relationship, etc. can be observed through the two coordinate values, even when the scanned image data of the smiley face is matched to the three-dimensional CT scan image data or overlaps with the three-dimensional CT scan image data, 3D visualization data combined with the smiley face can be generated by converting the CT image data based on the conversion relationship.

[0070] For example, assume that the above-mentioned marking (or detection point detection) operation is performed manually. Since the operation is performed manually rather than automatically, if a problem occurs, the operation may need to be re-performed. In addition, due to manual implementation, a lot of time is spent on data processing. In contrast, as in an embodiment of the inventive concept, when artificial intelligence is applied, 10-point marking (such as the tip of the nostril, etc.) can be completed with one click, and in addition, marking can be performed at consistent positions, or consistent positions can be detected. Therefore, a CT image of a smiling face can be observed, so that the accuracy of diagnosis is improved. 3D visualization of the CT image can be performed by combining the tomographic imaging data with the depth information. Details related to the AI-based automatic 3D facial scan matching unit 130 will be discussed later.

[0071] The storage unit 140 temporarily stores data processed under the control of the control unit 120. When receiving the scout image (or image) data and the 3D face scan data from the communication interface unit 110, the control unit 120 temporarily stores the received data on the storage unit 140, and then retrieves and provides the data to the AI-based automatic 3D face scan matching unit 130, thereby enabling an automatic matching operation to which artificial intelligence is applied.

[0072] In addition to the above, Figure 1 The communication interface unit 110 , the control unit 120 , the AI-based automatic 3D facial scan matching unit 130 , and the storage unit 140 may perform various operations, and other details will be discussed later.

[0073] Although the embodiments of the present invention Figure 1 The communication interface unit 110, the control unit 120, the AI-based automatic 3D facial scan matching unit 130, and the storage unit 140 are configured with hardware modules physically separated from each other, but each module may store software therein to implement and execute the operations. However, since the software is a set of software modules and each module may be formed by hardware, it may not be particularly limited to the configuration of software or hardware. For example, the storage unit 140 may be a storage or memory as hardware. However, since information may also be stored (stored) in a software manner, it is not particularly limited to the above.

[0074] Meanwhile, in another embodiment of the inventive concept, the control unit 120 may include a central processing unit (CPU) and a memory that may be formed in one chip. The CPU may include a control circuit, an operation unit (arithmetic logic unit (ALU)), an instruction interpretation unit, a register, etc., and the memory may include a random access memory (RAM). The control circuit may implement a control operation, and the operation unit may implement an operational operation of binary bit information, and the instruction interpretation unit may include an interpreter, a compiler, etc. for converting a high-level language into a machine language or converting a machine language into a high-level language, and the register may involve storing data in a software manner. According to the above configuration, for example, the AI-based automatic 3D face scan matching device 100 copies a program stored in the AI-based automatic 3D face scan matching unit 130 at an initial stage of operation to load the program onto a memory (ie, RAM) and execute the program, thereby quickly increasing a data operation processing speed. For the case of a deep learning model, the program is loaded onto a graphics processing unit (GPU) memory instead of a RAM, thereby executing the program by using a GPU to accelerate the execution speed.

[0075] Figure 3 It is shown Figure 1 A block diagram showing an example of the detailed structure of an AI-based automatic 3D facial scan matching unit. Figure 4 It is a view for explaining the comparison between a manual method and an automatic method to which artificial intelligence is applied when marking (or detecting) detection points is performed. Figure 5 It is used to explain Figure 3 A diagram showing the detailed operation of the face and keypoint detection unit. Figure 6 It is used to explain Figure 5 A diagram showing operations S500 to S520. Figure 7 It is used to explain Figure 5 A view showing operation S540 is shown.

[0076] Figure 8 It is used to explain Figure 3 A view showing the detailed operation of the 3D point detection unit. Fig. 9 It is used to explain Figure 8 A diagram showing operations S800 to S850 .

[0077] like Figure 3 As shown in FIG. 1 , according to an embodiment of the present invention Figure 1The AI-based automatic 3D facial scan matching unit 130 may include some or all of the components in a data receiving unit 300 , an AI-based detection point marking unit 310 , a detection point matching unit 320 , and a 3D visualization unit.

[0078] Here, the wording “including some or all” means that the AI-based automatic 3D facial scan matching unit 130 is configured by omitting some components such as the data receiving unit 300 or the 3D visualization unit, or some components such as the detection point matching unit 320 are integrated into the AI-based detection point marking unit 310, etc. In order to help fully understand the concept of the present invention, in this description, it is assumed that all components are included.

[0079] The data receiving unit 300 may receive scout image data from the CBCT scanning device and receive face scan image data from the face scanning device. In addition, the data receiving unit 300 may also receive three-dimensional CT image data for 3D visualization (i.e., visualization data generation). In an embodiment of the inventive concept, by matching the two-dimensional face scan image data to the three-dimensional CT image data, a CT image of a patient's smiling face may be observed, thereby enabling accurate diagnosis.

[0080] If available Figure 3 It is observed that the AI-based detection point labeling unit 310 includes some or all of the components in the data input unit 311, the face and key point detection unit 312, the 3D detection point (or feature point) detection unit 313, and the result output unit 314. Here, the wording "including some or all" is not much different from the above meaning. The data input unit 311, the face and key point detection unit 312, the 3D detection point detection unit 313, and the result output unit 314 can be configured with hardware, software, or a combination thereof, and in an embodiment of the inventive concept, a configuration with SW may be preferred, but is not particularly limited to any one form.

[0081] If available Figures 4 to 7 It is observed in FIG. 1 that the face and key point detection unit 312 can automatically match the reconnaissance data with the face scan data through artificial intelligence. In other words, there may be approximately 27 detection points or feature points in the face. In this state, in an embodiment of the inventive concept, points that do not move even when blinking (i.e., points whose positions in the face do not change) are used as key points, and Figures 5 to 7 In other words, the face and key point detection unit 312 performs a facial region detection operation to perform Figure 5 Operation S520 is shown. Figure 6The feature point detection process for automatic matching in texture images is shown. The process can be used when the overlap of the input correct answer area (i.e., the area predicted by the reference information and the artificial intelligence program (or algorithm)) is 50% or more. The artificial intelligence is repeated as follows Figure 6 In other words, for facial recognition, in an embodiment of the inventive concept, with respect to input scout image data, feature points can be detected by setting a plurality of regions or changing the set regions, and by learning the process using artificial intelligence, the facial region can be accurately identified by the detected feature points.

[0082] Next, the face and key point detection unit 312 may perform Figure 5 The operation of operation S540 is shown. In other words, when the face region is recognized, for example, points that do not move even when smiling or blinking, that is, key points, are detected from the corresponding face region. Figure 7 is a view for explaining key point detection AI. In an embodiment of the inventive concept, among the 51 feature points detectable in a face, the feature points that can be used as key points can be limited to 4 to 6 for use. Figure 7 As can be observed in FIG. 1 , the exact center portion between the two inner corners of the eye, the tip of the nostril, and the eyebrow can be used. Therefore, the key point detection accuracy can be improved by determining or learning the shapes about the six points. Therefore, the face and key point detection unit 312 can output the two-dimensional coordinates of the six key points from the scout image data and the face scan image data. In other words, it can be said that since the scout image is identical to the actual facial ratio of the patient, the face and key point detection unit 312 detects key points from the face scan image data using the scout image.

[0083] also, Figure 3 The 3D detection point detection unit 313 performs a recalculation operation, that is, converting the 2D coordinates of the key points into 3D coordinates. Figure 8 and Fig. 9 Shown respectively in Figure 3 The detailed operation flow and shape implemented in the 3D detection point detection unit 313 are shown. The 3D detection point detection unit 313 converts n (e.g., 6) key point coordinates into 3D coordinates based on the data of converting the facial scan image data into UV mapping coordinates. To this end, the 3D detection point detection unit 313 may calculate a shape or form made of V (i.e., three-dimensional coordinates) or check VT (i.e., color information about the position in the texture image). More precisely, the facial scan image data and the 3D facial scan image data converted after the facial scan image data is converted into UV mapping coordinates include coordinate values, and therefore, the two-dimensional coordinates of the key points are converted into three-dimensional coordinates. This process is Fig. 9 Well shown in.

[0084] In the process of converting into three-dimensional coordinates, various operations may be further performed. For example, in the process of calculating V, when the number of vertices of OBJ does not match the number of texture coordinate values, it is determined to be an inappropriate 3D coordinate and is thus identified as a warning. In this case, the corresponding coordinate values ​​are ignored, and the operation for calculating the coordinate values ​​may be re-performed. The UV mapping coordinates may be mapping coordinates that may be obtained after overlaying the two-dimensional facial scan image data into three-dimensional image data using a separate algorithm, etc. In other words, UV mapping is a three-dimensional modeling process that creates a two-dimensional image into a three-dimensional model, and for example, the simplest UV mapping may consist of three steps: a mesh decomposition step, a texture creation step, and a texture application step. Therefore, for example, as may be seen in Fig. 9 It is observed in FIG. 3 that the 3D facial scan data may overlap with the UV map data obtained by converting the 2D facial scan image data, and thus, the 3D coordinate values ​​of the 2D coordinate values ​​of the key points may be extracted.

[0085] As described above, the 3D detection point detection unit 313 converts n key point coordinates in the three-dimensional coordinates through UV mapping, calculates V and VT of the converted three-dimensional coordinates, and converts the three-dimensional coordinates into 3D key point coordinates when it is determined that there is no problem in the calculation (for example, the number of vertices of the OBJ is the same as the number of texture coordinate values). For example, when the number of vertices of the OBJ does not match the number of texture coordinate values, the three-dimensional coordinates are determined to be inappropriate 3D coordinates, so that a separate conversion operation may not be performed. Fig. 9 The process of coordinate conversion performed by the 3D detection point detection unit 313 is shown.

[0086] The detection point matching unit 320 matches the facial scan image data to the three-dimensional CT image (or overlaps the facial scan image data with the three-dimensional CT image) based on the detection points of the key points converted into three-dimensional coordinate values ​​to observe the CT image of the patient's smiling face. Therefore, by matching the facial scan image data to the three-dimensional CT image, when the patient smiles, the bone structure, the degree of exposure of the teeth, etc., change, so that accurate diagnosis can be achieved.

[0087] Therefore, although the scout image has the actual ratio of the patient's face, in dental or cosmetic treatment, more attention may be paid to accurately identifying the gum shape of the patient's smiling face and the effect of the prosthesis and denture in this state. To this end, based on the scout image data, key points (i.e., detection points) that do not move even when blinking are found in the 3D facial scan image data, and by matching the facial scan image to the CT scan image based on these detection points, the CT image of the smiling face can be visualized in 3D. In this process, since the coordinate values ​​of the detection points of the key points detected from the facial scan image data are two-dimensional, the corresponding coordinates are converted into three-dimensional and matched to the CT image through the converted three-dimensional coordinate values, thereby enabling 3D visualization.

[0088] In this process, in an embodiment of the inventive concept, artificial intelligence is applied to recognize a face, and key point detection points are detected by feature points detected from the face. Through artificial intelligence, key point detection points are detected at precise positions in the face. Therefore, detection or marking of consistent positions is performed so that the accuracy of matching can be improved. In addition, in the case of matching or detection, the facial scan image is converted into a UV map, and therefore, based on this, three-dimensional coordinate values ​​of two-dimensional key point detection points are obtained. In the process of determining whether to convert into three-dimensional coordinate values, by determining a shape formed by the coordinate values ​​or checking color information, the conversion of the three-dimensional coordinate values ​​of the coordinate values ​​of the two-dimensional key point detection points is finally determined based on the inspection or determination result.

[0089] When converted into three-dimensional coordinates through this process, 3D visualization of a smiling face through a CT image is achieved by matching the facial scan data to the CT scan image data. Since the precise coordinate value can only be detected when the detection point of the key point is accurately marked or detected, the corresponding detection operation is performed by analyzing the pixel information (e.g., reference information) or the like that is a feature of the area of ​​interest using artificial intelligence or learning. Therefore, the detection point of the key point can be marked or detected at a precise position. Through artificial intelligence, a precise detection corresponding to it is performed based on a reference point or a reference value. For example, since the inner corner of the eye as a reference point can have a specified pixel characteristic, the above information can be preset, and the artificial intelligence can detect the precise position by comparing the result of the pixel analysis with the preset information.

[0090] Fig.10 It is shown Figure 1 Flowchart of the driving process of the AI-based automatic 3D facial scanning matching device.

[0091] For the sake of explanation, see Fig.10 and Figure 1 , according to an embodiment of the present invention Figure 1The AI-based automatic 3D face scanning matching device 100 receives first image data and second image data, wherein the first image data is obtained by capturing the central part of the face during computer tomography of the user's face, and the second image data is obtained by scanning the user's face with a 3D scanning method (S1000). Here, it can be seen that the first image data is Figure 2 The scout image data shown in (a) is a captured image of the central part of the face during CT scanning, and the second image data is as shown in Figure 2 In this process, the three-dimensional image data of the CT image and the three-dimensional image data of the face scan can be received or generated.

[0092] In addition, the AI-based automatic 3D facial scan matching device 100 visualizes the user's face in 3D through the following steps: analyzing the first image data and the second image data by applying artificial intelligence (AI) to find or detect detection points of key points detected together, converting the two-dimensional coordinate values ​​of the detection points found on the second image data into three-dimensional coordinate values, and automatically matching the second image data to the CT scan image data based on the converted three-dimensional coordinate values ​​(S1010).

[0093] More specifically, the AI-based automatic 3D facial scan matching device 100 can accurately detect the detection points of the reconnaissance image data (i.e., the detection points of the key points) and the detection points of the key points in the facial scan image data by executing an artificial intelligence program based on preset reference points or reference information. The AI-based automatic 3D facial scan matching device 100 can use the facial scan image data (e.g., converted into UV mapping data) or the three-dimensional data of the facial scan image to convert the coordinate values ​​extracted from the detection points based on the key points into three-dimensional coordinates, and match the facial scan image data to the three-dimensional CT scan data based on the converted three-dimensional coordinates, so that a CT image of the patient's smiling face can be observed. This can be achieved through 3D visualization.

[0094] In addition to the above, Figure 1 The AI-based automatic 3D facial scan matching device 100 shown can also perform various operations, and since other details are fully explained above, they will not be repeated here.

[0095] Although the embodiments have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and changes made by those skilled in the art using the basic concepts of the present disclosure defined in the appended claims also fall within the scope of the present disclosure, and such modified embodiments should not be individually interpreted based on the technical concepts or prospects of the concepts of the present invention.

[0096] Meanwhile, although all components of the embodiments of the inventive concept have been described above as being coupled as a single unit or coupled to operate as a single unit, the present disclosure is not necessarily limited to such embodiments. That is, within the purpose of the present disclosure, one or more of the components may be selectively coupled to operate as one or more units. In addition, although each of the components may be implemented as independent hardware, some or all of the components may be selectively combined with each other so that they may be implemented as a computer program with one or more program modules for implementing some or all of the functions combined in one or more hardware. A person of ordinary skill in the technical field of the present disclosure may easily think of the code and code segments that form the computer program. Such a computer program may implement the embodiments of the present disclosure by being stored in a non-temporary computer-readable medium and being read and executed by a computer.

[0097] Here, the non-transitory readable medium does not refer to a medium that stores data for a short time (e.g., a register, a cache memory, a memory, etc.), but refers to a medium that stores data semi-permanently and can be read by a device. Specifically, the above-mentioned program can be stored and provided on a non-transitory readable recording medium (e.g., a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a universal serial bus (USB), a memory card, a read-only memory (ROM), etc.).

[0098] Although the concept of the present invention has been specifically shown and described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made to it in form and detail without departing from the spirit and scope of the concept of the present invention as defined by the appended claims. Therefore, the scope of the concept of the present invention is not limited by the detailed description of the concept of the present invention, but by the appended claims, and all differences within the scope will be interpreted as being included in the concept of the present invention.

[0099] Industrial Applicability

[0100] The concept of the present invention can be used in the medical industry, especially in the dental industry.

Claims

1. An automatic three-dimensional facial scanning and matching device using artificial intelligence, characterized in that: The automatic three-dimensional facial scanning and matching device comprises: a communication interface unit that receives first image data and second image data, the first image data being obtained by capturing a central portion of a face during a computer tomography (CT) scan of a user's face, and the second image data being obtained by scanning the user's face in a three-dimensional scanning method; and A control unit visualizes the user's face into three dimensions by the following steps: analyzing the first image data and the second image data by applying artificial intelligence (AI) to find detection points of key points detected together, converting two-dimensional coordinate values ​​of the detection points found on the second image data into three-dimensional coordinate values, and automatically matching the second image data to computer tomography scan image data based on the converted three-dimensional coordinate values.

2. The device according to claim 1, characterized in that The control unit analyzes the first image data and the second image data by applying the artificial intelligence to extract multiple detection points related to feature points from the first image data and the second image data respectively, and extracts multiple key point detection points that are common to each other from the extracted multiple detection points.

3. The device according to claim 2, characterized in that The control unit extracts detection points where the feature points are not changed in the facial part during the computer tomography scanning and the facial part during the three-dimensional scanning as the key point detection points.

4. The device according to claim 2, characterized in that The control unit presets reference information for detecting the feature points and the plurality of key point detection points, and detects the feature points and the key point detection points by applying the artificial intelligence based on the preset reference information.

5. The device according to claim 4, characterized in that The control unit determines a shape formed using the feature points as the reference information, and finally determines the key point detection point based on a result of the determination.

6. The device according to claim 1, characterized in that The control unit converts the second image data into three-dimensional UV mapping data, converts the two-dimensional coordinate value of the detection point into the three-dimensional coordinate value using the converted UV mapping data and the three-dimensional data generated during the facial scanning, determines a shape formed by three coordinates during the conversion of the three-dimensional coordinate value and color information of each coordinate, and performs the conversion of the three-dimensional coordinate value based on the result of the determination.

7. A method for driving an automatic three-dimensional facial scanning matching device using artificial intelligence, characterized in that: The method includes: receiving, by a communication interface unit, first image data and second image data, the first image data being obtained by capturing a central portion of a face during a computer tomography scan of a user's face, and the second image data being obtained by scanning the user's face in a three-dimensional scanning method; and The control unit implements visualizing the user's face into three dimensions by the following steps: analyzing the first image data and the second image data by applying artificial intelligence to find detection points of key points detected together, converting the two-dimensional coordinate values ​​of the detection points found on the second image data into three-dimensional coordinate values, and automatically matching the second image data to computer tomography scan image data based on the converted three-dimensional coordinate values.

8. The method according to claim 7, characterized in that The visualization includes: analyzing the first image data and the second image data by applying the artificial intelligence, thereby extracting a plurality of detection points related to feature points from the first image data and the second image data, respectively; and A plurality of key point detection points common to each other are extracted from the plurality of extracted detection points.

9. The method according to claim 8, characterized in that The extracting of the plurality of key point detection points comprises: Detection points where the feature points have not changed are extracted from the facial part during the computer tomography scanning and the facial part during the three-dimensional scanning as the key point detection points.

10. The method according to claim 8, characterized in that The visualization includes: Presetting reference information for detecting the feature points and the plurality of key point detection points; and The feature points and the key point detection points are detected by applying the artificial intelligence based on the preset reference information.

11. The method according to claim 10, characterized in that The detecting of the key points includes: A shape formed using the feature points is determined as the reference information, and the key point detection point is finally determined based on a result of the determination.

12. The method according to claim 7, characterized in that The visualization includes: Converting the second image data into three-dimensional UV mapping data; converting the two-dimensional coordinate values ​​of the detection points into the three-dimensional coordinate values ​​using the converted UV map data and the three-dimensional data generated during the face scanning; and A shape formed by three coordinate values ​​and color information of each coordinate during the conversion of the three-dimensional coordinate values ​​is determined, and conversion of the three-dimensional coordinate values ​​is carried out based on a result of the determination.

13. A computer program stored in a medium for implementing an automatic three-dimensional facial scan matching method using artificial intelligence, characterized in that: The automatic three-dimensional facial scanning matching method using artificial intelligence includes: receiving first image data and second image data, the first image data being obtained by capturing a central portion of a face during a computer tomography scan of a user's face, and the second image data being obtained by scanning the user's face in a three-dimensional scanning method; and The user's face is visualized in three dimensions by the following steps: analyzing the first image data and the second image data by applying artificial intelligence to find detection points of key points detected together, converting the two-dimensional coordinate values ​​of the detection points found on the second image data into three-dimensional coordinate values, and automatically matching the second image data to computer tomography scan image data based on the converted three-dimensional coordinate values.

Images