Method for obtaining intraoral model and electronic device for performing same

Abstract

Disclosed is a method for obtaining an intraoral model. The method comprises the steps of: obtaining depth data including a plurality of depth frames for the inside of a patient's oral cavity by using an intraoral scanner; matching the depth data; obtaining pose information of the intraoral scanner on the basis of the matching; and generating a 3D model representing the inside of the oral cavity on the basis of the depth data and the pose information, wherein the step of matching the depth data comprises matching a pair of data points corresponding to each other, included in two different depth frames, a weight is applied to the matching of the pair of data points, and the weight is determined on the basis of the type of an object corresponding to the pair of data points.

Description

Method for obtaining an intraoral model and an electronic device for performing the same

[0001] The present disclosure relates to a method for obtaining an intraoral model and an electronic device for performing the same, and more particularly, to a method for obtaining a 3D model representing the interior of the oral cavity using an intraoral scanner.

[0002] Digital Impression Scanning (DIS) is a technology that replaces the traditional physical impression method (which involves taking a model of the oral cavity). It involves using an intraoral scanner to precisely scan a patient's teeth, gums, and oral structure, then converting them into a digital 3D model. Compared to physical methods, digital impression scanning offers greater precision and the ability to process data in real time.

[0003] Creating a high-precision 3D model representing the oral cavity requires iteratively aligning multiple depth data points acquired from an intraoral scanner. For example, alignment algorithms such as the Iterative Closest Point (ICP) algorithm can be used. However, intraoral structures contain regions with different characteristics, such as teeth and gingiva, and not all structures are equally important.

[0004] In particular, the accuracy of tooth alignment in a 3D model significantly impacts dental treatments such as prosthesis design, orthodontic treatment planning, and implant placement. Conversely, the accuracy of alignment of flexible structures such as the gingiva and tongue is less critical. Therefore, to efficiently utilize limited resources, it is necessary to prioritize alignment of critical structures such as teeth during the 3D model creation process.

[0005] The technical problem that the present disclosure seeks to solve is to improve the alignment accuracy of objects with high importance in a 3D model by varying the weights according to the importance of the intraoral objects.

[0006] Another technical challenge that the present disclosure seeks to address is to efficiently align depth data in a limited resource environment to optimize the quality of the entire 3D model.

[0007] According to one embodiment of the present disclosure, a method for obtaining an intraoral model may be provided, including the steps of: obtaining depth data including a plurality of depth frames for the inside of a patient's oral cavity using an oral scanner; matching the depth data; obtaining pose information of the oral scanner based on the matching; and generating a 3D model representing the inside of the oral cavity based on the depth data and the pose information, wherein the step of matching the depth data matches pairs of corresponding data points included in two different depth frames, and a weight is applied to the matching of the pairs of data points, and the weight is determined based on the type of an object corresponding to the pair of data points.

[0008] The above object includes a static object and a dynamic object, and a weight applied to the static object may be greater than a weight applied to the dynamic object.

[0009] The static object may include at least one of a tooth, a gingiva, a scanbody, and an abutment, and the dynamic object may include at least one of a tongue, a cheek, a lip, a finger, and an oral tool.

[0010] The weight applied to at least one of the teeth, scan bodies, and abutments among the above static objects may be greater than the weight applied to the gingiva.

[0011] The above weight may be greater when the object is a tooth than when the object is a gum.

[0012] The above weight may be greater when the object is a tooth to be treated than when the object is a tooth not to be treated.

[0013] The type of the above object can be obtained using a neural network model trained to identify the type of object inside the oral cavity.

[0014] The above neural network model can output an identification result for the type of object inside the oral cavity and a confidence value for the identification result.

[0015] The above weight can be determined based on the above reliability value.

[0016] The teeth to be treated can be designated in advance by the user.

[0017] The step of matching the depth data includes the steps of identifying a data point pair composed of a first data point and a second data point in the two different depth frames, obtaining transformation information for converting the first data point to be as close as possible to the second data point, and calculating an error based on a distance between the first data point and the second data point converted using the transformation information, wherein the weight may be applied to at least one of the steps of identifying the data point pair, obtaining the transformation information, and calculating the error.

[0018] The pose information of the oral scanner can be obtained based on transformation information for transforming two data points constituting the data point pair to be close to each other and internal parameters of the oral scanner.

[0019] Each depth frame constituting the above depth data may include a depth map, a point cloud, or RGB-D data.

[0020] According to another embodiment of the present disclosure, an electronic device may be provided, comprising: a communication interface; a memory including at least one instruction; and a processor; wherein the processor, by executing the at least one instruction, obtains depth data including a plurality of depth frames for the inside of a patient's oral cavity using an oral scanner, aligns the depth data, obtains pose information of the oral scanner based on the alignment, and generates a 3D model representing the inside of the oral cavity based on the depth data and the pose information, wherein the processor aligns pairs of corresponding data points included in two different depth frames, and a weight is applied to the alignment of the pair of data points, and the weight is determined based on the type of an object corresponding to the pair of data points.

[0021] According to another embodiment of the present disclosure, a method for obtaining a 3D intraoral model may be provided, including: obtaining depth data for the inside of a patient's oral cavity; obtaining a first key depth frame group and a second key depth frame group composed of a plurality of key depth frames based on a plurality of low depth frames included in the depth data; and generating a 3D oral model based on at least one of the first key depth frame group and the second key depth frame group; wherein the generating the 3D oral model comprises: generating the 3D oral model based on the first key depth frame group and the second key depth frame group when the first key depth frame group and the second key depth frame group overlap by a predetermined standard or more, and generating the 3D intraoral model based on the first key depth frame group when the first key depth frame group and the second key depth frame group do not overlap by a predetermined standard or more.

[0022] The first key depth frame group and the second key depth frame group are defined as sequentially acquired key depth frame groups, and the second key depth frame group may be defined as a group that defines a key frame as a first key frame when the acquired key frame does not overlap with a plurality of key frames constituting the first key depth frame group, and includes the first key frame as a first key frame.

[0023] The second key depth frame group may include an additional key frame that overlaps a previous key frame within the second key depth frame group, and when the additional key frame overlaps one of a plurality of key frames constituting the first key depth frame group, the second key depth frame group may be reflected in the generation of the 3D intraoral model.

[0024] In the step of acquiring the depth data, the key depth frame group acquired is displayed in real time, and the first key depth frame group and the second key depth frame group are displayed spaced apart from each other, and when the second key depth frame group is reflected in the generation of the 3D oral model, the first key depth frame group and the second key depth frame group can be displayed so as to be connected.

[0025] The first key frame is generated by merging at least one low depth frame of a first number, and the second key frame of the second key depth frame group is generated by merging a second number of low depth frames, and the first number and the second number may be different from each other.

[0026] The first number is a predetermined number, and the second number can be determined based on the degree of overlap of the low depth frames.

[0027] The low depth frame constituting the second key frame is composed of at least one frame between the first key frame and the target frame, and the target frame can be selected as a low depth frame having a difference from the first key frame that is greater than or equal to a preset level.

[0028] The above exemplary embodiments and other exemplary embodiments will be explained or clarified by the detailed description set forth below of exemplary embodiments to be read in conjunction with the accompanying drawings.

[0029] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment must include all or only the following effects, and thus the scope of the disclosed technology should not be construed as being limited thereby.

[0030] According to one embodiment of the present disclosure, the alignment accuracy of objects with high importance is preferentially improved, so that important objects can be expressed more precisely in the final 3D model.

[0031] Additionally, in a resource-constrained environment, the weight-based alignment method can improve data processing efficiency and optimize the quality and computational efficiency of the 3D model generation process.

[0032] The above is not intended to be an exhaustive list of all aspects of the present disclosure. It should be understood that the present disclosure encompasses all methods, devices, and systems capable of being implemented from all appropriate combinations of the various aspects disclosed in the detailed description and claims below, as well as the summaries herein. Furthermore, the detailed description of the embodiments of the present disclosure may directly or implicitly disclose any benefits that may be achieved or anticipated by the embodiments of the present disclosure. For example, various anticipated benefits resulting from the embodiments of the present disclosure will be disclosed in the detailed description that follows.

[0033] Aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0034] FIG. 1 is a schematic diagram illustrating an oral scanning system according to one embodiment of the present disclosure.

[0035] FIG. 2 is a block diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0036] FIG. 3 is a flowchart illustrating a method for obtaining a 3D model according to one embodiment of the present disclosure.

[0037] FIG. 4 is a flowchart illustrating a depth frame alignment method according to one embodiment of the present disclosure.

[0038] FIG. 5 is a schematic diagram illustrating a depth frame alignment method according to one embodiment of the present disclosure.

[0039] FIG. 6 is a flowchart illustrating a method for obtaining a 3D oral model according to one embodiment of the present disclosure.

[0040] FIG. 7 is a diagram for explaining a method for obtaining a key depth frame according to one embodiment of the present disclosure.

[0041] FIG. 8 is a flowchart illustrating a method for obtaining a 3D oral model according to one embodiment of the present disclosure.

[0042] FIG. 9 is a schematic diagram illustrating a key depth frame group according to one embodiment of the present disclosure.

[0043] FIG. 10 is a block diagram showing the configuration of an oral scanning system according to one embodiment of the present disclosure.

[0044] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0045] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on their meanings and the overall content of this disclosure.

[0046] The embodiments of the present disclosure are capable of various modifications and multiple embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the scope of the disclosed concepts and techniques. In describing the embodiments, detailed descriptions of related known technologies will be omitted if they are deemed to obscure the main point.

[0047] Terms such as "first" and "second" may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0048] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description of the present disclosure, and similar parts are designated with similar reference numerals throughout the specification.

[0050] FIG. 1 is a schematic diagram illustrating an oral scanning system according to one embodiment of the present disclosure.

[0051] Referring to FIG. 1, an oral scanning system (1000) may include an oral scanner (100) and an electronic device (200). A user (1) may scan the inside of a patient's (2) oral cavity using the oral scanner (100). The user (1) may be a dental staff member. For example, dental staff members may include dentists, dental hygienists, dental assistants, orthodontists, dental technicians, and oral surgeons.

[0052] The oral scanner (100) can transmit scan data acquired during the scanning process to an electronic device (200). The electronic device (200) can generate a 3D model representing the inside of the oral cavity based on the scan data. In addition, the electronic device (200) can display the 3D model on the display. The user (1) can continue the scanning process while checking the 3D model displayed on the electronic device (200). The scanning target of the oral scanner (100) is not limited to the inside of the patient's (2) oral cavity, and may also include structures that are scheduled to be installed or applied to the inside of the oral cavity in the future or are intermediate results therefor, even if they are not currently located in the oral cavity.

[0053] The electronic device (200) can be connected to the oral scanner (100) via a wired or wireless connection to exchange data. The electronic device (200) may be a desktop, laptop, tablet PC, or a computer designed as an embedded system specifically for the purpose of using the oral scanner (100). The electronic device (200) may be implemented by being physically or functionally integrated with the oral scanner (100).

[0054] FIG. 2 is a block diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0055] Referring to FIG. 2, the electronic device (200) can obtain scan data (21). Specifically, the electronic device (200) can receive scan data (21) from an oral scanner (100).

[0056] Scan data (21) refers to raw data acquired by the oral scanner (100) during the oral scanning process. For example, the scan data (21) may include an RGB image photographing the inside of the oral cavity, an infrared image, an intensity image indicating the reflectivity of the tooth surface, a polarization image, etc. The scan data (21) may be a monocular image captured by a single camera, or a pair of stereo images captured from different angles. In addition, the scan data (21) may include a structured light pattern projected onto the tooth surface.

[0057] Scan data (21) is composed of a plurality of scan frames acquired in a time series manner. Here, the scan frame may be a unit data including scan information of the inside of the oral cavity captured by the oral scanner (100) at a specific point in time, and may be composed of one of the forms of the scan data (21) described above.

[0058] Depth data (22) refers to data containing depth information about the inside of the oral cavity. For example, the depth data (22) may take the form of a depth map that stores distance information to teeth and surrounding tissues in a two-dimensional array format, a point cloud composed of three-dimensional coordinates of teeth and surrounding tissues, a triangle mesh of the tooth surface, etc.

[0059] Depth data (22) is composed of multiple depth frames acquired in a time-series manner. Here, a depth frame may be unit data including depth information in a three-dimensional space acquired at a specific point in time, and may be composed of one of the forms of the depth data (22) described above.

[0060] Depth data (22) can be acquired in various ways. In one embodiment, the electronic device (200) can derive depth data (22) based on scan data (21). For example, if the scan data (21) is a stereo image, the electronic device (200) can analyze the disparity between the stereo images to generate a depth map. Additionally, the electronic device (200) can acquire a point cloud based on the internal parameters of the oral scanner (100) and the depth map.

[0061] In another embodiment, depth data (22) may be acquired by an oral scanner (100). For example, the oral scanner (100) may directly acquire depth data (22) using a Time-of-Flight (TOF) sensor or an active stereo method. That is, if the scan data (21) already includes three-dimensional shape information, the scan data (21) itself may be utilized as depth data (22).

[0062] The electronic device (200) can reconstruct an intraoral model (23) based on the depth data (22). Specifically, the electronic device (200) can register a plurality of depth frames included in the depth data (22). For example, the electronic device (200) can perform local registration between time-series adjacent depth frames. Here, each depth frame can be a 3D point cloud. The electronic device (200) can generate a transformation matrix including rotation and translation components for registering two depth frames through registration.

[0063] The electronic device (200) can obtain pose information of the oral scanner (100) based on a transformation matrix. The pose information can include position and posture information of the oral scanner (100). The electronic device (200) can calculate a relative pose change of the oral scanner (100) based on the transformation matrix. The electronic device (200) can use this pose information to align 3D point clouds of each depth frame to create an intraoral model.

[0064] In one embodiment, the electronic device (200) can generate a pose graph in which the pose information of the oral scanner (100) at the time of acquisition of each depth frame is used as nodes and the relative transformation relationship between the pose information is used as edges. The electronic device (200) can perform pose graph optimization to globally optimize the pose information of each node. The electronic device (200) can generate a more precise intraoral model by realigning the 3D point cloud of each depth frame using the optimized pose information. In addition, when the oral scanner (100) rescans an area that has been previously scanned, the electronic device (200) can detect a loop closure and add it as a new edge of the pose graph, thereby effectively correcting accumulated drift errors.

[0065] The intraoral model (23) refers to a three-dimensional model that digitally represents the oral structure of a patient. The intraoral model (23) may include a maxillary model, a mandibular model, and an occlusal model. The maxillary model is a digital model representing the three-dimensional shape of the patient's maxillary arch. The mandibular model is a digital model representing the three-dimensional shape of the patient's mandibular arch. The occlusal model is a digital model representing the three-dimensional shape of the patient's maxillary model and mandibular model in an occluded state. The occlusal model reflects the patient's actual occlusal state, which may include the contact relationship and occlusal gap between the maxillary and mandibular teeth.

[0066] The type of intraoral model (23) may vary depending on the scan mode. The electronic device (200) may operate in multiple scan modes. The multiple scan modes may include an upper jaw scan mode, a lower jaw scan mode, and an occlusion scan mode.

[0067] The maxillary scan mode is a scanning mode for scanning the maxilla. In the maxillary scan mode, the user (1) can scan the maxilla using the oral scanner (100). The oral scanner (100) can collect scan data for the maxilla. The electronic device (200) can receive scan data for the maxilla from the oral scanner (100) and create a maxillary model.

[0068] The mandibular scan mode is a scan mode for scanning the mandible. In the mandibular scan mode, the user (1) can scan the mandible using the oral scanner (100). The oral scanner (100) can collect scan data for the mandible. The electronic device (200) can receive scan data for the mandible from the oral scanner (100) and create a mandibular model.

[0069] The occlusion scan mode is a scan mode for scanning occlusion. In the occlusion scan mode, the user (1) can scan the upper and lower jaws in occlusion using the oral scanner (100). The oral scanner (100) can collect scan data on occlusion. The electronic device (200) can receive scan data on occlusion from the oral scanner (100) and create an occlusion model.

[0070] Meanwhile, in the present disclosure, a "scan session" may refer to a period of time during which a scanning operation is performed in a specific scan mode. For example, a scan session in the maxillary scan mode may be the period from the time a user (1) begins scanning the maxilla using an oral scanner (100) to the time the maxillary scan is completed.

[0071] FIG. 3 is a flowchart illustrating a method for obtaining a 3D model according to an embodiment of the present disclosure. FIG. 4 is a flowchart illustrating a depth frame matching method according to an embodiment of the present disclosure. FIG. 5 is a schematic diagram illustrating a depth frame matching method according to an embodiment of the present disclosure. Hereinafter, a method for obtaining a 3D model according to an embodiment of the present disclosure will be described with reference to FIGS. 3 to 5.

[0072] Referring to FIG. 3, the electronic device (200) can acquire depth data including multiple depth frames for the patient's oral cavity (S210). The depth data can include multiple depth frames. The depth frames can include depth information for the oral cavity. For example, the depth frames can be a depth map, a point cloud, or RGB-D data.

[0073] The electronic device (200) can obtain depth data using an oral scanner. For example, the electronic device (200) can generate depth data based on scan data received from the oral scanner. Alternatively, the electronic device (200) can receive depth data from the oral scanner.

[0074] The electronic device (200) can align a plurality of depth frames included in depth data (S220). Referring to FIG. 4, the step of aligning a plurality of depth frames (S220) may include a step of identifying corresponding data points among a plurality of data points included in each of the plurality of depth frames as data point pairs (S221), a step of obtaining transformation information between the plurality of depth frames based on the data point pairs (S222), a step of aligning the plurality of depth frames using the transformation information (S223), and a step of evaluating a matching error between the plurality of depth frames (S224). Hereinafter, each step will be described in detail with reference to FIG. 5.

[0075] The electronic device (200) can identify corresponding data points among a plurality of data points included in each of a plurality of depth frames as a data point pair (S221). For each data point included in one depth frame, the electronic device (200) can find the data point located at the closest distance among the data points included in another depth frame, identify it as a corresponding point, and identify the two data points thus identified as one data point pair. Here, the corresponding data points can be understood as corresponding to the same specific point inside the oral cavity.

[0076] As an example, referring to FIG. 5, the depth data may include a first depth frame (41) and a second depth frame (42). The first depth frame (41) may include a first data point (P1), a second data point (P2), and a third data point (P3), and the second depth frame (42) may include a fourth data point (P4), a fifth data point (P5), and a sixth data point (P6). At this time, the electronic device (200) may find a data point located at a closest distance in the second depth frame (42) for each data point of the first depth frame (41) and identify it as a corresponding point. For example, the electronic device (200) can identify that the first data point (P1) and the fourth data point (P4) correspond to each other, the second data point (P2) and the fifth data point (P5) correspond to each other, and the third data point (P3) and the sixth data point (P6) correspond to each other.

[0077] The electronic device (200) can obtain transformation information between a plurality of depth frames based on data point pairs (S222). Here, the transformation information may mean transformation parameters (or transformation matrix) including rotation parameters and translation parameters. The rotation parameters are values ​​that define rotation in a three-dimensional space, and can generally be expressed in the form of three rotation angles (rotation around the x, y, and z axes), a rotation matrix, or a quaternion. The translation parameters are values ​​that define movement in a three-dimensional space, and can be composed of three values ​​representing the amount of movement in the x, y, and z directions.

[0078] The electronic device (200) can calculate optimal rotation and translation transformation parameters for aligning one depth frame to another depth frame based on the identified data point pairs. In this process, the electronic device (200) can derive a transformation matrix that minimizes the distance difference between corresponding data points. In one embodiment, referring to FIG. 5, the electronic device (200) can calculate optimal rotation and translation transformation parameters for aligning the second depth frame (42) to the first depth frame (41) based on the identified data point pairs (P1-P4, P2-P5, P3-P6).

[0079] Here, the step of obtaining transformation information (S222) may include a process of estimating an initial transformation between multiple depth frames. The initial transformation may be estimated by utilizing external sensor information or performing feature-based matching. Specifically, the electronic device (200) may estimate the initial transformation between two consecutive depth frames using odometry information acquired through a sensor provided in the oral scanner (100). For example, the electronic device (200) may use information on the distance and angle by which the oral scanner (100) moved between the immediately previous depth frame and the current depth frame as the initial transformation value.

[0080] Alternatively, the electronic device (200) can estimate the initial transformation through feature-based matching. In one embodiment, a 3D feature point matching method can be used. Specifically, the electronic device (200) can obtain a global initial transformation by extracting 3D feature points (e.g., FPFH, SHOT, ISS, SIFT-3D, etc.) from each depth frame, matching descriptors of the extracted feature points, and then applying the RANSAC algorithm to remove outliers.

[0081] In another embodiment, the electronic device (200) may use a 2D image feature matching method. Specifically, the electronic device (200) may extract and match feature points (e.g., SIFT, ORB, etc.) from a 2D image acquired through an image sensor of the oral scanner (100), and then convert the extracted feature points into 3D coordinates using depth values ​​corresponding to the matched feature points, thereby calculating an initial transformation.

[0082] The electronic device (200) can align a plurality of depth frames using the conversion information (S223). In this process, the electronic device (200) can improve the overall consistency by spatially aligning data points between different depth frames based on the acquired conversion information. This alignment process can be performed in a direction that minimizes the distance between corresponding data points. As an example, referring to FIG. 5, the electronic device (200) can apply the conversion parameter to all data points (P4, P5, P6) of the second depth frame (42). Accordingly, the distance between the first data point (P1) and the fourth data point (P4), the distance between the second data point (P2) and the fifth data point (P5), and the distance between the third data point (P3) and the sixth data point (P6) can be reduced overall.

[0083] The electronic device (200) can evaluate the alignment error between multiple depth frames (S224). Here, the alignment error may refer to a value quantitatively representing the degree to which different depth frames do not match even after applying transformation information. The alignment error may be calculated based on the distance between data points identified as corresponding to each other in two different depth frames. For example, the electronic device (200) can calculate the Euclidean distance corresponding to each pair of multiple data points and quantify the alignment error by calculating the sum or average (Mean Squared Error) of the squared Euclidean distances.

[0084] As an example, referring to FIG. 5, the electronic device (200) can calculate a first distance between a first data point (P1) and a fourth data point (P4), a second distance between a second data point (P2) and a fifth data point (P5), and a third distance between a third data point (P3) and a sixth data point (P6) after applying the transformation information. In addition, the electronic device (200) can calculate a matching error through the sum of the squares of the first distance, the second distance, and the third distance.

[0085] The electronic device (200) can repeatedly perform the above steps (S221, S222, S223, S224) until the alignment error becomes below a threshold value. Such operations can be performed based on an iterative closest point (ICP) algorithm.

[0086] Meanwhile, when performing at least one of steps S221, S222, and S224, the electronic device (200) may apply a weight determined according to the type of object.

[0087] In one embodiment, when applying weights in S224, the electronic device (200) may reflect weights according to the type of intraoral object when evaluating the alignment error between multiple depth frames. For example, the electronic device (200) may calculate a weighted error by multiplying the error of each data point pair by the weight of the corresponding object type. Here, the error of each data point pair may mean the distance difference between the two corresponding data points after applying the transformation information. The electronic device (200) may calculate the average of these weighted errors to evaluate the overall alignment quality. In this case, since the error in the area with a high weight has a greater proportion in the overall alignment quality evaluation, it is possible to perform an evaluation that prioritizes the alignment accuracy of clinically important structures.

[0088] Mathematical expression 1 is an example of a cost function when weights are applied in S224.

[0089]

[0090] Here, is the weighted matching error, is the weight according to the type of object, R is the rotation matrix, and t is the translation vector. and are data points that correspond to each other. is an individual source data point belonging to the set of source data points (P), is an individual target data point belonging to the set of target data points (Q). The transformation matrix (R, t) is used to transform the source data points (P) so that the set of source data points (P) matches the set of target data points (Q). ) can be applied.

[0091] The electronic device (200) can repeatedly perform S221, S222, S223 and S224 until the alignment error becomes less than a threshold value. In this process, the weight ( ) has a greater influence on the matching result, so matching can be performed with priority given to objects with high weights.

[0092] In another embodiment, when applying weights in S221, the electronic device (200) may calculate a weighted distance based on the type of object when calculating the distance between two data points. For example, the electronic device (200) may calculate the weighted distance by dividing the actual distance between the two data points by the weight. Then, the electronic device (200) may identify the two data points at the closest distance as a data point pair based on the weighted distance. In this case, since the weighted distance decreases as the weight increases, the data point with the higher weight is more likely to be selected as a data point pair. This allows data points corresponding to objects with relatively higher weights to be given more priority in the matching process, thereby obtaining accurate matching results.

[0093] In another embodiment, when applying weights in S222, the electronic device (200) may consider weights according to the type of intraoral object when acquiring transformation information based on data point pairs. For example, when deriving a transformation matrix that minimizes the distance difference between corresponding data points, the electronic device (200) may apply different weights according to the type of object to which each data point pair belongs. In this case, a high weight may be assigned to a data point pair belonging to a hard tissue, such as a tooth, and a low weight may be assigned to a data point pair belonging to a soft tissue, such as the gums. This weighting has the effect of controlling the influence of each data point pair when calculating the transformation parameters. Specifically, the alignment error of a data point pair assigned a high weight will account for a larger proportion of the overall error function, so that the transformation parameters can be calculated in a way that preferentially reduces the alignment error of the corresponding data point pair during the transformation parameter optimization process. Therefore, by assigning a high weight to a data point pair belonging to an important structure (e.g., a tooth), the alignment accuracy of the corresponding structure can be preferentially considered when calculating the transformation parameters.

[0094] Objects can include static objects and dynamic objects. Dynamic objects refer to objects whose position or shape can change during the oral scanning process. For example, dynamic objects can be classified as follows according to their characteristics. First, there are soft-tissue dynamic objects that can both change position and change shape, such as the tongue, cheeks, and lips. These soft-tissue dynamic objects can move spontaneously during the scanning process and their shape can be changed by external pressure. Second, there are rigid-tissue dynamic objects that can change position but not change shape, such as dental mirrors, dental mirrors, and oral tools, or the practitioner's fingers. Rigid-tissue dynamic objects maintain their shape but can freely change position during the scanning process.

[0095] The characteristics of these dynamic objects are an important consideration during the depth data alignment process. Assigning high weights to data from dynamic objects that undergo positional changes or shape deformations can increase alignment errors. Therefore, the present disclosure applies relatively low weights to dynamic objects to improve alignment accuracy.

[0096] Additionally, different weights may be applied depending on the type of static object. For example, the weights applied to teeth, prostheses, scan bodies, and abutments among static objects may be greater than the weight applied to gingiva. Here, the scan body may refer to a prosthetic component used to accurately determine the position and direction of the implant placement through 3D scanning. In addition, the prosthesis refers to an artificial replacement for a tooth or related tissue, such as inlay or onlay treatment, and may include, for example, ceramic, resin, or zirconia.

[0097] In particular, the weight for objects that are teeth may be greater than for objects that are gingiva. Therefore, the accuracy of tooth registration may be improved more preferentially than that of gingiva. This is because the 3D structure of teeth in a 3D model representing the oral cavity directly affects precise dental tasks such as prosthesis design, occlusion analysis, or implant simulation. Registration accuracy can refer to a value that quantitatively evaluates the geometric consistency between two depth frames. Registration accuracy can indicate how well two depth frames spatially match.

[0098] Additionally, different weights may be applied depending on the tooth type. For example, the weight for an object that is a treatment target tooth may be greater than for an object that is not a treatment target tooth. Therefore, the alignment accuracy of the treatment target tooth may be improved preferentially over that of the non-treatment target tooth. A treatment target tooth refers to a tooth currently undergoing treatment, and may include, for example, a prep tooth. A non-treatment target tooth refers to a tooth that is not currently a treatment target, and may include a tooth that has completed treatment in the past or a tooth that has never been designated as a treatment target.

[0099] For example, the teeth to be treated may be pre-designated by the user. If the teeth to be treated are pre-designated by the user using a tooth type number or the like, the electronic device (200) can identify the teeth corresponding to the designated tooth type number as the teeth to be treated through analysis of the scan frame acquired during the scanning process or the constructed 3D model.

[0100] As another example, a tooth to be treated can be automatically identified by the electronic device (200). Even without information previously specified by the user, the electronic device (200) can identify the determined tooth as a tooth to be treated by determining whether preparation is required and whether a lesion exists through analysis of the scan frame acquired during the scanning process or the constructed 3D model.

[0101] When a tooth to be treated is identified, the electronic device (200) can provide visual feedback regarding the tooth to be treated identified on the 3D model. The electronic device (200) can provide different visual feedbacks depending on the reliability of the tooth to be treated, i.e., the degree of estimation as to whether the tooth is to be treated.

[0102] The visual feedback may include a highlighter that emphasizes the outline of the identified treatment target tooth with a specific color, an indicator that indicates that the tooth corresponds to the treatment target tooth, and the like, and may also include various other methods that allow the user to recognize that the tooth corresponds to the treatment target tooth. The visual feedback may be provided in real time during a scan session that generates a 3D model, and may also be provided in a 3D model displayed on a display during a standby session when scanning is completed or paused. In addition, visual feedback may be provided in a dental chart displayed on a display during an order generation session that generates an order to indicate that the tooth corresponds to the treatment target tooth. Meanwhile, although the present embodiment has been described with a focus on a treatment target tooth identified by an electronic device, the above description may of course be applied to various identification targets identified by an electronic device.

[0103] In one embodiment, referring to FIG. 5, a first data point (P1) and a fourth data point (P4) may correspond to a normal tooth, a second data point (P2) and a fifth data point (P5) may correspond to a prep tooth, and a third data point (P3) and a sixth data point (P6) may correspond to a gingiva. The normal tooth may be a non-target tooth for treatment, and the prep tooth may be a target tooth for treatment. At this time, the electronic device (200) may apply a first weight to a first data point pair composed of the first data point (P1) and the fourth data point (P4), a second weight to a second data point pair composed of the second data point (P2) and the fifth data point (P5), and a third weight to a third data point pair composed of the third data point (P3) and the sixth data point (P6). Here, each of the first weight and the second weight may be greater than the third weight. That is, teeth can be aligned with priority over the gingiva. Furthermore, the second weighting may be greater than the first weighting. This means that teeth being treated can be aligned with priority over teeth not being treated.

[0104] Meanwhile, the electronic device (200) can identify the target area for treatment through various analysis methods.

[0105] In one embodiment, the electronic device (200) may apply various methods to identify a carious (cavity) area. For example, the electronic device (200) may use color or reflectivity information provided by the oral scanner (100) to detect areas with reduced discoloration (brown, black) or glossiness or transparency compared to normal teeth (enamel). At this time, the electronic device (200) may detect color tone changes based on a specific threshold value or utilize a deep learning classification model. In addition, to detect areas where caries has progressed and the tooth surface has locally sunken (cavity), the electronic device (200) may analyze curvature outliers or groove depths in a 3D point cloud. In particular, the electronic device (200) may utilize color and curvature information together to distinguish them from occlusal surface fossa. In addition, the electronic device (200) can automatically segment the caries area by comprehensively analyzing features such as color, reflectivity, curvature, and texture through a CNN model learned using training data with caries labels.

[0106] In another embodiment, the electronic device (200) may also apply a method for identifying a fracture (crack) area. For example, the electronic device (200) may detect a linear discontinuity by searching for adjacent points with a large difference in normals or edge areas with a sharp change in curvature in a 3D point cloud. In addition, the electronic device (200) may analyze differences in light reflection, shadows, reflection distortion, etc. occurring in a thin crack to detect speckle abnormalities or deviations in reflection intensity. In particular, in the case of microcracks (fracture lines), the electronic device (200) may automatically detect them by combining a high-resolution scanner and AI-based post-processing.

[0107] In another embodiment, the electronic device (200) can analyze various characteristics to identify the boundary of the prosthesis. For example, since a temporary prosthesis made of metal or resin has different color, gloss, and transparency than a natural tooth, the electronic device (200) can recognize the boundary through the differences in color spectrum and reflectivity. In addition, the electronic device (200) can detect the edge by detecting changes in curvature or normal line due to a minute step that occurs at the connection between the prosthesis and the tooth. In addition, the electronic device (200) can estimate the boundary through a model that learns material classification and shape change together by utilizing learning data including the prosthesis boundary label.

[0108] In another embodiment, the electronic device (200) may also apply a method for identifying the surroundings of an implant abutment. For example, the electronic device (200) may identify the implant abutment in a 3D point cloud based on its characteristic shape (cone, hexagon, etc.) and the reflective properties of its metal material. In addition, the electronic device (200) may analyze the color and reflectivity differences unique to implant materials such as titanium or zirconia. In addition, the electronic device (200) may implement an automatic / semi-automatic system that identifies the position and angle of an implant abutment through a model that has learned the shape of a specific manufacturer's abutment, and inspects the fit of the bone and the prosthesis boundary, etc.

[0109] Meanwhile, the electronic device (200) can differentially apply weights in the alignment process by considering the structural features and curvature of the teeth. Specifically, the electronic device (200) can perform 3D transformation (Back-projection) by backprojecting the pixels (x, y) and depth (z) of the depth map provided by the oral scanner (100) into 3D coordinates (X, Y, Z) according to the camera model. The electronic device (200) can configure this as a point cloud or form a 3D surface by local triangulation.

[0110] In one embodiment, the electronic device (200) may apply differential weights to various characteristic structures of the occlusal surface. For example, the electronic device (200) may apply a high weight to a point where the curvature changes sharply, which is the highest protrusion compared to the surrounding surface among the multiple cusps existing on the occlusal surface of molars and premolars, and is called a cusp tip. In addition, the electronic device (200) may also apply a high weight to a part where the surface is sharply inclined up and down along the edge in the form of an elongated protrusion, such as a ridge extending between cusps or from a cusp to the center of a tooth, and a point where the curvature difference is noticeable because one side is upward and the other side is downward, such as a groove, fissure, or fossa on the occlusal surface.

[0111] In another embodiment, the electronic device (200) may also assign differential weights to characteristic structures of the cross-section area. For example, the electronic device (200) may apply high weights to areas where the surface slope changes abruptly in one or both directions in the form of thin and sharp lines, such as the incisor edge of a canine tooth, or areas where the teeth protrude like pillars and have a large curvature, such as the cusp of a canine tooth.

[0112] In another embodiment, the electronic device (200) may differentially apply weights to characteristic structures on the side of a tooth. For example, the electronic device (200) may assign a high weight to a protrusion that serves as a boundary line where the side of a tooth meets the occlusal surface, such as a marginal ridge, which is a marginal (border) ridge of a tooth, or a locally protruding area, such as a boundary of a prosthesis (such as a metal crown) on the side of a tooth (buccal surface, lingual surface), or a case where a tooth is partially fractured.

[0113] In another embodiment, the electronic device (200) may also apply a high weight to areas where there is local deformation due to tooth fracture or caries. For example, the electronic device (200) may also apply a high weight to areas where the surface shape shows an abrupt slope, such as a crack line where a tooth is partially broken or cracked, or areas where the difference between the groove and the slope is large in areas where the tooth surface is corroded and pitted, such as around areas where caries has progressed and become sunken.

[0114] Referring back to FIG. 3, the electronic device (200) can obtain pose information of the oral scanner based on alignment of a plurality of depth frames (S230). The pose information can include information on the position and orientation of the oral scanner (100). The electronic device (200) can obtain pose information based on transformation information used to transform two different depth frames to be as close as possible and internal parameters of the oral scanner (100). The internal parameters of the oral scanner can include optical parameters (e.g., focal length, field of view, pixel size).

[0115] The electronic device (200) can generate a 3D model representing the oral cavity based on depth data and pose information (S240). The electronic device (200) can associate each depth frame with pose information. The electronic device (200) can use the pose information to align multiple depth frames in a single coordinate system to generate a 3D model.

[0116] As described above, when weights are applied in the depth data alignment step (S220), objects with higher weights can be represented with relatively higher accuracy in the 3D model. Therefore, teeth can be represented more accurately than gums, or teeth subject to treatment can be represented more accurately than non-target teeth.

[0117] Meanwhile, the type of object can be acquired using a neural network model trained to identify the type of object inside the oral cavity. For example, the neural network model can be trained to perform a segmentation task. The electronic device (200) can identify the type of object by inputting depth data or scan data corresponding to the depth data into the neural network model. In one embodiment, the neural network model can output an identification result for the type of object inside the oral cavity and a confidence value for the identification result. At this time, the electronic device (200) can determine a weight corresponding to the object based on the confidence value. For example, even if the type of object is the same, a higher weight can be assigned if the confidence is higher.

[0118] In another embodiment, the electronic device (200) may perform segmentation using a 2D image acquired through an image sensor included in the oral scanner (100) and apply weights to depth data (e.g., point cloud or RGB-D data) corresponding to the segmented area. In another embodiment, the electronic device (200) may perform segmentation in a post-processing step after the 3D model creation is completed and apply weights based thereon to optimize the 3D model.

[0119] Meanwhile, the above-described alignment method can be performed only on a select number of depth frames, rather than all depth frames, and 3D models can also be generated based on these selected depth frames. This can reduce overall computational resources. Here, the selected depth frames can refer to key depth frames, and key depth frames will be described below.

[0120]

[0121] *110 is a flowchart illustrating a method for obtaining a 3D oral model according to one embodiment of the present disclosure.

[0122] Referring to FIG. 6, the electronic device (200) can acquire a plurality of raw depth frames containing depth information about the inside of the patient's oral cavity (S510). The electronic device (200) can generate raw depth frames based on scan data received from the oral scanner (100) or receive raw depth frames from the oral scanner (100). Here, the raw depth frames can be depth maps.

[0123] The electronic device (200) can obtain a key depth frame based on a plurality of low depth frames (S520). The electronic device (200) can merge a predetermined number (e.g., 10) of low depth frames to generate a key depth frame. At this time, different weights can be applied to each low depth frame according to quality. For example, a high weight can be applied to a low depth frame with high quality, and a low weight can be applied to a low depth frame with low quality. Here, the quality can be determined based on the accuracy, resolution, or noise level of the depth information included in each frame. In addition, the electronic device (200) can obtain a key depth frame by merging a predetermined number of low depth frames and then excluding at least one of them with a quality lower than a predetermined standard from the merging.

[0124] Meanwhile, the method for acquiring the first key depth frame and the method for acquiring the remaining subsequent key depth frames may differ. For example, the first key depth frame may be acquired by merging a predetermined number of low depth frames as described above. Alternatively, the first low depth frame may be selected as the first key depth frame.

[0125] On the other hand, the remaining subsequent key depth frames, i.e., non-initial key depth frames, can be acquired in a different manner. FIG. 7 is a diagram illustrating a method for acquiring a key depth frame according to an embodiment of the present disclosure, and a method for acquiring subsequent key depth frames will be described with reference thereto.

[0126] Referring to FIG. 7, the first frame (61) may be the first key depth frame, the second frame (62), the third frame (63), the fourth frame (64), and the fifth frame (65) may be subsequent low depth frames, and the sixth frame (66) may be the second key depth frame.

[0127] After acquiring the first frame (61), the electronic device (200) can identify a frame among the subsequent frames (62, 63, 64, 65) whose degree of overlap with the first frame (61) is less than or equal to a predetermined standard. The frame thus identified may be referred to as a reference frame or a target frame. For example, the electronic device (200) can identify the fifth frame (65). Then, the electronic device (200) can merge the frames (62, 63, 64) between the first frame (61) and the fifth frame (65) and the fifth frame (65) to generate a sixth frame (66). At this time, a high weight may be assigned to a frame with high quality, and a low weight may be assigned to a frame with low quality. For example, if the second frame (62) has higher quality than the third frame (63), the second frame (62) may be assigned a higher weight than the third frame (63).

[0128] In this way, by generating a key depth frame when a low depth frame whose degree of overlap with a previous key depth frame is less than or equal to a predetermined standard is identified, excessive duplication of depth information between key depth frames can be prevented. In addition, the total number of key depth frames can be controlled to efficiently manage the entire resources. In another embodiment, even if a low depth frame whose degree of overlap with a previous key depth frame is less than or equal to a predetermined standard is not identified, if a predetermined number of low depth frames are acquired, the electronic device (200) can generate a new key depth frame. For example, if there is no low depth frame whose degree of overlap with a previous key depth frame is less than or equal to a predetermined standard during a predetermined N frames, the electronic device (200) can generate a new key depth frame every Nth frame. Here, N is an integer and can be reset to 0 each time a new key depth frame is acquired.

[0129] The degree of overlap between two frames can be determined based on the position and orientation of the two frames and the degree of overlap between the depth information contained in the two frames. Specifically, the greater the difference in the position and orientation of the two frames, the smaller the degree of overlap. Furthermore, the smaller the degree of overlap between the depth information contained in the two frames, the smaller the degree of overlap between the two frames. The position and orientation of the frames can be determined based on the pose information of the oral scanner (100) corresponding to the corresponding frame.

[0130] For example, the electronic device (200) can determine the degree of overlap between the first frame (61) and the second frame (62). At this time, the electronic device (200) can determine whether the positional difference between the first frame (61) and the second frame (62) is greater than or equal to a predetermined value. In addition, the electronic device (200) can determine whether the directional difference between the first frame (61) and the second frame (62) is greater than or equal to a predetermined value. The electronic device (200) can determine whether the difference between the depth information included in the first frame (61) and the second frame (62) is less than or equal to a predetermined value. Based on the positional and directional differences and the difference between the depth information, the electronic device (200) can determine the degree of overlap between the first frame (61) and the second frame (62) and determine whether the degree of overlap is less than or equal to a predetermined standard.

[0131] Meanwhile, when determining the degree of overlap between depth information between two frames, different weights may be applied depending on the object type. For example, when calculating the degree of overlap between depth information for teeth, a first weight may be applied, while when calculating the degree of overlap between depth information for gums, a second weight, smaller than the first weight, may be applied. Accordingly, the degree of overlap between depth information for teeth may be more significantly reflected in the calculation of the degree of overlap.

[0132] Referring again to FIG. 6, the electronic device (200) can generate a 3D oral cavity model based on the key depth frame (S530). The 3D oral cavity model is a 3D model representing the patient's oral cavity, and may include a maxillary model, a mandibular model, and an occlusion model.

[0133] Meanwhile, creating a 3D oral model using an intraoral scanner typically requires scan data covering the entire oral cavity. However, the limited field of view of an intraoral scanner makes it difficult to capture the entire oral cavity with a single scan. Therefore, the user can scan the oral cavity in multiple segments by moving the intraoral scanner.

[0134] At this time, it is desirable for the continuously acquired scan data to include overlapping areas. In conventional oral scanner systems, if there is no overlapping area between continuously acquired scan data, the entire scanning process is treated as an error, requiring the scanning process to be restarted from the beginning.

[0135] In one embodiment of the present disclosure, the electronic device (200) can store depth frames that are not continuous among the depth frames acquired from the scan data by organizing them into different key depth frame groups. For example, a plurality of key depth frames acquired from scan data obtained by scanning the left part of the upper jaw can be organized into a first key depth frame group, and a plurality of key depth frames acquired from scan data obtained by scanning the right part of the upper jaw can be organized into a second key depth frame group. If an overlapping area occurs between the first key depth frame group and the second key depth frame group due to an additional scan by the user, the electronic device (200) can utilize this to integrate the depth frames of the two groups into a single 3D model. This method eliminates the inconvenience of the user having to perform the scan again from the beginning, and enables efficient utilization of already acquired scan data.

[0136] Below, we describe the key depth frame group and how the key depth frame group is reflected when creating a 3D oral model.

[0137] FIG. 8 is a flowchart illustrating a method for obtaining a 3D oral model according to one embodiment of the present disclosure.

[0138] Referring to FIG. 8, the electronic device (200) can obtain a first key depth frame group and a second key depth frame group composed of a plurality of key depth frames (S710). A key depth frame group may mean a set including a plurality of key depth frames.

[0139] The electronic device (200) can determine whether the first key depth frame group and the second key depth frame group overlap by a predetermined standard or more (S720). At this time, the electronic device (200) can determine whether the degree of overlap between the first key depth frame group and the second key depth frame group is greater than the predetermined standard. The degree of overlap between the first key depth frame group and the second key depth frame group can be calculated based on the degree of overlap between key depth frames included in each key depth frame group. For example, the higher the degree of overlap between individual key depth frames, the higher the degree of overlap between key depth frame groups can be.

[0140] When the first key depth frame group and the second key depth frame group overlap by a predetermined standard or more, that is, when the degree of overlap between the first key depth frame group and the second key depth frame group is a predetermined standard or more (S720: Yes), the electronic device (200) can generate a 3D oral model based on the first key depth frame group and the second key depth frame group (S730). At this time, the electronic device (200) can generate the 3D oral model using both the first key depth frame group and the second key depth frame group. In addition, the electronic device (200) can define the first key depth frame group and the second key depth frame group as one group.

[0141] If the first key depth frame group and the second key depth frame group overlap less than a predetermined standard, that is, if the degree of overlap between the first key depth frame group and the second key depth frame group is less than the predetermined standard (S720: No), the electronic device (200) may generate a 3D oral cavity model based on the first key depth frame group or the second key depth frame group (S740). At this time, the electronic device (200) may select one of the first key depth frame group and the second key depth frame group, and generate the 3D oral cavity model using only the selected group.

[0142] In one embodiment, the electronic device (200) may select a key depth frame group that includes a greater number of key depth frames. For example, if the number of key depth frames included in a first key depth frame group is greater than the number of key depth frames included in a second key depth frame group, the electronic device (200) may select the first key depth frame group. In this case, the electronic device (200) may generate a 3D oral cavity model using the first key depth frame group. Additionally, the electronic device (200) may delete a second key depth frame group that is not used to generate the 3D oral cavity model.

[0143] In another embodiment, the electronic device (200) may select a key depth frame group with better depth quality. For example, if the depth quality of the first key depth frame group is better than the depth quality of the second key depth frame group, the electronic device (200) may select the first key depth frame group.

[0144] Meanwhile, even if the degree of overlap between the first key depth frame group and the second key depth frame group is less than a predetermined standard (S720: No), the electronic device (200) can generate an independent 3D oral model corresponding to each key depth frame group using both key depth frame groups. For example, the electronic device (200) can generate a first 3D oral model using the first key depth frame group and a second 3D oral model using the second key depth frame group. In this case, the first 3D oral model and the second 3D oral model can represent independent areas that do not overlap each other.

[0145] FIG. 9 is a schematic diagram illustrating a key depth frame group according to one embodiment of the present disclosure.

[0146] Referring to FIG. 9, the electronic device (200) can obtain a first key depth frame group (G1) and a second key depth frame group (G2). The first key depth frame group (G1) can include a plurality of key depth frames, including a first frame (81), a second frame (82), a third frame (83), and a fourth frame (84). In addition, the second key depth frame group (G2) can include a plurality of key depth frames, including a fifth frame (85), a sixth frame (86), a seventh frame (87), and an eighth frame (88).

[0147] In order to be included in a key depth frame group, the degree of overlap with at least one frame included in the key depth frame group may be greater than or equal to a predetermined standard. Accordingly, a frame included in a key depth frame group may have a degree of overlap with at least one frame included in the key depth frame group greater than or equal to a predetermined standard. For example, a fourth frame (84) may have a degree of overlap with at least one of the first frame (81), the second frame (82), and the third frame (83) greater than or equal to a predetermined standard.

[0148] If a specific frame cannot be included in any existing key depth frame group, the frame may be defined as the first key depth frame of a new key depth frame group. For example, a fifth frame (85) may be acquired after a first key depth frame group (G1) is acquired. The fifth frame (85) may have an overlap degree with all frames included in the first key depth frame group (G1) that is less than a predetermined standard. In this case, the electronic device (200) may define the fifth frame (85) as the first frame of a second key depth frame group (G2) that is separate from the first key depth frame group (G1).

[0149] While the user is performing a scan operation, the electronic device (200) can update the key depth frame group in real time. When a new key depth frame is acquired, the electronic device (200) can determine the degree of overlap between the key depth frame and frames included in another key depth frame group already defined. Based on the degree of overlap, the electronic device (200) can include the new key depth frame in the already defined key depth frame group or define it as the first frame of the new key depth frame group.

[0150] Meanwhile, a single key depth frame may be associated with multiple key depth frame groups. For example, the eighth frame (88) may be included in the second key depth frame group (G2), and the degree of overlap with the first frame (81) included in the first key depth frame group (G1) may be greater than a predetermined standard. In this case, the first key depth frame group (G1) and the second key depth frame group (G2) may be connected via the eighth frame (88) and defined as a single group.

[0151] In this way, when two different key depth frame groups are connected, that is, when they are defined as one key depth frame group, 3D oral models corresponding to each key depth frame group can also be displayed so as to be connected. For example, the electronic device (200) can display a first 3D oral model generated based on a first key depth frame group (G1) and a second 3D model generated based on a second key depth frame group (G2). At this time, the first 3D model oral cavity and the second 3D oral cavity model can represent independent areas that do not overlap each other. Thereafter, when the eighth frame (88) included in both the first key depth frame group (G1) and the second key depth frame group (G2) is acquired, the electronic device (200) can display the first 3D model oral cavity and the second 3D oral cavity model so as to be connected to each other.

[0152] FIG. 10 is a block diagram showing the configuration of an oral scanning system according to one embodiment of the present disclosure.

[0153] Referring to FIG. 10, an oral scanning system (1000) may include an oral scanner (100) and an electronic device (200). The oral scanner (100) may acquire scan data regarding the inside of a patient's oral cavity and transmit the scan data to the electronic device (200). The oral scanner (100) may acquire scan data using structured light, Time-of-Flight (ToF), LiDAR, Computed Tomography (CT), or ultrasound. The oral scanner (100) may include an IMU sensor. IMU information acquired through the IMU sensor may be used to acquire pose information or movement speed of the oral scanner (100).

[0154] The electronic device (200) may include a communication interface (210), memory (220), processor (230), and display (240). The electronic device (200) may be a personal computer (PC), tablet, workstation, or smartphone.

[0155] The communication interface (210) may include at least one communication circuit. The communication interface (210) may receive scan data from the oral scanner (100). The communication interface (210) may include a wired interface and a wireless interface. The wired interface may include USB, Ethernet, HDMI, and Thunderbolt. The wireless interface may include Wi-Fi, Bluetooth, Zigbee, and NFC.

[0156] The memory (220) may store an operating system (OS) for controlling the overall operation of components of the electronic device (200) and commands or data related to components of the electronic device (200). In particular, the memory (220) may include instructions for controlling oral scanning software. The memory (220) may be implemented as a non-volatile memory (e.g., a hard disk, a solid state drive (SSD), a flash memory) or a volatile memory.

[0157] The processor (230) is electrically connected to the memory (220) and can control the overall functions and operations of the electronic device (200). The processor (230) can control the overall functions and operations of the electronic device (200) by executing instructions stored in the memory (220). In addition, the processor (230) can control the display (240) to display various screens. For example, the processor (230) can control the display (240) to display a 3D oral model.

[0158] The processor (230) can obtain depth data including multiple depth frames for the inside of the patient's oral cavity using the oral scanner (100).

[0159] The processor (230) can align depth data. For example, the processor (230) can align depth data using an ICP algorithm.

[0160] The processor (230) can align pairs of corresponding data points contained in two different depth frames. At this time, a weight may be applied to the alignment of the data point pairs. The weight may be determined based on the type of object corresponding to the data point pair. For example, the weight when the object is a tooth may be greater than when the object is a gum. In another example, the weight when the object is a tooth to be treated may be greater than when the object is a non-target tooth.

[0161] The type of object can be obtained using a neural network model trained to identify the type of object inside the oral cavity. For example, the neural network model can output an identification result for the type of object inside the oral cavity and a confidence value for the identification result. At this time, a weight can be determined based on the confidence value. The type of object inside the oral cavity can be specified by the user. For example, the tooth to be treated can be specified in advance by the user. If the tooth to be treated is specified in advance by the user using a tooth type number or the like, the processor (230) can identify the tooth corresponding to the specified tooth type number as the tooth to be treated through analysis of the scan frame acquired during the scanning process or the constructed 3D model.

[0162] As another example, a tooth to be treated can be automatically identified by the processor (230). Even in the absence of information previously specified by the user, the processor (230) can identify the determined tooth as a tooth to be treated by determining whether preparation has been performed and whether a lesion exists through analysis of the scan frame acquired during the scanning process or the constructed 3D model.

[0163] When a tooth to be treated is identified, the processor (230) can provide visual feedback regarding the tooth to be treated identified on the 3D model. The processor (230) can provide different visual feedbacks depending on the reliability of the tooth to be treated, i.e., the degree of estimation as to whether the tooth is to be treated.

[0164] Visual feedback may include a highlighter that emphasizes the outline of an identified tooth to be treated with a specific color, an indicator that indicates that the tooth corresponds to the tooth to be treated, and various other methods that allow the user to recognize that the tooth corresponds to the tooth to be treated. The visual feedback may be provided in real time during a scanning session that generates a 3D model, or may be provided in a 3D model displayed on a display during a standby session when scanning is completed or paused. Additionally, visual feedback may be provided in an esophagogram displayed on a display during an order generation session that generates an order to indicate that the tooth corresponds to the tooth to be treated.

[0165] The processor (230) can identify a data point pair consisting of a first data point and a second data point in two different depth frames. Furthermore, the processor (230) can obtain transformation information for transforming the first data point to be as close as possible to the second data point. Furthermore, the processor (230) can calculate an error based on the distance between the transformed first and second data points using the transformation information. The processor (230) can apply a weight to at least one of the identification of the data point pair, the acquisition of the transformation information, and the calculation of the error.

[0166] The processor (230) can obtain pose information of the oral scanner (100) based on the alignment of depth data. The processor (230) can obtain pose information of the oral scanner (100) based on the conversion information obtained through the alignment of the depth data with the internal parameters of the oral scanner (100) stored in the memory (220).

[0167] The processor (230) can generate a 3D model representing the inside of the oral cavity based on depth data and pose information.

[0168] The display (240) can display various screens under the control of the processor (230). In particular, the display (240) can output a 3D oral cavity model. Furthermore, the display (240) can output a UI element that can designate a tooth to be treated. Furthermore, the display (240) can output a UI element that can rotate or move the 3D model.

[0169] Although not shown, the electronic device (200) may include an input / output interface (I / O interface) that is connected to an input / output device. For example, the I / O interface may include a USB and Bluetooth that are connected to a mouse. The processor (230) may receive user input through the I / O interface. The user may input a command to start or switch the scan mode or a command to start an optimization task through the I / O interface. In addition, the user may designate a treatment target tooth and / or a non-treatment target tooth.

[0170] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

[0171] The various embodiments described above may be implemented in a computer-readable recording medium using software, hardware, or a combination thereof, or a computer or similar device. In some cases, the embodiments described herein may be implemented in the processor itself. When implemented in software, the embodiments, such as the procedures and functions described herein, may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0172] Computer instructions for performing processing operations according to the various embodiments of the present disclosure described above may be stored on a non-transitory computer-readable medium. When executed by a processor, the computer instructions stored on the non-transitory computer-readable medium may cause a specific device to perform processing operations according to the various embodiments described above.

[0173] A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0174] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0175] The methods according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. A step of acquiring depth data including multiple depth frames of the inside of a patient's oral cavity using an oral scanner; A step of aligning the plurality of depth frames included in the depth data; A step of obtaining pose information of the oral scanner based on the above alignment; and A step of generating a 3D model representing the inside of the oral cavity based on the depth data and the pose information; The step of aligning the above multiple depth frames is: Matches pairs of corresponding data points contained in two different depth frames, The matching of the above data point pairs is weighted, The above weights are determined based on the type of object corresponding to the data point pair. How to obtain an intraoral model.

2. In paragraph 1, The above objects include static objects and dynamic objects, The weight applied to the above static object is greater than the weight applied to the above dynamic object. How to obtain an intraoral model.

3. In paragraph 2, The above static object includes at least one of a tooth, a gingiva, a prosthesis, a scan body, and an abutment, The above dynamic object comprises at least one of a tongue, a cheek, a lip, a finger, and an oral tool. How to obtain an intraoral model.

4. In paragraph 2, The weight applied to at least one of the teeth, scan bodies and abutments among the above static objects is greater than the weight applied to the gingiva. How to obtain an intraoral model.

5. In paragraph 1, The above weights are, If the above object is a tooth to be treated, it is larger than if the above object is a tooth not to be treated. How to obtain an intraoral model.

6. In paragraph 1, The type of the above object is, Obtained using a neural network model trained to identify the type of object inside the oral cavity. How to obtain an intraoral model.

7. In paragraph 6, The above neural network model is, Output the identification result for the type of object inside the oral cavity and the confidence value for the identification result, The above weights are, Determined based on the above reliability value, How to obtain an intraoral model.

8. In paragraph 1, The above two different depth frames are, A first depth frame including a first data point and a second depth frame including a second data point, The step of aligning the above multiple depth frames is: A step of identifying the first data point and the second data point as a data point pair; A step of obtaining transformation information for transforming the first data point to be as close as possible to the second data point, and A step of evaluating a matching error between the first depth frame and the second depth frame based on a distance between the first data point and the second data point converted using the above conversion information, The above weights are, Applies to at least one of the steps of identifying the data point pair, obtaining the transformation information, and evaluating the matching error. How to obtain an intraoral model.

9. In paragraph 1, The above weights are, If the object is a tooth, it is larger than if the object is a gum. How to obtain an intraoral model.

10. In paragraph 1, Each depth frame that constitutes the above depth data is Containing depth map, point cloud or RGB-D data How to obtain an intraoral model.

11. In electronic devices, communication interface; memory containing at least one instruction; and including a processor; The processor, by executing the at least one instruction, Using an oral scanner, depth data including multiple depth frames of the patient's oral cavity is acquired, Aligning the plurality of depth frames included in the above depth data, Acquire pose information of the oral scanner based on the above alignment, A 3D model representing the inside of the oral cavity is created based on the depth data and pose information, The above processor, Matches pairs of corresponding data points contained in two different depth frames, The matching of the above data point pairs is weighted, The above weights are determined based on the type of object corresponding to the data point pair. Electronic devices.

12. In paragraph 11, The above objects include static objects and dynamic objects, The weight applied to the above static object is greater than the weight applied to the above dynamic object. Electronic devices.

13. In paragraph 12, The above static object includes at least one of a tooth, a gingiva, a scan body, and an abutment, The above dynamic object comprises at least one of a tongue, a cheek, lips, a finger, and an oral tool, Electronic devices.

14. In paragraph 12, The weight applied to at least one of the teeth, scan bodies and abutments among the above static objects is greater than the weight applied to the gingiva. How to obtain an intraoral model.

15. In paragraph 11, The above weights are, If the above object is a tooth to be treated, it is larger than if the above object is a tooth not to be treated. Electronic devices.

Images