Method for processing intraoral depth data and electronic device for performing same

Abstract

Disclosed is a method for generating an intraoral model. The method for generating an intraoral model comprises the steps of: generating an intraoral model on the basis of first data obtained by capturing an intraoral surface; determining a common region of the first data and second data obtained by capturing the intraoral surface, and identifying first elements of the first data and second elements of the second data corresponding to the first elements in the common region; classifying the second elements into similar elements or dissimilar elements on the basis of a statistical analysis of depth information of the first elements and the second elements; processing the depth information of the second elements on the basis of the classification result; and updating the intraoral model on the basis of the processing result.

Description

Method for processing intraoral depth data and electronic device for performing the same

[0001] The present disclosure relates to a method for processing intraoral depth data and an electronic device for performing the same, and more particularly, to a method for refining depth data.

[0002] Digital Impression Scanning (DIS) is the process of digitizing the interior of the oral cavity using a 3D scanner, replacing the traditional physical impression (which involves taking a mold of the oral cavity's interior). This method uses an intraoral scanner (usually an intraoral scanner) to precisely scan a patient's teeth, gums, and oral structure, then converts them into a digital 3D model.

[0003] While the use of intraoral scanners is expanding, the acquired depth frames may contain noise or inaccurate data due to the physical limitations of the sensor, environmental factors, and other interferences. Especially when scanning complex structures such as the oral cavity, the accuracy of the depth data can be reduced by factors such as the user's fingers, saliva, light reflections, or patient movement.

[0004] Existing depth frame processing techniques primarily apply filtering or correction to the entire frame. This approach applies the same processing to every pixel within a frame, which can unnecessarily alter accurate data that doesn't actually require correction. Furthermore, the inability to effectively leverage correlations between multiple depth frames limits accurate 3D modeling of complex structures.

[0005] Due to these issues, there is a need for improved techniques that can accurately identify only the parts that require correction in a multi-depth frame and apply an appropriate correction strategy based on the status of each pixel.

[0006] The technical challenge that the present disclosure seeks to address is to improve the accuracy and quality of depth data.

[0007] Another technical challenge that the present disclosure seeks to address is to provide an effective data processing method for generating a more accurate intraoral model.

[0008] Another technical challenge that the present disclosure seeks to address is to provide a function that can flexibly improve an intraoral model according to the user's intention.

[0009] According to one embodiment of the present disclosure, a method for generating an oral cavity model performed by an electronic device may be provided, the method comprising: generating an oral cavity model based on first data capturing an intraoral surface; determining a common area with first data for second data capturing the intraoral surface, and identifying first elements of the first data and second elements of the second data corresponding to the first elements in the common area; classifying the second elements into similar elements or dissimilar elements based on statistical analysis of depth information of the first elements and the second elements; processing depth information of the second elements based on a result of the classification; and updating the oral cavity model based on a result of the processing.

[0010] The step of determining the common area may be performed when the difference in pose information between the first data and the second data is within a preset threshold value.

[0011] The step of identifying the first elements and the second elements can be performed by coordinate system transformation or projection using pose information of the first data and the second data.

[0012] The above statistical analysis may include a process of calculating a depth difference between the first elements and the second elements, performing a statistical analysis on the depth difference, and determining each of the second elements as a normal element or an abnormal element.

[0013] The above judgment may be performed by comparing the number of normal elements with the number of abnormal elements. Additionally, the judgment may be performed based on a weighted count comparison that assigns greater weight to second elements of frames acquired later within the second data.

[0014] The depth information of the second elements classified as the above similar elements may be maintained or corrected as an average or weighted average with the depth information of the first elements.

[0015] The depth information of the second elements classified as the above-mentioned dissimilar elements may be set to an invalid value or excluded from subsequent processing, and the invalid value may be at least one of 0, a maximum depth value, an undefined value, or a flag value.

[0016] While the second data is continuously acquired, the common area determination step, the element identification step, the classification step, the processing step, and the update step can be repeatedly performed in real time.

[0017] At least some of the common area determination step and the update step and the visualization of the oral model can be performed in parallel.

[0018] Elements of the second data located in an area other than the common area may be partially or not reflected when updating the oral model.

[0019] The first data may include a plurality of frames, and the second data may include at least one frame acquired later in time than the first data.

[0020] According to one embodiment of the present disclosure, at least one processor including a processing circuit may be provided, wherein the processing circuit may be configured to generate an oral cavity model based on first data capturing an intraoral surface, determine a common area with the first data for second data capturing the intraoral surface, identify first elements of the first data and second elements of the second data corresponding to the first elements in the common area, classify the second elements into similar elements or dissimilar elements based on statistical analysis of depth information of the first elements and the second elements, process depth information of the second elements based on a result of the classification, and update the oral cavity model based on a result of the processing.

[0021] According to one embodiment of the present disclosure, an oral scanning system may be provided, including an oral scanner for acquiring scan data of an oral surface, and an electronic device for receiving and processing the scan data, wherein the electronic device may be configured to generate an oral model based on first data acquired from the scan data, determine a common area with the first data for second data acquired from the scan data, identify first elements of the first data and second elements of the second data corresponding to the first elements in the common area, classify the second elements into similar elements or dissimilar elements based on statistical analysis of depth information of the first elements and the second elements, process depth information of the second elements based on a classification result, and update the oral model based on a processing result.

[0022] According to one embodiment of the present disclosure, a computer-readable, non-transitory recording medium may be provided, wherein the recording medium may store instructions that cause a computer to perform the steps of: generating an oral cavity model based on first data capturing an oral cavity surface; determining a common area with first data for second data capturing the oral cavity surface, and identifying first elements of the first data and second elements of the second data corresponding to the first elements in the common area; classifying the second elements into similar elements or dissimilar elements based on statistical analysis of depth information of the first elements and the second elements; processing depth information of the second elements based on a result of the classification; and updating the oral cavity model based on a result of the processing.

[0023] The above exemplary embodiments and other exemplary embodiments will be explained or clarified by the detailed description set forth below of exemplary embodiments to be read in connection with the accompanying drawings.

[0024] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment must include all or only the following effects, and therefore the scope of the disclosed technology should not be construed as being limited thereby.

[0025] According to one embodiment of the present disclosure, depth data with improved accuracy and quality can be obtained.

[0026] According to another embodiment of the present disclosure, the accuracy of an intraoral model can be improved through an effective data processing method.

[0027] According to another embodiment of the present disclosure, the intraoral model can be flexibly improved according to the user's intention, thereby improving user convenience.

[0028] The above disclosure does not contain an exhaustive list of all aspects of the present disclosure. It should be understood that the present disclosure encompasses all methods, devices, and systems capable of being implemented from all appropriate combinations of the various aspects disclosed in the detailed description and claims below, as well as the matters summarized above. Furthermore, the detailed description of the embodiments of the present disclosure will directly or implicitly disclose any benefits that may be obtained or anticipated from the embodiments. For example, various anticipated benefits of the embodiments of the present disclosure will be disclosed in the detailed description that follows.

[0029] Aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0030] FIG. 1 is a schematic diagram illustrating an oral scanning system according to one embodiment of the present disclosure.

[0031] FIG. 2 is a block diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 3 is a flowchart illustrating a depth data processing method according to one embodiment of the present disclosure.

[0033] FIG. 4 is a flowchart illustrating a method for processing a depth value of a first pixel according to one embodiment of the present disclosure.

[0034] FIG. 5 is a schematic diagram illustrating a corresponding pixel identification method according to one embodiment of the present disclosure.

[0035] FIG. 6 is a schematic diagram illustrating a pixel classification method according to one embodiment of the present disclosure.

[0036] FIG. 7 is a schematic diagram illustrating an abnormal pixel processing method according to one embodiment of the present disclosure.

[0037] FIG. 8 is a schematic diagram illustrating a normal pixel processing method according to one embodiment of the present disclosure.

[0038] FIG. 9 is a schematic diagram showing how an intraoral model is updated according to one embodiment of the present disclosure.

[0039] FIG. 10 is a block diagram showing the configuration of an oral scanning system according to one embodiment of the present disclosure.

[0040] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0041] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0042] The embodiments of the present disclosure are capable of various modifications and multiple embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the scope of the disclosed concepts and techniques. In describing the embodiments, detailed descriptions of related known technologies will be omitted if they are deemed to obscure the main point.

[0043] Terms such as "first" and "second" may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0044] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0045] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description of the present disclosure, and similar parts are designated with similar reference numerals throughout the specification.

[0046] FIG. 1 is a schematic diagram illustrating an oral scanning system according to one embodiment of the present disclosure.

[0047] Referring to FIG. 1, an oral scanning system (1000) may include an oral scanner (100) and an electronic device (200). A user (1) may scan the inside of a patient's (2) oral cavity using the oral scanner (100). The user (1) may be a dental staff member. For example, dental staff members may include dentists, dental hygienists, dental assistants, orthodontists, dental technicians, and oral surgeons.

[0048] The oral scanner (100) can transmit scan data acquired during the scanning process to an electronic device (200). The electronic device (200) can generate a 3D model representing the inside of the oral cavity based on the scan data. In addition, the electronic device (200) can display the 3D model on the display. The user (1) can continue the scanning process while checking the 3D model displayed on the electronic device (200). The scanning target of the oral scanner (100) is not limited to the inside of the patient's (2) oral cavity, and may also include structures that are scheduled to be installed or applied to the inside of the oral cavity in the future or are intermediate results therefor, even if they are not currently located in the oral cavity.

[0049] The electronic device (200) can be connected to the oral scanner (100) via a wired or wireless connection to exchange data. The electronic device (200) may be a desktop, laptop, tablet PC, or a computer designed as an embedded system specifically for the purpose of using the oral scanner (100). The electronic device (200) may be implemented by being physically or functionally integrated with the oral scanner (100).

[0050] FIG. 2 is a block diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0051] Referring to FIG. 2, the electronic device (200) can obtain scan data (21). Specifically, the electronic device (200) can receive scan data (21) from an oral scanner (100).

[0052] Scan data (21) refers to raw data acquired by the oral scanner (100) during the oral scanning process. For example, the scan data (21) may include an RGB image photographing the inside of the oral cavity, an infrared image, an intensity image indicating the reflectivity of the tooth surface, a polarization image, etc. The scan data (21) may be a monocular image captured by a single camera, or a pair of stereo images captured from different angles. In addition, the scan data (21) may include a structured light pattern projected onto the tooth surface.

[0053] Scan data (21) is composed of a plurality of scan frames acquired in a time series manner. Here, the scan frame may be a unit data including scan information of the inside of the oral cavity captured by the oral scanner (100) at a specific point in time, and may be composed of one of the forms of the scan data (21) described above.

[0054] Depth data (22) refers to data containing depth information about the inside of the oral cavity, and may be referred to as 'intraoral depth data' for short. For example, depth data (22) may have the form of a depth map that stores distance information to teeth and surrounding tissues in a two-dimensional array format, a point cloud composed of three-dimensional coordinates of teeth and surrounding tissues, a triangle mesh of the tooth surface, etc.

[0055] Depth data (22) is composed of multiple depth frames acquired in a time-series manner. Here, a depth frame may be unit data including depth information in a three-dimensional space acquired at a specific point in time, and may be composed of one of the forms of the depth data (22) described above.

[0056] Depth data (22) can be acquired in various ways. In one embodiment, the electronic device (200) can derive depth data (22) based on scan data (21). For example, if the scan data (21) is a stereo image, the electronic device (200) can analyze the disparity between the stereo images to generate a depth map. Additionally, the electronic device (200) can acquire a point cloud based on the internal parameters of the oral scanner (100) and the depth map.

[0057] In another embodiment, depth data (22) may be acquired by an oral scanner (100). For example, the oral scanner (100) may directly acquire depth data (22) using a Time-of-Flight (TOF) sensor or an active stereo method. That is, if the scan data (21) already includes three-dimensional shape information, the scan data (21) itself may be utilized as depth data (22).

[0058] "First data" may refer to a set of multiple first scan frames or depth frames that capture the surface of the oral cavity. The first data serves as reference data used to create an initial oral cavity model, and may include multiple frames acquired sequentially or discontinuously in time.

[0059] "Second data" may refer to at least one second scan frame or depth frame acquired after the first data. The second data is used to update or renew the oral model and may include an oral region that partially overlaps with the first data.

[0060] An 'element' is a basic unit that constitutes data and can include pixels, points, or mesh vertices. In a depth map, an element can be a pixel, in a point cloud, an individual point, or in mesh data, a vertex.

[0061] A "similar element" may refer to an element of the second data whose depth value difference from the corresponding elements of the first data is within a threshold value. A "dissimilar element" may refer to an element of the second data whose depth value difference from the corresponding elements of the first data exceeds a threshold value.

[0062] The electronic device (200) may compare the depth values ​​of each element of the second data with a plurality of corresponding first data elements. If the number of first elements showing differences within a threshold value in the comparison results is greater than or equal to the number of first elements showing differences exceeding the threshold value, the electronic device (200) may classify the second element as a similar element. In the opposite case, the electronic device (200) may classify the second element as a dissimilar element.

[0063] This classification can be used as a criterion for determining whether each element of the second data accurately reflects the actual oral structure (similar element) or has a distorted value due to noise or foreign substances (dissimilar element).

[0064] A 'common area' may mean a portion where the first data and the second data capture the same physical area within the oral cavity.

[0065] 'Corresponding element' may mean an element of first data and an element of second data that are in a corresponding relationship with each other within a common area.

[0066] The electronic device (200) can reconstruct an intraoral model (23) based on the depth data (22). Specifically, the electronic device (200) can register a plurality of depth frames included in the depth data (22). For example, the electronic device (200) can perform local registration between time-series adjacent depth frames. Here, each depth frame can be a 3D point cloud. The electronic device (200) can generate a transformation matrix including rotation and translation components for registering two depth frames through registration.

[0067] The electronic device (200) can obtain pose information of the oral scanner (100) based on a transformation matrix. The pose information can include position and posture information of the oral scanner (100). The electronic device (200) can calculate a relative pose change of the oral scanner (100) based on the transformation matrix. The electronic device (200) can use this pose information to align 3D point clouds of each depth frame to create an intraoral model.

[0068] In one embodiment, the electronic device (200) can generate a pose graph in which the pose information of the oral scanner (100) at the time of acquisition of each depth frame is used as nodes and the relative transformation relationship between the pose information is used as edges. The electronic device (200) can perform pose graph optimization to globally optimize the pose information of each node. The electronic device (200) can generate a more precise intraoral model by realigning the 3D point cloud of each depth frame using the optimized pose information. In addition, when the oral scanner (100) rescans an area that has been previously scanned, the electronic device (200) can detect a loop closure and add it as a new edge of the pose graph, thereby effectively correcting accumulated drift errors.

[0069] The intraoral model (23) refers to a three-dimensional model that digitally represents the oral structure of a patient. The intraoral model (23) may include a maxillary model, a mandibular model, and an occlusal model. The maxillary model is a digital model representing the three-dimensional shape of the patient's maxillary arch. The mandibular model is a digital model representing the three-dimensional shape of the patient's mandibular arch. The occlusal model is a digital model representing the three-dimensional shape of the patient's maxillary model and mandibular model in an occluded state. The occlusal model reflects the patient's actual occlusal state, which may include the contact relationship and occlusal gap between the maxillary and mandibular teeth.

[0070] The type of intraoral model (23) may vary depending on the scan mode. The electronic device (200) may operate in multiple scan modes. The multiple scan modes may include an upper jaw scan mode, a lower jaw scan mode, and an occlusion scan mode.

[0071] The maxillary scan mode is a scanning mode for scanning the maxilla. In the maxillary scan mode, the user (1) can scan the maxilla using the oral scanner (100). The oral scanner (100) can collect scan data for the maxilla. The electronic device (200) can receive scan data for the maxilla from the oral scanner (100) and create a maxillary model.

[0072] The mandibular scan mode is a scan mode for scanning the mandible. In the mandibular scan mode, the user (1) can scan the mandible using the oral scanner (100). The oral scanner (100) can collect scan data for the mandible. The electronic device (200) can receive scan data for the mandible from the oral scanner (100) and create a mandibular model.

[0073] The occlusion scan mode is a scan mode for scanning occlusion. In the occlusion scan mode, the user (1) can scan the upper and lower jaws in occlusion using the oral scanner (100). The oral scanner (100) can collect scan data on occlusion. The electronic device (200) can receive scan data on occlusion from the oral scanner (100) and create an occlusion model.

[0074] Meanwhile, in the present disclosure, a "scan session" may refer to a period of time during which a scanning operation is performed in a specific scan mode. For example, a scan session in the maxillary scan mode may be the period from the time a user (1) begins scanning the maxilla using an oral scanner (100) to the time the maxillary scan is completed.

[0075] FIG. 3 is a flowchart illustrating a depth data processing method according to an embodiment of the present disclosure. FIG. 4 is a flowchart illustrating a method for processing a depth value of a first pixel according to an embodiment of the present disclosure.

[0076] Referring to FIG. 3, the electronic device (200) can acquire multiple depth frames containing depth information about the oral cavity (S310). Each depth frame can include multiple pixels. Each pixel can have a depth value.

[0077] The electronic device (200) can identify a plurality of second depth frames among a plurality of depth frames, wherein the difference in pose information between the first depth frame and the second depth frame is within a first threshold value (S320). Here, the pose information of the depth frame may refer to pose information of the oral scanner (100) that corresponds temporally to the depth frame. The pose information of the oral scanner (100) may include position and posture information.

[0078] The first threshold may refer to a reference value for determining the similarity between pose information. For example, if the difference between the pose information of two depth frames is within the first threshold, the pose information of the two depth frames may be considered similar. Otherwise, the pose information of the two depth frames may be considered dissimilar.

[0079] The difference between pose information can be calculated by considering the difference between the position information and the difference between the posture information included in the two pieces of pose information. For example, the electronic device (200) can calculate the difference between the position information and the difference between the posture information, respectively. Then, the electronic device (200) can calculate the difference between the position information (e.g., Euclidean distance) and determine whether the value is within a preset distance threshold. In addition, the electronic device (200) can calculate the difference between the posture information (e.g., angular difference) and determine whether the value is within a preset angular threshold. Then, the electronic device (200) can determine that the pose information is similar when both differences are within their respective thresholds, or can determine that the pose information is similar when a weighted average value obtained by assigning weights to the position difference and the posture difference is within an integrated threshold.

[0080] That is, the electronic device (200) can identify at least one second depth frame having pose information similar to the pose information of the first depth frame. However, identifying the second depth frame is for the purpose of identifying depth frames including depth information for the same area inside the oral cavity as the first depth frame, and identification is not necessarily based on whether the pose information is similar. Such second depth frames including depth information for the same area inside the oral cavity as the first depth frame may be referred to as 'corresponding frames' in the present disclosure.

[0081] In another embodiment, it is also possible to identify multiple second depth frames, i.e., corresponding frames, using information other than pose information. For example, the electronic device (200) may identify frames acquired within a specific time range as second depth frames based on their temporal proximity to the first depth frame. Alternatively, the electronic device (200) may identify frames in which the number of common feature points exceeds a specific threshold as second depth frames based on the matching results of feature points extracted from the first depth frame and other depth frames.

[0082] The electronic device (200) can determine a common area between the first data and the second data. The common area refers to a portion where the first data and the second data capture the same physical area inside the oral cavity.

[0083] The determination of a common area can be performed through various methods. In one embodiment, the electronic device (200) can determine the common area based on pose information. Specifically, the electronic device (200) can compare the pose information at the time of first data acquisition with the pose information at the time of second data acquisition, and identify frames in which the pose information difference is within a first threshold value. At this time, the pose information includes the position information and posture information of the oral scanner (100), and the electronic device (200) can set individual threshold values ​​for the position information and posture information, or apply a weighted integrated threshold value that assigns weights to the two pieces of information.

[0084] In another embodiment, the electronic device (200) may determine a common area based on temporal proximity. For example, the electronic device (200) may determine that frames acquired within a specific time range have a common area.

[0085] In another embodiment, the electronic device (200) may determine a common area based on feature point matching. The electronic device (200) may extract feature points from each of the first and second data, match the extracted feature points, and determine the area as a common area if the number of common feature points exceeds a certain threshold.

[0086] Additionally, the electronic device (200) can determine a common area using the Iterative Closest Point (ICP) algorithm. Matching between two data sets can be performed using the ICP algorithm, and the overlapping area resulting from the matching can be determined as a common area.

[0087] Once the common area is determined, the electronic device (200) can identify corresponding elements through projection or coordinate system transformation. Specifically, the electronic device (200) can transform the coordinate system of the first data into the coordinate system of the second data, or vice versa, to match elements corresponding to the same physical location. For example, the electronic device (200) can calculate three-dimensional spatial coordinates for a specific element of the first data, transform these coordinates into the coordinate system of the second data, and then determine which element of the second data the transformed coordinates correspond to. Through this process, the electronic device (200) can obtain element correspondence relationships between data acquired at different points in time or poses.

[0088] Meanwhile, the electronic device (200) can enhance processing efficiency by performing depth information processing limited to a plurality of first scan frames and second scan frames associated with a common area. By processing only data corresponding to the common area, rather than all data, the amount of computation can be significantly reduced and processing speed can be improved.

[0089] Furthermore, the electronic device (200) can process depth information more quickly by prioritizing pose information when determining common areas. Using pose information allows for faster prediction of common areas and elimination of unnecessary inter-area matching operations. This can provide a particularly important advantage in oral scanning environments requiring real-time processing.

[0090] The electronic device (200) can identify a plurality of second pixels of a plurality of second depth frames corresponding to a first pixel of a first depth frame (S330). This identification process can be performed based on pose information of the first depth frame and pose information of each of the plurality of second depth frames. Specifically, the electronic device (200) can identify a plurality of corresponding second pixels by projecting each pixel of the first depth frame onto each of the plurality of second depth frames.

[0091] Here, the projection process may include a step of transforming the coordinate system of the first depth frame into the coordinate system of each second depth frame to determine a corresponding pixel location. For example, the electronic device (200) may calculate a three-dimensional spatial coordinate for a specific pixel of the first depth frame, transform the coordinate into the coordinate system of the second depth frame, and then determine which pixel of the second depth frame the transformed coordinate corresponds to. Through this process, the electronic device (200) may obtain a pixel correspondence relationship between depth frames acquired at different poses. In the present disclosure, pixels that are in a spatial correspondence relationship may be referred to as 'corresponding pixels'. For example, a second pixel corresponding to a first pixel may be referred to as a corresponding pixel of the first pixel.

[0092] The electronic device (200) can process the depth value of the first pixel based on a comparison between the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels (S340). Referring to FIG. 4, the step of processing the depth value of the first pixel (S340) may include a step of calculating a difference between the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels (S341), a step of classifying the first pixel as a normal pixel or an abnormal pixel based on a statistical analysis of the calculated difference (S342), and a step of processing the depth value of the first pixel based on the classification (S343).

[0093] In step S341, the electronic device (200) can calculate the difference between the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels. At this time, the difference between the two depth values ​​can be calculated in a simple difference based on the absolute value, a square difference, a normalized difference, or the like.

[0094] In step S342, the electronic device (200) can classify the first pixel as a normal pixel or an abnormal pixel based on a statistical analysis of the difference between the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels. A 'normal pixel' refers to a pixel that accurately reflects the depth value of an actual oral structure. On the other hand, an 'abnormal pixel' refers to a pixel that has a depth value different from the depth value of the actual oral structure. Abnormal pixels can be caused by various reasons. For example, during an oral scanning process, a foreign substance such as a finger or a dental instrument may be scanned, and a depth value for the foreign substance may be obtained instead of a depth value for the actual teeth or oral structure at the corresponding location. In addition, a case where a depth value that does not match the actual oral structure is recorded due to a malfunction of the scanner, sensor noise, reflection by saliva or blood in the oral cavity, etc., is also considered an abnormal pixel.

[0095] The electronic device (200) may distinguish the second pixel as a similar pixel or a dissimilar pixel based on the difference between the depth values. For example, the electronic device (200) may distinguish the second pixel, in which the difference between the depth value of the first pixel and the second pixel is within a second threshold value, as a similar pixel, and the second pixel, in which the difference between the depth value of the first pixel and the second threshold value exceeds the second threshold value, as a dissimilar pixel. Here, the second threshold value may mean a reference value for determining whether depth information is similar.

[0096] The electronic device (200) can determine whether each pixel is normal through a systematic statistical analysis pipeline. This pipeline may consist of steps for calculating depth differences, performing statistical analysis, and determining normality / abnormality.

[0097] First, the electronic device (200) can calculate the depth value difference between the elements of the first data and the corresponding elements of the second data. At this time, the difference between the depth values ​​can be calculated using a simple difference based on absolute values, a square difference, or a normalized difference.

[0098] Next, the electronic device (200) can perform a statistical analysis on the calculated differences. This may include calculating statistics such as the mean, variance, standard deviation, and median. For example, the electronic device (200) can calculate the mean and standard deviation of the depth differences between a plurality of second pixels and the first pixel to determine the characteristics of the depth value distribution.

[0099] The electronic device (200) can determine each element as a normal element or an abnormal element based on statistical analysis results. This determination can be made using various criteria, either singly or in combination.

[0100] The first criterion may be a comparison of the number of similar / dissimilar pixels, i.e., a majority vote. The electronic device (200) may determine that a pixel is normal if the number of similar pixels is greater than the number of dissimilar pixels, and may determine that a pixel is abnormal if the number of similar pixels is less than the number of dissimilar pixels.

[0101] The second criterion may be a weighted count comparison. The electronic device (200) may calculate a weighted count by assigning greater weight to frames acquired later in time. For example, the pixels of the most recently acquired frame may be assigned the greatest weight, and the pixels of the most recently acquired frame may be assigned the least weight. The electronic device (200) may compare the weighted count of similar pixels and the weighted count of dissimilar pixels, thus calculating normality / abnormality.

[0102] The third criterion may be a threshold number criterion. The electronic device (200) may determine that the number of similar pixels is normal if it is greater than or equal to a preset threshold number C, or that the number of dissimilar pixels is abnormal if it is greater than or equal to a preset threshold number D. This threshold number may be a fixed value or may be set as a percentage of the total number of corresponding pixels.

[0103] Additionally, the electronic device (200) may apply a combination of these criteria. For example, a combination rule may be applied, such as determining a normal condition only when the number of similarities is greater than the number of dissimilarities and at the same time the number of similarities is greater than or equal to a threshold number C, or determining an abnormal condition regardless of other conditions when the number of dissimilarities is greater than or equal to a threshold number D.

[0104] Meanwhile, the number of similar / dissimilar pixels can be calculated based not only on the number of pixels but also on the number of frames containing the corresponding type. For example, if similar pixels appear in three of five corresponding frames and dissimilar pixels appear in two, it can be determined that similar pixels occupy the majority of the frames.

[0105] This type of statistical analysis method can provide significantly higher accuracy than single-frame or simple threshold-based judgments. By comprehensively analyzing data acquired from multiple frames, the effects of transient noise and measurement errors can be minimized. Furthermore, by combining various statistical indices and judgment criteria, stable and accurate depth information processing can be achieved even under diverse scanning environments and conditions. This ultimately leads to the creation of a more precise and reliable intraoral model.

[0106] In addition, the electronic device (200) can classify the first pixel as a normal pixel or an abnormal pixel by comparing the number of similar pixels and dissimilar pixels. For example, if the number of similar pixels is greater than or equal to the number of dissimilar pixels, the electronic device (200) can classify the first pixel as a normal pixel. Conversely, if the number of similar pixels is less than the number of dissimilar pixels, the electronic device (200) can classify the first pixel as an abnormal pixel. That is, the electronic device (200) can determine whether the first pixel is a normal pixel or an abnormal pixel based on the consistency or tendency of depth values ​​appearing in a plurality of depth frames for the same intraoral region. This method can be based on the principle of detecting outliers by statistically utilizing a plurality of observation data.

[0107] In one embodiment, the electronic device (200) may classify the first pixel as a normal pixel or an abnormal pixel by applying other comparison criteria instead of a simple comparison between the number of similar pixels and the number of dissimilar pixels.

[0108] For example, a first pixel may be classified as a normal pixel or an abnormal pixel based on the number of dissimilar pixels. Specifically, the electronic device (200) may classify the pixel as a normal pixel if the number of dissimilar pixels is less than a threshold number, and may classify the pixel as an abnormal pixel if the number of dissimilar pixels is greater than or equal to a threshold number. The threshold number may be a preset constant value or may be determined by applying a weight to the number of similar pixels. The present embodiment may enable high computational efficiency and fast filtering when abnormal depth deviations observed between multiple depth frames frequently occur.

[0109] The various comparison criteria described above can be applied independently or in combination. The application of these various comparison criteria can be appropriately adjusted based on factors such as the characteristics of the scanned area, surrounding pixel density, scan timing, and device performance. This allows for a more flexible and adaptive outlier removal criterion than a standardized classification method.

[0110] The electronic device (200) can adaptively set classification parameters. Classification parameters such as weights, margins, and threshold numbers (C, D) can be automatically set based on statistical information from previous scan sessions.

[0111] Specifically, the electronic device (200) can analyze the similarity / dissimilarity ratio (a / b ratio) from the previous scan session. For example, if the ratio of similar pixels to dissimilar pixels was 7:3 in the previous session, the electronic device (200) can adjust the threshold by expecting a similar ratio in the current session.

[0112] Additionally, the electronic device (200) can calculate the variance or standard deviation of depth values ​​in the previous session. If the variance is large, it is determined that the variability of the depth values ​​is large, and thus the second threshold value can be set large. If the variance is small, the second threshold value can be set small.

[0113] The electronic device (200) can utilize a margin value Z to establish a confidence interval. For example, by applying a Z value corresponding to a 95% confidence interval, the reliability of normal pixel determination can be increased. This margin value can be dynamically adjusted depending on the noise level or data quality of the scanning environment.

[0114] Using this statistical information, the electronic device (200) can automatically determine parameters optimized for the current scanning environment. For example, in environments with a high concentration of oral saliva, the criteria for determining dissimilar pixels can be relaxed in anticipation of increased noise due to reflections. In cases where orthodontic appliances are present, parameters can be adjusted to account for metal reflections. This can enable more accurate detection of abnormal pixels.

[0115] Meanwhile, the number of similar pixels may be equal to the number of depth frames including similar pixels, and the number of dissimilar pixels may be equal to the number of depth frames including dissimilar pixels. Accordingly, the electronic device (200) may classify the first pixel as a normal pixel or an abnormal pixel not only by comparing the number of similar pixels and dissimilar pixels, but also by comparing the number of depth frames including similar pixels and the number of depth frames including dissimilar pixels.

[0116] Additionally, the electronic device (200) can calculate a weighted number of similar pixels and dissimilar pixels using weights according to the temporal acquisition order of a plurality of second depth frames, and classify the first pixel as a normal pixel or an abnormal pixel based on the weights.

[0117] In one embodiment, the electronic device (200) may assign a weight to each of a plurality of second pixels according to the temporal acquisition order of the plurality of second depth frames. In this case, the weight may be assigned a larger value to a second pixel included in a second depth frame acquired later in time.

[0118] The electronic device (200) can calculate the weighted number of similar pixels and the weighted number of dissimilar pixels by applying weights to the number of similar pixels and the number of dissimilar pixels, respectively. Then, the electronic device (200) can compare the weighted number of similar pixels and the weighted number of dissimilar pixels to classify the first pixel as a normal pixel or an abnormal pixel. For example, if the weighted number of similar pixels is greater than or equal to the weighted number of dissimilar pixels, the electronic device (200) can classify the first pixel as a normal pixel. Conversely, if the weighted number of similar pixels is less than the weighted number of dissimilar pixels, the electronic device (200) can classify the first pixel as an abnormal pixel.

[0119] In step S343, the electronic device (200) may process the depth value of the first pixel based on the classification of the first pixel. If the first pixel is classified as a normal pixel, the electronic device (200) may maintain or correct the depth value of the first pixel.

[0120] When correcting the depth value of the first pixel, the electronic device (200) can correct the depth value of the first pixel by reflecting the depth values ​​of a plurality of second pixels. For example, the electronic device (200) can calculate an average value of the depth value of the first pixel and the depth values ​​of the plurality of second pixels. At this time, the average value may mean a value obtained by adding up the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels and then dividing the sum by the total number of pixels. That is, when there are a depth value of the first pixel and depth values ​​of three second pixels (the 2nd-1st pixel, the 2nd-2nd pixel, and the 2nd-3rd pixel), the average value may be obtained by adding up the depth values ​​of the four pixels and dividing the sum by 4. The electronic device (200) can replace the depth value of the first pixel with the average value calculated in this way. In this way, the electronic device (200) can perform correction to further improve the reliability of the depth value even for pixels determined to be normal pixels.

[0121] Meanwhile, the average of the depth value of the first pixel and the depth values ​​of multiple second pixels may be a weighted average. For example, when calculating the weighted average, a greater weight may be assigned to the depth value of the second pixel included in the second depth frame acquired later in time.

[0122] If the first pixel is classified as an abnormal pixel, the electronic device (200) may invalidate the depth value of the first pixel. For example, the electronic device (200) may set the depth value of the first pixel to a predefined invalid value. Alternatively, the electronic device (200) may exclude the depth value of the first pixel from subsequent processing (e.g., generating an intraoral model).

[0123] The electronic device (200) can process the depth value in various ways depending on the classification result of the pixel.

[0124] If classified as a normal pixel, the electronic device (200) can maintain or correct the depth value. If the depth value is maintained, the original depth value can be used as is to create an intraoral model. If the depth value is corrected, the electronic device (200) can adjust the depth value of the corresponding pixels to a more accurate value.

[0125] The correction method may use either an average or a weighted average. In the case of a simple average, the electronic device (200) may calculate a new depth value by averaging the depth values ​​of the first pixel and all corresponding pixels with equal weighting. In the case of a weighted average, the electronic device (200) may calculate the average by applying weights based on the frame acquisition order. For example, pixels from frames acquired later in time may be given greater weighting. This may be because recent data is more likely to accurately reflect the current oral condition.

[0126] If classified as an abnormal pixel, the electronic device (200) can invalidate the depth value or exclude it from subsequent processing. As an invalidation method, the electronic device (200) can set the depth value to a predefined invalid value. The invalid value can be 0, the maximum depth value (e.g., 65535 in a 16-bit system), a negative value (e.g., -1), an undefined value, NaN (Not a Number), or a specific flag value defined by the system. Pixels set to such invalid values ​​can be automatically identified and excluded from subsequent processing.

[0127] Additionally, the electronic device (200) can manage abnormal pixels by setting separate flags. For example, a validation bitmap can be generated for each pixel to indicate the location of the abnormal pixel. This allows for efficient filtering of the pixels during subsequent processing.

[0128] If the depth value of the second pixel is set to an invalid value, the corresponding area may be referred to as a 'depth value blank area'. Special processing may be performed on the depth value blank area. The electronic device (200) may set an invalid value for this blank area or replace it with statistics of surrounding similar pixels. For example, the blank area may be filled by calculating the average or median of normal pixels around the blank area. In addition, the electronic device (200) may estimate the depth value of the blank area based on the depth values ​​of surrounding pixels by applying an interpolation technique. Various interpolation methods such as linear interpolation, bilinear interpolation, or spline interpolation may be applied.

[0129] This depth value processing process is performed in real time, allowing the user to continuously see an improved oral cavity model while scanning.

[0130] Meanwhile, while the processing process for the depth value of the first pixel has been described above, the processing process can be applied equally or similarly to all pixels included in the first depth frame. Similarly, the processing method described for the first depth frame can be applied sequentially or in parallel to each of the multiple depth frames, so that processing can be performed on the entire depth frame.

[0131] Meanwhile, although the above description focuses on the pixels of the depth map, the depth data processing method of the present disclosure can be equally applied to various types of 3D data elements.

[0132] When depth data is in the form of a point cloud, each point can be an element. In this case, a correspondence between points in the first point cloud and points in the second point cloud is established, and the differences in depth values ​​between corresponding points can be compared to classify them as similar or dissimilar. Processing processes such as determining whether points are normal or abnormal and maintaining / correcting / invalidating depth values ​​can be performed in the same manner as described for pixels.

[0133] Similarly, if the depth data is in mesh form, the vertices of the mesh can be elements. The electronic device (200) can establish a correspondence between the vertices of the first mesh and the vertices of the second mesh, and perform statistical analysis by comparing the depth information of the corresponding vertices. The classification and processing of the vertices can also be applied in the same manner as described for pixels.

[0134] As such, the depth data processing method of the present disclosure can be universally applied to various elements containing three-dimensional spatial information, regardless of the data representation format. Regardless of the element type, such as pixels, points, or vertices, the basic principles of corresponding element identification, depth value comparison, statistical analysis, classification, and processing remain the same.

[0135] The series of tasks for individually improving depth frames through the processing of the depth values ​​of each pixel described above can be referred to as "depth data refining" or "depth map refining." The results of this depth data refining can be reflected in the creation and update of an intraoral model.

[0136] The electronic device (200) can separately process elements of the second data located in an area outside the common area. The area outside the common area may refer to a new scan area that is not included in the first data but is included in the second data.

[0137] For elements in these areas, the electronic device (200) can apply various processing methods. In one embodiment, the electronic device (200) may restrictively reflect elements outside the common area. For example, the electronic device (200) may assign low reliability weights to these elements and only partially reflect them in the oral cavity model. This may be to account for the possibility that data from new areas may not have been sufficiently verified.

[0138] In another embodiment, the electronic device (200) may defer processing of elements outside the common area. The electronic device (200) may delay processing until additional data is acquired and a duplicate scan of the area is performed. Once sufficient corresponding data is available, the statistical analysis method described above can be applied to determine normality / abnormality.

[0139] In another embodiment, the electronic device (200) may employ a conditional addition approach. The electronic device (200) may add elements outside the common area to the intraoral model only if certain conditions are met. For example, the electronic device may assess whether the elements maintain continuity with the existing model, meet quality criteria (e.g., noise level, signal strength), or exhibit consistency with surrounding elements. The elements may be added to the model only if these conditions are met.

[0140] In some cases, the electronic device (200) may completely exclude elements outside the common area. In particular, if data in the relevant area is judged to be noise or suspected to be a scan artifact, the electronic device (200) may not reflect it at all in the intraoral model.

[0141] The electronic device (200) can adaptively select this processing method based on the scan situation, data quality, user settings, etc. For example, in the initial scan phase, more lenient criteria can be applied to collect more data, and as the scan progresses, increasingly stricter criteria can be applied to increase model accuracy. Furthermore, if the user desires a quick scan, a limited reflection method can be applied, and if high accuracy is desired, a hold and verification method can be applied.

[0142] The electronic device (200) can update the intraoral model using the refined depth data. For example, if the initially generated intraoral model contains a foreign substance (e.g., a finger), the electronic device (200) can update the intraoral model using the refined depth data, thereby removing the area containing the foreign substance.

[0143] The timing of reflecting the refined depth data into the intraoral model can be determined by various implementation methods. In one embodiment, the refined depth data can be reflected in real time while the intraoral model is being generated. In another embodiment, after the refining process for all depth frames is completed, the intraoral model can be generated in batches using all refined depth frames. In this manner, the depth data processing method of the present disclosure can enable the generation of a more accurate and reliable intraoral model.

[0144] During the oral scanning process, if outliers (abnormal pixels or noise) remain in the depth data, serious problems can arise when acquiring subsequent scan data. Specifically, errors can accumulate during the alignment or registration process between the previous scan containing the outliers and the new scan data. These errors can lead to distortion of the entire oral model, and in the worst case, registration failure can require re-scanning from the beginning.

[0145] Therefore, outliers may need to be detected and removed in real time. The oral scanning system of the present disclosure can support real-time, streaming-based processing and parallel processing to address this need.

[0146] The oral scanning system of the present disclosure can support real-time processing and parallel processing in a streaming manner.

[0147] The electronic device (200) can perform real-time processing while the second data is continuously input in a streaming manner during the oral scanning process. Specifically, each step of determining common areas, identifying corresponding elements, statistical classification, processing depth values, and updating the model can be repeatedly performed in real time. The electronic device (200) can begin processing immediately upon receiving a frame, allowing the user to view the continuously improving oral model in real time even during the scanning process.

[0148] For example, if the oral scanner (100) transmits data at a rate of 30 frames per second, the electronic device (200) can immediately execute the processing pipeline as each frame arrives. If a common area with previous frames is identified, a statistical analysis of the pixels in that area can be performed, and the results can be immediately reflected in the intraoral model.

[0149] Additionally, the electronic device (200) can utilize a parallel processing architecture. Model generation and update processes, as well as rendering and visualization processes, can be performed in parallel. If the processor (230) includes multiple cores, each core can perform different tasks simultaneously.

[0150] For example, the first core can perform statistical analysis of depth data and pixel classification. Simultaneously, the second core can handle updating and optimizing the intraoral model. The third core can handle 3D model rendering and display output. This parallel processing architecture can improve system responsiveness and minimize latency.

[0151] Furthermore, the electronic device (200) can utilize a GPU (Graphics Processing Unit) to perform parallel operations on a large number of pixels or points. For example, depth value comparison and statistical analysis for thousands of pixels can be performed simultaneously using the parallel processing capabilities of the GPU.

[0152] The electronic device (200) can independently configure an update pipeline. Model updates based on the processing results of the second data can be performed in a separate thread or process. This update pipeline runs in parallel with the model generation process, and the update results can be immediately provided to the user through real-time visualization.

[0153] Through this real-time and parallel processing method, the electronic device (200) can generate a high-quality intraoral model without compromising scanning speed. Users can enjoy a smooth scanning experience without processing delays and can confirm that foreign substances and noise are removed in real time.

[0154] FIG. 5 is a schematic diagram illustrating a corresponding pixel identification method according to one embodiment of the present disclosure.

[0155] Referring to FIG. 5, the electronic device (200) can identify a second depth frame (F2) and a third depth frame (F3) corresponding to the first depth frame (F1). That is, the second depth frame (F2) and the third depth frame (F3) may be corresponding frames of the first depth frame (F1).

[0156] The electronic device (200) can identify corresponding depth frames (F2, F3) of the first depth frame (F1) using pose information. For example, the electronic device (200) can determine whether the difference between the first pose information of the first depth frame (F1) and the second pose information of the second depth frame (F2) is within a first threshold. If the difference between the first pose information and the second pose information is within the first threshold, the electronic device (200) can determine the second depth frame (F2) as a corresponding frame of the first depth frame (F1). Similarly, the electronic device (200) can determine the third depth frame (F3) as a corresponding frame of the first depth frame (F1) by comparing the first pose information with the third pose information of the third depth frame (F3).

[0157] The electronic device (200) can identify the corresponding pixels (p2, p3) of the first pixel (p1) by projecting the first depth frame (F1) onto each of its corresponding frames (F2, F3). At this time, the electronic device (200) can use the pose information of the first depth frame (F1) and the pose information of each of the corresponding frames (F2, F3) to calculate the 3D spatial coordinates of the first pixel (p1), convert these coordinates into the coordinate system of the corresponding frames (F2, F3), and then determine the positions of the corresponding pixels (p2, p3) corresponding to the converted coordinates.

[0158] FIG. 6 is a schematic diagram illustrating a pixel classification method according to one embodiment of the present disclosure.

[0159] Referring to FIG. 6, the electronic device (200) can identify second to sixth depth frames (F2, F3, F4, F5, F6), which are corresponding frames of the first depth frame (F1). In addition, the electronic device (200) can identify second to sixth pixels (p2, p3, p4, p5, p6), which are corresponding pixels of the first pixel (p1).

[0160] The electronic device (200) can compare the depth value of the first pixel (p1) with the depth values ​​of each corresponding pixel. The electronic device (200) can classify a corresponding pixel whose difference from the depth value of the first pixel (p1) is within a second threshold value as a similar pixel. In addition, the electronic device (200) can classify a corresponding pixel whose difference from the depth value of the first pixel (p1) exceeds the second threshold value as a dissimilar pixel. For example, the fourth pixel (p4) and the fifth pixel (p5) can be classified as similar pixels of the first pixel (p1), and the second pixel (p2), the third pixel (p3), and the sixth pixel (p6) can be classified as dissimilar pixels of the first pixel (p1).

[0161] The electronic device (200) can compare the number of similar pixels and the number of dissimilar pixels to determine whether the first pixel (p1) is a normal pixel. In one embodiment, if the number of similar pixels is greater than or equal to the number of dissimilar pixels, the electronic device (200) can determine that the first pixel (p1) is a normal pixel. Otherwise, the electronic device (200) can determine that the first pixel (p1) is an abnormal pixel. In FIG. 6, since the number of similar pixels (p4, p5) is less than the number of dissimilar pixels (p2, p3, p6), the electronic device (200) can classify the first pixel (p1) as an abnormal pixel.

[0162] In another embodiment, in addition to the number of similar pixels and the number of dissimilar pixels, the electronic device (200) may determine whether the first pixel (p1) is a normal pixel by considering the weight assigned to each pixel. The electronic device (200) may assign a weight to each corresponding pixel according to the temporal acquisition order of the corresponding pixels (p2, p3, p4, p5, p6). At this time, the weight may be assigned a larger value to a pixel included in a depth frame acquired later in time. For example, the largest weight may be assigned to the sixth pixel (p6) of the sixth depth frame (F6) acquired most recently in time, and the smallest weight may be assigned to the second pixel (p2) of the second depth frame (F2) acquired most early in time.

[0163] The electronic device (200) can calculate a weighted number of similar pixels and a weighted number of dissimilar pixels by applying weights assigned to each of the number of similar pixels and the number of dissimilar pixels. For example, the weighted number can be defined as the weighted sum of pixels classified as pixels of the same type (similar or dissimilar). Based on the calculated weighted number, the electronic device (200) can determine whether the first pixel (p1) is a normal pixel.

[0164] Specifically, if the weighted number of similar pixels is greater than or equal to the weighted number of dissimilar pixels, the electronic device (200) can classify the first pixel (p1) as a normal pixel. On the other hand, if the weighted number of similar pixels is less than the weighted number of dissimilar pixels, the electronic device (200) can classify the first pixel (p1) as an abnormal pixel.

[0165] For example, weights of 1, 2, 3, 4, and 5 may be assigned to the second pixel (p2) to the sixth pixel (p6) respectively according to the temporal acquisition order. In this case, the sum of the weights of the fourth pixel (p4) and the fifth pixel (p5), which are similar pixels, is 7, and the sum of the weights of the second pixel (p2), the third pixel (p3), and the sixth pixel (p6), which are dissimilar pixels, is 8. Therefore, since the number of weights of similar pixels is smaller than the number of weights of dissimilar pixels, the electronic device (200) can classify the first pixel (p1) as an abnormal pixel.

[0166] Through this weight-based classification method, the electronic device (200) can improve the accuracy of pixel classification by giving greater importance to pixel information of depth frames acquired more recently in time.

[0167] FIG. 7 is a schematic diagram illustrating an abnormal pixel processing method according to one embodiment of the present disclosure.

[0168] Referring to FIG. 7, the first depth frame (F1) may include pixels classified as abnormal pixels, including the first pixel (p1). For example, all pixels corresponding to the foreign matter area (71) may be classified as abnormal pixels. In this case, the electronic device (200) may invalidate the depth values ​​of the pixels corresponding to the foreign matter area (71).

[0169] In one embodiment, the electronic device (200) may set the depth values ​​of pixels corresponding to the foreign matter area (71) to a predefined invalid value. Examples of the predefined invalid value may include 0, a maximum depth value (e.g., 65535), a negative value (e.g., -1), or a special flag value specifically defined by the system.

[0170] In another embodiment, the electronic device (200) may exclude the depth values ​​of pixels corresponding to the foreign matter area (71) from subsequent processing. For example, the electronic device (200) may flag abnormal pixels and automatically filter them out during subsequent depth data processing, such as 3D model generation.

[0171] FIG. 8 is a schematic diagram illustrating a normal pixel processing method according to one embodiment of the present disclosure.

[0172] Referring to FIG. 8, the first depth frame (F1) may include a seventh pixel (p7) classified as a normal pixel. The fourth depth frame (F4) and the fifth depth frame (F5) may be corresponding frames of the first depth frame (F1), and the eighth pixel (p8) and the ninth pixel (p9) may be corresponding pixels of the seventh pixel (p7).

[0173] In one embodiment, the electronic device (200) can maintain the depth value of the seventh pixel (p7).

[0174] In another embodiment, the electronic device (200) may correct the depth value of the seventh pixel (p7) based on the depth values ​​of the eighth pixel (p8) and the ninth pixel (p9). Specifically, the electronic device (200) may calculate an average depth value by adding up the depth value of the seventh pixel (p7), the depth value of the eighth pixel (p8), and the depth value of the ninth pixel (p9), and then dividing the total number of pixels (3). The electronic device (200) may replace the depth value of the seventh pixel (p7) with the average depth value.

[0175] At this time, the average depth value may be a weighted average depth value. When calculating the weighted average depth value, the electronic device (200) may assign a greater weight to the depth value of a pixel included in a depth frame acquired later in time. For example, if the fifth depth frame (F5) is acquired later in time than the fourth depth frame (F4), a greater weight may be assigned to the depth value of the ninth pixel (p9) than to the depth value of the eighth pixel (p8).

[0176] FIG. 9 is a schematic diagram showing how an intraoral model is updated according to one embodiment of the present disclosure.

[0177] Referring to FIG. 9, the electronic device (200) can display a first model (M1) including a foreign substance area (91) through a display (240).

[0178] According to the aforementioned depth data processing process, the first model (M1) can be updated to a second model (M2) from which the foreign matter region (91) has been removed. As the depth data that formed the basis for creating the foreign matter region (91) is invalidated or corrected, the shape of the region can be changed and removed from the second model (M2).

[0179] The electronic device (200) performs depth data processing in the background while generating the first model (M1), and can update the first model (M1) in real time using the processed depth data. Accordingly, the user (1) can check the second model (M2) from which foreign substances have been removed while performing an oral scan.

[0180] In conventional oral scanning systems, if a shape distortion occurs in a 3D model, such as a foreign substance area (91), the scanning session ends and the user (1) must start the oral scanning again from the beginning. However, the oral scanning system (1000) according to the present disclosure allows the user (1) to check the foreign substance area (91) in real time, and naturally guides the user (1) to perform an additional scan, thereby providing an improved model. This can increase the efficiency of the scanning operation and significantly enhance user convenience.

[0181] FIG. 10 is a block diagram showing the configuration of an oral scanning system according to one embodiment of the present disclosure.

[0182] Referring to FIG. 10, an oral scanning system (1000) may include an oral scanner (100) and an electronic device (200). The oral scanner (100) may acquire scan data of the inside of a patient's oral cavity and transmit the scan data to the electronic device (200). The oral scanner (100) may include an optical sensor (e.g., a structured light projector, a stereo camera, or other optical depth sensor) for scanning the inside of the oral cavity, an inertial sensor (IMU, Inertial Measurement Unit) for estimating the movement of the oral scanner, or a separate imaging sensor.

[0183] The oral scanner (100) can acquire scan data using structured light, Time-of-Flight (ToF), LiDAR, Computed Tomography (CT), or ultrasound. The oral scanner (100) can include an IMU sensor. IMU information acquired through the IMU sensor can be used to acquire pose information or movement speed of the oral scanner (100).

[0184] The electronic device (200) may include a communication interface (210), memory (220), processor (230), and display (240). The electronic device (200) may be a personal computer (PC), tablet, workstation, or smartphone.

[0185] The communication interface (210) may include at least one communication circuit. The communication interface (210) may receive scan data from the oral scanner (100). The communication interface (210) may include a wired interface and a wireless interface. The wired interface may include USB, Ethernet, HDMI, and Thunderbolt. The wireless interface may include Wi-Fi, Bluetooth, Zigbee, and NFC.

[0186] The memory (220) may store an operating system (OS) for controlling the overall operation of components of the electronic device (200) and commands or data related to the components of the electronic device (200). In particular, the memory (220) may include instructions for controlling oral scanning software. In addition, the memory (220) may store a neural network model trained to identify objects in a 2D image or a 3D model. The memory (220) may be implemented as a non-volatile memory (e.g., a hard disk, a solid state drive (SSD), a flash memory) or a volatile memory.

[0187] The processor (230) is electrically connected to the memory (220) and can control the overall functions and operations of the electronic device (200). The processor (230) can control the overall functions and operations of the electronic device (200) by executing instructions stored in the memory (220).

[0188] The processor (230) may include a processing circuit. The processing circuit may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a digital signal processor (DSP), or a combination thereof. The processing circuit may execute instructions stored in the memory (220) to perform depth data processing and oral model generation operations according to various embodiments of the present disclosure.

[0189] The processor (230) can acquire a plurality of depth frames containing depth information about the inside of the oral cavity. Each depth frame can include a plurality of pixels.

[0190] The processor (230) can identify a plurality of second depth frames among a plurality of depth frames, the difference in pose information between the first depth frame and the second depth frame being within a first threshold value.

[0191] The processor (230) can identify a plurality of second pixels of a plurality of second depth frames corresponding to a first pixel of a first depth frame. The processor (230) can identify a plurality of second pixels by projecting each pixel of the first depth frame onto each of the plurality of second depth frames based on pose information of the first depth frame and pose information of each of the plurality of second depth frames. At this time, the processor (230) can determine a corresponding pixel position by converting a coordinate system of the first depth frame into a coordinate system of each of the second depth frames.

[0192] The processor (230) may process the depth value of the first pixel based on a comparison between the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels. The processor (230) may calculate a difference between the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels. The processor (230) may classify the first pixel as a normal pixel or an abnormal pixel based on a statistical analysis of the calculated difference. The processor (230) may process the depth value of the first pixel based on the classification.

[0193] The processor (230) can classify a second pixel whose difference from the depth value of the first pixel is within a second threshold as a similar pixel, and a second pixel whose difference from the depth value of the first pixel exceeds the second threshold as a dissimilar pixel. The processor (230) can compare the number of similar pixels and dissimilar pixels to classify the first pixel as a normal pixel or an abnormal pixel. If the number of similar pixels is greater than or equal to the number of dissimilar pixels, the processor (230) can classify the first pixel as a normal pixel. If the number of similar pixels is less than the number of dissimilar pixels, the processor (230) can classify the first pixel as an abnormal pixel.

[0194] The processor (230) may assign a weight to each of a plurality of second pixels according to the temporal acquisition order of the plurality of second depth frames. In this case, the weight may be assigned a larger value to a second pixel included in a second depth frame acquired later in time.

[0195] The processor (230) can calculate the weighted number of similar pixels and the weighted number of dissimilar pixels by applying weights assigned to the number of similar pixels and the number of dissimilar pixels, respectively. If the weighted number of similar pixels is greater than or equal to the weighted number of dissimilar pixels, the processor (230) can classify the first pixel as a normal pixel. If the weighted number of similar pixels is less than the weighted number of dissimilar pixels, the processor (230) can classify the first pixel as an abnormal pixel.

[0196] If the first pixel is classified as a normal pixel, the processor (230) can maintain the depth value of the first pixel or correct it by reflecting the depth values ​​of a plurality of second pixels.

[0197] The processor (230) can calculate an average value of the depth value of the first pixel and the depth values ​​of a plurality of second pixels. The processor (230) can replace the depth value of the first pixel with the calculated average value. In this case, the average value may be a weighted average value. When calculating the weighted average value, the processor (230) can assign a greater weight to the depth value of the second pixel included in the second depth frame acquired later in time.

[0198] If the first pixel is classified as an abnormal pixel, the processor (230) may invalidate the depth value of the first pixel. For example, the processor (230) may set the depth value of the first pixel to a predefined invalid value or exclude the depth value of the first pixel from subsequent processing.

[0199] The processor (230) can generate a three-dimensional model of the oral cavity based on a plurality of depth frames. The three-dimensional model can be updated in real time based on processing of the depth value of the first pixel.

[0200] Although not shown, the electronic device (200) may include an input / output interface (I / O interface) that is connected to an input / output device. For example, the I / O interface may include a USB and Bluetooth connection to a mouse. The processor (230) may receive user input through the I / O interface.

[0201] The oral scanning system (1000) can be implemented as an integrated system that acquires and processes scan data in real time. While the oral scanner (100) continuously acquires scan data, the electronic device (200) receives the data in real time, processes it into first and second data, and can create and update an oral cavity model. This integrated system configuration allows the user to perform efficient scanning while observing the oral cavity model being improved in real time during the scanning process.

[0202] According to one embodiment of the present disclosure, a method for processing intraoral depth data, performed by an electronic device, may be provided, comprising: acquiring a plurality of depth frames including depth information about an intraoral area, each of the depth frames including a plurality of pixels; identifying a plurality of second depth frames among the plurality of depth frames, wherein a difference in pose information between a first depth frame and a first depth frame is within a first threshold value; identifying a plurality of second pixels among the plurality of second depth frames corresponding to a first pixel of the first depth frame; and processing a depth value of the first pixel based on a comparison between a depth value of the first pixel and a depth value of each of the plurality of second pixels.

[0203] The step of processing the depth value of the first pixel may include the step of calculating a difference between the depth value of the first pixel and the depth values ​​of each of the plurality of second pixels, the step of classifying the first pixel as a normal pixel or an abnormal pixel based on statistical analysis of the calculated difference, and the step of processing the depth value of the first pixel based on the classification.

[0204] The step of classifying the first pixel may include a step of classifying the second pixel, in which the difference between the depth value of the first pixel and the second pixel is within a second threshold value, as a similar pixel, and a step of classifying the second pixel, in which the difference between the depth value of the first pixel and the second pixel is greater than the second threshold value, as a dissimilar pixel, and a step of classifying the first pixel as the normal pixel or the abnormal pixel by comparing the number of the similar pixels and the dissimilar pixels.

[0205] The step of classifying the first pixel may include classifying the first pixel as the normal pixel if the number of similar pixels is greater than or equal to the number of dissimilar pixels, and classifying the first pixel as the abnormal pixel if the number of similar pixels is less than the number of dissimilar pixels.

[0206] The step of classifying the first pixel includes the step of assigning a weight to each of the plurality of second pixels according to a temporal acquisition order of the plurality of second depth frames, wherein the weight is assigned with a larger value to the second pixel included in the second depth frame acquired later in time, and the step of calculating a weighted number of the similar pixels and the dissimilar pixels by applying the assigned weight to each of the number of similar pixels and the number of dissimilar pixels, and when the weighted number of the similar pixels is greater than or equal to the weighted number of the dissimilar pixels, the first pixel can be classified as the normal pixel, and when the weighted number of the similar pixels is less than the weighted number of the dissimilar pixels, the first pixel can be classified as the abnormal pixel.

[0207] The step of processing the depth value of the first pixel may include a step of maintaining the depth value of the first pixel or correcting it by reflecting the depth values ​​of the plurality of second pixels when the first pixel is classified as the normal pixel.

[0208] The above correction step may include a step of calculating an average value of the depth value of the first pixel and the depth values ​​of the plurality of second pixels, and replacing the depth value of the first pixel with the calculated average value.

[0209] The above average value is a weighted average value, and when calculating the weighted average value, a greater weight may be given to the depth value of the second pixel included in the second depth frame acquired later in time.

[0210] The step of processing the depth value of the first pixel may include a step of invalidating the depth value of the first pixel if the first pixel is classified as the abnormal pixel.

[0211] The step of invalidating the depth value of the first pixel may set the depth value of the first pixel to a predefined invalid value or exclude the depth value of the first pixel from a subsequent processing step.

[0212] The plurality of second pixels can be identified by projecting each pixel of the first depth frame onto each of the plurality of second depth frames based on pose information of the first depth frame and pose information of each of the plurality of second depth frames.

[0213] The projection may include a step of transforming the coordinate system of the first depth frame into the coordinate system of each of the second depth frames to determine a corresponding pixel location.

[0214] The above method may further include a step of generating a three-dimensional model of the inside of the oral cavity based on the plurality of depth frames.

[0215] The above three-dimensional model can be updated in real time according to processing of the depth value of the first pixel.

[0216] The method may further include a step of aligning the plurality of depth frames to obtain pose information of each of the plurality of depth frames. The pose information of each depth frame may include position and posture information of an oral scanner temporally corresponding to the corresponding depth frame.

[0217] According to another embodiment of the present disclosure, an electronic device may be provided, comprising: a communication interface; a memory including at least one instruction; and a processor; wherein the processor, by executing the at least one instruction, obtains a plurality of depth frames including depth information about an inside of an oral cavity, each of the depth frames including a plurality of pixels; identifies a plurality of second depth frames among the plurality of depth frames in which a difference in pose information between a first depth frame and a first depth frame is within a first threshold value; identifies a plurality of second pixels of the plurality of second depth frames corresponding to a first pixel of the first depth frame; and processes a depth value of the first pixel based on a comparison between a depth value of the first pixel and a depth value of each of the plurality of second pixels.

[0218] The processor may calculate a difference between a depth value of the first pixel and a depth value of each of the plurality of second pixels, classify the first pixel as a normal pixel or an abnormal pixel based on a statistical analysis of the calculated difference, and process the depth value of the first pixel based on the classification.

[0219] The processor may classify the second pixel, in which the difference between the depth value of the first pixel and the second pixel is within a second threshold value, as a similar pixel, and the second pixel, in which the difference between the depth value of the first pixel and the second pixel is greater than the second threshold value, as a dissimilar pixel, and classify the first pixel as the normal pixel or the abnormal pixel by comparing the number of the similar pixels and the dissimilar pixels.

[0220] The processor may classify the first pixel as the normal pixel if the number of similar pixels is greater than or equal to the number of dissimilar pixels, and may classify the first pixel as the abnormal pixel if the number of similar pixels is less than the number of dissimilar pixels.

[0221] The processor may assign a weight to each of the plurality of second pixels according to a temporal acquisition order of the plurality of second depth frames, wherein the weight is assigned with a larger value to the second pixel included in the second depth frame acquired later in time, and may apply the assigned weight to each of the number of similar pixels and the number of dissimilar pixels to calculate a weighted number of the similar pixels and the dissimilar pixels, and classify the first pixel as the normal pixel when the weighted number of the similar pixels is greater than or equal to the weighted number of the dissimilar pixels, and classify the first pixel as the abnormal pixel when the weighted number of the similar pixels is less than the weighted number of the dissimilar pixels.

[0222] The processor may maintain the depth value of the first pixel or correct it by reflecting the depth values ​​of the plurality of second pixels when the first pixel is classified as the normal pixel.

[0223] The processor can calculate an average value of the depth value of the first pixel and the depth values ​​of the plurality of second pixels, and replace the depth value of the first pixel with the calculated average value.

[0224] According to one embodiment of the present disclosure, a method for generating an oral cavity model based on data capturing an oral cavity surface, performed by an electronic device, may be provided, comprising: generating an oral cavity model based on first scan data including a plurality of first scan frames capturing an oral cavity surface; updating the oral cavity model based on second scan data including at least one second scan frame capturing the oral cavity surface; and identifying first points of the plurality of first scan frames and corresponding second points of the second scan frame in a common area between the first scan data and the second scan data, and processing depth information of the second points based on a statistical analysis of depth information of the first points and the second points, wherein the step of processing the depth information is performed in parallel with the step of updating the oral cavity model.

[0225] The step of processing the depth information may be performed in parallel with the step of updating the oral model based on third scan data acquired subsequent to the second scan data.

[0226] The plurality of first scan frames related to the common area may be frames in which the pose information difference with the second scan frame is within a first threshold value.

[0227] The step of processing the depth information of the second points may include the step of calculating a difference between the depth value of the first point of each of the plurality of first scan frames and the depth value of the second point of the corresponding second scan frame; the step of determining the second point as a normal point or an abnormal point based on a statistical analysis of the calculated difference; and the step of processing the depth value of the second point based on the determination result.

[0228] The step of determining the second point as a normal point or an abnormal point may include the step of classifying the first point, in which the difference between the depth value of the second point and the first point is within a second threshold value, as a similar point, and the first point, in which the difference exceeds the second threshold value, as a dissimilar point; and the step of classifying the second point by comparing the number of the similar points and the number of the dissimilar points.

[0229] The step of classifying the second point may classify the second point as a normal point if the number of similar points is greater than or equal to the number of dissimilar points, and may classify the second point as an abnormal point if the number of similar points is less than the number of dissimilar points.

[0230] The number of similar points and the number of dissimilar points may be weighted numbers that apply weights according to the temporal acquisition order of the plurality of first scan frames, and the weights may be assigned with a larger value to points of frames acquired later in time.

[0231] The step of classifying the second point may classify the second point as a normal point if the number of dissimilar points is less than a threshold number, and as an abnormal point if the number is greater than or equal to the threshold number, and the threshold number may be determined based on the number of similar points.

[0232] If the second point is classified as a normal point, the depth value of the second point may be maintained or corrected to an average value with the depth values ​​of the plurality of first points. The average value may be a weighted average value according to the temporal acquisition order of the plurality of first scan frames and the second scan frame.

[0233] If the second point is classified as an abnormal point, the depth value of the second point may be set to an invalid value or excluded from subsequent processing.

[0234] The method may further include a step of updating the oral model based on the processed depth information, wherein the updating step may be performed in parallel with the processing of subsequent scan data.

[0235] If the second point is determined to be a depth value blank area, the depth value of the second point may be set to an invalid value or replaced with the depth value of an adjacent normal point.

[0236] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

[0237] The various embodiments described above may be implemented in a computer-readable recording medium using software, hardware, or a combination thereof, or a computer or similar device. In some cases, the embodiments described herein may be implemented in the processor itself. When implemented in software, the embodiments, such as the procedures and functions described herein, may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0238] Computer instructions for performing processing operations according to the various embodiments of the present disclosure described above may be stored on a non-transitory computer-readable medium. When executed by a processor, the computer instructions stored on the non-transitory computer-readable medium may cause a specific device to perform processing operations according to the various embodiments described above.

[0239] A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0240] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0241] The methods according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. A step of creating an oral model based on first data capturing an intraoral surface; A step of determining a common area with the first data for the second data that captures the surface within the oral cavity, and identifying first elements of the first data and second elements of the second data corresponding to the first elements in the common area; A step of classifying the second elements into similar elements or dissimilar elements based on statistical analysis of depth information of the first elements and the second elements; A step of processing depth information of the second elements based on the classification result; and Comprising a step of updating the oral model based on the processing result, How to create an oral model.

2. In paragraph 1, The step of determining the above common area is: When the difference in pose information between the first data and the second data is within a preset threshold value, How to create an oral model.

3. In paragraph 1, The step of identifying the first elements and the second elements comprises: It is performed by coordinate system transformation or projection using pose information of the first data and the second data. How to create an oral model.

4. In paragraph 1, The above statistical analysis, Calculating the depth difference between the first elements and the second elements, and performing a statistical analysis on the depth difference to determine each of the second elements as a normal element or an abnormal element. How to create an oral model.

5. In paragraph 4, The above judgment is, It is performed by comparing the number of normal elements and the number of abnormal elements. How to create an oral model.

6. In paragraph 4, The above judgment is, Based on a weighted number comparison that gives greater weight to the second elements of the later acquired frame within the second data, How to create an oral model.

7. In paragraph 1, Depth information of the second elements classified as the above similar elements, maintained or corrected by the average or weighted average of the depth information of the first elements, How to create an oral model.

8. In paragraph 1, The depth information of the second elements classified as the above dissimilar elements is Set to an invalid value or excluded from subsequent processing, How to create an oral model.

9. In paragraph 8, The above invalid value is at least one of 0, maximum depth value, undefined value or flag value. How to create an oral model.

10. In paragraph 1, While the above second data is continuously acquired, The common area determination step, the element identification step, the classification step, the processing step, and the update step are repeatedly performed in real time. How to create an oral model.

11. In paragraph 1, At least some of the common area determination step, the element identification step, the classification step, the processing step and the update step and the visualization of the oral model are performed in parallel. How to create an oral model.

12. In paragraph 1, Elements of the second data located in an area other than the common area are Partially or not reflected when updating the above oral model, How to create an oral model.

13. In paragraph 1, The above first data includes a plurality of frames, The second data includes at least one frame acquired later in time than the first data, How to create an oral model.

14. In at least one processor including a processing circuit, The above processing circuit, Generate an oral model based on the first data capturing the oral surface, Determine a common area with the first data for the second data that captures the surface within the oral cavity, and identify first elements of the first data and second elements of the second data corresponding to the first elements in the common area, Based on statistical analysis of the depth information of the first and second elements, the second elements are classified into similar elements or dissimilar elements, Processing the depth information of the second elements based on the above classification results, Updating the oral model based on the processing results, At least one processor.

15. An oral scanner that acquires scan data of the oral surface; and An electronic device for receiving and processing the scan data, The above electronic device, Generating an oral model based on first data obtained from the above scan data, Determine a common area with the first data for the second data obtained from the scan data, and identify first elements of the first data and second elements of the second data corresponding to the first elements in the common area, Based on statistical analysis of the depth information of the first and second elements, the second elements are classified into similar elements or dissimilar elements, Processing the depth information of the second elements based on the above classification results, Updating the oral model based on the processing results, Oral scanning system.

Images