Head-mounted display-based digital impression taking method, system, and head-mounted display

Abstract

The present application discloses a digital impression-taking method, system, and head-mounted display (HMD) based on a head-mounted display (HMD). The method comprises the following steps: stitching and matching multiple frames of feature point data collected in real time by a HMD camera to obtain coded scanning rod frame point data; stitching corresponding standard data in the HMD based on feature points in the coded scanning rod frame point data that match standard data to generate first stitching data; and generating corresponding adjustment suggestions based on unmatched feature points in the coded scanning rod frame point data when the number of scanning rods in the first stitching data does not match the number of installed scanning rods. The method utilizes a HMD to replace an extraoral scanner, utilizes feature points in the coded scanning rod frame point data that match standard data to generate first stitching data to assist in tracking the scanning progress of the coded scanning rod, and utilizes unmatched feature points to generate adjustment suggestions to assist dentists in detecting whether any coded scanning rods have been omitted from scanning, thereby lowering the operational threshold of the impression-taking process.

Description

Technical Field

[0001] The present application relates to the field of digital modeling technology, and in particular to a head-mounted display (HMD)-based digital modeling method, system, and HMD. Background Art

[0002] Oral dental implants are a type of tooth restoration technology. In cases of tooth loss, functional impairment, aesthetic needs, or other specific situations, dental implants are an effective restoration method. The entire process of dental implant technology is as follows: local anesthesia, gingival incision, exposure of the alveolar bone, preparation of an implant socket on the alveolar bone (i.e. drilling a hole that matches the implant), implantation of the implant, waiting for bone integration (after the implant is implanted, it is necessary to wait for a period of time (usually 3-6 months) to allow the implant to gradually integrate with the surrounding bone tissue and become stable), installation of the healing abutment, mold taking, production of the crown, and installation of the crown. The purpose of the mold taking step is to determine the positional relationship of the implant in the mouth, as well as the relative positional relationship between multiple implants. Based on these positional relationships, bridges and crowns can be made.

[0003] Depending on the method used to take the impression, there are traditional and digital impressions. Traditional impressions typically use alginate for the primary impression, which is then used to create individual trays for the secondary impression. Traditional impression technology has several limitations, including cumbersome and time-consuming procedures, difficulty controlling the impression pressure, high technical sensitivity, and the risk of deformation and difficulty preserving the impression. Compared with digital impression technology, traditional impression technology may lack accuracy and efficiency.

[0004] Digital impression technology uses an oral digital impression device to take impressions of implants. Since the implant is located within the alveolar bone and cannot be directly scanned, a scanning rod is installed on the implant. The scanning rod is scanned with an impression device to determine the position of the scanning rod and, therefore, the position of the implant.

[0005] Scanning rods are divided into ordinary scanning rods and coded scanning rods. The scanning modes and accuracy of these two types of scanning rods are different. Ordinary scanning rods are small rods made of metal or other materials. They are generally cylindrical and have some features such as steps and bevels to facilitate feature recognition when the impression device is scanning. During scanning, the scanning rod is installed on the intraoral implant and scans together with the jaw to obtain grid data. Since ordinary scanning rods are small in size and are generally made of metal, they are highly reflective, and the data quality is often poor during scanning. In addition, when multiple ordinary scanning rods are scanned together with the jaw, the overall deformation of the jaw data may be large, resulting in inaccurate positional relationships between multiple ordinary scanning rods, and the error can reach hundreds of microns. In actual use, the digital impression-taking process of ordinary scanning rods is as follows: 1. Use an intraoral scanner to scan the gums to obtain a three-dimensional model of the gums; 2. The patient wears a mouth expander, connects the ordinary scanning rod to the implant or abutment, and then scans.

[0006] The coding scanning rod has characteristic points (such as Figure 8 (As shown in the figure), scanning only identifies feature points; the entire rod model does not need to be scanned. Multiple rods are connected by crossbars, eliminating the need for gingival data to facilitate transitions. Consequently, the final rod positioning accuracy can reach tens of microns. In practice, the digital impression acquisition process for an encoded scanning rod is as follows: 1. Scan the gingiva using an intraoral scanner to obtain a 3D gingival model A. 2. The patient wears a mouth expander and attaches the encoded scanning rod to the implant or abutment. Then, using an intraoral scanner (or extraoral scanner), scan the feature point data B on the rods to determine the relative positions of the individual rods. 3. Using an intraoral scanner (or extraoral scanner), scan a small portion of gingival data C and feature point data D simultaneously. This data is used to align the 3D model A with the feature point data B to determine the positional relationship between the rods and the gingiva. Steps 3 and 2 can be performed simultaneously, meaning both gingival data C and feature point data B are scanned simultaneously. Gingival data C is used to assemble the 3D model A, while feature point data B is used directly as the final rod data.

[0007] Traditional impression-taking is complex, its accuracy is greatly affected by the operator's experience level, and the process is difficult to digitize. The digital impression-taking process of ordinary scanning rods solves the shortcomings of traditional impression-taking, but the accuracy is still not good enough, so dentists are required to have a high level of professional knowledge to effectively complete the impression-taking work. Summary of the Invention

[0008] The present application provides a head-mounted display (HMD)-based digital impression-taking method, system, and HMD, the purpose of which is to use the HMD to assist dentists in taking impressions, thereby lowering the operational threshold for impression-taking.

[0009] In order to achieve the above objectives, this application provides the following technical solutions:

[0010] A digital impression taking method based on a head-mounted display, comprising:

[0011] The multi-frame feature point data collected in real time by the head-mounted display camera is spliced and matched to obtain the frame point data of the encoding scanning rod;

[0012] Based on the feature points in the encoded scanning rod frame point data that match the standard data, the corresponding standard data are spliced in the head display to generate first spliced data;

[0013] When the number of scanning rods in the first splicing data does not match the number of installed scanning rods, corresponding adjustment suggestions are generated based on unmatched feature points in the encoded scanning rod frame point data.

[0014] Optionally, the method further includes:

[0015] Obtaining data to be spliced based on oral scan data collected by an intraoral scanner;

[0016] Obtaining oral cavity data of multiple areas captured by the head-mounted display camera; the oral cavity data includes feature point data of the encoding scanning rod within the area, and the oral cavity data is image data and / or model data;

[0017] Splicing the data to be spliced with the plurality of oral cavity data respectively, so that the data to be spliced and the plurality of oral cavity data are aligned;

[0018] The feature point data in the plurality of oral data are respectively spliced with the encoding scanning rod frame point data so that the data to be spliced is aligned with the encoding scanning rod frame point data.

[0019] Optionally, the data to be spliced are respectively spliced with the plurality of oral data so that the data to be spliced are aligned with the plurality of oral data, including:

[0020] For each of the oral data, determining whether the oral data matches the data to be spliced;

[0021] If the oral cavity data matches the data to be spliced, the data to be spliced and the oral cavity data are spliced so that the data to be spliced and the oral cavity data are aligned.

[0022] Optionally, the method further includes:

[0023] If the oral data does not match the data to be spliced, the head-mounted display camera is triggered to collect new oral data of the multiple areas; the new oral data is used to re-match with the data to be spliced.

[0024] Optionally, based on feature points in the encoded scanning rod frame point data that match the standard data, corresponding standard data are spliced in the head display to generate first spliced data, including:

[0025] When obtaining feature point data of each frame, determining a target encoding scanning rod that matches a feature point in the feature point data of the historical frame;

[0026] For each feature point in each frame of feature point data, if the feature point belongs to the target encoding scanning rod, mark the feature point as a matched point; if the feature point does not belong to the target encoding scanning rod, mark the feature point as an unmatched point;

[0027] Matching the unmatched points with the characteristic points of the encoded scanning rod pre-stored in the scanning rod database to obtain corresponding matching results;

[0028] If the matching result is a successful match, re-mark the unmatched point as a matched point;

[0029] Based on the matched points in the feature point data of each frame and the feature points in the feature point data of the historical frame, the feature points in the frame point data of the encoding scanning rod are determined to match the standard data;

[0030] Based on the feature points in the encoded scanning rod frame point data that match the standard data, the corresponding standard data is spliced in the head display to generate first spliced data.

[0031] Optionally, when the number of scanning rods in the first splicing data does not match the number of installed scanning rods, generating corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data includes:

[0032] When the number of scanning rods in the first spliced data does not match the number of installed scanning rods, determining attribute information of unmatched feature points in the encoded scanning rod frame point data; the attribute information includes at least position, distance, clarity, and number of occurrences;

[0033] generating corresponding adjustment suggestions based on the attribute information of the unmatched feature points;

[0034] The adjustment suggestions include adjusting the patient's mouth posture, adjusting the distance between the encoding scanning rods, prompting to re-download the scanning rod database and / or prompting to adjust the distance between the head display and the encoding scanning rod.

[0035] Optionally, the method further includes:

[0036] The matched points and the unmatched points are displayed on a head display; wherein the display effect of the matched points is different from that of the unmatched points.

[0037] Optionally, the standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

[0038] Optionally, the adjustment suggestion is used to prompt the head-mounted display wearer to perform corresponding adjustment actions so that the coding scanning rod frame point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing coding scanning rod.

[0039] Optionally, the data to be spliced includes gum model data, and the gum model data is used to represent the corresponding gum model.

[0040] Optionally, the oral data further includes light spot features pre-added on the area, and the light spot features include optical features projected on the area using a third-party optical device.

[0041] A head-mounted display, comprising:

[0042] A data determination unit is used to splice and match multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the frame point data of the encoding scanning rod;

[0043] a data splicing unit, configured to splice corresponding standard data in the head-mounted display based on feature points in the encoded scanning rod frame point data that match the standard data, to generate first spliced data;

[0044] The suggestion generating unit is configured to generate corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data when the number of scanning rods in the first splicing data does not match the number of scanning rods to be installed.

[0045] A head-mounted display (HMD)-based digital impression-taking system includes an intraoral scanner and a HMD. The intraoral scanner is used to collect oral scan data of a patient, and the HMD is used to execute the above-mentioned head-mounted display-based digital impression-taking method.

[0046] A storage medium includes a stored program, wherein the program, when executed by a processor, executes the head-mounted display-based digital modeling method.

[0047] The technical solution provided by the present application is to splice and match the multi-frame feature point data collected in real time by the head-mounted display camera to obtain the coded scanning rod frame point data. Based on the feature points in the coded scanning rod frame point data that match the standard data, the corresponding standard data is spliced in the head-mounted display to generate the first spliced data. When the number of scanning rods in the first spliced data does not match the number of scanning rods installed, the corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data. The present application uses a head-mounted display to replace the extraoral scanner to obtain the coded scanning rod frame point data, and uses the feature points in the coded scanning rod frame point data that match the standard data to generate the first spliced data to assist in tracking the scanning progress of the coded scanning rod, and uses the unmatched feature points to generate adjustment suggestions to assist the dentist in detecting whether there are any missing coded scanning rods that have not been scanned, thereby lowering the operational threshold of the molding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A schematic diagram of the architecture of a head-mounted display-based digital imaging system provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a flow chart of a head-mounted display-based digital modeling method provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a flow chart of another head-mounted display-based digital imaging method provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a flow chart of another head-mounted display-based digital imaging method provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of a flow chart of another head-mounted display-based digital imaging method provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of a flow chart of another head-mounted display-based digital imaging method provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of the architecture of a head-mounted display provided in an embodiment of the present application;

[0056] Figure 8A schematic diagram of a coding scanning rod provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] In this application, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element.

[0059] like Figure 1 As shown, it is a schematic diagram of the architecture of a head-mounted display-based digital modeling system provided in an embodiment of the present application, which includes the following modules.

[0060] The intraoral scanner 100 and the head display 200, the intraoral scanner 100 is used to collect the patient's gum scanning data, and the head display 200 is used to execute the head display-based digital impression taking method provided in this application.

[0061] In some examples, types of HMD 200 include, but are not limited to, head-mounted displays (e.g., virtual reality devices, augmented display devices, hybrid display devices) and smart glasses. Generally speaking, HMD 200 and its operating system provide the basic hardware and computing resources required for 3D modeling, including, but not limited to, cameras (e.g., binocular cameras), lidar, and structured light scanning components.

[0062] It should be noted that the embodiment of the present application utilizes the head display 200 to replace the extraoral scanner to perform the scanning work of the coding scanning rod on the patient's gums. To this end, it is necessary to ensure that the camera of the head display 200 has reliable accuracy. If the head display 200 has not been calibrated or a long time has passed since the last camera calibration, the head display 200 needs to be calibrated before using the head display 200 to assist the dentist in taking impressions.

[0063] In some examples, the purpose of camera calibration is to calculate the intrinsic and extrinsic parameters of the head-mounted display camera. When using a binocular reconstruction algorithm to reconstruct and stitch multiple frames of feature point data to obtain the encoding scanning rod frame point data, the intrinsic and extrinsic parameters of the head-mounted display camera are required.

[0064] In a possible implementation, the steps for implementing camera calibration are as follows.

[0065] Step 1: Prepare the calibration object, usually a calibration plate with known size and pattern (such as a checkerboard or dot array).

[0066] Step 2: Trigger the head-mounted display camera to capture calibration images. Specifically, place the calibration plate at different angles and positions, and capture multiple calibration images to ensure that the calibration plate is clearly visible in each calibration image.

[0067] Step 3: Process the calibration image. Specifically, use image processing software (such as OpenCV) to detect the calibration image to obtain feature points.

[0068] Step 4: Use the feature points to calculate the camera parameters (i.e., intrinsic and extrinsic parameters). Specifically, based on the obtained feature points, a calibration algorithm is used to calculate the intrinsic and extrinsic parameters of the head-mounted display camera. The intrinsic and extrinsic parameters include but are not limited to focal length, principal point coordinates, distortion coefficients, etc.

[0069] Step 5: Verify the calibration results. Specifically, verify the accuracy of the calibration results (intrinsic and extrinsic parameters) through indicators such as reprojection error. If the error is large, retake the calibration image or adjust the calibration parameters.

[0070] After confirming the calibration of the head-mounted display camera, the dentist can wear the head-mounted display to take impressions. The head-mounted display camera will capture the image from the dentist's current perspective and display the image on the screen. In the embodiment of this application, the digital impression-taking method based on the head-mounted display guides the dentist to use the head-mounted display to replace the extraoral scanner to achieve the work of scanning the coding scanning rod. Figure 2 The steps are shown and the corresponding explanations.

[0071] like Figure 2 As shown, it is a flow chart of a head-mounted display-based digital modeling method provided in an embodiment of the present application, which includes the following steps.

[0072] S201: Splicing and matching multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the frame point data of the encoding scanning rod.

[0073] Among them, after the patient wears the mouth expander and installs the coded scanning rod on the implant or abutment, the headset can reconstruct and splice the feature point data of multiple frames through the feature point reconstruction and matching algorithm to obtain the coded scanning rod frame point data. The implementation principle can be understood as using multiple pictures or multiple groups of local data (i.e. frame feature point data, or scanning rod image frames) to form a large picture or complete data (i.e. coded scanning rod frame point data, complete scanning rod image). Each time, the headset will only display a part of the large picture or complete data, so that the dentist can find the location of all pictures or data in the fewest possible times.

[0074] In some examples, the encoding scanning rod frame point data includes information such as the location of feature points on the encoding scanning rod and whether they match. Generally speaking, before obtaining the first frame of feature point data, the encoding scanning rod frame point data is empty.

[0075] In a possible implementation, the head-mounted display uses a binocular camera to capture images of the scanning rod. Since the coded scanning rod carries some specially designed landmark features (which can be regarded as feature points), the landmarks of the coded scanning rod have coded information, and their corresponding position coordinates can be obtained during the intraoral scanning process. Therefore, binocular stereo vision and three-dimensional reconstruction algorithms can be used to reconstruct and extract the feature points on the coded scanning rod to obtain real-time multi-frame feature point data, thereby achieving precise positioning of the scanning rod.

[0076] It should be noted that when obtaining the feature point data of any frame, if the current encoding scanning rod frame point data is not empty, the current frame feature point data and the current encoding scanning rod frame point data (obtained after splicing and matching based on the historical frame feature point data) need to be spliced and matched to obtain more complete encoding scanning rod frame point data.

[0077] S202: Based on feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data is spliced in the head display to generate first spliced data.

[0078] The standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

[0079] It should be noted that the encoder scan bar is unique. Only a certain number of feature points (e.g., six or more) need to be identified to match the corresponding encoder scan bar from the scan bar database, thereby determining the encoder scan bar's position and orientation. Therefore, the encoder scan bar frame point data necessarily contains redundant feature points. For real-time updated encoder scan bar frame point data, feature points in the current frame feature point data may be redundant feature points of previously matched encoder scan bars. Therefore, it is possible to pre-traverse the target encoder scan bars that have been determined to be matched. If the feature points in the current frame feature point data belong to the target encoder scan bar, the feature points in the current frame feature point data can be marked as matched points, thereby determining the feature points in the encoder scan bar frame point data that match the standard data.

[0080] Optionally, based on the feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data is spliced in the head display to generate the first spliced data. For details, see Figure 3 The steps are shown and the corresponding explanations.

[0081] S203: When the number of scanning rods in the first splicing data does not match the number of installed scanning rods, generating corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data.

[0082] Among them, the adjustment suggestion is used to prompt the wearer of the headset to perform corresponding adjustment actions so that the framework point data adds feature points that match the missing data, and the missing data includes the standard data part corresponding to the missing encoding scanning rod.

[0083] It can be understood that when the number of scanning rods in the first splicing data does not match the number of scanning rods installed, it can be considered that some of the coding scanning rods installed in the patient's gums have failed to be successfully modeled, that is, there are scanning rods that have not been spliced together. The coding scanning rods that have not been successfully modeled can be regarded as missing coding scanning rods.

[0084] In some examples, the reasons for missing coding scanning rods during the scanning process include but are not limited to: the feature points on the missing coding scanning rods are blocked, the head-mounted display camera fails to capture the feature points on the missing coding scanning rods due to shooting angle problems, etc.

[0085] To this end, when the number of scanning rods in the first splicing data does not match the number of scanning rods installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data. This can provide corresponding solutions to the reasons for the missing coded scanning rods during the scanning process, thereby helping dentists to complete the modeling of the missing coded scanning rods in a timely and effective manner.

[0086] Optionally, when the number of scanning rods in the first splicing data does not match the number of scanning rods installed, the implementation process of generating corresponding adjustment suggestions based on the unmatched feature points in the encoded scanning rod frame point data can be found in Figure 4 The steps are shown and the corresponding explanations.

[0087] It should be noted that in the digital impression process, in addition to scanning the encoding scanning rod frame point data corresponding to all the encoding scanning rods on the patient's gums, the patient's gum model and the encoding scanning rod frame point data must be aligned to determine the positional relationship between the encoding scanning rod and the patient's gums.

[0088] Optionally, the process of aligning the patient's gum model with the framework point data using a head-mounted display can be found in Figure 5 The steps are shown and the corresponding explanations.

[0089] In the process shown in S201-S203 above, a head-mounted display is used to replace the extraoral scanner to obtain the frame point data of the coded scanning rod, and the feature points in the frame point data of the coded scanning rod that match the standard data are used to generate the first spliced data to assist in tracking the scanning progress of the coded scanning rod. Furthermore, the unmatched feature points in the frame point data of the coded scanning rod are used to generate corresponding adjustment suggestions to assist the dentist in detecting whether any coded scanning rod has been missed and not scanned, thereby effectively lowering the operational threshold of the impression-taking process.

[0090] like Figure 3 As shown, it is a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0091] S301: When obtaining feature point data of each frame, determining a target encoding scanning rod that matches a feature point in feature point data of a historical frame.

[0092] For each frame of feature point data, the target encoder rod that matches a feature point in the historical frame feature point data can be understood as the target encoder rod that matches the current encoder rod frame point data. In other words, at least one feature point in the current encoder rod frame point data belongs to the target encoder rod, and there are one or more target encoder rods.

[0093] In a possible implementation, the target code scanning rod may be regarded as a code scanning rod that has been determined to be matched in a historical time.

[0094] S302: For each feature point in each frame of feature point data, determine whether the feature point belongs to the target encoding scanning rod.

[0095] If the feature point belongs to the target coding scanning rod, execute S303; if the feature point does not belong to the target coding scanning rod, execute S304.

[0096] S303: Mark the feature point as a matched point.

[0097] After executing S303 , continue to execute S307 .

[0098] S304: Mark the feature point as an unmatched point.

[0099] After executing S304, continue to execute S305.

[0100] Optionally, the matched points and the unmatched points may also be displayed on the head display, wherein the display effect of the matched points is different from that of the unmatched points.

[0101] In some examples, the display effect of the matched points may be: displayed in the form of green dots. The display effect of the unmatched points may be: displayed in the form of red dots.

[0102] It can be understood that displaying matched points and unmatched points on the head display allows dentists to know the scanning effects of the feature points on the coded scanning rod in real time.

[0103] S305: Match the unmatched points with the feature points of the encoded scanning rod pre-stored in the scanning rod database to obtain corresponding matching results.

[0104] The unmatched points are matched with the feature points of the coded scanbars stored in the scanbar database. If a new coded scanbar matching the unmatched point exists in the scanbar database, the matching result corresponding to the unmatched point can be determined as a successful match. If a new coded scanbar matching the unmatched point does not exist in the scanbar database, the matching result corresponding to the unmatched point can be determined as a failed match. For the determined set of unmatched points, each unmatched point in the set of unmatched points can be sequentially matched with a feature point shown in the scanbar database until no new coded scanbar can be determined from the scanbar database.

[0105] S306: If the matching result is successful, re-mark the unmatched points as matched points.

[0106] After executing S306 , continue to execute S307 .

[0107] S307: Based on the matched points in each frame of feature point data and the feature points in the historical frame feature point data, feature points in the encoding scanning rod frame point data that match the standard data are determined.

[0108] Among them, based on the matched points in each frame feature point data and the feature points that match the target encoding scanning rod in the historical frame feature point data, the feature points in the encoding scanning rod frame point data that match the standard data are determined, which can effectively determine the matched points in the real-time updated encoding scanning rod frame point data.

[0109] S308: Based on the feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data is spliced in the head display to generate first spliced data.

[0110] Among them, the feature points in the coded scanning rod frame point data that match the standard data are spliced in the head display to generate first splicing data. The purpose is to splice the corresponding standard data in the scanning rod image displayed by the head display, so as to identify the relative position relationship between multiple coded scanning rods in the scanning rod image.

[0111] It should be noted that splicing the corresponding standard data onto the scanning rod image displayed on the head-mounted display can help dentists track the scanning progress of the coded scanning rod. Generally speaking, after the standard data corresponding to each coded scanning rod installed on the patient's gums are determined and spliced, the resulting first spliced data can be saved to determine the relative positional relationship between the coded scanning rods.

[0112] The process shown in S301-S308 above utilizes the matched points in the real-time updated coded scanning rod frame point data to splice the corresponding standard data in the head display to generate the first spliced data, thereby assisting the dentist in tracking the scanning progress of the coded scanning rod.

[0113] like Figure 4 As shown, it is a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0114] S401: When the number of scanning rods in the first splicing data does not match the number of installed scanning rods, determine the attribute information of unmatched feature points in the encoded scanning rod frame point data.

[0115] The attribute information includes at least position, distance, clarity, and number of occurrences. During the splicing process of the first spliced data, the encoded scanning rod frame point data may contain unmatched feature points (i.e., unmatched points). These unmatched points are not involved in the splicing process of the first spliced data. Therefore, by analyzing the attribute information of these unmatched points, the cause of the missing encoded scanning rods during the scanning process can be identified.

[0116] S402: Generate corresponding adjustment suggestions based on the attribute information of the unmatched feature points.

[0117] The adjustment suggestions include adjusting the patient's mouth posture, adjusting the distance between the encoding scanning rods, prompting to re-download the scanning rod database and / or prompting to adjust the distance between the head display and the encoding scanning rods.

[0118] It should be noted that, based on the attribute information of the unmatched feature points, the reasons why the unmatched points do not match the standard data can be analyzed and determined, such as the patient's mouth posture is incorrect, the distance between two adjacent encoding scanning rods is too far, the head display is too far away from the encoding scanning rod on the patient's gums, and the unmatched points do not meet the feature point standards in the scanning rod database.

[0119] In some examples, based on the attribute information of the unmatched feature points, it is determined that the patient's mouth posture is incorrect, so an adjustment suggestion can be generated to prompt the patient to adjust the mouth posture.

[0120] In some examples, based on the attribute information of the unmatched feature points, it is determined that the distance between two adjacent encoding scanning rods is too far, so an adjustment suggestion can be generated to prompt the dentist to adjust the distance between the encoding scanning rods so that the two adjacent encoding scanning rods are closer to each other.

[0121] In some examples, based on the attribute information of the unmatched feature points, it is determined that the headset is too far away from the encoding scanning rod on the patient's gums, so an adjustment suggestion can be generated to prompt the dentist to move closer to the encoding scanning rod to shorten the distance between the headset and the encoding scanning rod.

[0122] In some examples, based on the attribute information of the unmatched feature points, it is determined that the unmatched points do not meet the feature point standards in the scanning rod database. For example, if the coded scanning rods of another set of coded scanning rods are scanned together, the missing coded scanning rods cannot be identified. Therefore, an adjustment suggestion can be generated to prompt the user to re-download the scanning rod database of another set of coded scanning rods.

[0123] The process shown in S401-S402 above guides the dentist to perform corresponding adjustment actions according to the adjustment suggestions based on the attribute information of the unmatched feature points in the coding scanning rod frame point data, ensuring that all the coding scanning rods on the patient's gums can complete the corresponding standard data splicing and realize the complete modeling of each coding scanning rod.

[0124] like Figure 5 As shown, it is a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0125] S501: Obtaining data to be spliced based on oral scan data collected by an intraoral scanner.

[0126] The data to be spliced includes gum model data, and the gum model data is used to represent the corresponding gum model.

[0127] It is important to note that if the patient's gums are edentulous, due to the lack of characteristic features, an intraoral scanner can be used to scan the patient's mouth using structured light to obtain oral scan data. A transmission link is pre-established between the intraoral scanner and the headset, allowing the intraoral scanner to transmit the oral scan data to the headset via this link. After the headset obtains the oral scan data, the headset's rendering engine renders the oral scan data in real time to obtain the data to be stitched.

[0128] In some examples, the headset can display the patient's gum model data in real time to assist the dentist in understanding the patient's mouth.

[0129] S502: Obtain oral data of multiple areas collected by the head-mounted display camera.

[0130] The oral cavity data includes feature point data of the encoding scanning rod within the area, and the oral cavity data is image data and / or model data.

[0131] In some examples, the so-called area can be understood as part of the gum area.

[0132] During the digital impression process, in order to determine the positional relationship between the encoding scanning rod and the patient's gums, it is also necessary to obtain a small section of the gum model (i.e., oral data) and the encoding scanning rod frame point data for splicing.

[0133] Optionally, the oral data also includes pre-applied light spot features on the area. These light spot features include optical features projected onto the area using a third-party optical device. The third-party optical device has a pre-set light projection structure that can project the corresponding optical features onto the area, helping the head-mounted display camera collect oral data from multiple areas.

[0134] It should be noted that because the gums of edentulous jaws lack features, a preset light projection structure is used to add light spot features to the area so that the oral data of multiple areas collected by the head-mounted display camera have corresponding feature data. Parallel light and the rendering engine on the head-mounted display are used to draw the feature data in the oral data in real time to obtain the oral data of the corresponding area.

[0135] In some examples, the third-party optical device can be a light pen. The light point projected by the light pen can be moved to the corresponding area first, and then the oral data of multiple areas can be collected using a head display. At this time, the oral data includes feature point data of the gums and the encoding scanning rod, so the oral data and the encoding scanning rod frame point data can be spliced simultaneously.

[0136] In addition, since the oral data comes from the intraoral scanner, there is a deviation between the data to be spliced and the oral data (that is, the coordinate systems of the data to be spliced and the oral data are different). In order to reduce the error, oral data of multiple areas (such as the right side of the gum, the middle of the gum, and the right side of the gum) are collected. By subdividing the oral data of multiple areas, more reference feature points are provided to ensure that the splicing between the data to be spliced and the oral data is more stable and reliable.

[0137] In some examples, technicians can define multiple regions based on actual conditions. Specifically, the multiple regions may include the left side of the gum, the middle of the gum, and the right side of the gum. Multiple stitching of different regions provides more constraints, which can be used to optimize the alignment between the data to be stitched and the oral cavity data, minimizing the overall stitching error.

[0138] S503: Splicing the data to be spliced with the multiple oral cavity data respectively, so that the data to be spliced and the multiple oral cavity data are aligned.

[0139] The data to be spliced are spliced with the multiple oral data respectively so that the data to be spliced are aligned with the multiple oral data. In essence, the multiple oral data and the data to be spliced are converted into a unified coordinate system.

[0140] Optionally, the data to be spliced are spliced with multiple oral data respectively, so that the data to be spliced are aligned with the multiple oral data. Figure 6 The steps are shown and the corresponding explanations.

[0141] S504: The feature point data in the plurality of oral cavity data are spliced and matched with the encoding scanning rod frame point data respectively, so that the data to be spliced is aligned with the encoding scanning rod frame point data.

[0142] Among them, the feature point data shown in multiple oral data are respectively spliced and matched with the encoding scanning rod frame point data, so that the feature point data in each oral data can be aligned with the encoding scanning rod frame point data. When the feature point data in each oral data is aligned with the encoding scanning rod frame point data, since the feature point data in each oral data is reconstructed in the head-mounted display camera coordinate system, it can be ensured that each oral data is aligned with the encoding scanning rod frame point data. Furthermore, since the data to be spliced is aligned with multiple oral data, when it is determined that the feature point data in multiple oral data are aligned with the encoding scanning rod frame point data, it can be determined that the data to be spliced and the encoding scanning rod frame point data are aligned to the same coordinate system.

[0143] The process shown in S501-S504 above utilizes the splicing matching between the oral data of multiple regions and the data to be spliced, as well as the splicing matching between the feature point data in the multiple oral data and the frame point data of the encoding scanning rod, so that the data to be spliced and the frame point data of the encoding scanning rod are aligned to the same coordinate system. After multiple splicing on the left side of the gum, the middle of the gum, and the right side of the gum, more constraints are obtained, which can optimize the alignment relationship between the gum and the scanning rod, minimize the overall error, and thus obtain the positional relationship between the patient's gum and the encoding scanning rod.

[0144] like Figure 6 As shown, it is a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0145] S601: For each oral cavity data, determine whether the oral cavity data matches the data to be spliced.

[0146] If the oral cavity data matches the data to be spliced, then S602 is executed; if the oral cavity data does not match the data to be spliced, then S603 is executed.

[0147] S602: Splicing the data to be spliced with the oral cavity data so that the data to be spliced and the oral cavity data are aligned.

[0148] Among them, after the data to be spliced and the oral data are successfully spliced, the data to be spliced and the oral data will be converted into a unified coordinate system.

[0149] S603: Triggering the head-mounted display camera to collect new oral cavity data of multiple areas.

[0150] The new oral data is used to rematch with the data to be spliced.

[0151] In the above process S601-S603, when any oral data does not match the data to be spliced, new oral data of multiple regions are recollected to ensure that the data to be spliced matches the new oral data of multiple regions and is successfully spliced.

[0152] like Figure 7 , which is a schematic diagram of the architecture of a head-mounted display provided in an embodiment of the present application, including the units shown below.

[0153] The data determination unit 100 is used to splice and match multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the encoded scanning rod frame point data.

[0154] The data splicing unit 200 is used to splice the corresponding standard data in the head display based on the feature points in the encoded scanning rod frame point data that match the standard data to generate first spliced data.

[0155] Optionally, the data splicing unit 200 is specifically used to: when obtaining the feature point data of each frame, determine the target encoding scanning rod that matches the feature point in the historical frame feature point data; for each feature point in each frame feature point data, if the feature point belongs to the target encoding scanning rod, mark the feature point as a matched point; if the feature point does not belong to the target encoding scanning rod, mark the feature point as an unmatched point; match the unmatched point with the feature points of the encoding scanning rod pre-stored in the scanning rod database to obtain a corresponding matching result; if the matching result is a successful match, re-mark the unmatched point as a matched point; based on the matched points in the feature point data of each frame and the feature points in the historical frame feature point data, determine the feature points in the encoding scanning rod frame point data that match the standard data; based on the feature points in the encoding scanning rod frame point data that match the standard data, splice the corresponding standard data in the head display to generate first spliced data.

[0156] Optionally, the standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

[0157] The suggestion generating unit 300 is configured to generate corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data when the number of scanning rods in the first splicing data does not match the number of installed scanning rods.

[0158] Optionally, the suggestion generation unit 300 is specifically used to: when the number of scanning rods in the first spliced data does not match the number of scanning rods installed, determine the attribute information of the unmatched feature points in the encoded scanning rod frame point data; the attribute information includes at least position, distance, clarity and number of occurrences; based on the attribute information of the unmatched feature points, generate corresponding adjustment suggestions; wherein the adjustment suggestions include adjusting the patient's mouth posture, adjusting the encoding scanning rod spacing, prompting to re-download the scanning rod database and / or prompting to adjust the distance between the head display and the encoding scanning rod.

[0159] Optionally, the adjustment suggestion is used to prompt the head-mounted display wearer to perform corresponding adjustment actions so that the encoding scanning rod frame point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing encoding scanning rod.

[0160] The gingival alignment unit 400 is used to: obtain data to be spliced based on oral scanning data collected by the intraoral scanner; obtain oral data of multiple areas collected by the head-mounted display camera; the oral data includes feature point data of the encoding scanning rod in the area, and the oral data is image data and / or model data; splice the data to be spliced with the multiple oral data respectively to align the data to be spliced with the multiple oral data; splice the feature point data in the multiple oral data respectively with the encoding scanning rod frame point data to align the data to be spliced with the encoding scanning rod frame point data.

[0161] Optionally, the gum alignment unit 400 is specifically used to: for each oral data, determine whether the oral data matches the data to be spliced; if the oral data matches the data to be spliced, splice the data to be spliced with the oral data so that the data to be spliced is aligned with the oral data.

[0162] Optionally, the gum alignment unit 400 is further used to: if the oral data does not match the data to be spliced, trigger the head-mounted camera to collect new oral data of multiple areas; the new oral data is used to re-match with the data to be spliced.

[0163] Optionally, the data to be spliced includes gum model data, and the gum model data is used to represent the corresponding gum model.

[0164] Optionally, the oral data further includes light spot features pre-added to the area, where the light spot features include optical features projected onto the area using a third-party optical device.

[0165] The point display unit 500 is used to display matched points and unmatched points on the head display; wherein the display effect of the matched points is different from that of the unmatched points.

[0166] Each of the units shown above uses a head-mounted display to replace the extraoral scanner to obtain the coded scanning rod framework point data, and uses the feature points in the coded scanning rod framework point data that match the standard data to generate the first spliced data to assist in tracking the scanning progress of the coded scanning rod, and uses the unmatched feature points to generate adjustment suggestions to assist the dentist in detecting whether there are any omissions in the coded scanning rod scanning, thereby lowering the operational threshold of the impression-taking process.

[0167] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the head-mounted display-based digital modeling method provided by the present application.

[0168] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0169] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the disclosure herein is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A digital modeling method based on a head-mounted display, characterized in that: include: The multi-frame feature point data collected in real time by the head-mounted display camera is spliced and matched to obtain the frame point data of the encoding scanning rod; Based on the feature points in the encoded scanning rod frame point data that match the standard data, the corresponding standard data are spliced in the head display to generate first spliced data; When the number of scanning rods in the first splicing data does not match the number of scanning rods to be installed, generating corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data; Obtaining data to be spliced based on oral scan data collected by an intraoral scanner; Obtaining oral cavity data of multiple areas captured by the head-mounted display camera; the oral cavity data includes feature point data of a coding scanning rod within the areas, and the oral cavity data is image data and / or model data; Splicing the data to be spliced with the plurality of oral cavity data respectively, so that the data to be spliced and the plurality of oral cavity data are aligned; The feature point data in the plurality of oral data are respectively spliced with the encoding scanning rod frame point data so that the data to be spliced is aligned with the encoding scanning rod frame point data.

2. The method according to claim 1, characterized in that Splicing the data to be spliced with the plurality of oral data respectively so that the data to be spliced and the plurality of oral data are aligned, comprising: For each of the oral data, determining whether the oral data matches the data to be spliced; If the oral cavity data matches the data to be spliced, the data to be spliced and the oral cavity data are spliced so that the data to be spliced and the oral cavity data are aligned.

3. The method according to claim 2, characterized in that The method further comprises: If the oral data does not match the data to be spliced, the head-mounted display camera is triggered to collect new oral data of the multiple areas; the new oral data is used to re-match with the data to be spliced.

4. The method according to claim 1, wherein Based on feature points in the encoded scanning rod frame point data that match the standard data, corresponding standard data are spliced in the head display to generate first spliced data, including: When obtaining feature point data of each frame, determining a target encoding scanning rod that matches a feature point in the feature point data of the historical frame; For each feature point in each frame of feature point data, if the feature point belongs to the target encoding scanning rod, mark the feature point as a matched point; if the feature point does not belong to the target encoding scanning rod, mark the feature point as an unmatched point; Matching the unmatched points with the characteristic points of the encoded scanning rod pre-stored in the scanning rod database to obtain corresponding matching results; If the matching result is a successful match, re-mark the unmatched point as a matched point; Based on the matched points in the feature point data of each frame and the feature points in the feature point data of the historical frame, the feature points in the frame point data of the encoding scanning rod are determined to match the standard data; Based on the feature points in the encoded scanning rod frame point data that match the standard data, the corresponding standard data is spliced in the head display to generate first spliced data.

5. The method according to claim 1, wherein When the number of scanning rods in the first splicing data does not match the number of installed scanning rods, generating corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data, including: When the number of scanning rods in the first spliced data does not match the number of installed scanning rods, determining attribute information of unmatched feature points in the encoded scanning rod frame point data; the attribute information includes at least position, distance, clarity, and number of occurrences; generating corresponding adjustment suggestions based on the attribute information of the unmatched feature points; The adjustment suggestions include adjusting the patient's mouth posture, adjusting the distance between the encoding scanning rods, prompting to re-download the scanning rod database and / or prompting to adjust the distance between the head display and the encoding scanning rods.

6. The method according to claim 4, characterized in that The method further comprises: The matched points and the unmatched points are displayed on a head display; wherein the display effect of the matched points is different from that of the unmatched points.

7. The method according to claim 1, characterized in that The standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

8. The method according to claim 1, characterized in that The adjustment suggestion is used to prompt the head-mounted display wearer to perform corresponding adjustment actions so that the coding scanning rod frame point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing coding scanning rod.

9. The method according to claim 1, characterized in that The data to be spliced includes gum model data, and the gum model data is used to represent the corresponding gum model.

10. The method according to claim 1, characterized in that The oral data further includes light spot features pre-added to the area, where the light spot features include optical features projected onto the area using a third-party optical device.

11. A head-mounted display, characterized in that: include: A data determination unit is used to splice and match multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the frame point data of the encoding scanning rod; a data splicing unit, configured to splice corresponding standard data in the head-mounted display based on feature points in the encoded scanning rod frame point data that match the standard data, to generate first spliced data; A suggestion generating unit, configured to generate corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data when the number of scanning rods in the first splicing data does not match the number of scanning rods to be installed; An alignment unit is configured to: obtain data to be spliced based on oral scanning data acquired by an intraoral scanner; obtain oral data of multiple areas acquired by the head-mounted display camera; the oral data includes feature point data of a coding scanning rod within the area, and the oral data is image data and / or model data; splice the data to be spliced with the multiple oral data respectively, so that the data to be spliced are aligned with the multiple oral data; splice the feature point data in the multiple oral data respectively with the coding scanning rod frame point data, so that the data to be spliced are aligned with the coding scanning rod frame point data.

12. A digital imaging system based on a head-mounted display, characterized in that: It comprises an intraoral scanner and a head display, wherein the intraoral scanner is used to collect oral scan data of a patient, and the head display is used to execute the head display-based digital impression taking method described in any one of claims 1 to 10.

13. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program is executed by a processor to execute the head-mounted display-based digital modeling method according to any one of claims 1 to 10.

Images