Digital modulus taking method and system based on head-mounted display and head-mounted display

Abstract

The invention discloses a digital modulus taking method and system based on a head-mounted display and the head-mounted display, and the method comprises the steps: carrying out the splicing and matching of multi-frame feature point data collected by a head-mounted display camera in real time, so as to obtain coding scanning rod frame point data; splicing corresponding standard data in the head-mounted display based on feature points matched with the standard data in the coded scanning rod frame point data to generate first spliced data; and when the number of the scanning rods in the first spliced data does not accord with the installation number of the scanning rods, generating a corresponding adjustment suggestion based on unmatched feature points in the coded scanning rod frame point data. According to the method, a head-mounted display is used for replacing an extraoral scanner, feature points matched with standard data in frame point data of a coding scanning rod are used for generating first splicing data so as to assist in tracking the scanning progress of the coding scanning rod, and unmatched feature points are used for generating adjustment suggestions so as to assist a dentist to detect whether the coding scanning rod is not scanned or not; therefore, the operation threshold of the mold taking process is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of digital modeling, and in particular to a head-mounted display-based digital modeling method, system, and head-mounted display. Background Art

[0002] Oral dental implants are a type of tooth restoration technology. In cases of tooth loss, impaired function, aesthetic needs or other specific situations, dental implants are an effective restoration method. The entire process of dental implant technology is as follows: local anesthesia, gingival incision, exposure of the alveolar bone, preparation of an implant socket on the alveolar bone (i.e. drilling a hole that matches the implant) to implant the implant, waiting for bone integration (after the implant is implanted, it takes a while (usually 3-6 months) to allow the implant to gradually integrate with the surrounding bone tissue and become stable), installation of a healing abutment, mold taking, crown making, and crown installation. The purpose of the mold taking step is to determine the positional relationship of the implant in the mouth and the relative positional relationship between multiple implants. Based on these positional relationships, bridges and crowns can be made.

[0003] According to the different ways of taking impressions, they are divided into traditional impressions and digital impressions. Traditional impressions usually use alginate to make the primary impression, and then use this primary impression to make individual trays for secondary impressions. Traditional impression technology has some limitations, such as cumbersome operating procedures, time-consuming, difficult to control impression pressure, high technical sensitivity, and the impression has problems of deformation and difficulty in preservation. Compared with digital impression technology, traditional impression technology may be insufficient in terms of accuracy and efficiency.

[0004] Digital impression technology uses an oral digital impression device to take implant impressions. Since the implant is in the alveolar bone and cannot be scanned directly, a scanning rod needs to be installed on the implant. The scanning rod is scanned with an impression device to obtain the position of the scanning rod, which also obtains the position of the implant.

[0005] The scanning rod is divided into ordinary scanning rod and coded scanning rod. The scanning modes and precisions of these two scanning rods are different. The ordinary scanning rod is a small rod made of metal or other materials. It is generally cylindrical and has some features such as steps and beveled surfaces, which are convenient for the impression device to identify features when scanning. During scanning, the scanning rod is installed on the intraoral implant and scanned together with the jaw to obtain grid data. Since the ordinary scanning rod is small in size and is generally made of metal, it has serious reflections, and the data quality is often poor during scanning. In addition, when multiple ordinary scanning rods are scanned together with the jaw, the overall deformation of the jaw data may be large, resulting in the positional relationship between multiple ordinary scanning rods being inaccurate, and the error can reach hundreds of microns. In actual use, the digital impression process of the ordinary scanning rod is as follows: 1. Use an intraoral scanner to scan the gums to obtain a three-dimensional model of the gums; 2. The patient wears a mouth expander, connects the ordinary scanning rod to the implant or abutment, and then scans.

[0006] The coding scanning rod has characteristic points (such as Figure 8 As shown in the figure, only feature points are identified during scanning, and the overall model data of the rod does not need to be scanned. Multiple rods are close to each other through crossbars, and there is no need to splice and transition through gum data. Therefore, the final positioning accuracy of the rod can reach tens of microns. In actual use, the digital modeling process of the coding scanning rod is as follows: 1. Use an intraoral scanner to scan the gums to obtain a three-dimensional model A of the gums; 2. The patient wears a mouth expander, connects the coding scanning rod to the implant or base, and then uses an intraoral scanner (or an extraoral scanner) to scan the feature point data B on the rod to determine the relative position relationship between each rod; 3. Use an intraoral scanner (or an extraoral scanner) to scan a small part of the gum data C and feature point data D at the same time, which is used to align the three-dimensional model A with the feature point data B to determine the position relationship between the rod and the gum; among them, the third step and the second step can be performed at the same time, that is, the gum data C and the feature point data B are scanned at the same time, the gum data C is used to splice the three-dimensional model A, and the feature point data B is directly used as the final rod data.

[0007] Traditional impression-taking is complicated, its accuracy is greatly affected by the operator's experience level, and the process is difficult to digitize. The digital impression-taking process of ordinary scanning rods solves the shortcomings of traditional impression-taking, but the accuracy is still not good enough, so it requires dentists to have a high level of professional knowledge to effectively complete the impression-taking work. Summary of the invention

[0008] The present application provides a head-mounted display-based digital impression-taking method, system and head-mounted display, the purpose of which is to use the head-mounted display to assist dentists in impression-taking, so as to lower the operational threshold for impression-taking.

[0009] In order to achieve the above objectives, this application provides the following technical solutions:

[0010] A digital modeling method based on head display, comprising:

[0011] The multi-frame feature point data collected in real time by the head-mounted display camera are spliced ​​and matched to obtain the frame point data of the encoding scanning rod;

[0012] Based on the feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data are spliced ​​in the head display to generate first spliced ​​data;

[0013] When the number of scanning rods in the first splicing data does not match the number of scanning rods to be installed, a corresponding adjustment suggestion is generated based on unmatched feature points in the encoded scanning rod frame point data.

[0014] Optionally, the method further includes:

[0015] Based on the oral scan data collected by the intraoral scanner, data to be spliced ​​are obtained;

[0016] Obtaining oral cavity data of multiple areas collected by the head-mounted display camera; the oral cavity data includes feature point data of the encoding scanning rod in the area, and the oral cavity data is image data and / or model data;

[0017] Splicing the data to be spliced ​​with the plurality of oral cavity data respectively, so that the data to be spliced ​​is aligned with the plurality of oral cavity data;

[0018] The feature point data in the plurality of oral data are spliced ​​with the encoding scanning rod framework point data respectively, so that the data to be spliced ​​is aligned with the encoding scanning rod framework point data.

[0019] Optionally, the data to be spliced ​​are spliced ​​with the plurality of oral cavity data respectively so that the data to be spliced ​​are aligned with the plurality of oral cavity data, including:

[0020] For each of the oral data, determining whether the oral data matches the data to be spliced;

[0021] If the oral cavity data matches the data to be spliced, the data to be spliced ​​and the oral cavity data are spliced ​​so that the data to be spliced ​​and the oral cavity data are aligned.

[0022] Optionally, the method further includes:

[0023] If the oral cavity data does not match the data to be spliced, the head-mounted display camera is triggered to collect new oral cavity data of the multiple areas; the new oral cavity data is used to re-match with the data to be spliced.

[0024] Optionally, based on feature points in the encoding scanning rod frame point data that match the standard data, corresponding standard data are spliced ​​in the head display to generate first spliced ​​data, including:

[0025] When obtaining feature point data of each frame, determining a target encoding scanning rod that matches a feature point in the feature point data of the historical frame;

[0026] For each feature point in each frame of feature point data, if the feature point belongs to the target coding scanning rod, the feature point is marked as a matched point; if the feature point does not belong to the target coding scanning rod, the feature point is marked as an unmatched point;

[0027] Matching the unmatched points with the feature points of the encoded scanning rod pre-stored in the scanning rod database to obtain corresponding matching results;

[0028] If the matching result is a successful match, re-mark the unmatched point as a matched point;

[0029] Based on the matched points in the feature point data of each frame and the feature points in the feature point data of the historical frame, feature points in the frame point data of the encoding scanning rod that match the standard data are determined;

[0030] Based on the feature points in the encoded scanning rod frame point data that match the standard data, the corresponding standard data is spliced ​​in the head display to generate first spliced ​​data.

[0031] Optionally, when the number of scanning rods in the first splicing data does not match the number of scanning rod installations, generating corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data includes:

[0032] When the number of scanning rods in the first splicing data does not match the number of scanning rod installations, determining the attribute information of the unmatched feature points in the coded scanning rod frame point data; the attribute information includes at least position, distance, clarity, and number of occurrences;

[0033] generating corresponding adjustment suggestions based on the attribute information of the unmatched feature points;

[0034] The adjustment suggestions include adjusting the patient's mouth posture, adjusting the distance between the encoding scanning rods, prompting to re-download the scanning rod database and / or prompting to adjust the distance between the head display and the encoding scanning rod.

[0035] Optionally, the method further includes:

[0036] The matched points and the unmatched points are displayed on a head display; wherein the display effect of the matched points is different from that of the unmatched points.

[0037] Optionally, the standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

[0038] Optionally, the adjustment suggestion is used to prompt the head display wearer to perform corresponding adjustment actions so that the coding scanning rod framework point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing coding scanning rod.

[0039] Optionally, the data to be spliced ​​includes gum model data, and the gum model data is used to characterize the corresponding gum model.

[0040] Optionally, the oral data further includes light spot features pre-added on the area, and the light spot features include optical features projected on the area using a third-party optical device.

[0041] A head display, comprising:

[0042] A data determination unit, used for splicing and matching multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the frame point data of the encoding scanning rod;

[0043] A data splicing unit, configured to splice corresponding standard data in the head display based on feature points in the encoding scanning rod frame point data that match the standard data, to generate first spliced ​​data;

[0044] A suggestion generating unit is used to generate corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data when the number of scanning rods in the first splicing data does not match the number of scanning rod installations.

[0045] A head-mounted display-based digital impression-taking system comprises an intraoral scanner and a head-mounted display, wherein the intraoral scanner is used to collect oral scan data of a patient, and the head-mounted display is used to execute the above-mentioned head-mounted display-based digital impression-taking method.

[0046] A storage medium includes a stored program, wherein the program is executed by a processor to execute the head-mounted display-based digital modeling method.

[0047] The technical solution provided by the present application is to splice and match multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the coded scanning rod framework point data. Based on the feature points in the coded scanning rod framework point data that match the standard data, the corresponding standard data is spliced ​​in the head-mounted display to generate the first spliced ​​data. When the number of scanning rods in the first spliced ​​data does not match the number of scanning rods installed, the corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod framework point data. The present application uses a head-mounted display to replace the extraoral scanner to obtain the coded scanning rod framework point data, and uses the feature points in the coded scanning rod framework point data that match the standard data to generate the first spliced ​​data to assist in tracking the scanning progress of the coded scanning rod, and uses the unmatched feature points to generate adjustment suggestions to assist dentists in detecting whether there are any missing coded scanning rods that have not been scanned, thereby lowering the operational threshold of the molding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of the architecture of a head-mounted display-based digital modeling system provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a flow chart of a head-mounted display-based digital modeling method provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of a flow chart of another head-mounted display-based digital modeling method provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of the architecture of a head-mounted display provided in an embodiment of the present application;

[0056] Figure 8A schematic diagram of a coding scanning rod provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0059] like Figure 1 As shown, it is a schematic diagram of the architecture of a head-mounted display-based digital modeling system provided in an embodiment of the present application, which includes the following modules.

[0060] Intraoral scanner 100 and head display 200, the intraoral scanner 100 is used to collect the patient's gum scanning data, and the head display 200 is used to execute the head display-based digital impression taking method provided in the present application.

[0061] In some examples, the types of the head display 200 include, but are not limited to, head-mounted displays (such as virtual reality devices, augmented display devices, mixed display devices) and smart glasses. Generally speaking, the head display 200 and its operating system provide basic hardware and computing resources required for three-dimensional modeling, and the basic hardware includes, but is not limited to, cameras (such as binocular cameras), laser radars, and structured light scanning components.

[0062] It should be noted that the embodiment of the present application utilizes the head display 200 to replace the extraoral scanner to perform the scanning work of the coding scanning rod on the patient's gums. To this end, it is necessary to ensure that the camera of the head display 200 has reliable accuracy. If the head display 200 has not been calibrated or a long time has passed since the last camera calibration, the head display 200 needs to be calibrated before using the head display 200 to assist the dentist in taking impressions.

[0063] In some examples, the purpose of camera calibration is to calculate the intrinsic and extrinsic parameters of the head-mounted display camera. When using a binocular reconstruction algorithm to reconstruct and stitch multiple frames of feature point data to obtain the encoding scanning rod frame point data, the intrinsic and extrinsic parameters of the head-mounted display camera are required.

[0064] In a possible implementation, the steps for implementing camera calibration are as follows.

[0065] Step 1: Prepare the calibration object, usually a calibration plate with known size and pattern (such as a checkerboard or dot array).

[0066] Step 2: Trigger the head-mounted display camera to capture calibration images. Specifically, place the calibration plate at different angles and positions, capture multiple calibration images, and ensure that the calibration plate is clearly visible in each calibration image.

[0067] Step 3: Process the calibration image. Specifically, use image processing software (such as OpenCV) to detect the calibration image to obtain feature points.

[0068] Step 4: Calculate the camera parameters (i.e., intrinsic and extrinsic parameters) using the feature points. Specifically, based on the obtained feature points, use a calibration algorithm to calculate the intrinsic and extrinsic parameters of the head-mounted display camera. The intrinsic and extrinsic parameters include but are not limited to focal length, principal point coordinates, distortion coefficients, etc.

[0069] Step 5: Verify the calibration results. Specifically, verify the accuracy of the calibration results (intrinsic and extrinsic parameters) through indicators such as reprojection error. If the error is large, retake the calibration image or adjust the calibration parameters.

[0070] After the calibration of the head-mounted display camera is confirmed, the dentist can wear the head-mounted display to take impressions. The head-mounted display camera will capture the image from the dentist's current perspective and display the image on the screen. In the embodiment of the present application, the head-mounted display-based digital impression taking method guides the dentist to use the head-mounted display to replace the extraoral scanner to achieve the work of the scanning coding scanning rod. Figure 2 The steps are shown and the corresponding explanations.

[0071] like Figure 2 As shown, it is a flow chart of a head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0072] S201: Splicing and matching multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the frame point data of the encoding scanning rod.

[0073] Among them, after the patient wears the mouth expander and installs the coded scanning rod on the implant or abutment, the headset can reconstruct and splice and match multiple frames of feature point data through feature point reconstruction and matching algorithms to obtain the coded scanning rod framework point data. The implementation principle can be understood as using multiple pictures or multiple groups of local data (i.e. frame feature point data, or scanning rod image frames) to form a large picture or complete data (i.e. coded scanning rod framework point data, complete scanning rod image). Each time, the headset will only display a part of the large picture or complete data, so that the dentist can find the location of all pictures or data in the fewest possible times.

[0074] In some examples, the encoding scanning rod frame point data includes information such as the location of feature points on the encoding scanning rod and whether they match. Generally speaking, before obtaining the first frame of feature point data, the encoding scanning rod frame point data is empty.

[0075] In a possible implementation, the head display collects the scanning rod image through a binocular camera. Since the coded scanning rod carries some specially designed marker point features (which can be regarded as feature points), the marker points of the coded scanning rod have coded information, and their corresponding position coordinate points can be obtained during the intraoral scanning process. Therefore, binocular stereo vision and three-dimensional reconstruction algorithms can be used to reconstruct and extract the feature points on the coded scanning rod to obtain real-time multi-frame feature point data, thereby realizing precise positioning of the scanning rod.

[0076] It should be noted that when obtaining feature point data of any frame, if the current encoding scanning rod framework point data is not empty, it is necessary to splice and match the current frame feature point data and the current encoding scanning rod framework point data (obtained after splicing and matching the historical frame feature point data) to obtain more complete encoding scanning rod framework point data.

[0077] S202: Based on feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data is spliced ​​in the head display to generate first spliced ​​data.

[0078] The standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

[0079] It should be noted that the coding scanning rod is unique. It is only necessary to determine more than a certain number of feature points, such as more than 6 feature points, to match the corresponding coding scanning rod from the scanning rod database, thereby determining the position and direction of the coding scanning rod. For this reason, the coding scanning rod frame point data must contain redundant feature points. For the real-time updated coding scanning rod frame point data, the feature points in the current frame feature point data may be redundant feature points of the previously matched coding scanning rod, so the target coding scanning rod that has been determined to be matched can be traversed in advance. If the feature points in the current frame feature point data belong to the target coding scanning rod, the feature points in the current frame feature point data can be marked as matched points, thereby determining the feature points in the coding scanning rod frame point data that match the standard data.

[0080] Optionally, based on the feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data is spliced ​​in the head display to generate the first spliced ​​data. For details, see Figure 3 The steps are shown and the corresponding explanations.

[0081] S203: When the number of scanning rods in the first splicing data does not match the number of scanning rod installations, generating corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data.

[0082] Among them, the adjustment suggestion is used to prompt the wearer of the headset to perform corresponding adjustment actions so that the framework point data adds feature points that match the missing data, and the missing data includes the standard data part corresponding to the missing encoding scanning rod.

[0083] It can be understood that when the number of scanning rods in the first splicing data does not match the number of scanning rods installed, it can be considered that some of the coding scanning rods installed in the patient's gums have failed to be modeled successfully, that is, there are scanning rods that have not been spliced ​​successfully. The coding scanning rods that have not been successfully modeled can be regarded as missing coding scanning rods.

[0084] In some examples, the reasons for missing coding scanning rods during the scanning process include but are not limited to: the feature points on the missing coding scanning rods are blocked, the head display camera fails to capture the feature points on the missing coding scanning rods due to shooting angle problems, etc.

[0085] To this end, when the number of scanning rods in the first stitching data does not match the number of scanning rods installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data. This can provide corresponding solutions to the reasons for the missing coded scanning rods during the scanning process, thereby helping dentists to complete the modeling of missing coded scanning rods in a timely and effective manner.

[0086] Optionally, when the number of scanning rods in the first splicing data does not match the number of scanning rods installed, the implementation process of generating corresponding adjustment suggestions based on the unmatched feature points in the coded scanning rod frame point data can be seen in Figure 4 The steps are shown and the corresponding explanations.

[0087] It should be noted that in the process of digital impression taking, in addition to scanning the encoding scanning rod frame point data corresponding to all the encoding scanning rods on the patient's gums, the patient's gum model must be aligned with the encoding scanning rod frame point data to determine the positional relationship between the encoding scanning rod and the patient's gums.

[0088] Optionally, the process of aligning the patient's gum model with the framework point data using a head mounted display can be seen in Figure 5 The steps are shown and the corresponding explanations.

[0089] In the process shown in the above S201-S203, a head-mounted display is used to replace the extraoral scanner to obtain the coding scanning rod framework point data, and the feature points in the coding scanning rod framework point data that match the standard data are used to generate the first spliced ​​data to assist in tracking the scanning progress of the coding scanning rod, and the unmatched feature points in the coding scanning rod framework point data are used to generate corresponding adjustment suggestions to assist the dentist in detecting whether there are any omissions in the coding scanning rod that have not been scanned, thereby effectively lowering the operational threshold of the mold taking process.

[0090] like Figure 3 As shown, it is a schematic diagram of the process of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0091] S301: When obtaining feature point data of each frame, determining a target encoding scanning rod that matches a feature point in feature point data of a historical frame.

[0092] Among them, for each frame of feature point data, the target coding scanning rod that matches the feature point in the historical frame feature point data can be understood as the target coding scanning rod that matches the current coding scanning rod frame point data. That is to say, at least one feature point in the current coding scanning rod frame point data belongs to the target coding scanning rod, and the number of target coding scanning rods is one or more.

[0093] In a possible implementation manner, the target code scanning rod may be regarded as a code scanning rod that has been determined to be matched in the historical time.

[0094] S302: For each feature point in each frame of feature point data, determine whether the feature point belongs to the target encoding scanning rod.

[0095] If the feature point belongs to the target coding scanning rod, execute S303; if the feature point does not belong to the target coding scanning rod, execute S304.

[0096] S303: Mark the feature point as a matched point.

[0097] After executing S303, continue to execute S307.

[0098] S304: Mark the feature point as an unmatched point.

[0099] After executing S304, continue to execute S305.

[0100] Optionally, the matched points and the unmatched points may also be displayed in the head display, wherein the display effect of the matched points is different from that of the unmatched points.

[0101] In some examples, the display effect of the matched points may be: displayed in the form of green dots. The display effect of the unmatched points may be: displayed in the form of red dots.

[0102] It can be understood that displaying matched points and unmatched points in the head display allows dentists to know the scanning effects of the feature points on the encoded scanning rod in real time.

[0103] S305: Match the unmatched points with the feature points of the encoded scanning rod pre-stored in the scanning rod database to obtain corresponding matching results.

[0104] Among them, the unmatched points are matched with the feature points of the coded scanning rods pre-stored in the scanning rod database. If there is a new coded scanning rod matching the unmatched point in the scanning rod database, the matching result corresponding to the unmatched point can be determined as a successful match. If there is no new coded scanning rod matching the unmatched point in the scanning rod database, the matching result corresponding to the unmatched point can be determined as a failed match. For the determined unmatched point set, each unmatched point in the unmatched point set can be matched with the feature points shown in the scanning rod database in turn until a new coded scanning rod can no longer be determined from the scanning rod database.

[0105] S306: If the matching result is successful, re-mark the unmatched points as matched points.

[0106] After executing S306, continue to execute S307.

[0107] S307: Based on the matched points in each frame of feature point data and the feature points in the historical frame feature point data, feature points in the encoding scanning rod frame point data that match the standard data are determined.

[0108] Among them, based on the matched points in each frame feature point data and the feature points in the historical frame feature point data that match the target coding scanning rod, the feature points in the coding scanning rod framework point data that match the standard data are determined, which can effectively determine the matched points in the real-time updated coding scanning rod framework point data.

[0109] S308: Based on the feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data is spliced ​​in the head display to generate first spliced ​​data.

[0110] Among them, the feature points in the coded scanning rod frame point data that match the standard data are spliced ​​in the head display to generate the first splicing data, in order to splice the corresponding standard data in the scanning rod image displayed by the head display, so as to identify the relative position relationship between multiple coded scanning rods in the scanning rod image.

[0111] It should be noted that splicing the corresponding standard data on the scanning rod image displayed on the head display can help the dentist track the scanning progress of the coding scanning rod. Generally speaking, after the standard data corresponding to each coding scanning rod installed on the patient's gums are determined and spliced, the first spliced ​​data obtained by splicing can be saved to determine the relative position relationship between each coding scanning rod.

[0112] The process shown in S301-S308 above utilizes the matched points in the real-time updated encoding scanning rod frame point data to splice the corresponding standard data in the head display to generate the first spliced ​​data, thereby assisting the dentist to track the scanning progress of the encoding scanning rod.

[0113] like Figure 4 As shown, it is a schematic diagram of the process of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0114] S401: When the number of scanning rods in the first splicing data does not match the number of installed scanning rods, determine the attribute information of unmatched feature points in the coded scanning rod frame point data.

[0115] The attribute information includes at least position, distance, clarity and number of occurrences. In the splicing process of the first splicing data, there are unmatched feature points (i.e., unmatched points) in the frame point data of the coding scanning rod, and the unmatched points do not participate in the splicing process of the first splicing data. Therefore, the reason for the missing coding scanning rod in the scanning process can be found by analyzing the attribute information of the unmatched points.

[0116] S402: Generate corresponding adjustment suggestions based on the attribute information of the unmatched feature points.

[0117] The adjustment suggestions include adjusting the patient's mouth posture, adjusting the encoding scanning rod spacing, prompting to re-download the scanning rod database, and / or prompting to adjust the distance between the head display and the encoding scanning rod.

[0118] It should be noted that based on the attribute information of the unmatched feature points, the reasons why the unmatched points do not match the standard data can be analyzed and determined, such as incorrect patient mouth posture, too far distance between two adjacent encoding scanning rods, too far distance between the head display and the encoding scanning rod on the patient's gums, and the unmatched points do not meet the feature point standards in the scanning rod database.

[0119] In some examples, based on the attribute information of the unmatched feature points, it is determined that the patient's mouth posture is incorrect, so an adjustment suggestion can be generated to prompt the patient to adjust the mouth posture.

[0120] In some examples, based on the attribute information of the unmatched feature points, it is determined that the distance between two adjacent coding scanning rods is too far, so an adjustment suggestion can be generated to prompt the dentist to adjust the distance between the coding scanning rods so that the two adjacent coding scanning rods are closer to each other.

[0121] In some examples, based on the attribute information of the unmatched feature points, it is determined that the head display is too far away from the encoding scanning rod on the patient's gums, so an adjustment suggestion can be generated to prompt the dentist to get closer to the encoding scanning rod to shorten the distance between the head display and the encoding scanning rod.

[0122] In some examples, based on the attribute information of the unmatched feature points, it is determined that the unmatched points do not meet the feature point standards in the scanning rod database. For example, the coded scanning rods of another set of coded scanning rods are scanned together, resulting in the inability to identify the missing coded scanning rods. Therefore, an adjustment suggestion can be generated to prompt you to re-download the scanning rod database of another set of coded scanning rods.

[0123] The process shown in the above S401-S402 guides the dentist to perform corresponding adjustment actions according to the adjustment suggestions based on the attribute information of the unmatched feature points in the coding scanning rod framework point data, thereby ensuring that all the coding scanning rods on the patient's gums can complete the corresponding standard data splicing and realize the complete modeling of each coding scanning rod.

[0124] like Figure 5 As shown, it is a schematic diagram of the process of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0125] S501: Obtaining data to be spliced ​​based on oral scan data collected by an intraoral scanner.

[0126] The data to be spliced ​​include gum model data, and the gum model data is used to represent the corresponding gum model.

[0127] It should be noted that if the patient's gums are edentulous, since the edentulous gums lack features, an intraoral scanner can be used to scan the patient's mouth by projecting structured light to obtain oral scan data. A transmission link between the intraoral scanner and the head display is established in advance so that the intraoral scanner can transmit the oral scan data to the head display through the transmission link. After the head display obtains the oral scan data, the rendering engine on the head display is used to draw the oral scan data in real time to obtain the data to be spliced.

[0128] In some examples, the headset can display the patient's gum model data in real time to assist the dentist in understanding the patient's mouth.

[0129] S502: Obtain oral cavity data of multiple areas collected by a head-mounted display camera.

[0130] The oral cavity data includes feature point data of the encoding scanning rod in the area, and the oral cavity data is image data and / or model data.

[0131] In some examples, the so-called area may be understood as part of the gum area.

[0132] In the process of digital impression taking, in order to determine the positional relationship between the encoding scanning rod and the patient's gums, it is also necessary to obtain a small section of the gum model (ie, oral data) and the encoding scanning rod frame point data for splicing.

[0133] Optionally, the oral data also includes light spot features pre-added to the area, and the light spot features include optical features projected on the area using a third-party optical device. The third-party optical device is preset with a light projection structure that can project corresponding optical features on the area to help the head-mounted display camera collect oral data from multiple areas.

[0134] It should be noted that because the gums of edentulous jaws lack features, a preset light projection structure is used to add light spot features to the area so that the oral data of multiple areas collected by the head-mounted display camera have corresponding feature data. Parallel light and the rendering engine on the head-mounted display are used to draw the feature data in the oral data in real time to obtain the oral data of the corresponding area.

[0135] In some examples, the third-party optical device may be a light pen, and the light spot projected by the light pen may be moved to the corresponding area first, and then the head display may be used to collect oral data of multiple areas. At this time, the oral data includes feature point data of the gums and the encoding scanning rod, so the oral data and the encoding scanning rod frame point data may be spliced ​​simultaneously.

[0136] In addition, since the oral data comes from the intraoral scanner, there is a deviation between the data to be spliced ​​and the oral data (that is, the coordinate systems of the data to be spliced ​​and the oral data are different). In order to reduce the error, oral data of multiple areas (such as the right side of the gums, the middle of the gums, and the right side of the gums) are collected. By subdividing the oral data of multiple areas to provide more reference feature points, the splicing between the data to be spliced ​​and the oral data is ensured to be more stable and reliable.

[0137] In some examples, the division of multiple regions can be set by the technician according to the actual situation. Specifically, the multiple regions can include the left side of the gum, the middle of the gum, and the right side of the gum. After multiple splicing of different regions, more constraints are obtained, which can be used to optimize the alignment relationship between the data to be spliced ​​and the oral data, so that the overall splicing error is minimized.

[0138] S503: splicing the data to be spliced ​​with the multiple oral cavity data respectively, so that the data to be spliced ​​and the multiple oral cavity data are aligned.

[0139] The data to be spliced ​​are spliced ​​with the multiple oral data respectively so that the data to be spliced ​​are aligned with the multiple oral data. In essence, the multiple oral data and the data to be spliced ​​are converted into a unified coordinate system.

[0140] Optionally, the data to be spliced ​​are spliced ​​with the multiple oral data respectively, so that the data to be spliced ​​are aligned with the multiple oral data. Figure 6 The steps are shown and the corresponding explanations.

[0141] S504: The feature point data in the plurality of oral cavity data are respectively spliced ​​and matched with the frame point data of the encoding scanning rod, so that the data to be spliced ​​is aligned with the frame point data of the encoding scanning rod.

[0142] Among them, the feature point data shown in multiple oral data are spliced ​​and matched with the encoding scanning rod frame point data respectively, so that the feature point data in each oral data can be aligned with the encoding scanning rod frame point data. When the feature point data in each oral data is aligned with the encoding scanning rod frame point data, since the feature point data in each oral data is reconstructed in the head-mounted display camera coordinate system, it can be ensured that each oral data is aligned with the encoding scanning rod frame point data. Furthermore, since the data to be spliced ​​is aligned with multiple oral data, when it is determined that the feature point data in multiple oral data is aligned with the encoding scanning rod frame point data, it can be determined that the data to be spliced ​​and the encoding scanning rod frame point data are aligned to the same coordinate system.

[0143] The process shown in S501-S504 above utilizes the oral data of multiple regions and the splicing matching between the data to be spliced, as well as the splicing matching between the feature point data in the multiple oral data and the frame point data of the encoding scanning rod, so that the data to be spliced ​​and the frame point data of the encoding scanning rod are aligned to the same coordinate system. After multiple splicing on the left side of the gum, the middle of the gum, and the right side of the gum, more constraints are obtained, and the alignment relationship between the gum and the scanning rod can be optimized to minimize the overall error, thereby obtaining the positional relationship between the patient's gums and the encoding scanning rod.

[0144] like Figure 6 As shown, it is a schematic diagram of the process of another head-mounted display-based digital modeling method provided in an embodiment of the present application, including the following steps.

[0145] S601: For each oral cavity data, determine whether the oral cavity data matches the data to be spliced.

[0146] If the oral cavity data matches the data to be spliced, S602 is executed; if the oral cavity data does not match the data to be spliced, S603 is executed.

[0147] S602: Splicing the data to be spliced ​​with the oral cavity data so that the data to be spliced ​​and the oral cavity data are aligned.

[0148] Among them, after the data to be spliced ​​and the oral data are successfully spliced, the data to be spliced ​​and the oral data will be converted into a unified coordinate system.

[0149] S603: Trigger the head-mounted display camera to collect new oral cavity data of multiple areas.

[0150] The new oral data is used to rematch the data to be spliced.

[0151] In the above process S601-S603, when any oral data does not match the data to be spliced, new oral data of multiple regions are re-collected to ensure that the data to be spliced ​​matches and splices successfully with the new oral data of multiple regions.

[0152] like Figure 7 , which is a schematic diagram of the architecture of a head display provided in an embodiment of the present application, including the units shown below.

[0153] The data determination unit 100 is used to splice and match multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the encoding scanning rod frame point data.

[0154] The data splicing unit 200 is used to splice the corresponding standard data in the head display based on the feature points in the encoding scanning rod frame point data that match the standard data to generate first spliced ​​data.

[0155] Optionally, the data splicing unit 200 is specifically used to: when obtaining feature point data of each frame, determine the target encoding scanning rod that matches the feature point in the feature point data of the historical frame; for each feature point in the feature point data of each frame, if the feature point belongs to the target encoding scanning rod, mark the feature point as a matched point, if the feature point does not belong to the target encoding scanning rod, mark the feature point as an unmatched point; match the unmatched point with the feature points of the encoding scanning rod pre-stored in the scanning rod database to obtain a corresponding matching result; if the matching result is a successful match, re-mark the unmatched point as a matched point; based on the matched points in the feature point data of each frame and the feature points in the feature point data of the historical frame, determine the feature points in the encoding scanning rod framework point data that match the standard data; based on the feature points in the encoding scanning rod framework point data that match the standard data, splice the corresponding standard data in the head display to generate first spliced ​​data.

[0156] Optionally, the standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

[0157] The suggestion generating unit 300 is used to generate corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data when the number of scanning rods in the first splicing data does not match the number of scanning rod installations.

[0158] Optionally, the suggestion generation unit 300 is specifically used to: when the number of scanning rods in the first stitching data does not match the number of scanning rods installed, determine the attribute information of the unmatched feature points in the encoded scanning rod frame point data; the attribute information includes at least position, distance, clarity and number of occurrences; based on the attribute information of the unmatched feature points, generate corresponding adjustment suggestions; wherein the adjustment suggestions include adjusting the patient's mouth posture, adjusting the encoding scanning rod spacing, prompting to re-download the scanning rod database and / or prompting to adjust the distance between the head display and the encoding scanning rod.

[0159] Optionally, the adjustment suggestion is used to prompt the head display wearer to perform corresponding adjustment actions so that the encoding scanning rod framework point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing encoding scanning rod.

[0160] The gingival alignment unit 400 is used to: obtain data to be spliced ​​based on oral scanning data collected by an intraoral scanner; obtain oral data of multiple areas collected by a head-mounted display camera; the oral data includes feature point data of a coding scanning rod in the area, and the oral data is image data and / or model data; splice the data to be spliced ​​with the multiple oral data respectively, so that the data to be spliced ​​is aligned with the multiple oral data; splice the feature point data in the multiple oral data respectively with the coding scanning rod frame point data, so that the data to be spliced ​​is aligned with the coding scanning rod frame point data.

[0161] Optionally, the gingival alignment unit 400 is specifically used to: for each oral data, determine whether the oral data matches the data to be spliced; if the oral data matches the data to be spliced, splice the data to be spliced ​​with the oral data so that the data to be spliced ​​is aligned with the oral data.

[0162] Optionally, the gingival alignment unit 400 is also used to: if the oral data does not match the data to be spliced, trigger the head-mounted display camera to collect new oral data of multiple areas; the new oral data is used to re-match the data to be spliced.

[0163] Optionally, the data to be spliced ​​includes gum model data, and the gum model data is used to characterize the corresponding gum model.

[0164] Optionally, the oral data also includes light spot features pre-added to the area, and the light spot features include optical features projected onto the area using a third-party optical device.

[0165] The point display unit 500 is used to display the matched points and the unmatched points in the head display; wherein the display effect of the matched points is different from that of the unmatched points.

[0166] Each of the units shown above utilizes a head display to replace the extraoral scanner to obtain the coded scanning rod framework point data, utilizes the feature points in the coded scanning rod framework point data that match the standard data, generates the first spliced ​​data to assist in tracking the scanning progress of the coded scanning rod, and utilizes the unmatched feature points to generate adjustment suggestions to assist the dentist in detecting whether there are any omissions in the coded scanning rod that have not been scanned, thereby lowering the operational threshold of the impression-taking process.

[0167] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the head-mounted display-based digital modeling method provided by the present application.

[0168] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0169] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A digital modeling method based on head-mounted display, characterized in that: include: The multi-frame feature point data collected in real time by the head-mounted display camera are spliced ​​and matched to obtain the frame point data of the encoding scanning rod; Based on the feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data are spliced ​​in the head display to generate first spliced ​​data; When the number of scanning rods in the first splicing data does not match the number of scanning rods to be installed, a corresponding adjustment suggestion is generated based on unmatched feature points in the encoded scanning rod frame point data.

2. The method according to claim 1, characterized in that The method further comprises: Based on the oral scan data collected by the intraoral scanner, data to be spliced ​​are obtained; Obtaining oral cavity data of multiple areas collected by the head-mounted display camera; the oral cavity data includes feature point data of the encoding scanning rod in the area, and the oral cavity data is image data and / or model data; Splicing the data to be spliced ​​with the plurality of oral cavity data respectively, so that the data to be spliced ​​is aligned with the plurality of oral cavity data; The feature point data in the plurality of oral data are spliced ​​with the encoding scanning rod framework point data respectively, so that the data to be spliced ​​is aligned with the encoding scanning rod framework point data.

3. The method according to claim 2, characterized in that Splicing the data to be spliced ​​with the plurality of oral data respectively so that the data to be spliced ​​is aligned with the plurality of oral data, comprising: For each of the oral data, determining whether the oral data matches the data to be spliced; If the oral cavity data matches the data to be spliced, the data to be spliced ​​and the oral cavity data are spliced ​​so that the data to be spliced ​​and the oral cavity data are aligned.

4. The method according to claim 3, characterized in that The method further comprises: If the oral cavity data does not match the data to be spliced, the head-mounted display camera is triggered to collect new oral cavity data of the multiple areas; the new oral cavity data is used to re-match with the data to be spliced.

5. The method according to claim 1, characterized in that Based on the feature points in the encoding scanning rod frame point data that match the standard data, the corresponding standard data is spliced ​​in the head display to generate first spliced ​​data, including: When obtaining feature point data of each frame, determining a target encoding scanning rod that matches a feature point in the feature point data of the historical frame; For each feature point in each frame of feature point data, if the feature point belongs to the target coding scanning rod, the feature point is marked as a matched point; if the feature point does not belong to the target coding scanning rod, the feature point is marked as an unmatched point; Matching the unmatched points with the feature points of the encoded scanning rod pre-stored in the scanning rod database to obtain corresponding matching results; If the matching result is a successful match, re-mark the unmatched point as a matched point; Based on the matched points in the feature point data of each frame and the feature points in the feature point data of the historical frame, feature points in the frame point data of the encoding scanning rod that match the standard data are determined; Based on the feature points in the encoded scanning rod frame point data that match the standard data, the corresponding standard data is spliced ​​in the head display to generate first spliced ​​data.

6. The method according to claim 1, characterized in that When the number of scanning rods in the first splicing data does not match the number of scanning rod installations, generating corresponding adjustment suggestions based on unmatched feature points in the coded scanning rod frame point data, including: When the number of scanning rods in the first splicing data does not match the number of scanning rod installations, determining the attribute information of the unmatched feature points in the coded scanning rod frame point data; the attribute information includes at least position, distance, clarity, and number of occurrences; generating corresponding adjustment suggestions based on the attribute information of the unmatched feature points; The adjustment suggestions include adjusting the patient's mouth posture, adjusting the encoding scanning rod spacing, prompting to re-download the scanning rod database and / or prompting to adjust the distance between the head display and the encoding scanning rod.

7. The method according to claim 5, characterized in that The method further comprises: The matched points and the unmatched points are displayed on a head display; wherein the display effect of the matched points is different from that of the unmatched points.

8. The method according to claim 1, characterized in that The standard data includes standard scanning rod data, and the standard scanning rod data is used to identify the relative position relationship between multiple encoding scanning rods.

9. The method according to claim 1, characterized in that: The adjustment suggestion is used to prompt the head display wearer to perform corresponding adjustment actions so that the encoding scanning rod framework point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing encoding scanning rod.

10. The method according to claim 2, characterized in that The data to be spliced ​​includes gum model data, and the gum model data is used to characterize the corresponding gum model.

11. The method according to claim 2, characterized in that The oral cavity data further includes light spot features pre-added on the area, wherein the light spot features include optical features projected on the area using a third-party optical device.

12. A head mounted display, characterized in that: include: A data determination unit, used for splicing and matching multiple frames of feature point data collected in real time by the head-mounted display camera to obtain the frame point data of the encoding scanning rod; A data splicing unit, configured to splice corresponding standard data in the head display based on feature points in the encoding scanning rod frame point data that match the standard data, to generate first spliced ​​data; A suggestion generating unit is used to generate corresponding adjustment suggestions based on unmatched feature points in the encoded scanning rod frame point data when the number of scanning rods in the first splicing data does not match the number of scanning rod installations.

13. A digital imaging system based on head-mounted display, characterized in that: It comprises an intraoral scanner and a head display, wherein the intraoral scanner is used to collect oral scan data of a patient, and the head display is used to execute the head display-based digital impression taking method described in any one of claims 1-11.

14. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the head display-based digital modeling method described in any one of claims 1-11.

Images