AR-assisted multi-generation tooth extraction positioning method

Abstract

The invention discloses an AR-assisted multi-generation tooth extraction positioning method, and relates to the technical field of oral treatment, the positioning method combines a virtual reality technology and an oral and maxillofacial surgical operation, and uses an augmented reality technology to obtain a multi-generation tooth extraction positioning result. A digital model of anatomical structures, which cannot be directly seen, such as jaw bones, multiple teeth and tooth roots of a patient is superposed on the oral cavity and maxillofacial region, which can be directly seen in reality, of the patient, so that the patient can be directly observed before and during an operation; when the oral cavity or maxillofacial skin of a patient is directly viewed, the position, shape and size of structures, such as jaw bones, multiple teeth and tooth roots, which cannot be directly viewed, of the patient can be displayed, so that a doctor is assisted in preoperative evaluation and intraoperative positioning of the patient, multiple wounds caused by inaccurate positioning are avoided, the injury to the patient is reduced, and the success rate of the operation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of oral treatment, and particularly to an AR-assisted supernumerary tooth extraction positioning method. Background Art

[0002] Augmented Reality (AR) technology is a technology that integrates virtual information with the real world. It widely uses various technical means such as multimedia, 3D modeling, real-time tracking, intelligent interaction, and sensing. After simulating and emulating virtual information such as text, images, 3D models, and videos generated by a computer, it is applied to the real world. The two types of information complement each other, thus realizing the "enhancement" of the real world. Virtual reality technology has been applied in many fields, but it is less used in oral and maxillofacial surgery. Oral and maxillofacial surgery is a surgery that requires high operating requirements, surgical vision, and understanding of anatomical structures. Many operations require sufficient exposure of the vision, and the operator is required to operate under direct vision. However, more exposed vision means more trauma. For some surgeries with difficult positioning, such as the extraction of impacted supernumerary teeth, impacted impacted teeth, and the removal of foreign bodies in soft tissues, if the positioning is inaccurate, it often requires turning over a larger flap or even removing a large amount of bone. However, if the operator knows the position of the supernumerary tooth, impacted tooth, or foreign body under direct vision, it can greatly reduce the unnecessary intraoperative trauma caused by inaccurate positioning of the operator. For some surgeries with high precision, such as orthognathic surgery and tumor osteotomy, it is often necessary to make a guide plate to assist in positioning the osteotomy line. However, the cost of making the guide plate is high and the time is long, and it may interfere with the operator's operation during the surgery. If the operator can see the preoperatively designed osteotomy line under direct vision, it can reduce the surgical cost and will not interfere with the operator's operation. For some surgeries that require precise knowledge of anatomical structures, such as flap transplantation, if the operator can observe the position and course of the blood vessels in the surgical area under direct vision before and during the surgery, it can be more convenient for the preoperative design and intraoperative operation of the surgery.

[0003] Therefore, to solve the above problems, this article proposes an AR-assisted supernumerary tooth extraction positioning method. Summary of the Invention

[0004] The purpose of the present invention is to provide an AR-assisted supernumerary tooth extraction positioning method that combines virtual reality technology with oral and maxillofacial surgery, can assist doctors to successfully complete the surgery while reducing the intraoperative trauma of patients, and improve the surgical success rate.

[0005] To achieve the above technical effects, the present invention is realized through the following technical solutions: An AR-assisted supernumerary tooth extraction positioning method, characterized by comprising the following steps:

[0006] S1. Model construction: Three-dimensional reconstruction is performed on the data model of the visible parts in the patient's oral cavity and maxillofacial region, as well as the models of the non-visible parts such as the jawbone, supernumerary teeth, tooth roots, nerve canals, blood vessels, deep tumors, and deep foreign bodies.

[0007] S2. Generation of model recognition data packet: The STL format of the optical scan model is imported into Model TargetGenerator for target model recognition and training to generate a target recognition model toolkit for this model that can be used in Unity.

[0008] S3. Generation of display model prefab: The dental arch and supernumerary teeth in STL format of the CT scan model are imported into Blender software, the supernumerary teeth and dental arch are rendered and colored, and then exported in FBX format to obtain the display model prefab. Then, the "dental arch + supernumerary teeth" model of the size and positional relationship between the maxillary teeth and supernumerary teeth in STL format of the CT scan model is imported, rendered and colored to obtain the intermediate model prefab, and then exported as FBX for later manual registration.

[0009] S4. Import the target recognition model data packet and display model prefab into the Vuforia SDK for enhancement, and then import them into Unity for editing. Specifically, the target recognition model data packet, display model prefab, intermediate model prefab, and vuforia SDK are imported into unity for editing.

[0010] The vuforia is a software development kit with AR recognition function. After importing it into unity, its function can be used to edit the target recognition model data packet and display model prefab, and then a software with AR function is exported.

[0011] S5. Manual registration: In Unity, manually register the coordinates of the target recognition model data packet and display model prefab to ensure that the display model is displayed under direct vision at the correct size, position, and angle; obtain the positioning model.

[0012] S6. Export of application program: The positioning model obtained in S5 is digitalized, and then the digitalized positioning model is exported to the application program adapted to the device to be applied. Specifically, after being adjusted in unity, the application program can be directly exported. After this program is installed and opened, it can call the camera of the device, and then when the camera can recognize the target object in the real world, the display model with the adjusted size and position is superimposed and displayed.

[0013] S7. Test and verification: After installing the application program on the device, the surgeon observes the non-visible anatomical structures such as the jawbone, supernumerary teeth, tooth roots, nerve canals, blood vessels, deep tumors, and deep foreign bodies under direct vision according to the device's field of view, so as to realize the preoperative evaluation and intraoperative positioning of the surgery.

[0014] Furthermore, in S1, 3D reconstruction is performed on the data model of the visible parts inside the patient's oral cavity and maxillofacial region, as well as the models of the invisible parts such as the jawbone, supernumerary teeth, tooth roots, nerve canals, blood vessels, deep tumors, and deep foreign bodies. The steps are as follows:

[0015] S1.1 3D reconstruction of the data model of the visible parts inside the patient's oral cavity and maxillofacial region: Optically scan the surface morphology inside the patient's oral cavity or the plaster model to obtain optical scan data;

[0016] Based on the optical scan data, establish an optical scan model of the visible parts inside the patient's oral cavity and maxillofacial region, and export it in STL format to complete 3D modeling, which is the structure directly visible to the operator during the operation;

[0017] S1.2 3D reconstruction of the models of the invisible parts such as the jawbone, supernumerary teeth, tooth roots, nerve canals, blood vessels, deep tumors, and deep foreign bodies: Perform a CT scan on the deep tissues such as the supernumerary teeth and jawbone of the patient, collect the X-ray data of the jawbone, supernumerary teeth, and tooth roots that the patient needs for virtual display to obtain CT scan data,

[0018] Import the obtained CT scan data into mimics, edit it in mimics, establish a CT scan model of the invisible parts of the jawbone, supernumerary teeth, and tooth roots, and export it in STL format to complete 3D modeling, that is, the anatomical structures that cannot be directly seen. Import the obtained CT scan data into mimics, edit it in mimics, establish a CT scan model of the invisible parts of the jawbone, supernumerary teeth, and tooth roots, and export it in STL format to complete 3D modeling, that is, the anatomical structures that cannot be directly seen.

[0019] Furthermore, the target model recognition and training specifically use Model Target Generator to extract the surface landmark points and outer contours of the target model (intraoral plaster model), and then perform recognition and training through the Vuforia cloud. Specifically, import the digital model of the target recognition object into the model target generator software, and then debug the model direction, size, color, complexity, optimized tracking, and guiding view in the software. After the debugging is completed, upload it to the vuforia cloud for recognition. After successful recognition, generate a target model data packet.

[0020] Furthermore, in S5, the manual registration includes the following steps:

[0021] S5.1. Using the dental arch of the target recognition model as the standard, manually register the "dental arch + supernumerary tooth" model with clear positional relationships to the target recognition model until the dental arches of the target recognition model and the "dental arch + supernumerary tooth" model widely overlap; then match the display model prefabricated body with the "dental arch + supernumerary tooth" model until the dental arches of the display model prefabricated body and the "dental arch + supernumerary tooth" model widely overlap; thus, using the "dental arch + supernumerary tooth" as a medium, determine the positional and size relationships between the display model prefabricated body and the target recognition model.

[0022] S5.2. After wide overlap, fix the positional relationship between the display model prefabricated body and the target model; then, according to the part to be displayed, hide the remaining objects and make the part to be displayed a sub-object of the target model; thus, the manual registration can be completed to obtain the positioning model.

[0023] Furthermore, in S6, the digitalization of the positioning model specifically includes: after completing the editing in unity, debug the export format and export it as an apk file for use on mobile phones, or a program software for display on computers or head-mounted display devices; for example, the function can be realized after installing the exported APK file on an Android mobile phone, the function can be realized on HoloLens2 after exporting on the UWP platform, the function can be realized on a computer after exporting on the Windows platform, and the function can be realized on apple version series products after exporting on the versionOS platform.

[0024] Furthermore, in S6, the device is one of a mobile phone, a computer, an AR glasses, or a VR glasses.

[0025] The beneficial effects of the present invention are:

[0026] The present invention utilizes augmented reality technology to superimpose the digital models of the anatomical structures that cannot be directly visualized, such as the patient's jawbone, supernumerary teeth, and tooth roots, on the patient's oral cavity and maxillofacial region that can be directly visualized in reality. Thus, during the preoperative and intraoperative periods, when the surgeon directly visualizes the patient's oral cavity or maxillofacial skin, the position, shape, and size of the structures that cannot be directly visualized, such as the patient's jawbone, supernumerary teeth, and tooth roots, can be displayed, thereby assisting the doctor in the preoperative assessment and intraoperative positioning of the patient's surgery, avoiding multiple traumas caused by inaccurate positioning, reducing the damage to the patient, and improving the success rate of the surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0028] Figure 1Schematic diagram of the construction of the optical scanning model of the present invention;

[0029] Figure 2 Schematic diagram of the construction of the CT scanning model of the present invention;

[0030] Figure 3 Schematic diagram of the production interface of the model recognition data packet of the present invention;

[0031] Figure 4 Schematic diagram of the target model tool kit generated by the present invention;

[0032] Figure 5 Schematic diagram of the supernumerary tooth structure after rendering and coloring of the present invention;

[0033] Figure 6 Schematic diagram of the supernumerary tooth + dentition model of the present invention including supernumerary teeth and the size and position relationship of the dentition;

[0034] Figure 7 Schematic diagram of the target model recognition of the present invention;

[0035] Figure 8 Schematic diagram of the model structure after manual registration of the present invention;

[0036] Figure 9 Schematic diagram of the export application program according to the application scenario to be applied by the present invention;

[0037] Figure 10 Schematic diagram of the model display during the test and verification of the present invention;

[0038] Figure 11 Schematic diagram of the specific use interface of the application program exported by the present invention on the plaster model and the human body. Detailed implementation manners

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] Embodiment 1

[0041] The use of an AR-assisted supernumerary tooth extraction positioning method in practice is as follows:

[0042] Taking the extraction of supernumerary teeth as an example, the size, position and shape of the supernumerary teeth buried in the jaw bone that cannot be directly seen are displayed in the directly visible field of view in the patient's mouth.

[0043] Step 1: Model construction

[0044] Optically scan the inner surface morphology of the patient's oral cavity or the plaster model to obtain optical scan data;

[0045] Based on the optical scan data, establish an optical scan model of the visible parts of the patient's oral cavity and maxillofacial region, and export it in STL format to complete 3D modeling, which is the structure directly visible to the operator during the operation; as Figure 1 shown.

[0046] S1.2 Three-dimensional reconstruction of the non-visible part models of the jawbone, supernumerary teeth, and tooth roots:

[0047] Perform a CT scan on these deep tissues such as the patient's supernumerary teeth and jawbone, collect the X-ray data of the jawbone, supernumerary teeth, and tooth roots that the patient needs for virtual display, and obtain CT scan data;

[0048] Import the obtained CT scan data into mimics, edit it in mimics, establish a CT scan model of the non-visible parts of the jawbone, supernumerary teeth, and tooth roots, and export it in STL format to complete 3D modeling, which is the anatomical structure that cannot be directly visualized; as Figure 2 shown.

[0049] Step Two: Generation of the model recognition data packet

[0050] Import the STL format of the optical scan model into the Model Target Generator for target model recognition and training to generate a target recognition model toolkit for this model that can be used in Unity; as Figure 4 shown.

[0051] As Figure 3 shown, the specific process of target model recognition and training in the Model Target Generator is as follows: Use the Model Target Generator to extract the surface landmark points and outer contours of the target model (intraoral plaster model), and then through the recognition and training of the Vuforia cloud, generate a target recognition model toolkit for this model that can be used in Unity.

[0052] Step Three: Generation of the display model prefab

[0053] Import the dentition and supernumerary teeth in STL format of the CT scan model into the Blender software, render and color the supernumerary teeth and dentition, and export it in FBX format (as Figure 5 shown); then import the "dentition + supernumerary teeth" model in STL format of the CT scan model with the size and position relationship of the maxillary teeth and supernumerary teeth, render and color it to obtain the display model prefab (as Figure 6 shown), and then export it as FBX for later manual registration;

[0054] Step 4: Import the target recognition model data package and display model prefab into Vuforia SDK for enhancement, and then import them into Unity for editing. Specifically, import the target recognition model data package, display model prefab, and vuforiaSDK into Unity for editing; (e.g. Figure 7 shown);

[0055] Among them, the above-mentioned vuforia is a software development kit, which has the function of AR recognition. After being imported into unity, its function can be used to edit the target recognition model data package and display model prefab, and then export the software with AR function.

[0056] Figure 7 In Figure A, the left side is the prefabricated model of "dentition + supernumerary teeth" with determined positional relationship, and the right side is the target model that needs to be identified; Figure B is a schematic diagram of importing Vuforia SDK into Unity.

[0057] Step 5: Manual Registration

[0058] like Figure 8 As shown, taking the dentition of the target recognition model as the standard, the "dentition + supernumerary teeth" model with a clear positional relationship is manually registered to the target recognition model, and the dentition of the target model is based on the extensive overlap between the dentition of the "dentition + supernumerary teeth" model; then the display model preform is matched with the "dentition + supernumerary teeth" model, and the dentition of the display model preform is based on the extensive overlap between the dentition of the "dentition + supernumerary teeth" model; thus, the position and size relationship between the display model and the target model is determined with "dentition + supernumerary teeth" as the medium;

[0059] After extensive overlap, the positional relationship between the display model prefabricated body and the target model is fixed; then, according to the part you want to display, the remaining objects are hidden, and the part you want to display becomes a sub-object of the target model; manual alignment is completed to obtain a positioning model.

[0060] Step 6: Export the application

[0061] The positioning model obtained in step 5 is digitized, and then the digitized positioning model is exported to an application that is compatible with the device to be applied. Specifically, the application can be directly exported after the deployment in Unity. After the program is installed and opened, the device's camera can be called, and then the camera can recognize the target object in the real world and overlay the display model with the deployed size and position;

[0062] Step 7: Test and Verify

[0063] like Figure 10As shown, after installing the application on the device, the operator can observe the non-directly visible anatomical structures such as the jawbone, supernumerary teeth, and tooth roots under direct vision according to the device's field of view, so as to achieve preoperative evaluation and intraoperative positioning of the surgery;

[0064] Figure 10 In Figure A, it is a simulated target model to be recognized; in Figure B, it is the interface displayed after the generated Android application is opened; in Figure C, it is the "dentition + supernumerary teeth" part displayed on the application interface when the camera recognizes the target model.

[0065] As Figure 11 shown, when the program is applied to a specific patient, the position of the supernumerary teeth can be observed under direct vision through the mobile phone screen or the head display device, which helps intraoperative positioning, is conducive to reducing trauma, and helps to achieve minimally invasive surgery.

Claims

1. An AR-assisted supernumerary tooth extraction positioning method, characterized in that: The following steps are involved: S1. Model construction: 3D reconstruction of the data model of the visible part of the patient's mouth and maxillofacial area and the models of the parts that cannot be seen directly, such as the jaw, supernumerary teeth, tooth roots, neural tubes, blood vessels, deep tumors, and deep foreign bodies; S2. Model recognition data package generation: Import the STL format of the optical scanning model into the Model Target Generator, perform target model recognition and training, and generate a target recognition model toolkit of this model that can be used in Unity; S3. Display model prefab generation: import the dentition and supernumerary teeth in STL format of the CT scan model into Blender software, render and color the supernumerary teeth and dentition, and export them to FBX format to obtain a display model prefab; then import the "dentition + supernumerary teeth" model of the maxillary teeth and supernumerary teeth in STL format of the CT scan model, render and color it, obtain an intermediate model prefab, and then export it to FBX for later manual registration; S4, importing the target recognition model data package and the display model prefab into Vuforia SDK for enhancement, and then importing them into Unity for editing after enhancement, specifically by importing the target recognition model data package, the display model prefab, the intermediate model prefab, and VuforiaSDK into Unity for editing; S5. Manual registration: In Unity, manually register the target recognition model data package and the display model prefab coordinates to ensure that the display model is displayed in the correct size, position and angle under direct viewing; Get the positioning model, S6, application export: digitize the positioning model obtained in S5, and then export the digitized positioning model to an application compatible with the desired application device; Specifically, after the deployment is completed in Unity, the application can be directly exported. After the program is installed and opened, the device's camera can be called. Then, when the camera recognizes the target object in the real world, it can overlay and display the display model with the deployed size and position; S7. Test verification: After installing the application on the device, the surgeon can directly observe the anatomical structures that cannot be directly viewed, such as the jaw, supernumerary teeth, tooth roots, neural canals, blood vessels, deep tumors, and deep foreign bodies, based on the device's field of view, to achieve preoperative evaluation and intraoperative positioning of the surgery.

2. The AR-assisted supernumerary tooth extraction positioning method according to claim 1, characterized in that: In S1, the data models of the visible parts of the patient's mouth and maxillofacial area and the models of the non-visible parts such as the jawbone, supernumerary teeth, tooth roots, neural tubes, blood vessels, deep tumors, and deep foreign bodies are 3D reconstructed. The steps are as follows: S1.

1. 3D reconstruction of the data model of the patient's oral and maxillofacial parts that can be directly seen: Optically scanning the intraoral surface morphology of the patient or the plaster model to obtain optical scanning data; Based on the optical scanning data, an optical scanning model of the patient's oral and maxillofacial parts that can be directly seen is established and exported in STL format to complete the 3D modeling, that is, the structure that the surgeon sees directly during the operation; S1.2, 3D reconstruction of the model of the parts that cannot be directly viewed, such as the jaw, supernumerary teeth, tooth roots, neural tubes, blood vessels, deep tumors, and deep foreign bodies: Perform CT scans on the deep tissues of the patient's supernumerary teeth and jaws, collect X-ray data of the jaws, supernumerary teeth, tooth roots, neural tubes, blood vessels, deep tumors, and deep foreign bodies that the patient needs to virtually display, and obtain CT scan data; The obtained CT scan data was imported into mimics, edited in mimics, and CT scan models of the parts of the jaw, supernumerary teeth, and tooth roots that could not be directly viewed were established, and exported into STL format to complete the 3D modeling, that is, the anatomical structures that could not be directly viewed.

3. The AR-assisted supernumerary tooth extraction positioning method according to claim 1, characterized in that: The target model recognition and training specifically involves extracting the surface landmarks and outer contours of the target model (intraoral plaster model) through the Model Target Generator, and then recognizing and training through the Vuforia cloud; specifically, the digital model of the target recognition object is imported into the model target generator software, and then the model direction, size, color, complexity, optimized tracking, and guided view are debugged in the software. After the debugging is completed, it is uploaded to the Vuforia cloud for recognition, and a target model data packet is generated after successful recognition.

4. The AR-assisted supernumerary tooth extraction positioning method according to claim 1, characterized in that: In S5, manual registration includes the following steps: S5.

1. Using the dentition of the target recognition model as the standard, manually align the "dentition + supernumerary teeth" model with a clear positional relationship to the target recognition model, with the target model dentition and the "dentition + supernumerary teeth" model's dentition widely overlapping; then match the display model prefabricated body with the "dentition + supernumerary teeth" model, with the display model prefabricated body dentition and the "dentition + supernumerary teeth" model's dentition widely overlapping; thus, using "dentition + supernumerary teeth" as the medium, determine the position and size relationship between the display model and the target model; S5.

2. After extensive overlap, fix the positional relationship between the display model preform and the target model; then, based on the part you want to display, hide the remaining objects and make the part you want to display a sub-object of the target model; this completes the manual alignment and obtains the positioning model.

5. The AR-assisted supernumerary tooth extraction positioning method according to claim 1, characterized in that: In S6, the positioning model is digitized specifically including: after editing in Unity, debugging the export format, exporting to an APK file used on a mobile phone, or a program software displayed on a computer or head display device; for example, exporting an APK file to implement the function after installation on an Anshun mobile phone, exporting to the UWP platform to implement the function on HoloLens2, exporting to the Windows platform to implement the function on a computer, and exporting to the versionOS platform to implement the function on the Apple version series products.

6. The AR-assisted supernumerary tooth extraction positioning method according to claim 1, characterized in that: In S6, the device is one of a mobile phone, a computer, an AR glasses or a VR glasses.

Images