Learning-based ar-assisted dental treatment automatic calibration and navigation method

Abstract

A learning-based AR-assisted dental treatment automatic calibration and navigation method, comprising: using a virtual reality display device to acquire a 3D mandible model of a target; inputting the 3D mandible model into a trained feature point detection network to identify a plurality of corresponding feature points; aligning the plurality of feature points with feature points of a corresponding real 3D mandible model so as to calibrate and navigate onto the real model the virtual model under the virtual reality display device, wherein the feature points of the real 3D mandible model are detected by using a depth camera; and on the basis of a multi-feature iterative nearest point algorithm, combining a virtual environment and the real environment to complete the display of the virtual projection in the real environment. The method improves the efficiency and accuracy of automatic calibration and navigation of dental treatment.

Description

A learning-based automatic calibration and navigation method for AR-assisted dental treatment Technical Field

[0001] The present invention relates to the field of biomedical engineering technology, and more particularly to a learning-based AR-assisted dental treatment automatic calibration and navigation method. Background Art

[0002] Oral and maxillofacial surgery is a discipline focused on surgical treatment, with the primary focus on the prevention and treatment of certain diseases of the oral cavity, facial soft tissues, maxillofacial bones, temporomandibular joint, and neck. Computer-assisted therapy (CAT) is a commonly used treatment method that has transformed treatments in many different medical fields, including digital dentistry, improving efficiency and accuracy, reducing patient impact, and assisting with preoperative and intraoperative procedures. Developed augmented reality (AR) technology is becoming increasingly popular in the field of CAT. AR technology enables surgical navigation by overlaying virtual anatomical structures on real patients.

[0003] To improve the accuracy and reliability of display calibration procedures, existing solutions are based on optical tracking systems. Optical tracking systems use multiple markers as a dynamic reference frame. These markers are firmly fixed to the target anatomy to track the direction and posture of the target during the movement of the dynamic reference frame. During the tracking process, it is usually necessary to calibrate the relative posture between the dynamic reference frame and the offline anatomy. Since the errors in the marker-to-target registration may be distributed throughout the process and may cause unnecessary errors during the marker insertion process, error point-based or contour-based registration can be utilized. Therefore, it is challenging to achieve safety and avoid invasiveness when using markers in computer-assisted treatment.

[0004] In the prior art, Kellner et al. proposed a two-stage geometric calibration method (Kellner F, Bolte B, Bruder G, et al. Geometric calibration of head-mounted displays and its effects on distance estimation [J]. IEEE transactions on visualization and computer graphics, 2012, 18 (4): 589-596.). This method tracks six-degree-of-freedom head-attached markers and three-degree-of-freedom manual markers, improving the user's interaction method. However, this method requires manual marking, which reduces the efficiency and accuracy of calibration. For example, Jun et al. proposed a calibration method (Jun H, Kim GA calibration method for optical see-through head-mounted displays with a depth camera [C] / / 2016 IEEE Virtual Reality (VR). IEEE, 2016: 103-111.). Using a low-cost time-of-flight depth camera, the method performed two stages of full calibration and simplified calibration to calculate key calibration parameters. However, this method requires the user to point to a virtual circle with their fingertips, which is not only cumbersome, but also has a high probability of drawing an incorrect virtual circle, thereby reducing the accuracy of calibration.

[0005] After analysis, in the field of digital dental diagnosis and treatment in oral and maxillofacial surgery, traditional AR-based treatment relies on optical markers and trackers, which makes the treatment process monotonous and expensive, and is prone to errors when using fiducial markers, retaining reference markers, or creating tooth models when using large optical tracking devices. These fiducial markers and bulky tracking devices bring difficulties when using them, such as limiting the doctor's line of sight or causing inaccuracies due to marker displacement, which increases the complexity of subsequent technology and requires major modifications to surgical arrangements. In addition, traditional computer-assisted treatment relies on two-dimensional (2D) imaging rather than perceiving three-dimensional (3D) images for guidance and navigation, which results in the doctor's lack of perception of the depth information of the oral cavity and causes problems with hand-eye coordination. Therefore, surgical navigation in the field of dental treatment remains challenging.

[0006] Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a learning-based AR-assisted dental treatment automatic calibration and navigation method. The method comprises the following steps:

[0008] A 3D mandibular model of the target is obtained using a virtual reality display device;

[0009] Inputting the 3D mandibular model into a trained feature point detection network to identify a plurality of corresponding feature points;

[0010] aligning the plurality of feature points with feature points of a corresponding real 3D mandibular model to calibrate and navigate the virtual model on the virtual reality display device onto the real model, wherein the feature points of the real 3D mandibular model are detected using a depth camera;

[0011] Based on the multi-feature iterative closest point algorithm, the virtual environment and the real environment are combined to complete the display of virtual projection in the real environment.

[0012] Compared with the existing technology, the advantage of the present invention is that it realizes markerless calibration and navigation based on augmented reality (AR) in digital dental treatment, and can be used for visual augmented reality based on head-mounted display. By improving the quality of the depth map between virtual and real distances, the virtual model calibration process in the head-mounted display is fully automated, thereby improving the efficiency and accuracy of calibration.

[0013] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0015] FIG1 is a flowchart of a learning-based automatic calibration and navigation method for AR-assisted dental treatment according to one embodiment of the present invention;

[0016] FIG2 is a schematic diagram of a structure of a feature point detection network using depth RGB data according to an embodiment of the present invention;

[0017] FIG3 is a schematic diagram of three-dimensional model alignment according to an embodiment of the present invention;

[0018] In the attached figure, Landmarks detection-feature point detection; Conv-convolution layer; Max pool-maximum pooling; dropout-discard; upsampling-up sampling; Output Probability Map-output probability map; Pixel-wise labeling-pixel-by-pixel labeling. DETAILED DESCRIPTION

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0022] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0023] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0024] This invention provides a learning-based automatic calibration and navigation method for AR-assisted dental treatment. This AR-assisted markerless calibration and navigation solution utilizes an attached deep RGB stereo camera to automatically calibrate a virtual 3D mandibular model within a head-mounted display (HMD). In general, a pretrained convolutional neural network model is first used to detect anatomical feature points using a deep RGB image. Next, the virtual model's coordinate system is automatically aligned with the world coordinate system based on these feature points, calibrating the HMD's virtual model. Finally, markerless navigation is performed to overlay the virtual and real environments, enabling accurate surgical navigation for dental treatment. This solution primarily includes three core components: virtual image or environment modeling, virtual image and real-world space registration, and a display technology that integrates the virtual and real environments. This final display technology is implemented using a HMD. To enhance deep navigation and tracking of the 3D model, a deep RGB camera is integrated on top of the HMD. This deep RGB camera utilizes active stereo imaging with unstructured light to achieve accurate reconstruction within a restricted field of view.

[0025] Specifically, referring to FIG1 , the provided learning-based AR-assisted dental treatment automatic calibration and navigation method includes the following steps:

[0026] Step S110 : constructing a training data set, where the training data set includes an RGB image of the mandible, depth information, and multiple feature points of the mandible.

[0027] In one embodiment, during the preoperative surgical planning phase, patient sample data is acquired using CBCT (oral and maxillofacial cone-beam CT). The data is segmented and reconstructed based on regions of interest (RoIs) for diagnosis and treatment planning. A deep RGB stereo camera is then used to record and generate a dataset for training a deep neural network for anatomical landmark detection.

[0028] For example, a dataset contains the correspondence between the RGB image and depth information of the mandible and multiple feature points of the mandible. Feature points are used to identify basic features of the mandible, such as the head, neck, base, and beak. The number of feature points can be set based on accuracy and efficiency requirements.

[0029] Furthermore, to enhance the dataset's diversity, the virtual geometry can be constructed using a variety of textures and materials. To ensure that the training dataset encompasses a variety of camera poses, a moderate randomization of camera position and rotation offsets is introduced before capturing each image. Furthermore, various lighting configurations with varying intensities, color temperatures, and positions are employed to maximize the dataset's diversity.

[0030] Step S120: construct a neural network model as a feature point detection network, and perform training using a training data set.

[0031] In one embodiment, the mandibular anatomical model feature point detection network is constructed based on FCN (fully convolutional network), as shown in Figure 2. The feature point detection network as a whole includes an encoder and a decoder. The encoder is mainly composed of convolution layers and maximum pooling layers, which gradually reduce the size of the feature map and capture higher-level semantic information through convolution operations. The decoder mainly contains upsampling layers and convolution layers, which gradually restore image detail information through upsampling or deconvolution. And by designing skip connections, global information and local information are connected to produce more accurate and detailed detection results. The purpose of detecting the feature points of the anatomical model is to identify the basic features of the three-dimensional model of the mandible, such as the head, neck, base and beak.

[0032] Furthermore, for feature point detection, an FCN (fully convolutional network) was optimized to predict the 3D tooth model by generating a dense probability map after separating the foreground from the background. The output of the FCN is used to mark the feature points of the mandible and create labels for real-time pixel-by-pixel tracking. The FCN encoder uses convolution and pooling operations to calculate feature maps with decreasing spatial resolution and increasing depth information. The decoder uses transposed convolution and element-by-element fusion to generate a class score map with the same spatial dimensions as the input image.

[0033] To perform initial calibration of the HMD (head-mounted display), a depth RGB camera mounted on the HMD captures and identifies features of a mandibular phantom. The captured frames are processed to conform to the input modality required by an FCN (fully convolutional network), which identifies a 3-channel probability map with probability values ​​for each class (e.g., background = 0, mandible = 1, and landmark = 2) at each pixel location. By combining the probabilities with depth information from the input frames, the mandibular landmarks are densely labeled. These labeled pixels are then used to create data correspondences for model-based position tracking. This tracking method matches the object model to the labeled data, enabling the determination of the object's 6D pose.

[0034] The network was then trained using deep RGB images generated from the 3D model, while also being tested using the real 3D model. Furthermore, the invention employed data augmentation techniques to improve performance.

[0035] In step S130 , the trained feature point detection network is used to detect anatomical feature points, and the anatomical feature points are aligned with the feature points of the real 3D model to achieve markerless calibration between the virtual model and the real model.

[0036] Because the feature point detection network is pre-trained for mandibular feature detection, when the feature points of the real model are captured using a head-mounted display, the feature point detection network can align and navigate the virtual model onto the real model to complete markerless calibration and navigation.

[0037] Specifically, the detected feature points can describe the characteristics of the mandible. Using these detected anatomical feature points, a method for automatically calibrating the virtual model and navigation was further designed. After using a depth stereo camera to detect feature points on a real three-dimensional object, multiple feature points of the virtual model are aligned with the feature points of its real 3D model. Through this method, the distance between the feature points of the virtual model and the real object is preserved, and accurate calibration is given. As shown in Figure 3, in a virtual environment, the six detected feature points are automatically aligned with the real object, where Figure 3(a) corresponds to the real 3D model and Figure 3(b) corresponds to the virtual 3D model.

[0038] It should be noted that, in order to determine the pose of a 3D virtual object, three non-collinear points in space are sufficient, but it is preferred to use six points for easier alignment and to provide better depth cues.

[0039] In one embodiment, the corresponding transformation matrix is ​​calculated and the RANSAC (Random Sample Consensus) algorithm is used to remove outliers with reprojection errors and refine the most accurate transformation. RANSAC can iteratively estimate the parameters of a mathematical model from a set of observation data containing "outliers" and remove "outliers" or "noise" by setting relevant thresholds.

[0040] Failure to consider parallax may lead to misalignment due to the depth difference between the target and the virtual display. (s) and the observed 3D real point V (M) The spatial consistency between them was calibrated using the following equation:

[0041] Among them, p (s) To display the position of the pixel, K is the projection matrix of the camera, is the camera stereo pose, is the internal transformation matrix of the HMD (head-mounted display), M (D) To track the posture, V (M) is a 3D real point.

[0042] In step S140 , based on the feature points, a multi-feature iterative closest point algorithm is used to combine the virtual environment and the real environment to achieve markerless navigation.

[0043] An important step in developing head-mounted display (HMD) applications is to obtain the transformation matrix between the real-world stereo camera and the 3D virtual model. In one embodiment, feature point detection technology using a depth RGB camera is used to obtain the overall transformation and reprojection matrix.

[0044] For example, a multi-feature iterative closest point algorithm (ICP) based on point-to-point navigation of detected feature points is provided. This method uses a three-dimensional mandibular model. The corresponding feature points of the real object are used to automatically estimate the navigation error between the feature points of the real object and the feature points of the 3D virtual model detected from the depth RGB camera. The coordinates of the feature points of the 3D virtual model have been obtained from the depth camera coordinate system. This coordinate system displays and aligns the 3D model in the coordinate system of the real object to complete markerless navigation. The following transformation matrix is ​​used:

[0045] in, is the re-projected feature point matrix, is the internal transformation matrix of the head-mounted display, is the original feature point matrix, n is the number of feature points

[0046] Specifically, the RANSAC algorithm is used to remove outlier samples based on the reprojection error to achieve a more accurate transformation. The reprojection error is calculated as follows:

[0047] In order to further verify the effect of the present invention, an experiment was conducted. In the experiment, a depth RGB camera was connected to a commercial head-mounted AR display to detect the mandibular feature points in real time and perform calibration. It has been verified that the system developed by the present invention achieved a virtual model to real object reprojection error of 1.09±0.23mm pixels. Moreover, the calibration achieved a display error of 5.33±1.89arcmin. In addition, a markerless navigation was performed on a dental treatment experiment (mandible) based on real anatomical structure to verify the use of the system of the present invention in digital dentistry. According to user feedback, surgical navigation based on a commercial head-mounted display has the characteristics of reliable and stable tracking, display delay, and rapid alignment with the real anatomical structure, achieving overall translation and rotation surgical navigation errors of 3.85±0.62mm and 2.65±1.47°, respectively. It has been verified that the present invention successfully achieved learning-based automatic calibration and navigation of AR-assisted dental treatment.

[0048] In summary, compared with the prior art, the present invention has the following advantages:

[0049] 1) A new calibration and navigation method for a highly reconfigurable head-mounted display virtual model is proposed. A depth RGB sensing stereo camera is used to automatically detect the mandibular region of interest (RoI) feature points, thus avoiding the use of optical markers and sensors. The method can track overlapping virtual models and real objects.

[0050] 2) The present invention designs a large-scale deep RGB dataset with multiple part occlusions and random regions of different postures, which conforms to real scenes to overcome the data problem.

[0051] 3) In contrast to current marker-based calibration and navigation methods, this paper proposes a new learning-based markerless head-mounted display AR system for computer-assisted treatment in digital dentistry. By improving the quality of depth maps between virtual and real distances, the virtual model calibration process in the head-mounted display is fully automated, improving calibration efficiency and accuracy.

[0052] 4) The present invention eliminates user input and avoids external sensors, making it more accurate in the measurement process and avoiding complex equipment.

[0053] 5) Based on the identified feature points, the present invention proposes a navigation method, which shortens the re-projection time and improves the accuracy, and can quickly and accurately project and quantitatively analyze.

[0054] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0055] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0056] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0057] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0058] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0059] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0060] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0061] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0062] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A learning-based AR-assisted dental treatment automatic calibration and navigation method, comprising the following steps: A 3D mandibular model of the target is obtained using a virtual reality display device; Inputting the 3D mandibular model into a trained feature point detection network to identify a corresponding plurality of feature points; Aligning the plurality of feature points with feature points of a corresponding real 3D mandibular model to calibrate and navigate the virtual model under the virtual reality display device onto the real model, wherein the feature points of the real 3D mandibular model are detected using a depth camera; Based on the multi-feature iterative closest point algorithm, the virtual environment and the real environment are combined to complete the display of virtual projection in the real environment.

2. The method according to claim 1, characterized in that The feature point detection network is a fully convolutional network model, including an encoder and a decoder, and a jump connection is set between the encoder and the decoder. The encoder uses convolution and pooling operations to calculate a feature map with decreasing spatial resolution and increasing depth information. The decoder uses transposed convolution and element-by-element fusion to generate a class score map with the same spatial dimension as the input image. The output of the fully convolutional network model is used to mark the feature points of the 3D mandibular model.

3. The method according to claim 2, characterized in that The fully convolutional network model identifies a multi-channel probability map of multiple category probability values ​​at each pixel position for the input RGB image, and marks the mandibular feature points by combining the probability with the depth information from the input frame. The multiple categories include background, mandibular and feature points.

4. The method according to claim 1, characterized in that Display pixel p is realized according to the following formula (s) and 3D real point V (M) Registration between: Among them, p (s) To display the position of the pixel, K is the projection matrix of the depth camera, is the stereo pose of the depth camera, is the internal transformation matrix of the virtual reality display device, M (D) To track the posture, V (M) is a 3D real point.

5. The method according to claim 1, characterized in that The following formula is used to display the virtual projection in the real environment: in, is the reprojected feature point matrix, is the internal transformation matrix of the virtual reality display device, is the original feature point matrix, and n is the number of feature points.

6. The method according to claim 5, characterized in that In the process of displaying the virtual projection in the real environment, points with a reprojection error greater than a set threshold are removed as outlier samples. The reprojection error is calculated according to the following formula:

7. The method according to claim 1, characterized in that The virtual reality display device is a head-mounted AR display, and a depth RGB camera is integrated on the top of the head-mounted display.

8. The method according to claim 1, characterized in that The training data set of the feature point detection network is constructed according to the following steps: Oral and maxillofacial cone beam CT was used to obtain sample data of the patient's mandible; Based on the region of interest, the sample data is segmented and reconstructed according to the diagnosis and treatment plan; Generate a data set using a deep RGB stereo camera, wherein the data set includes an RGB image of the mandible, depth information, and multiple feature points of the mandible; Performing enhancement processing on the data set to construct a training data set, wherein the enhancement processing includes: using a variety of different textures and materials for the virtual geometric body; For RGB images, different camera poses are included and various lighting configurations with different intensities, color temperatures, and positions are set.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images