Oral and maxillofacial lesion positioning and registering method and system based on mixed reality

Abstract

The invention relates to the technical field of mixed reality, in particular to an oral and maxillofacial lesion positioning and registering method and system based on mixed reality, and the method comprises the steps: obtaining the oral and maxillofacial surface of a patient, carrying out the three-dimensional reconstruction, recognizing a three-dimensional reconstruction model to obtain the specific position of each part of the oral and maxillofacial surface of the patient, and carrying out the marking. A label is preset on the oral maxillofacial region of a patient, MR is used for scanning the label, a label image is obtained, the coordinate position of the label in an MR coordinate system is obtained according to a slam algorithm, world coordinate system conversion is carried out on the coordinate position of the label in the MR coordinate system, and the coordinate position of the label in the MR coordinate system is obtained. The coordinate position of the label under the world coordinate is registered with the oral and maxillofacial mark point in the three-dimensional reconstruction model to obtain the position coordinate of the oral and maxillofacial mark point under the world coordinate, the oral and maxillofacial can be effectively and accurately positioned, and through the three-dimensional reconstruction of the oral and maxillofacial, the oral and maxillofacial recognition accuracy is improved. And in combination with a label presetting mode of the oral and maxillofacial surface of a patient, the diseased region is effectively and accurately positioned and matched.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality display technology, and in particular to a mixed reality-based oral and maxillofacial lesion positioning and registration method and system. Background Art

[0002] Mixed reality (MR) refers to a new visual environment created by merging the real and virtual worlds, in which physical and digital objects coexist and interact in real time. MR has the characteristics of deep fitting between the real and virtual worlds, precise matching of three-dimensional virtual models with the real world, and real-time interaction between the environment and the user.

[0003] As a cutting-edge innovative technology, mixed reality technology has been clinically applied in the medical field. Medical imaging equipment usually uses MRI, CT or ultrasound technologies to obtain three-dimensional images of the patient's body. The computer processing system processes and analyzes these images, generates a surgical navigation model, and matches it with the real-time surgical scene to provide accurate positioning information. The navigation display presents this information to the surgeon in a visual form to help him accurately position and operate during the operation.

[0004] Clinically, the methods for locating the position of oral and maxillofacial lesions are prone to damage the maxillofacial area or result in large positioning errors, requiring doctors to make judgments based on their own experience, which results in many unfavorable factors. Summary of the invention

[0005] (I) Purpose of the invention

[0006] In order to solve the technical problems existing in the background technology, the present invention proposes an oral and maxillofacial lesion positioning and registration method based on mixed reality. The surgical field image from the perspective of the surgeon can be shared and converted in real time into the surgical field image from the perspective of the assistant's own position according to the relative spatial position of the wearer, which is helpful for the assistant to carry out work and observe and learn.

[0007] (II) Technical solution

[0008] In order to solve the above problems, the first aspect of the present invention provides a method for positioning and registering oral and maxillofacial lesions based on mixed reality, comprising:

[0009] Obtain the patient's oral and maxillofacial structure and perform three-dimensional reconstruction;

[0010] Identify the three-dimensional reconstructed model to obtain the specific positions of various parts of the patient's oral and maxillofacial area and mark them;

[0011] Pre-place labels on the patient's oral and maxillofacial area;

[0012] Scanning the label using MR to obtain a label image;

[0013] Obtain the coordinate position of the tag in the MR coordinate system according to the SLAM algorithm;

[0014] Convert the coordinate position of the tag in the MR coordinate system into a world coordinate system;

[0015] The coordinate position of the label in the world coordinates is aligned with the oral and maxillofacial marker points in the three-dimensional reconstructed model to obtain the position coordinates of the oral and maxillofacial marker points in the world coordinates.

[0016] As a technical solution of the present invention, obtaining the patient's oral and maxillofacial features and performing three-dimensional reconstruction includes:

[0017] Acquire CT image data of the patient's oral and maxillofacial region, and establish a virtual three-dimensional model based on the CT image data;

[0018] Obtaining image information of the patient's oral and maxillofacial region from various perspectives, fusing the image information, and establishing a simulated three-dimensional model;

[0019] Setting markers at various parts of the maxillofacial region in the virtual three-dimensional model and the simulated three-dimensional model;

[0020] The position coordinates of the markers are used for one-to-one matching to complete the three-dimensional reconstruction.

[0021] As a technical solution of the present invention, the image information at least includes key parts of the oral and maxillofacial regions in the virtual three-dimensional model.

[0022] As a technical solution of the present invention, the fusing of the image information includes:

[0023] Extracting image features of the images to be fused from each perspective of the oral and maxillofacial region, and clustering the images to be fused according to the image features;

[0024] Calculate the global center based on the obtained cluster centers, and sum up the distance M between the cluster centers and the global center;

[0025] Select the target cluster center and sum the distance N between the global center and the target cluster center;

[0026] Calculating the fusion coefficient of the oral and maxillofacial image to be fused corresponding to the target cluster center according to the ratio of the distance M to the distance N;

[0027] The fusion coefficient is used to fuse the oral and maxillofacial image information from different perspectives.

[0028] As a technical solution of the present invention, obtaining CT image data of the oral and maxillofacial region of a patient and establishing a virtual three-dimensional model according to the CT image data includes:

[0029] Extract feature data of the patient's oral and maxillofacial area and generate a grid map;

[0030] Performing format conversion on the grid map;

[0031] A virtual three-dimensional model is established according to the converted format content.

[0032] As a technical solution of the present invention, the coordinate position of the tag in the MR coordinate system is converted into the world coordinate system, including: using the conversion relationship between the binocular camera coordinate system and the MR unit coordinate system, converting the coordinate position of the tag in the binocular camera coordinate system into the coordinate position of the marker in the MR unit coordinate system;

[0033] According to the conversion relationship between the MR unit coordinate system and the world coordinate system, the coordinate position of the tag in the MR unit coordinate system is converted into the position coordinate of the tag in the world coordinate system.

[0034] As a technical solution of the present invention, the three-dimensional reconstructed model is identified to obtain the specific positions of various parts of the patient's oral and maxillofacial area and mark them, and the identification includes:

[0035] Identify the lesion location based on the 3D reconstruction model and depict the contour of the lesion;

[0036] Two points on the contour with the farthest straight line distance are selected, and a circle is drawn with the midpoint of the straight line as the center. The two points are distributed on the circumference so that the contour of the lesion is marked with a circle.

[0037] As a technical solution of the present invention, a simulated three-dimensional model is constructed according to the fused image, including:

[0038] Calculate the features of each pixel in the image;

[0039] Match the pixels of multiple images and estimate the camera parameters based on the features of each pixel to obtain sparse 3D information.

[0040] Dense point cloud reconstruction based on estimated camera parameters;

[0041] The point cloud is processed to generate a mesh to construct the simulated three-dimensional model.

[0042] As a technical solution of the present invention, the method of pre-setting labels on the oral and maxillofacial area of ​​a patient includes:

[0043] Obtain the maxillofacial contours near the lesion markers in the 3D reconstruction model, and select the two clearest positions as positioning points;

[0044] Calculate the distance L1 and the distance L2 from the positioning point to the center of the circle;

[0045] The tag preset position is determined according to the distance L1 and the distance L2.

[0046] The second aspect of the present invention proposes an oral and maxillofacial lesion positioning and registration system based on mixed reality, comprising:

[0047] An acquisition module, used to acquire the patient's oral and maxillofacial structures and perform three-dimensional reconstruction;

[0048] An identification module is used to identify the three-dimensional reconstructed model to obtain the specific positions of various parts of the patient's oral and maxillofacial area and mark them;

[0049] A label module, used for pre-setting labels on the patient's oral and maxillofacial area;

[0050] An MR unit, used for scanning the label by using MR to obtain a label image;

[0051] An algorithm module is used to obtain the coordinate position of the tag in the MR coordinate system according to the SLAM algorithm;

[0052] A first conversion module, used for converting the coordinate position of the tag in the MR coordinate system into a world coordinate system;

[0053] The second conversion module is used to align the coordinate position of the label in the world coordinates with the oral and maxillofacial marking points in the three-dimensional reconstructed model to obtain the position coordinates of the oral and maxillofacial marking points in the world coordinates.

[0054] The above technical solution of the present invention has the following beneficial technical effects:

[0055] The present invention can effectively and precisely locate the oral and maxillofacial region, assist doctors in performing efficient maxillofacial surgery, and through three-dimensional reconstruction of the oral and maxillofacial region, combined with the pre-setting of labels of the patient's oral and maxillofacial region, can effectively and precisely locate and match the lesion site. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the method of the present invention;

[0057] Figure 2 is a flow chart of three-dimensional reconstruction in the method of the present invention;

[0058] Figure 3 This is a flowchart of image information fusion in the method of the present invention;

[0059] Figure 4 This is a schematic diagram of presetting the raindrop label in the method of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0061] refer to Figure 1-4 .

[0062] The first aspect of the present invention provides a method for positioning and registering oral and maxillofacial lesions based on mixed reality, comprising:

[0063] Step S1, obtain the patient's oral and maxillofacial features and perform three-dimensional reconstruction:

[0064] In one embodiment, specifically, CT image data of the patient's oral and maxillofacial region is obtained, and a virtual three-dimensional model is established based on the CT image data, and image information of each angle of the patient's oral and maxillofacial region is obtained, and the image information is fused to establish a simulated three-dimensional model, and markers are set on each part of the maxillofacial region in the virtual three-dimensional model and the simulated three-dimensional model, and the position coordinates of the markers are used for one-to-one matching to complete the three-dimensional reconstruction. In addition, the image information at least includes key parts of the oral and maxillofacial region in the virtual three-dimensional model.

[0065] Among them, the fusing of the image information includes extracting image features of the oral and maxillofacial images to be fused from various perspectives, clustering the images to be fused according to the image features; calculating the global center according to the obtained cluster centers, and summing up the distance M between the cluster centers and the global center; selecting the target cluster center, and summing up the distance N between the global center and the target cluster center; calculating the fusion coefficient of the oral and maxillofacial images to be fused corresponding to the target cluster center according to the ratio of the distance M to the distance N; and using the fusion coefficient to fuse the image information of the oral and maxillofacial images from various perspectives.

[0066] More specifically, constructing a simulated 3D model based on the fused image includes:

[0067] Use SI FR to calculate the features of each pixel in the image; use SFM to match the pixels of multiple images, estimate the camera parameters based on the features of each pixel, and obtain sparse 3D information; reconstruct dense point cloud based on the estimated camera parameters; process the point cloud to generate a grid to build the simulated 3D model. Use Bundler and Visual SFM to generate sparse point cloud first, and then generate dense point cloud based on CMVS / PMVS integrated in the program.

[0068] Step S2, identifying the three-dimensional reconstructed model to obtain the specific positions of various parts of the patient's oral and maxillofacial area and marking them.

[0069] The above-mentioned identification of the three-dimensional reconstructed model to obtain the specific positions of various parts of the patient's oral and maxillofacial area and mark them includes:

[0070] Identify the lesion location based on the 3D reconstruction model and depict the contour of the lesion; Figure 4 The shaded area is the lesion area.

[0071] Select two points A and B on the contour that are farthest from each other, and draw a circle with the midpoint of the straight line as the center. The two points are distributed on the circumference so that the contour of the lesion is marked with a circle ( Figure 4 Since A and B are the two points farthest from each other, the circular mark can cover all the lesion areas.

[0072] It should be noted that, in some embodiments, all points on the clearly distinguishable contour line can be marked, and these points can be connected to form an irregular shape, but the irregular shape can easily cover the entire lesion, and the area is closer to the lesion. This embodiment of the present invention only expresses this calculation idea for the convenience of understanding the present invention.

[0073] Step S3, presetting labels on the patient's oral and maxillofacial area:

[0074] refer to Figure 4 In one embodiment, the maxillofacial contour near the lesion marker in the three-dimensional reconstructed model is obtained, and the two clearest positions are selected as positioning points C and D; the distance L1 and the distance L2 from the positioning points C and D to the center of the circle are calculated; and the label preset position is determined according to the distance L1 and the distance L2.

[0075] It should be noted that in the above scheme, the marking point has been determined to be a circle. Since the values ​​of L1 and L2 have been determined, the center of the circle (the midpoint of the straight line connecting the two farthest points A and B) can also be determined. Then, according to the actual oral and maxillofacial features of the patient, the label can be preset. Just as an example, for example, the two clearest positioning points C and D have been determined in the gum part. According to the values ​​of distance L1 and distance L2, the position of the center of the circle can be determined, and the label can be preset to the center of the circle. Therefore, the shape and size of the label should be the same as the circular marking point.

[0076] Step S4, scanning the label using MR to obtain a label image;

[0077] Step S5, obtaining the coordinate position of the tag in the MR coordinate system according to the SLAM algorithm;

[0078] Step S6, converting the coordinate position of the tag in the MR coordinate system into the world coordinate system;

[0079] Step S7, aligning the coordinate position of the label in the world coordinate system with the oral and maxillofacial landmarks in the three-dimensional reconstructed model to obtain the position coordinates of the oral and maxillofacial landmarks in the world coordinate system.

[0080] In one embodiment, obtaining CT image data of the oral and maxillofacial region of a patient and establishing a virtual three-dimensional model according to the CT image data includes:

[0081] Extract feature data of the patient's oral and maxillofacial area and generate a grid map;

[0082] Performing format conversion on the grid map;

[0083] A virtual three-dimensional model is established according to the converted format content.

[0084] In one embodiment, converting the coordinate position of the tag in the MR coordinate system to the world coordinate system includes: utilizing the conversion relationship between the binocular camera coordinate system and the MR unit coordinate system to convert the coordinate position of the tag in the binocular camera coordinate system into the coordinate position of the marker in the MR unit coordinate system; and converting the coordinate position of the tag in the MR unit coordinate system into the position coordinate of the tag in the world coordinate system according to the conversion relationship between the MR unit coordinate system and the world coordinate system.

[0085] It can be imagined that the coordinate position of the tag in the MR coordinate system is obtained according to the SLAM algorithm. The conversion method is as mentioned above and will not be repeated here.

[0086] The second aspect of the present invention provides a mixed reality-based oral and maxillofacial lesion positioning and registration system, comprising:

[0087] An acquisition module is used to acquire the patient's oral and maxillofacial area and perform three-dimensional reconstruction; an identification module is used to identify the three-dimensional reconstructed model to obtain the specific positions of various parts of the patient's oral and maxillofacial area and mark them; a label module is used to pre-set labels on the patient's oral and maxillofacial area; an MR unit is used to scan the labels using MR to obtain label images; an algorithm module is used to obtain the coordinate position of the label in the MR coordinate system according to the SLAM algorithm; a first conversion module is used to convert the coordinate position of the label in the MR coordinate system into a world coordinate system; and a second conversion module is used to align the coordinate position of the label in the world coordinate system with the oral and maxillofacial marker points in the three-dimensional reconstructed model to obtain the position coordinates of the oral and maxillofacial marker points in the world coordinate system.

[0088] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for positioning and registering oral and maxillofacial lesions based on mixed reality, characterized in that: include: Obtain the patient's oral and maxillofacial structure and perform three-dimensional reconstruction; Identify the three-dimensional reconstructed model to obtain the specific positions of various parts of the patient's oral and maxillofacial area and mark them; Pre-place labels on the patient's oral and maxillofacial area; Scanning the label using MR to obtain a label image; Obtain the coordinate position of the tag in the MR coordinate system according to the SLAM algorithm; Convert the coordinate position of the tag in the MR coordinate system into a world coordinate system; The coordinate position of the label in the world coordinates is aligned with the oral and maxillofacial marker points in the three-dimensional reconstructed model to obtain the position coordinates of the oral and maxillofacial marker points in the world coordinates.

2. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 1, characterized in that: Obtain the patient's oral and maxillofacial data and perform 3D reconstruction, including: Acquire CT image data of the patient's oral and maxillofacial region, and establish a virtual three-dimensional model based on the CT image data; Obtaining image information of the patient's oral and maxillofacial region from various perspectives, fusing the image information, and establishing a simulated three-dimensional model; Setting markers at various parts of the maxillofacial region in the virtual three-dimensional model and the simulated three-dimensional model; The position coordinates of the markers are used for one-to-one matching to complete the three-dimensional reconstruction.

3. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 2, characterized in that: The image information at least includes key parts of the oral and maxillofacial regions in the virtual three-dimensional model.

4. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 2, characterized in that: The fusing of the image information includes: Extracting image features of the images to be fused from each perspective of the oral and maxillofacial region, and clustering the images to be fused according to the image features; Calculate the global center based on the obtained cluster centers, and sum up the distance M between the cluster centers and the global center; Select the target cluster center and sum the distance N between the global center and the target cluster center; Calculating the fusion coefficient of the oral and maxillofacial image to be fused corresponding to the target cluster center according to the ratio of the distance M to the distance N; The fusion coefficient is used to fuse the oral and maxillofacial image information from different perspectives.

5. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 2, characterized in that: Acquiring CT image data of the patient's oral and maxillofacial region and establishing a virtual three-dimensional model according to the CT image data includes: Extract feature data of the patient's oral and maxillofacial area and generate a grid map; Performing format conversion on the grid map; A virtual three-dimensional model is established according to the converted format content.

6. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 1, characterized in that: The converting the coordinate position of the tag in the MR coordinate system into the world coordinate system includes: using the conversion relationship between the binocular camera coordinate system and the MR unit coordinate system to convert the coordinate position of the tag in the binocular camera coordinate system into the coordinate position of the marker in the MR unit coordinate system; According to the conversion relationship between the MR unit coordinate system and the world coordinate system, the coordinate position of the tag in the MR unit coordinate system is converted into the position coordinate of the tag in the world coordinate system.

7. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 2, characterized in that: The three-dimensional reconstructed model is identified to obtain the specific positions of various parts of the patient's oral and maxillofacial region and mark them. The identification includes: Identify the lesion location based on the 3D reconstruction model and depict the contour of the lesion; Two points on the contour with the farthest straight line distance are selected, and a circle is drawn with the midpoint of the straight line as the center. The two points are distributed on the circumference so that the contour of the lesion is marked with a circle.

8. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 4, characterized in that: Construct a simulated 3D model based on the fused images, including: Calculate the features of each pixel in the image; Match the pixels of multiple images and estimate the camera parameters based on the features of each pixel to obtain sparse 3D information. Dense point cloud reconstruction based on estimated camera parameters; The point cloud is processed to generate a mesh to construct the simulated three-dimensional model.

9. The method for positioning and registering oral and maxillofacial lesions based on mixed reality according to claim 7, characterized in that: The pre-placed labels on the patient's oral and maxillofacial area include: Obtain the maxillofacial contours near the lesion markers in the 3D reconstruction model, and select the two clearest positions as positioning points; Calculate the distance L1 and the distance L2 from the positioning point to the center of the circle; The tag preset position is determined according to the distance L1 and the distance L2.

10. Oral and maxillofacial lesion positioning and registration system based on mixed reality, characterized in that: include: An acquisition module, used to acquire the patient's oral and maxillofacial structures and perform three-dimensional reconstruction; An identification module is used to identify the three-dimensional reconstructed model to obtain the specific positions of various parts of the patient's oral and maxillofacial area and mark them; A label module, used for pre-setting labels on the patient's oral and maxillofacial area; An MR unit, used for scanning the label by using MR to obtain a label image; The algorithm module is used to obtain the coordinate position of the tag in the MR coordinate system according to the SLAM algorithm; A first conversion module, used for converting the coordinate position of the tag in the MR coordinate system into a world coordinate system; The second conversion module is used to align the coordinate position of the label in the world coordinates with the oral and maxillofacial marking points in the three-dimensional reconstructed model to obtain the position coordinates of the oral and maxillofacial marking points in the world coordinates.

Images