Ear canal inner ear high-precision three-dimensional modeling method based on monocular endoscope point cloud

Abstract

The embodiment of the invention discloses an ear canal inner ear high-precision three-dimensional modeling method based on monocular endoscope point cloud. The method comprises the steps that point cloud data of an ear canal inner ear area are obtained through scanning of a monocular endoscope; arranging the collected data into an endoscope point cloud file, and checking the integrity of the file; recognizing the internal structural features of the ear canal and the position and shape information of the inner ear through a deep learning feature extraction algorithm 3D-CNN; importing the sorted point cloud data into software, reconstructing to generate a triangular mesh, and exporting the triangular mesh as a three-dimensional general format file; importing the OBJ file into three-dimensional modeling software, and carrying out model modification, wiring adjustment and normal adjustment; according to the panorama, the model is split, and part facilities are added; and importing the model into a three-dimensional scene editor, and carrying out later authenticity processing and effect adding processing on a model material. According to the invention, three-dimensional entity modeling can be carried out in a high-quality, high-efficiency and high-stability manner for narrow closed spaces such as bronchus, nasal cavities, ear canals and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for high-precision three-dimensional modeling of an ear canal inner ear based on a monocular endoscope point cloud. Background Art

[0002] In the field of otolaryngology, especially in ear canal and inner ear surgery, accurate 3D anatomical information is particularly important for surgical navigation, disease diagnosis and surgical planning. Although traditional medical imaging technologies (such as CT and MRI) can provide detailed structural information, they have certain limitations in real-time surgical navigation and accurate positioning, such as limited soft tissue resolution, especially in capturing details of the internal structure of the ear canal. Due to the very complex geometric structure of the ear canal and inner ear, including tortuous passages, small structures and tiny features, traditional medical imaging technologies have limitations in capturing these details. Summary of the invention

[0003] The purpose of the present invention is to solve the problems raised in the above-mentioned background technology. The present invention provides a method for high-precision three-dimensional modeling of the inner ear of the ear canal based on a monocular endoscope point cloud, which can perform three-dimensional solid modeling with high quality, high efficiency and high stability for narrow and closed spaces such as the bronchus, nasal cavity and ear canal.

[0004] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0005] A high-precision three-dimensional modeling method for the inner ear of the ear canal based on a monocular endoscope point cloud includes:

[0006] Step 1: Data collection: obtaining point cloud data of the inner ear area of ​​the ear canal through monocular endoscope scanning;

[0007] Step 2: Data preprocessing: organize the collected data into an endoscope point cloud file and then check the integrity of the file;

[0008] Step 3: Feature extraction: Use the deep learning feature extraction algorithm 3D-CNN to identify the structural features inside the ear canal and the position and shape information of the inner ear;

[0009] Step 4: Reconstruction: import the sorted point cloud data into 3DReshape software, reconstruct and generate a triangular mesh, and then export the generated triangular mesh into a 3D general format file;

[0010] Step 5: Modeling: import the OBJ file into 3ds Max 3D modeling software to modify the model, adjust the wiring, and adjust the normals;

[0011] Step 6: Supplement: Compare the panorama, split the models of different materials, and add components and facilities;

[0012] Step seven, optimization, import the model into the 3D scene editor, perform post-processing on the model material to make it more realistic, and add effect processing.

[0013] Beneficial effects:

[0014] The present invention first organizes the monocular endoscope point cloud data, then performs data extraction and triangular mesh conversion, uses 3DReshape to repair and reconstruct the model, and finally imports the three-dimensional scene visualization engine to complete the later work. The present invention can perform three-dimensional solid modeling with high quality, high efficiency and high stability for narrow and closed spaces such as bronchi, nasal cavities and ear canals, providing an accurate basis for the field of otolaryngology. The present invention is simple and efficient, can reduce the workload of a large number of medical staff, and at the same time the data results are more intuitive, real and effective, and highly practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flow chart of a method for high-precision three-dimensional modeling of the inner ear of the ear canal based on a monocular endoscope point cloud according to an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a feature extraction method based on 3D-CNN according to an embodiment of the present invention;

[0018] Figure 3 This is a workflow diagram of the Deep Closest Point algorithm for point cloud registration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0020] As shown in the figure, an embodiment of the present invention discloses a method for high-precision three-dimensional modeling of the inner ear of the ear canal based on a monocular endoscope point cloud, comprising:

[0021] Step 1 S1, data collection, obtaining point cloud data of the inner ear area of ​​the ear canal through monocular endoscope scanning, ensuring that the density of the collected data is high enough to display the detailed structure of the inner ear of the ear canal and the detailed features of the inner ear;

[0022] Step 2 S2, data preprocessing, collating the collected data into an endoscope point cloud file, and then checking the integrity of the file. Before collating the collected data, the collected point cloud data can also be filtered, denoised and registered, which is conducive to improving the quality and accuracy of the data;

[0023] Step 3 S3, feature extraction, identifying the structural features inside the ear canal and the position and shape information of the inner ear through the deep learning feature extraction algorithm 3D-CNN;

[0024] Step 4 S4, reconstruction, importing the sorted point cloud data into 3DReshape software, reconstructing and generating a triangular mesh, and then exporting the generated triangular mesh into a 3D general format file;

[0025] Step 5 S5, modeling, importing the OBJ file into 3ds Max 3D modeling software, and performing model modification, wiring adjustment, and normal adjustment;

[0026] Step 6 S6, supplement, compare the panorama, split the models of different materials, make PBR materials for mapping, and add parts and facilities;

[0027] Step seven S7, optimization, the model is imported into the 3D scene editor, the model material is post-processed for authenticity, and effect processing is added. The added effects are not limited to the environment. At the same time, the progress of the modeling results can be evaluated by point cloud comparison or comparison with known measurement data, thereby improving the quality of the model.

[0028] In one embodiment of the present invention, in step 2, the point cloud registration Deep Closest Point algorithm is used to splice the data of multiple scanning positions to form a complete point cloud data file, and the integrity of the monocular endoscope point cloud file is checked again, including checking whether the point cloud coordinates are unified, and useless data is eliminated after the check.

[0029] In one embodiment of the present invention, in step 2, the steps of stitching multiple scanning positions using the point cloud registration Deep Closest Point algorithm are as follows:

[0030] S1: Feature extraction: input the ear canal point cloud data scanned by endoscope and perform feature extraction through 3D-CNN;

[0031] S2: Graph construction and matching: construct a k-nearest neighbor-based graph in the feature space and use the nearest neighbor strategy to match point pairs;

[0032] S3: Transformation estimation, solving the rotation and translation matrices through SVD decomposition;

[0033] S4: Iterative update, Deep Closest Point improves the registration effect through continuous iteration until convergence.

[0034] In one embodiment of the present invention, in step 2, the key function k nearest neighbor algorithm is used to calculate the distance between two points in the registration process using the point cloud registration Deep Closest Point algorithm to determine the distance between the source point cloud and the target point cloud. The Euclidean distance is used for distance measurement in the ear canal. The Euclidean distance is used to measure the absolute distance between two point clouds. Assuming that the dimension of the multidimensional space is N, X1=X 11 ,X 12 ,···,X 1N and X2 = X 21 ,X 22 ,···,X 2N are two points. Since the ear canal space is three-dimensional, N is 3, and the Euclidean distance between X1 and X2 is calculated as

[0035]

[0036] In one embodiment of the present invention, in step 2, during the registration process using the point cloud registration Deep Closest Point algorithm, the transformation estimation in step S3 uses SVD decomposition to solve the rotation and translation matrix. Assuming that there are two point clouds A and B, and the points therein correspond one to one, in order to find the rotation matrix R and translation matrix t between the two point clouds, the specific steps are as follows:

[0037] S11: Calculate the center point,

[0038] P = [xyz] T

[0039]

[0040] S12: To calculate the optimal rotation matrix R, we first need to re-center the point set to generate new A` and B`.

[0041] A i `={P i AU B}

[0042] B i `={P i BU B}

[0043] Calculate the covariance H between the point sets

[0044]

[0045] The matrices U, S and V are obtained by the SVD method, and the rotation matrix R is calculated.

[0046] [U, S, V] = SVD(H)

[0047] R=VU T

[0048] S13: Calculate the translation matrix t

[0049] t=-R*U A +U B .

[0050] In one embodiment of the present invention, in step 3, the specific steps of the deep learning feature extraction algorithm 3D-CNN are as follows:

[0051] S111: extracting features from the ear canal point cloud image scanned by the monocular endoscope by using a principal component analysis method;

[0052] S112: inputting the reference data and the extracted point cloud data into the 3D-CNN framework to extract deep spatial features;

[0053] S113: combining the point cloud image scanned by the monocular endoscope with the reference data to extract point cloud features based on a balanced local discriminant embedding algorithm;

[0054] S114: Superimpose the point cloud features and the depth features to extract the fusion features and input them into the classifier for feature classification.

[0055] In one embodiment of the present invention, in step 4, the sorted point cloud data is imported into the 3DReshape software, and the process of reconstructing and generating a triangulated network is as follows:

[0056] S1111: Import point cloud data, open 3DReshape software, find the file for importing point cloud data in the software interface, and select the data file to be imported;

[0057] S1112: Processing data: after importing point cloud data, removing outliers and performing point cloud data registration is beneficial to improving the quality and accuracy of data and providing a basis for the generation of triangular meshes;

[0058] S1113: Create a triangular network. Select the triangular network generation tool in the software. Set the parameters of the triangular mesh according to the distribution and characteristics of the point cloud data. In actual operation, select different algorithms and methods to generate a suitable triangular mesh according to different needs.

[0059] S1114: Mesh optimization. After generating the triangular mesh, the triangular mesh is optimized by smoothing, subdividing, correcting topology and error operations.

[0060] In one embodiment of the present invention, in step 4, the process of exporting the generated triangular mesh into a three-dimensional general format file is as follows:

[0061] S11111: Check the export options. In the 3D modeling software you are using, open the triangular mesh model and enter the export options.

[0062] S11112: Select the export format. In the export options, select the file format to be exported, OBJ;

[0063] S11113: Configure export settings. In the export options, set the coordinate system, unit, size, normal and direction as needed.

[0064] S11114: Determine the export path, select the export path, local file or specific device;

[0065] S11115: Export the triangular mesh. After complete configuration, click the Export button to export the triangular mesh.

[0066] S11116: Verify the export results. After the export is completed, you can use other 3D software that supports this format to open the exported file and verify whether the exported result is consistent with the original triangular mesh.

[0067] In one embodiment of the present invention, in the feature extraction step of step three, the structural features inside the ear canal and the position and shape information of the inner ear are identified by the deep learning feature extraction algorithm 3D-CNN, and the curvature and normal vector features are used to identify the ear canal wall and the inner ear surface area.

[0068] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high-precision three-dimensional modeling method for the inner ear of the ear canal based on a monocular endoscope point cloud, characterized in that: include: Step 1: Data collection: obtaining point cloud data of the inner ear area of ​​the ear canal through monocular endoscope scanning; Step 2: Data preprocessing: organize the collected data into endoscope point cloud files and check the integrity of the files; Step 3: Feature extraction: Use the deep learning feature extraction algorithm 3D-CNN to identify the structural features inside the ear canal and the position and shape information of the inner ear; Step 4: Reconstruction: import the sorted point cloud data into 3DReshape software, reconstruct and generate a triangular mesh, and then export the generated triangular mesh into a 3D general format file; Step 5: Modeling: import the OBJ file into 3ds Max 3D modeling software to modify the model, adjust the wiring, and adjust the normals; Step 6: Supplement, compare the panoramic view, split the models of different materials, and add components and facilities; Step seven, optimization, import the model into the 3D scene editor, perform post-processing on the model material to make it more realistic, and add effect processing.

2. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 1, characterized in that: In step 2, the point cloud registration Deep Closest Point algorithm is used to stitch the data of multiple scanning positions to form a complete point cloud data file. The integrity of the monocular endoscope point cloud file is checked again, including whether the point cloud coordinates are unified. After the check, useless data is eliminated.

3. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 2, characterized in that: In step 2, the steps for stitching multiple scan positions using the point cloud registration Deep Closest Point algorithm are as follows: S1: Feature extraction: input the ear canal point cloud data scanned by endoscope and perform feature extraction through 3D-CNN; S2: Graph construction and matching: construct a k-nearest neighbor-based graph in the feature space and use the nearest neighbor strategy to match point pairs; S3: Transformation estimation, solving the rotation and translation matrices through SVD decomposition; S4: Iterative update, Deep Closest Point improves the registration effect through continuous iteration until convergence.

4. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 3, characterized in that: In the step 2, the key function k nearest neighbor algorithm is used to calculate the distance between two points in the registration process using the point cloud registration Deep Closest Point algorithm to determine the distance between the source point cloud and the target point cloud. The Euclidean distance is used for distance measurement in the ear canal. The Euclidean distance is used to measure the absolute distance between two point clouds. Assuming that the dimension of the multidimensional space is N, X1=X 11 ,X 12 ,···,X 1N and X2 = X 21 ,X 22 ,···,X 2N are two points. Since the ear canal space is three-dimensional, N is 3. Then the Euclidean distance between X1 and X2 is calculated as 5. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 3, characterized in that: In the step 2, the Deep Closest Point algorithm is used for point cloud registration. The transformation estimation in step S3 uses SVD decomposition to solve the rotation and translation matrix. Assuming that there are two point clouds A and B, and the points therein correspond one to one, in order to find the rotation matrix R and translation matrix t between the two point clouds, the specific steps are as follows: S11: Calculate the center point, P=[x y z] T S12: To calculate the optimal rotation matrix R, we first need to re-center the point set to generate new A` and B`. A i `={P i A-U B } B i `={P i B-U B } Calculate the covariance H between the point sets The matrices U, S and V are obtained by the SVD method, and the rotation matrix R is calculated. [U, S, V] = SVD(H) R=VU T S13: Calculate the translation matrix t t=-R*U A +U B 。 6. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 1, characterized in that: In step 3, the specific steps of the deep learning feature extraction algorithm 3D-CNN are as follows: S111: extracting features from the ear canal point cloud image scanned by the monocular endoscope by using a principal component analysis method; S112: inputting the reference data and the extracted point cloud data into the 3D-CNN framework to extract deep spatial features; S113: combining the point cloud image scanned by the monocular endoscope with the reference data to extract point cloud features based on a balanced local discriminant embedding algorithm; S114: Superimpose the point cloud features and the depth features to extract the fusion features and input them into the classifier for feature classification.

7. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 1, characterized in that: In step 4, the sorted point cloud data is imported into the 3DReshape software, and the process of reconstructing the triangulated network is as follows: S1111: Import point cloud data, open 3DReshape software, find the file for importing point cloud data in the software interface, and select the data file to be imported; S1112: Processing data: after importing point cloud data, removing outliers and performing point cloud data registration is beneficial to improving the quality and accuracy of data and providing a basis for the generation of triangular meshes; S1113: Create a triangular network. Select the triangular network generation tool in the software. Set the parameters of the triangular mesh according to the distribution and characteristics of the point cloud data. In actual operation, select different algorithms and methods to generate a suitable triangular mesh according to different needs. S1114: Mesh optimization. After generating the triangular mesh, the triangular mesh is optimized by smoothing, subdividing, correcting topology and error operations.

8. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 1, characterized in that: In step 4, the process of exporting the generated triangular mesh to a 3D general format file is as follows: S11111: Check the export options. In the 3D modeling software you are using, open the triangular mesh model and enter the export options. S11112: Select the export format. In the export options, select the file format to be exported, OBJ; S11113: Configure export settings. In the export options, set the coordinate system, unit, size, normal and direction as needed. S11114: Determine the export path, select the export path, local file or specific device; S11115: Export the triangular mesh. After complete configuration, click the Export button to export the triangular mesh. S11116: Verify the export results. After the export is completed, you can use other 3D software that supports this format to open the exported file and verify whether the exported result is consistent with the original triangular mesh.

9. The method for high-precision three-dimensional modeling of the inner ear of the ear canal based on the monocular endoscope point cloud according to claim 1, characterized in that: In the feature extraction step of step three, the deep learning feature extraction algorithm 3D-CNN is used to identify the structural features inside the ear canal and the position and shape information of the inner ear, and the curvature and normal vector features are used to identify the ear canal wall and inner ear surface area.

Images