Fully automatic spatial registration system based on multimodal information fusion

Abstract

The present invention relates to a fully automatic spatial registration system based on multimodal information fusion, comprising: a preoperative planning module for fusing preoperative knee joint MRI and CT images to reconstruct a "bone + cartilage" combined navigation model; an intraoperative point cloud extraction module for scanning and automatically extracting intraoperative knee joint lesion cartilage surface point cloud data; and a spatial registration module for aligning the cartilage surface point cloud reconstructed in the preoperative CT image space with the cartilage surface point cloud data scanned intraoperatively, thereby achieving navigation registration between the intraoperative patient space and the preoperative image space. The present invention's point cloud-based registration method can achieve similar accuracy registration without selecting anatomical points, greatly reducing the degree of dependence on the doctor. This technology greatly assists doctors, allowing them to focus more on the surgery itself.

Description

Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a fully automatic spatial registration system based on multimodal information fusion. Background Art

[0002] Image-guided computer-assisted spatial registration (or registration) is a core technology for surgical navigation, enabling surgeons to better perform knee replacement surgery. Spatial registration is also a major factor limiting end-to-end clinical accuracy, impacting surgical navigation runtime.

[0003] Currently, spatial registration technologies in knee orthopedic navigation are mainly divided into three categories: reference point-based registration, anatomical point-based registration, and surface-based registration.

[0004] Fiducial makers based: Before the patient undergoes a CT scan, fiducials are implanted as reference points, which will be registered with the intraoperative data points. This method can simply and intuitively determine the correspondence between the image space and the patient space, and has the highest accuracy. However, this method requires the additional implantation of fiducials before surgery, and sometimes the fiducials need to be manually adjusted, and there may be risks of pain and infection. If marking is used on the skin surface, errors will be introduced due to the relative movement between the skin and bones, and the markers are easy to fall off. Therefore, this method is rarely used in clinical knee replacement surgery.

[0005] Since the reference points require additional preoperative operations, doctors use anatomical points (Landmark-based) instead of reference points. The registration method based on the probe picking up the surface points of the knee joint during surgery is very popular in clinical orthopedic navigation surgery. Before the operation, the patient's knee joint is collected with a small inter-slice spacing and high resolution medical tomographic image, and a high-precision three-dimensional medical tibiofemoral joint visualization model is segmented and reconstructed, and the anatomical points are marked on the model. During the intraoperative registration, only the corresponding anatomical points of the patient's tibiofemoral region during the operation need to be collected using a calibrated special probe [6], and the two are aligned for corresponding points. However, due to the small number of anatomical points, this method is easily affected by noise during the operation, the robustness of the registration is not strong, and the selection of anatomical points is time-consuming and labor-intensive and depends on the doctor's experience, so the cost is high.

[0006] Due to the above shortcomings of the anatomical point-based method, doctors use a surface point-based method for spatial registration. Surface-based registration methods can be divided into surface point set-based methods and surface point cloud methods. The surface point set-based method is currently the most commonly used method. During the operation, the doctor uses a digital probe to collect points in a specific area rather than just limited to bone anatomical points. During the operation, about 40 points are collected from both the femur and tibia, and the points are aligned with the points on the preoperative three-dimensional reconstructed model to complete the registration. However, this has the following problems: This manual digital point selection method is time-consuming and labor-intensive, and relies on the doctor's experience. For example, when blocked by bone spurs, it is difficult to locate the points or the located points are inaccurate, so the cost is high. It can be seen that registration takes up a certain amount of surgical time, which is also an urgent problem currently faced in knee replacement surgery navigation.

[0007] To save time in spatial registration, some studies have used surface-based registration methods using point clouds. Laser scanners are typically used to automatically and contactlessly acquire digital point cloud data from the joint surface during surgery. Extracting the point cloud of the lesion bone surface takes an average of 15 minutes, compared to invasive probe acquisition of bone surface data. Using a laser scanner only takes 4 minutes, and most of this time is spent on post-processing the scanned point cloud. Speeding up post-processing may further increase navigation accuracy and speed. However, this method has only been tested on bone surfaces and has not yet been effectively used in clinical practice.

[0008] In summary, the current clinical navigation system for knee replacement surgery often uses a marker-based intraoperative registration method for navigation, which involves the selection and confirmation of multiple markers, and the registration time is long, resulting in prolonged surgery.

[0009] This paper proposes a novel spatial registration system for knee replacement surgery based on multimodal fusion and point cloud registration. Based on preoperative multimodal image fusion information and the point cloud of knee lesions acquired by intraoperative scanners, the system registers the lesion point cloud to the preoperative images in real time. This enables rapid and accurate navigation of the intraoperative lesion from the preoperative images, significantly reducing intraoperative spatial registration time. Summary of the Invention

[0010] In order to solve the problem of long spatial registration time in computer-guided knee replacement surgery, the present invention provides a fully automatic spatial registration system based on multimodal information fusion, which can not only maintain high registration accuracy but also significantly reduce the time required for registration.

[0011] To achieve the above object, the present invention provides the following solutions:

[0012] A fully automatic spatial registration system based on multimodal information fusion, including:

[0013] Preoperative planning module: used to fuse preoperative knee joint MRI images and CT images to reconstruct a "bone + cartilage" combined navigation model;

[0014] Intraoperative point cloud extraction module: used to scan and automatically extract point cloud data of the cartilage surface of knee joint lesions during surgery;

[0015] Spatial registration module: used to align the cartilage surface point cloud reconstructed in the preoperative CT image space with the cartilage surface point cloud data scanned during the operation, so as to realize the navigation registration between the intraoperative patient space and the preoperative image space.

[0016] Preferably, the preoperative planning module includes:

[0017] An image fusion unit is configured to perform tissue segmentation on the acquired knee joint MRI image and the acquired CT image, obtain the tibia and femur through the MRI image, and obtain the tibia and femur and their corresponding cartilage through the CT image;

[0018] Model reconstruction unit: used to obtain the "bone + cartilage" navigation model through three-dimensional reconstruction, and extract the cartilage surface point cloud on the outer surface of the knee joint.

[0019] Preferably, the image fusion unit selects a target area based on the segmented tibia or femur, and maps the MRI image to the CT image using a mutual information registration method based on the target area to obtain a transformation matrix; the transformation matrix maps the femoral or tibial cartilage segmented from the MRI image to the CT image, respectively, for fusing cartilage information, and constructs the "bone + cartilage" navigation model through the model reconstruction unit, and extracts the cartilage surface point cloud on the outer surface of the knee joint based on the navigation model.

[0020] Preferably, the intraoperative point cloud extraction module includes:

[0021] Scanning extraction unit: used to collect the surface point cloud of the patient's knee joint lesion area during surgery, and automatically extract the cartilage surface point cloud of the knee joint lesion area based on the FPFH_PointNet neural network.

[0022] Preferably, collecting the surface point cloud of the patient's knee joint lesion area during surgery includes: obtaining the surface point cloud of the patient's knee joint lesion area during surgery through a scanner, converting the surface point cloud of the lesion area to obtain the surface point cloud of the lesion area in the locator space, that is, the patient space point cloud, and removing the background point cloud in the patient space point cloud.

[0023] Preferably, obtaining the surface point cloud of the lesion area in the locator space includes:

[0024] Solve the calibration transformation T of the scanner in the adapter space scan→adapter, the calibration transformation T of the adapter in the locator space adapter→polaris , is transformed by the following formula:

[0025] P lesion =P scan ×T scan→adapter ×T adapter→polaris

[0026] Among them, P lesion is the lesion surface point cloud in the locator space, P scan The surface point cloud of the lesion area obtained by the scanner.

[0027] Preferably, extracting the cartilage surface point cloud of the knee joint lesion area includes:

[0028] The PointNet neural network is improved by using the fast point feature histogram FPFH to construct an FPFH_PointNet network; based on the FPFH_PointNet network, the cartilage area and the non-cartilage area in the knee joint lesion area are distinguished, irrelevant background is removed, and the cartilage surface point cloud is extracted.

[0029] Preferably, the space registration module includes:

[0030] Hierarchical registration unit: used to register the cartilage surface point cloud reconstructed in the preoperative CT image space with the cartilage point cloud scanned during the operation.

[0031] Preferably, performing registration includes:

[0032] Several corresponding points were selected interactively using a graphical interface, and a coarse registration was performed based on the SVD method to ensure that the directions of the two sets of point clouds before and during surgery were consistent. After completing the coarse registration, the fine registration was initialized, and the surface registration of the two sets of point clouds was achieved using the ICP method, and the intraoperative patient space was registered with the preoperative image space.

[0033] The beneficial effects of the present invention are:

[0034] The present invention utilizes preoperative fusion of CT and MRI images to provide soft tissue information, thereby facilitating better preoperative planning and intraoperative incision location determination based on soft tissue information.

[0035] In the computer-assisted navigation knee replacement surgery provided by the present invention, the selection of anatomical points often requires experienced doctors. The more accurate the anatomical points are, the more precise the surgical registration is. However, the point cloud-based registration method can obtain registration with similar accuracy without the need to select anatomical points, greatly reducing the dependence on doctors. This technology greatly assists doctors, allowing them to focus more on the surgery itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is the overall framework diagram of the system of the present invention;

[0038] Figure 2 Schematic diagram of the conversion of a point cloud from scanner space to patient space according to an embodiment of the present invention; (i) is the conversion of a point cloud from scanner space to adapter space, (ii) is the conversion from adapter space to locator space, and (iii) is the locator space coordinate, which is considered to be patient space.

[0039] Figure 3 is the PFH local coordinate system of an embodiment of the present invention;

[0040] Figure 4 1 is a diagram showing the FPFH calculation principle of an embodiment of the present invention;

[0041] Figure 5 This is the FPFH_PointNet network model structure of an embodiment of the present invention;

[0042] Figure 6 This is a flowchart of the application of the system in clinical knee arthritis patients according to an embodiment of the present invention;

[0043] Figure 7 The bone and cartilage segmentation results of CT and MRI images of the same patient's knee joint are shown in an embodiment of the present invention. The first to third rows show the CT image and its bone segmentation results, and the MRI image and its bone and cartilage segmentation results from the axial, coronal, and sagittal planes, respectively.

[0044] Figure 8 A schematic diagram showing the registration results and navigation model of an embodiment of the present invention;

[0045] Figure 9 The results of automatically extracting intraoperative cartilage surface point clouds using different neural networks according to an embodiment of the present invention;

[0046] Figure 10 In the embodiment of the present invention, the surface registration method is applied to the registration error distribution of the distal femur and the proximal tibia respectively. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] The fully automatic spatial registration system based on multimodal information fusion provided by the present invention includes three modules: (I) a preoperative planning module (preoperative planning module) that fuses preoperative knee joint MRI and CT images and reconstructs a "bone + cartilage" combined navigation model; (II) an intraoperative point cloud extraction module (intraoperative point cloud extraction module) that scans and automatically extracts the surface point cloud of the knee joint lesion cartilage during surgery; and (III) a spatial registration module (spatial registration module) that aligns the surface point cloud, the intraoperative patient, and the preoperative image. The overall architecture of the system is as follows: Figure 1 As shown. Among them:

[0050] (I) A preoperative planning module for reconstructing a "bone + cartilage" combined navigation model using MRI and CT images of the knee joint before fusion surgery: for preoperative CT and MRI images of the knee joint of the same patient, tissue segmentation is performed on the two modal images, the tibia and femur are obtained from the CT image, and the tibia and femur and their corresponding cartilage are obtained from the MRI image; based on the segmented tibia or femur region of interest, the MRI image is mapped to the CT image using a mutual information registration method to obtain a transformation matrix; the transformation matrix maps the femoral or tibial cartilage segmented from the MRI image to the CT image, respectively, and the cartilage information is integrated into the bone structure; a "bone + cartilage" navigation model is obtained through three-dimensional reconstruction, and a cartilage surface point cloud on its outer surface can be extracted;

[0051] (II) an intraoperative point cloud extraction module for scanning and automatically extracting a point cloud of the cartilage surface of the intraoperative knee joint lesion: using a laser scanner to collect a surface point cloud of the patient's intraoperative knee joint lesion area, and then automatically extracting the cartilage surface point cloud using an FPFH_PointNet neural network;

[0052] (III) The hierarchical registration of surface point clouds and the spatial registration module of intraoperative patients and preoperative images: the cartilage surface point cloud reconstructed in the preoperative CT image space is registered with the cartilage point cloud scanned during the operation, including coarse registration based on singular value decomposition (SVD) and fine registration based on iterative conditional point (ICP), ultimately achieving navigation registration between the intraoperative patient space and the preoperative image space.

[0053] The three modules are further explained in detail below.

[0054] (1) The preoperative planning module (I) for reconstructing the "bone + cartilage" combined navigation model based on the MRI and CT images of the knee joint before fusion surgery includes the following tasks:

[0055] Segment the femur or tibia from the preoperative CT image, denoted as H ct , the femur or tibia and its corresponding cartilage were segmented from the preoperative MRI image, denoted as H mri and C mri ;

[0056] The bone structure H is segmented based on the preoperative CT image and MRI image of the patient. ct and H mri Perform linear registration based on mutual information and obtain the transformation matrix T mri→ct ;

[0057] The transformation matrix T mri→ct The cartilage area segmented on the MRI image can be transformed into the CT image, that is, The cartilage information on the CT image is enhanced, denoted as

[0058] H on CT images ct and Fusion, through 3D reconstruction, obtain the "bone + cartilage" combined navigation model of the preoperative image space

[0059] The surface of the navigation model is the cartilage surface, and its surface points can be automatically acquired as the navigation point cloud of the preoperative image space, denoted as P image .

[0060] (II) Intraoperative point cloud extraction module (II) scans and automatically extracts point clouds from the cartilage surface of the knee joint lesion during surgery. Its work includes:

[0061] The calibrated laser scanner is used to collect the intraoperative knee joint lesion area point cloud P in the locator space. lesion . The specific description is (reference Figure 2 ): The scanner scanned the surface point cloud P of the patient's knee joint lesion area during surgery scan , P scan Need to convert to locator space P polaris , to become a patient space point cloud. The conversion process involves solving the scanner calibration transformation T in the adapter space. scan→adapter , the calibration transformation T of the adapter in the locator space adapter→polaris , where the adapter is already fixed to the scanner. Once the calibration transformation is calculated experimentally, P can be converted to scan Transform to the locator space and get P lesion :

[0062] P lesion =P scan ×T scan→adapter ×T adapter→polaris

[0063] P lesion It is the point cloud of the lesion surface in the locator space, where the locator space is also the patient space.

[0064] P lesion The point cloud not only includes the cartilage surface point cloud, but also the point cloud of other irrelevant background such as muscles and ligaments around the cartilage. These background point clouds need to be removed to avoid affecting the accuracy of intraoperative navigation registration.

[0065] Based on the PointNet neural network, the Fast Point Feature Histogram (FPFH) is used to improve the PointNet neural network and construct the FPFH_PointNet network. This network can accurately distinguish P lesion The cartilage area and non-cartilage area in the surgery are automatically and accurately extracted by removing ligaments, muscles and other irrelevant backgrounds. patient .

[0066] The first is the point feature histogram description. Figure 3 The figure shows the local coordinate system of the common point cloud local feature descriptor, namely the point feature histogram (PFH). s and p t They are two points in the point cloud, n s and n t are their normal vectors, for example, p s =(x s ,y s , z s ), For point p s , construct the uvw coordinate system, namely:

[0067]

[0068] Then n s and n t The angular difference between can be expressed as (α, φ, θ, d), where

[0069]

[0070] Fast Point Feature Histograms (FPFH) Figure 4 As shown, the red center point p q , and their neighboring points are p k1 ~p k5 Establish a local coordinate system and obtain the quadruple (α, φ, θ, d) associated with each neighboring point. Then use p k1 ~p k5 Repeat the above process for the center. Each feature interval is divided into 11 parts for statistics and concatenated into a 33-dimensional vector. The FPFH complexity is O(nk).

[0071] Secondly, the FPFH is used to improve the PointNet neural network and establish the FPFH_PointNet neural network, which can automatically and accurately extract the cartilage surface point cloud. Since the PointNet network lacks sufficient local feature information, the FPFH descriptor that can reflect the local features of the point cloud is fused into the PointNet network, referred to as FPFH_PointNet. The overall calculation method of the network is shown in Figure 5 For the input point cloud, first calculate its FPFH feature. Since the calculation of FPFH feature requires the normal vector information of the point, the coordinates (x i ,y i , z i ) and normal vector information The 33-dimensional FPFH features are combined to form 39-dimensional input features. These features are then passed through the Multiple Level Perception (MLP) layer, which is a 5-layer MLP network with dimensionality increased, outputting feature maps with dimensions of (64, 128, 128, 512, 2048). A max-pooling operation is then performed on the n×2048 feature map to extract global features. The global features are 1×2048 vectors, which are expanded to n×2048 dimensions. Finally, the output features of the 5 intermediate layers are concatenated with the expanded global features and the originally calculated FPFH features to form n×4964-dimensional features. These features are then passed through three MLP layers, and finally a softmax function is used to output the predicted score for each point in the point cloud.

[0072] (III) The spatial registration module (III) for hierarchical registration of surface point clouds, intraoperative patient images, and preoperative images includes the following tasks:

[0073] (1) Image space point cloud P image and patient space point cloud P patient Each contains more than 10,000 points, with significant differences in orientation and position. Using a graphical interface, 3 to 5 pairs of corresponding points are interactively selected and a coarse registration based on SVD is performed to ensure the orientation of the two point clouds before and during surgery, and to initialize the subsequent fine registration.

[0074] (2) Once the initial positions of the two sets of point clouds are given, the ICP method can be used to quickly and accurately achieve surface registration of the two sets of point clouds, thereby registering the intraoperative patient space to the preoperative image space.

[0075] The present invention is further described below with reference to examples and drawings.

[0076] Figure 6 The figure is a flowchart of the application of the system of the present invention in clinical knee arthritis patients. Here, the femur is used as an example to explain the registration system of the present invention.

[0077] Module I is the process of enhancing the preoperative CT image and obtaining the spatial coordinates of the preoperative cartilage surface point cloud. The femur and its cartilage are segmented from the preoperative CT and MRI images respectively, and the registration transformation matrix T of CT and MRI is obtained based on the segmented femoral ROI area. mri→ct The transformation matrix obtained by registration acts on the cartilage label obtained by MRI image segmentation, and the femoral cartilage can be mapped to the corresponding structure of the CT image. The label fusion structure of the femur and its cartilage can be obtained. After three-dimensional reconstruction, a model with cartilage is obtained, from which the cartilage surface point cloud P is extracted. image , which will be used for intraoperative navigation.

[0078] Module II is the process of collecting and extracting the point cloud of the cartilage surface in the lesion area of ​​the knee joint in the patient space during surgery. First, the scanner is calibrated before surgery to obtain T scan→adapter Then the point cloud P of the lesion area is collected by the scanning system (composed of a locator and a scanner equipped with an adapter). scan , which will be transformed into the patient space point cloud P through coordinate space transformation lesion Then, the FPFH_PointNet network is used to remove irrelevant point clouds and obtain the patient space point cloud P patient .

[0079] Module III is the point cloud P obtained before surgery image Compared with the point cloud P obtained during the operation patientThe registration process is as follows: First, the SVD algorithm is used for rough registration to obtain a good initial position, but there is still a large error. Then, the ICP algorithm is used to further optimize the position of the two point clouds to achieve the optimal match between the two sets of point clouds.

[0080] Figure 7 The first to third rows show the CT image and its bone segmentation results, and the MRI image and its bone and cartilage segmentation results, respectively, from the axial, coronal, and sagittal planes.

[0081] Figure 8 To demonstrate the registration results and their navigation models, the distal femur and its cartilage, and the proximal tibia and its cartilage on the MRI image are registered and superimposed onto the corresponding structures on the CT image, shown in light gray. The fused registration results are then reconstructed into navigation models of the femur and tibia. Figure 8 3D printing is also used to demonstrate the system construction of intraoperative navigation.

[0082] Figure 9 Results of automatically extracting intraoperative cartilage surface point clouds using different neural networks. Compared to the gold standard segmentation point cloud (Ground Truth), both the PointNet and PointNet++ networks exhibit over-segmentation (green arrows) and under-segmentation (yellow arrows), while our FPFH_PointNet network achieves the best cartilage surface point cloud segmentation results.

[0083] The present invention was validated on MRI and CT images of four patients with knee arthritis. A "bone + cartilage" combined navigation model was constructed. In order to evaluate the performance of the present invention (surface registration) method, it was compared with the commonly used marker-based registration method (denoted as gold standard transformation). Here, in order to scan the smooth cartilage surface, five reference markers were set on the non-cartilage area of ​​each patient's distal femur model and proximal tibia model, rather than the cartilage surface of the distal femur and proximal tibia. Table 1 shows the average reference registration error (FRE) of the gold standard transformation and surface registration transformation of the four patients, as well as the total average registration error of the four subjects. The total average reference registration errors of the gold standard transformation and surface registration of the distal femur were 0.89 mm and 1.61 mm, respectively; and 0.74 mm and 1.85 mm, respectively, for the proximal tibia. Table 1 also lists the average surface registration error (SRE). The SRE of the distal femur and proximal tibia were 0.29 mm and 0.27 mm, respectively.

[0084] Figure 10 The registration error distributions of the surface registration method applied to the distal femur and proximal tibia are shown.

[0085] Table 2 shows the time taken for each step of the surface registration method, including scanning the point cloud, automatically segmenting the point cloud, coarse registration, and fine registration. Table 2 shows that the total time for the femur and tibia registration is less than 2 minutes, which is significantly lower than existing navigation systems.

[0086] Table 1

[0087]

[0088] Table 2

[0089]

[0090] The present invention has the following advantages:

[0091] Significantly reduces intraoperative registration time. Current surgical navigation systems require a probe to pick up paired points for intraoperative registration, typically selecting 15 to 17 points. This process of picking up anatomical points consumes a significant amount of intraoperative time. Scanner-based intraoperative navigation, however, can quickly scan a point cloud of the cartilage surface and align it with the preoperative image space point cloud, typically taking about two minutes, significantly reducing surgical time.

[0092] The use of preoperative fused CT and MRI images can provide soft tissue information, which helps doctors make better preoperative planning and determine the location of the incision during surgery based on the soft tissue information.

[0093] In computer-assisted navigation knee replacement surgery, selecting anatomical points often requires experienced surgeons. The more accurately these points are selected, the more precise the surgical registration. However, point cloud-based registration methods achieve similar registration accuracy without requiring anatomical points, significantly reducing the surgeon's reliance on the technology. This technology significantly assists surgeons, allowing them to focus more on the surgery itself.

[0094] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A fully automatic spatial registration system based on multimodal information fusion, characterized by: include: Preoperative planning module: used to fuse preoperative knee joint MRI and CT images to reconstruct a "bone + cartilage" combined navigation model; Intraoperative point cloud extraction module: used to scan and automatically extract point cloud data of the cartilage surface of knee joint lesions during surgery; Spatial registration module: used to align the cartilage surface point cloud reconstructed in the preoperative CT image space with the cartilage surface point cloud data scanned during the operation, so as to realize the navigation registration between the intraoperative patient space and the preoperative image space; The intraoperative point cloud extraction module includes: Scanning extraction unit: used to collect the surface point cloud of the knee joint lesion area during the patient's operation, and automatically extract the cartilage surface point cloud of the knee joint lesion area based on the FPFH_PointNet neural network; improve the PointNet neural network through the fast point feature histogram FPFH to construct the FPFH_PointNet network; for the input point cloud, first calculate its FPFH feature, since the calculation of the FPFH feature requires the normal vector information of the point, the coordinates of each point are converted to and normal vector information And 33-dimensional FPFH features are combined to form 39-dimensional input features, that is, the dimensions are upgraded in sequence through 5 layers of MLP network, and the output dimensions are (64, 128, 128, 512, 2048) feature maps; then the max-pooling operation is performed on the n×2048 feature map to extract the global features; the global feature is a 1×2048 vector, which is expanded to n×2048 dimensions. Finally, the output features of the 5 middle layers are spliced ​​with the expanded global features and the originally calculated FPFH features to form n×4964-dimensional features, and then passed through three layers of MLP layers respectively, and finally the softmax function is used to output the prediction score of each point in the point cloud.

2. The fully automatic spatial registration system based on multimodal information fusion according to claim 1 is characterized in that: The preoperative planning module includes: An image fusion unit is configured to perform tissue segmentation on the acquired knee joint MRI image and the acquired CT image, obtain the tibia and femur through the MRI image, and obtain the tibia and femur and their corresponding cartilage through the CT image; Model reconstruction unit: used to obtain the "bone + cartilage" navigation model through three-dimensional reconstruction and extract the cartilage surface point cloud on the outer surface of the knee joint.

3. The fully automatic spatial registration system based on multimodal information fusion according to claim 2 is characterized in that: The image fusion unit selects a target area based on the segmented tibia or femur, and maps the MRI image to the CT image using a mutual information registration method based on the target area to obtain a transformation matrix; The transformation matrix maps the femoral or tibial cartilage segmented from the MRI image onto the CT image, respectively, for fusing cartilage information. The "bone + cartilage" navigation model is constructed by the model reconstruction unit, and the cartilage surface point cloud on the outer surface of the knee joint is extracted based on the navigation model.

4. The fully automatic spatial registration system based on multimodal information fusion according to claim 1 is characterized in that: Collecting the surface point cloud of the patient's knee joint lesion area during surgery includes: obtaining the surface point cloud of the patient's knee joint lesion area during surgery through a scanner, converting the surface point cloud of the lesion area to obtain the surface point cloud of the lesion area in the locator space, that is, the patient space point cloud, and removing the background point cloud in the patient space point cloud.

5. The fully automatic spatial registration system based on multimodal information fusion according to claim 4 is characterized in that: Obtaining the surface point cloud of the lesion area in the locator space includes: Solve the calibration transformation of the scanner in the adapter space , calibration transformation of the adapter in the locator space , is transformed by the following formula: ， in, is the lesion surface point cloud in the locator space, The surface point cloud of the lesion area obtained by the scanner.

6. The fully automatic spatial registration system based on multimodal information fusion according to claim 1 is characterized in that: Extracting the cartilage surface point cloud of the knee joint lesion area includes: The cartilage area and the non-cartilage area in the knee joint lesion area are distinguished based on the FPFH_PointNet network, irrelevant background is removed, and the cartilage surface point cloud is extracted.

7. The fully automatic spatial registration system based on multimodal information fusion according to claim 1 is characterized in that: The space registration module includes: Hierarchical registration unit: used to register the cartilage surface point cloud reconstructed in the preoperative CT image space with the cartilage point cloud scanned during the operation.

8. The fully automatic spatial registration system based on multimodal information fusion according to claim 7 is characterized in that: The registration comprises: Several corresponding points were selected interactively using a graphical interface, and a coarse registration was performed based on the SVD method to ensure that the directions of the two sets of point clouds before and during surgery were consistent. After completing the coarse registration, the fine registration was initialized, and the surface registration of the two sets of point clouds was achieved using the ICP method, and the intraoperative patient space was registered with the preoperative image space.

Images