Multi-modal medical image 3D reconstruction and visualization system for surgical planning

Abstract

The invention discloses a multi-modal medical image 3D reconstruction and visualization system for surgical planning, and relates to the technical field of medical image processing, and the system comprises a multi-modal image collection and preprocessing module which is used for collecting and preprocessing medical images including CT, MRI and PET modalities; the multi-modal image segmentation module is used for accurately segmenting the multi-modal image; the 3D reconstruction and fusion module is used for reconstructing and fusing a 3D model through a multi-scale reconstruction algorithm; the intra-operation augmented reality visualization module projects the reconstructed 3D model to an operation area through a self-adaptive projection algorithm; the error analysis and correction module is used for ensuring that the real-time error correction precision in surgical navigation reaches the submicron level; the feature matching and fusion module ensures a multi-modal fusion effect through feature matching and performs nonlinear fusion by using feature splicing; according to the multi-modal medical image 3D reconstruction and visualization system for surgical planning, the limitation of a traditional surgical planning system is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multimodal medical image 3D reconstruction and visualization system for surgical planning. Background Art

[0002] With the rapid development of medical imaging technology, multimodal medical imaging is increasingly used in surgical planning. Multimodal medical image fusion and processing technology can provide more comprehensive patient anatomical structure and functional information by integrating imaging data of different modalities such as CT, MRI, PET, and X-ray, thereby helping doctors to develop more accurate surgical plans and guide surgical robots to complete high-precision surgical operations.

[0003] Traditional surgical planning systems mainly rely on preoperative 3D imaging data, such as CT or MRI images, to generate a virtual model of the patient's anatomical structure through preoperative image registration and 3D reconstruction, so that surgeons can make detailed plans before surgery. However, these systems usually have some limitations, such as the inability to update intraoperative images in real time, which may lead to the problem that anatomical changes may not be reflected in time during surgery.

[0004] In recent years, the application of augmented reality (AR) technology in surgical planning has gradually increased. AR technology can superimpose virtual images generated before surgery on the actual surgical site, helping surgeons to more intuitively identify the location and boundaries of lesions and improve the accuracy and safety of surgery. In addition, combined with real-time intraoperative imaging data, such as fluoroscopic X-ray images and ultrasound images, the real-time and dynamic adaptability of surgical navigation systems can be further improved.

[0005] In order to overcome the limitations of traditional surgical planning systems, this paper proposes a multimodal medical image 3D reconstruction and visualization system for surgical planning. The system not only supports offline analysis of preoperative images, but also processes intraoperative image data in real time, meeting the edge computing and cloud collaborative processing requirements in different surgical scenarios. Through the collaborative work of augmented reality navigation information and robots, fully automated surgical planning and execution are achieved, thereby greatly improving the safety and efficiency of surgery. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal medical image 3D reconstruction and visualization system for surgical planning to address the limitations of traditional surgical planning systems.

[0007] To achieve the above objectives, the present invention proposes a multimodal medical image 3D reconstruction and visualization system for surgical planning, comprising: Multimodal image acquisition and preprocessing module, used to acquire and preprocess medical images including CT, MRI, and PET modalities, while using the real-time processing module based on edge computing to optimize the real-time performance of image transmission and processing; Multimodal image segmentation module, used to accurately segment multimodal images and dynamically optimize segmentation results; 3D reconstruction and fusion module, which reconstructs and fuses 3D models through multi-scale reconstruction algorithms to generate accurate three-dimensional anatomical structure models; Intraoperative augmented reality visualization module, which projects the reconstructed 3D model onto the surgical area through an adaptive projection algorithm, and supports real-time adjustment of the projection viewing angle and depth perception; Error analysis and correction module, used to ensure sub-micron accuracy of real-time error correction in surgical navigation; Feature matching and fusion module,The feature matching module is used to perform feature matching between different modalities to ensure the multimodal fusion effect of the system. The feature fusion module uses feature splicing operations for nonlinear fusion to integrate the feature information of multiple modalities and improve the quality and accuracy of the reconstructed image.

[0008] Preferably, the multimodal image acquisition and preprocessing module integrates intelligent denoising and contrast enhancement technology, and automatically and dynamically adjusts the denoising and enhancement algorithms according to the image quality to optimize the image processing effect.

[0009] Preferably, the multimodal image segmentation module adopts a model that combines reinforcement learning and deep learning, extracts key features of single-modal images through a modality-preserving encoder, and fuses multimodal data through a modality fusion decoder. Through a neural network based on transfer learning, combined with intraoperative real-time images and historical image data, the weight of the segmentation model is automatically adjusted to improve the adaptability of image segmentation.

[0010] Preferably, the segmentation model of the multimodal image segmentation module is based on the following formula: ; in, represents the actual segmentation result, is the predicted output of the model.

[0011] Preferably, the 3D reconstruction and fusion module uses a multi-scale reconstruction algorithm to perform 3D reconstruction. The specific formula is as follows: ; in, represents the model after 3D reconstruction, is the volume data of each mode, is the reconstruction coefficient.

[0012] Preferably, the 3D reconstruction and fusion module uses a multimodal mutual information optimization algorithm, and the specific formula is as follows: ; in, The 3D reconstruction and fusion module uses this algorithm to fuse the information of different modal images, automatically balance the weights of different modal data, and reduce the multimodal image fusion error.

[0013] Preferably, the intraoperative augmented reality visualization module is based on holographic augmented reality technology. Based on the holographic augmented reality technology, the holographic image is aligned with the intraoperative image in real time. The formula for holographic augmented reality model projection is as follows: ; in, For holographic augmented reality projection, It is the projection angle transformation matrix. This module realizes the real-time floating display of holographic images in surgical navigation.

[0014] Preferably, the error analysis and correction module tracks the patient's real-time position through an optical-magnetic field joint positioning system, and combines image-patient registration technology to perform intraoperative error analysis and correction. The error analysis formula is as follows: ; in, represents the actual patient location, is the position of the projected image.

[0015] Preferably, the matching formula of the spectral feature and the texture feature of the feature matching and fusion module is: ; in, are the spectral and texture characteristics of the hyperspectral image, Spectral and texture features of pathological images; The matching formula of spatial features and morphological features is: ; in, is the spatial feature, For the morphological characteristics.

[0016] Preferably, the feature fusion of the feature matching and fusion module uses the formula of feature splicing for nonlinear fusion as follows: ; in, is the concatenated feature vector, is a nonlinear transformation function used to generate the final fusion features.

[0017] The present invention adopts the above-mentioned multimodal medical image 3D reconstruction and visualization system for surgical planning, which has the following beneficial effects: (1) Seamless integration, achieving fully automated surgical planning and execution through the collaborative work of augmented reality navigation information and robots; (2) Supports offline analysis of preoperative images and real-time processing of intraoperative images, meeting the needs of edge computing and cloud collaborative processing in different surgical scenarios.

[0018] (3) It integrates cutting-edge technologies such as multimodal image processing, deep learning, and augmented reality, especially real-time error correction through quantum computing, which greatly improves the accuracy and safety of surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a module diagram of a multimodal medical image 3D reconstruction and visualization system for surgical planning according to the present invention. DETAILED DESCRIPTION

[0020] Example In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, the present invention proposes a multimodal medical image 3D reconstruction and visualization system for surgical planning, including a multimodal image acquisition and preprocessing module, a multimodal image segmentation module, a 3D reconstruction and fusion module, an intraoperative augmented reality visualization module, an error analysis and correction module, and a feature matching and fusion module.

[0023] Multimodal image acquisition and preprocessing module,First, this system acquires multimodal medical images including CT, MRI, PET, etc., and performs real-time preprocessing through the integrated edge computing module. The preprocessing steps include but are not limited to noise removal, geometric correction, image standardization and other operations, aiming to improve image quality and ensure the accuracy of subsequent segmentation and reconstruction. After preprocessing, the image data is stored in a standardized DICOM format for subsequent analysis and processing.

[0024] Multimodal image segmentation module: The segmentation of multimodal images adopts a model that combines reinforcement learning and deep learning. The modality-preserving encoder extracts the key features of the single-modal image, and the modality fusion decoder fuses the multimodal data. The segmentation model is based on the following formula: ; in, represents the actual segmentation result, It is the predicted output of the model. By continuously iterating and optimizing the segmentation model, the system can gradually improve the segmentation accuracy. In addition, the system uses the reinforcement learning module to adaptively adjust the weight of the model to ensure that the segmentation results remain highly accurate during real-time surgery.

[0025] 3D reconstruction and fusion module, based on the segmentation results, the system uses a multi-scale reconstruction algorithm for 3D reconstruction. The specific formula is as follows: ; in, represents the model after 3D reconstruction, is the volume data of each mode, is the reconstruction coefficient. In order to ensure the accuracy of fusion, the system uses the mutual information optimization algorithm: ; in, Represents probability. This formula ensures that the fusion information of different modal images is maximized and the error is reduced.

[0026] Intraoperative augmented reality visualization module: During the operation, the system uses augmented reality technology to project the 3D model to the surgical area in real time, and generates a 3D suspended image based on holographic imaging technology. The intraoperative projection uses an adaptive projection algorithm to adjust the projection angle and depth in real time according to the changes in the patient's position to ensure accuracy. The formula for holographic augmented reality model projection is as follows: ; in, For holographic augmented reality projection, is the projection angle transformation matrix.

[0027] Error analysis and correction module: The system uses a parallel processing algorithm driven by quantum computing to analyze and correct navigation errors generated during surgery. The error analysis formula is as follows: ; in, represents the actual patient location, The system is able to correct the position of the projected image by minimizing the error value. In addition, quantum computing is used to process large amounts of real-time data, speeding up the correction and improving the error accuracy to the sub-micron level.

[0028] Feature matching and fusion module, in order to enhance the multimodal fusion effect of the system, feature matching is performed between different modalities. The matching formula of spectral features and texture features is: ; in, are the spectral and texture characteristics of the hyperspectral image, Spectral and texture features of pathological images.

[0029] The matching formula of spatial features and morphological features is: ; in, is the spatial feature, For the morphological characteristics.

[0030] Feature fusion uses feature concatenation for nonlinear fusion, and the formula is: ; in, is the concatenated feature vector, is a nonlinear transformation function used to generate the final fusion features.

[0031] Hyperspectral images and pathological images are used as input and feature extraction is performed through their respective encoders. The encoder compresses high-dimensional data into low-dimensional feature representations to extract key information. The encoded feature representation is further compressed through a 1×1 convolution operation to obtain a 128-dimensional feature vector.

[0032] The fusion module uses the Transformer self-attention mechanism to perform feature fusion through the following formula: ; Where Q, K and V are query, key and value matrices respectively. is the feature dimension. The Transformer module can capture the dependency between the hyperspectral image and the pathological image and generate a deeply fused feature representation.

[0033] The decoder remaps the fused features back to the high-dimensional space and undergoes a convolution operation to finally generate a fused feature map. The formula is: ; in, It is a 3×3 convolution operation. The generated fusion feature map includes comprehensive information from multimodal images and has higher discrimination ability.

[0034] The fused feature map output by the final system combines the spectral and spatial information of the hyperspectral image and the morphological, texture and depth features of the pathological image. This feature map can be used for applications such as disease diagnosis, image classification, and lesion detection. In order to enhance the interpretability of the diagnosis results, the system generates a diagnosis report containing important features and stores it in an encrypted database for pathologists to check.

[0035] Therefore, the present invention adopts the above-mentioned multimodal medical image 3D reconstruction and visualization system for surgical planning, which not only integrates cutting-edge technologies such as multimodal image processing, deep learning, and augmented reality, greatly improving the accuracy and safety of surgery, but also realizes fully automated surgical planning and execution, meeting the remote collaborative processing needs in different surgical scenarios.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal medical image 3D reconstruction and visualization system for surgical planning, characterized by: Multimodal image acquisition and preprocessing module, used to acquire and preprocess medical images including CT, MRI, and PET modalities, while using the real-time processing module based on edge computing to optimize the real-time performance of image transmission and processing; Multimodal image segmentation module, used to accurately segment multimodal images and dynamically optimize segmentation results; 3D reconstruction and fusion module, which reconstructs and fuses 3D models through multi-scale reconstruction algorithms to generate three-dimensional anatomical structure models; Intraoperative augmented reality visualization module, which projects the reconstructed 3D model onto the surgical area through an adaptive projection algorithm, adjusting the projection viewing angle and depth perception in real time; Error analysis and correction module, used to adjust the real-time error correction accuracy in surgical navigation to sub-micron level; Feature matching and fusion module,The feature matching module is used to perform feature matching between different modalities, and the feature fusion module uses feature splicing operations to perform nonlinear fusion and integrate the feature information of multiple modalities.

2. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 1, characterized in that: The multimodal image acquisition and preprocessing module integrates intelligent denoising and contrast enhancement, and dynamically adjusts the denoising and enhancement algorithms according to the image quality.

3. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 1, characterized in that: The multimodal image segmentation module adopts a model that combines reinforcement learning and deep learning. It extracts the key features of single-modal images through a modality-preserving encoder and fuses multimodal data through a modality fusion decoder.

4. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 3, characterized in that: The segmentation model of the multimodal image segmentation module is based on the following formula: ； in, represents the actual segmentation result, is the predicted output of the model.

5. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 1, characterized in that: The 3D reconstruction and fusion module uses a multi-scale reconstruction algorithm for 3D reconstruction. The specific formula is as follows: ； in, represents the model after 3D reconstruction, is the volume data of each mode, is the reconstruction coefficient.

6. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 5, characterized in that: The 3D reconstruction and fusion module uses a multimodal mutual information optimization algorithm. The specific formula is as follows: ； in, Represents probability.

7. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 5, characterized in that: The intraoperative augmented reality visualization module is based on holographic augmented reality technology, which aligns the holographic image with the intraoperative image in real time. The formula for holographic augmented reality model projection is as follows: ； in, For holographic augmented reality projection, is the projection angle transformation matrix.

8. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 1, characterized in that: The error analysis formula of the error analysis and correction module is as follows: ； in, represents the actual patient location, is the position of the projected image.

9. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 1, characterized in that: The matching formula of spectral features and texture features of the feature matching and fusion module is: ； in, are the spectral and texture characteristics of the hyperspectral image, Spectral and texture features of pathological images; The matching formula of spatial features and morphological features is: ； in, is the spatial feature, For the morphological characteristics.

10. The multimodal medical image 3D reconstruction and visualization system for surgical planning according to claim 9, characterized in that: The feature fusion of the feature matching and fusion module uses the formula of feature splicing for nonlinear fusion: ； in, is the concatenated feature vector, is a nonlinear transformation function used to generate the final fusion features.

Images