A three-dimensional reconstruction method and system based on multi-modal medical image fusion

Abstract

The application belongs to the field of medical image processing, and relates to a three-dimensional reconstruction method and system based on multi-modal medical image fusion, comprising: acquiring an original multi-modal image sequence; the original multi-modal image sequence comprises CT images and MRI images; performing spatial alignment on the original multi-modal image sequence to obtain a spatially aligned image pair; the spatially aligned image pair comprises an aligned CT image and a corresponding aligned MRI image; performing multi-scale layered cognitive feature extraction on the spatially aligned image pair to obtain a corrected feature atlas; performing space-time-frequency domain collaborative attention fusion on the corrected feature atlas to obtain a fused feature; and performing topological constraint three-dimensional diffusion reconstruction on the fused feature to obtain a medical image three-dimensional model; and the three-dimensional reconstruction precision of multi-modal fusion is improved, and more reliable three-dimensional model support is provided for clinical diagnosis and treatment.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and specifically discloses a three-dimensional reconstruction method and system based on multimodal medical image fusion. Background Technology

[0002] Existing technologies standardize and normalize CT data of the area to be examined, then use 3D reconstruction software to generate accurate 3D models. Virtual examinations are then performed on these 3D models, capturing virtual images containing the target structure along with corresponding sagittal, coronal, and axial views to form a multimodal dataset. DICOM format CT images are then fused with these captured images, and multimodal data features are incorporated to train the 3D-UNet model, enhancing its ability to segment complex structures. However, existing technologies have several limitations in 3D reconstruction using multimodal medical image fusion: feature extraction is often limited to a single scale, failing to fully capture temporal trends, spatial relationships of anatomical structures, and attentional features in key areas across multiple scales, making it difficult to form comprehensive cognitive features; and the accuracy of modeling complex anatomical structures is insufficient, failing to meet clinical needs for detailed 3D models.

[0003] In view of this, the present invention provides a three-dimensional reconstruction method and system based on multimodal medical image fusion. Through precise spatial alignment, multi-scale feature extraction and three-dimensional diffusion reconstruction, the accuracy of three-dimensional reconstruction based on multimodal fusion is improved, providing more reliable three-dimensional model support for clinical diagnosis and treatment. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional reconstruction method based on multimodal medical image fusion, addressing the problem of improving the accuracy of three-dimensional reconstruction through multimodal fusion and providing more reliable three-dimensional model support for clinical diagnosis and treatment. The specific solution includes:

[0005] Obtain the raw multimodal image sequence; the raw multimodal image sequence includes CT images and MRI images;

[0006] Spatial alignment is performed on the original multimodal image sequence to obtain spatially aligned image pairs; the spatially aligned image pairs include aligned CT images and corresponding aligned MRI images;

[0007] Multi-scale hierarchical cognitive feature extraction is performed on spatially aligned image pairs to obtain a corrected feature map;

[0008] Spatiotemporal frequency domain collaborative attention fusion is performed on the modified feature map to obtain the fused features;

[0009] Topological constraint 3D diffusion reconstruction is performed on the fused features to obtain a 3D model of the medical image.

[0010] Furthermore, spatial alignment is performed on the original multimodal image sequence to obtain spatially aligned image pairs, including:

[0011] A spatial transformation network is used to predict global transformation parameters;

[0012] Based on global transformation parameters, a bilinear interpolation space transformation function is determined. The original multimodal image sequence is then processed using the bilinear interpolation space transformation function to obtain a multimodal transformed image sequence.

[0013] Feature extraction was performed on the multimodal transformed image sequence to obtain multimodal contour features;

[0014] Local deformation corrections were performed on the multimodal contour features to obtain a multimodal corrected image sequence;

[0015] The dynamic time warping factor is obtained by processing the multimodal corrected image sequence through dynamic time warping.

[0016] Spatially aligned image pairs are obtained by registering multimodal transformed image sequences using a dynamic time warping factor.

[0017] Furthermore, multi-scale hierarchical cognitive feature extraction is performed on the spatially aligned image pairs to obtain a revised feature map, including:

[0018] A multi-scale graph convolutional neural network was used to capture spatially aligned image pairs to obtain multi-scale feature maps; the multi-scale feature maps include multi-scale CT feature maps and multi-scale MRI feature maps;

[0019] Feature encoding is performed on the highest-scale feature maps to obtain high-scale corrected feature maps.

[0020] Furthermore, multi-scale feature maps are obtained, including:

[0021] Capture the slice change trend of the feature map at the previous scale across multiple time series to obtain the change trend features;

[0022] Obtain the spatial positional relationship of anatomical structures in the feature atlas of the next scale to obtain spatial positional features;

[0023] Attention features are obtained by assigning attention to key regions in the feature map of the previous scale.

[0024] By integrating trend characteristics, spatial location characteristics, and attention characteristics, a feature map at the current scale is obtained.

[0025] Furthermore, a high-scale corrected feature map is obtained, including:

[0026] Capture the local interactions of the highest-scale feature maps to obtain local feature maps;

[0027] Global dependency modeling is performed on the highest-scale feature map to obtain the global feature map;

[0028] Construct an encoding loss function through a self-supervised masking task;

[0029] Based on the encoding loss function, cross-axis attention processing is performed on the local feature map and the global feature map to obtain a high-scale corrected feature map.

[0030] Furthermore, the formula for calculating the coding loss function is as follows:

[0031] ;

[0032] in, This represents the mask feature reconstruction function; This represents the feature map after masking and reconstructing the high-scale modified feature map; Represents a high-scale corrected feature map; This represents the Euclidean norm.

[0033] Furthermore, the fusion features are obtained, including:

[0034] Spatial threshold attention processing is performed on the high-scale modified feature maps to obtain spatial attention maps;

[0035] Frequency-domain adaptive weighting is performed on the high-scale modified feature map to obtain enhanced high-resolution structural information;

[0036] By processing the high-scale modified feature map using spatial attention maps and enhanced high-resolution structural information, fused features are obtained.

[0037] Furthermore, the formula for calculating the fused features is as follows:

[0038] ;

[0039] in, Indicates fusion characteristics; Represents a spatial attention map; This represents the modal interaction modeled by the Kronecker product; This indicates enhanced high-resolution structural information; Cat represents element-wise product; Cat represents cross-numerical matrix. Represents high-scale CT corrected feature maps; This represents a high-scale MRI modified feature map.

[0040] Furthermore, a three-dimensional voxel model is obtained, including:

[0041] Three-dimensional structures are generated based on fusion features and diffusion model.

[0042] Based on the diffusion model and the gold standard model, a reconstruction loss function is constructed;

[0043] A topological loss function is constructed based on the gradient of the diffusion model and the gradient of the gold standard model.

[0044] Construct the total loss function based on the reconstruction loss function and the topology loss function;

[0045] The generation of the three-dimensional structure of the diffusion model is guided by the total loss function, resulting in the final three-dimensional voxel model.

[0046] The present invention also provides a three-dimensional reconstruction system based on multimodal medical image fusion, including an acquisition module, an alignment module, an extraction module, a fusion module and a reconstruction module;

[0047] The acquisition module is used to acquire raw multimodal image sequences, which include CT images and MRI images.

[0048] The alignment module is used to spatially align the original multimodal image sequences to obtain spatially aligned image pairs; the spatially aligned image pairs include the aligned CT images and the corresponding aligned MRI images;

[0049] The extraction module is used to extract multi-scale hierarchical cognitive features from spatially aligned image pairs to obtain a corrected feature map;

[0050] The fusion module is used to perform spatiotemporal frequency domain collaborative attention fusion on the modified feature map to obtain fused features;

[0051] The reconstruction module is used to perform topological constraint 3D diffusion reconstruction on the fused features to obtain a 3D model of the medical image.

[0052] The present invention has the following advantages and beneficial effects:

[0053] This invention integrates CT and MRI multimodal images, and through spatial alignment, multi-scale feature extraction, spatiotemporal frequency domain fusion and topological constraint reconstruction, forms a complete three-dimensional reconstruction process. It not only utilizes the complementary information of different modal images, but also improves the integrity and accuracy of the three-dimensional model through multi-step collaborative processing. Compared with single-modal reconstruction, it can better reflect the comprehensive features of anatomical structures.

[0054] This invention achieves global transformation through a spatial transformation network, and combines bilinear interpolation, local deformation correction and dynamic time warping to effectively solve the spatial offset problem between CT and MRI modalities, thereby improving the accuracy of multimodal image alignment.

[0055] This invention employs a multi-scale graph convolutional neural network to extract features. By capturing and fusing temporal trends, spatial relationships, and key region attention, it achieves multi-scale, multi-dimensional feature recognition. This not only covers structural details at different scales but also highlights temporal dynamics and key region information, making the feature map more closely aligned with the essential characteristics of anatomical structures.

[0056] This invention utilizes spatial attention to focus on key regions and frequency-domain adaptive weighting to enhance high-resolution structures. Combined with modal interaction modeling operations such as the Kronecker product, it achieves synergistic fusion of the spatiotemporal and frequency domains. This not only highlights important spatial anatomical regions but also enhances fine structural information in the frequency domain, fully leveraging the modal complementarity of CT and MRI. Attached Figure Description

[0057] Figure 1 An exemplary flowchart of a three-dimensional reconstruction method based on multimodal medical image fusion provided by the present invention;

[0058] Figure 2 An exemplary block diagram of a three-dimensional reconstruction system based on multimodal medical image fusion provided by the present invention;

[0059] Figure 3 This is a schematic diagram of the medical image three-dimensional model of the blood vessels reconstructed by the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0061] Figure 1 This is an exemplary flowchart of a three-dimensional reconstruction method based on multimodal medical image fusion provided by the present invention. Figure 1 As shown, the present invention provides a three-dimensional reconstruction method based on multimodal medical image fusion, comprising the following:

[0062] Acquire raw multimodal image sequences; these raw multimodal image sequences include CT image sequences and MRI image sequences. A raw multimodal image sequence refers to a collection of medical image slices encompassing multiple modalities. A CT image sequence refers to a collection of image slices representing the CT modality. An MRI image sequence refers to a collection of image slices representing the MRI modality.

[0063] Spatially aligned images are generated from the original multimodal image sequences. A spatially aligned image pair refers to a set of spatially aligned images obtained after registration of the original multimodal image sequences. This can include aligned CT images and corresponding aligned MRI images. Aligned CT images refer to the images belonging to the CT modality within the spatially aligned image pair. Aligned CT images are obtained by processing the CT image sequences using a Spatial Transform Network (STN) and a Transformer-CAN model, achieving precise spatial matching between the aligned CT images and their corresponding aligned MRI images. Aligned MRI images refer to the images belonging to the MRI modality within the spatially aligned image pair. Aligned MRI images are generated by registering the MRI image sequences, maintaining spatial consistency with the aligned CT images.

[0064] In some embodiments, spatial alignment is performed on the original multimodal image sequence to obtain spatially aligned image pairs, including:

[0065] A Spatial Transform Network (STN) is used to predict global transform parameters. An STN is a convolutional neural network used for registration that determines global transform parameters from the input multimodal imagery to correct spatial misalignment. Global transform parameters, predicted by the STN, include rotation matrices and translation vectors. The rotation matrix describes the rotational transformation of the image in space, and the translation vector describes the translational transformation. The formula for calculating the global transform parameters is as follows:

[0066] ;

[0067] in, Represents the global transformation parameters, including the rotation matrix R and the translation vector t; Represents a spatial transformation network; Represents the original CT image; This represents the original MRI image.

[0068] Based on global transformation parameters, a bilinear interpolation spatial transformation function is determined using a bilinear interpolation algorithm. This bilinear interpolation spatial transformation function is then applied to the original multimodal image sequence to obtain a multimodal transformed image sequence. The bilinear interpolation spatial transformation function is the image transformation function that combines global transformation parameters with the bilinear interpolation algorithm. The multimodal transformed image sequence refers to the spatially aligned image set obtained after processing the original multimodal image sequence using the bilinear interpolation spatial transformation function. The calculation formula for the multimodal transformed image sequence is:

[0069] ;

[0070] in, This refers to multimodal transformed image sequences, including CT transformed image sequences and MRI transformed image sequences; Represents the bilinear interpolation space transformation function; This represents the image sequence of the i-th modality. .

[0071] Feature extraction was performed on the multimodal transformed image sequences to obtain multimodal contour features. Multimodal contour features refer to a set of features used to reflect the boundaries of anatomical structures (e.g., the contours of blood vessels) in different modalities of images. Multimodal contour features include CT contour features and MRI contour features.

[0072] Local deformation corrections were applied to the multimodal contour features to obtain a multimodal corrected image sequence. A multimodal corrected image sequence refers to a spatially aligned image sequence obtained by performing local deformation corrections on the extracted contour features.

[0073] Dynamic temporal warping is used to process multimodal corrected image sequences to obtain a dynamic temporal warping factor. The dynamic temporal warping factor is a factor that locally deforms the corrected contour features to reduce local alignment errors between multimodal images.

[0074] Spatially aligned image pairs are obtained by registering multimodal transformed image sequences using a dynamic time warping factor.

[0075] Multi-scale hierarchical cognitive feature extraction is performed on spatially aligned image pairs to obtain a revised feature atlas. The revised feature atlas refers to the multi-scale feature atlas obtained after processing spatially aligned image pairs through a hierarchical cognitive feature extraction network. The revised feature atlas includes a revised CT feature atlas. and corrected MRI feature maps , This indicates the feature level (i.e., scale).

[0076] In some embodiments, obtaining the modified feature map may include:

[0077] A multi-scale graphical convolutional neural network (MCN) is used to capture spatially aligned image pairs, resulting in a multi-scale feature atlas. This atlas includes multi-scale CT and MRI feature atlases. A multi-scale graphical MCN employs multi-scale convolutional kernels to capture features such as edges and textures at different scales from spatially aligned multimodal images. A multi-scale feature atlas is a set of features extracted from spatially aligned multimodal images, containing different feature levels. For example, a multi-scale feature atlas may include edge, texture, and semantic features. Edges refer to the boundary lines between different tissues or structures in a multimodal image, such as the boundary between bone and surrounding soft tissue. Texture reflects the subtle structural features within tissues. For example, texture can be determined by the spatial distribution pattern of pixel grayscale values ​​in an image, such as the branching pattern of pulmonary vessels and bronchi.

[0078] In some embodiments, obtaining a multi-scale feature map includes:

[0079] By capturing the slice change trends of the previous-scale feature atlas across multiple time series, trend features are obtained. These trend features can be used to reflect the dynamic evolution of anatomical structures in sequential images. For example, in the previous-scale feature atlas of the coronary arteries, the clarity and density of vascular branches in different slices are considered. The formula for calculating trend features is:

[0080] ;

[0081] in, Indicates the characteristics of the changing trend; express Feature map of scale; i represents modal variable; LSTM represents Long Short-Term Memory network; 1D-Conv represents element-wise addition; 1D-Conv represents one-dimensional convolution.

[0082] The spatial positional relationships of anatomical structures within a higher-scale feature map are obtained, yielding spatial positional features. Spatial positional relationships refer to the spatial associations of anatomical structures relative to each other within the feature map. These relationships can include interactions such as adjacency and contact within local regions, and dependencies such as distribution and subordination within a global scope. For example, in a coronary artery feature map, there are the connections between vascular branches and the main trunk (local), and the overall distribution of the blood vessels within the cardiac anatomy (global). The formula for calculating spatial positional features is:

[0083] ;

[0084] in, Represents spatial location characteristics; j represents the associated variable; Indicates the first The weight matrix of the j-th location association at the scale is used to quantify the importance of the j-th location association matrix in spatial location feature extraction, and can be dynamically adjusted through model training; The location correlation matrix includes the physical distance matrix of anatomical structures, the tissue biomechanical correlation matrix, and the functional synergy matrix. The physical distance matrix quantifies the actual spatial distance between anatomical structures. For example, the pixel distance between anatomical structures can be calculated based on image coordinates or anatomical maps. The tissue biomechanical correlation matrix quantifies the mechanical coupling strength between structures. For example, in coronary artery feature maps, elements in the tissue biomechanical correlation matrix can represent the mechanical correlation strength between main vessel segments and branch segments, reflecting the blood pressure transmission from branches to the main vessel. The functional synergy matrix reflects the functional similarity or synergy between structures and can be calculated through the consistency between CT density and MRI signals. The bias vector representing the spatial location features at the l-th scale can be obtained through training.

[0085] Attention features are obtained by assigning attention to key regions in the feature map at the previous scale. Key regions refer to areas with anatomical structures or lesions. For example, key regions may include tumors, plaques, bifurcations, and blood vessels. Attention features are used to enhance key regions. The formula for calculating attention features is:

[0086] ;

[0087] ;

[0088] ;

[0089] in, The functions represent attention features; DifferentialAttention represents the differential attention calculation function, which highlights the attention weights of key regions (e.g., vessel bifurcation) by comparing global dependencies and local interaction features; GA represents global dependency features, which aggregates anatomical relationships across the entire image using the GridAttn attention mechanism, including the overall course of blood vessels; WA represents local interaction features, which captures details of small regions using the WindowAttn attention mechanism. These details can include relationships between local branches of blood vessels; Cat represents the cross-numerical matrix, used to achieve interaction between local and global features. represents element-wise multiplication; Softmax represents the activation function; MLP represents a multilayer perceptron. express Scale-based CT feature maps; express Scale-based MRI feature map; i represents modal variables, including CT and MRI.

[0090] By integrating trend characteristics, spatial location characteristics, and attention characteristics, a feature map at the current scale is obtained. The formula for calculating the feature map at the current scale is:

[0091] ;

[0092] ;

[0093] in, express Scale-based feature map. express Feature maps at scale; LayerNorm indicates normalization processing; Represents the ReLU activation function; Indicates the first Scale of convolution kernels; Indicates composite features; Indicates the first The weight matrix of the convolution operation; Indicates the first Layer bias; This indicates element-wise addition; This indicates element-wise multiplication.

[0094] Feature encoding is performed on the highest-scale feature maps to obtain high-scale corrected feature maps. The highest-scale feature map refers to the highest-level feature map in hierarchical cognitive feature extraction. It can be obtained by extracting low-level features using a multi-scale CNN. Examples include feature maps of the main structures of large blood vessels. Feature encoding can be used to integrate multi-scale information. The high-scale corrected feature map refers to the high-scale feature map obtained after feature encoding. It is used to correct and enhance the semantic expression of the original high-scale features, integrating local and global feature dependencies. For example, high-scale CT corrected feature maps of blood vessels can more clearly distinguish the boundaries between coronary artery calcified plaques and the surrounding sternum and ribs; high-scale MRI corrected feature maps of blood vessels can more accurately reflect the difference between the lipid core infiltration area of ​​carotid atherosclerotic plaques and the normal vessel wall.

[0095] In some embodiments, obtaining a high-scale corrected feature map includes:

[0096] Local feature maps are obtained by capturing local interactions in the highest-scale feature maps. Local interactions refer to the feature associations and mutual influences between adjacent small-scale anatomical structures, which can be used to reflect the detailed dependencies within a local region. For example, the feature interactions between local branches of blood vessels and adjacent myocardial tissue. Local feature maps are feature maps obtained by capturing local interaction information, used to obtain detailed relational features within a local region. For example, the feature interaction map between a local bend in a blood vessel and a nearby plaque extracted using WindowAttn.

[0097] Global dependency modeling is performed on the highest-scale feature map to obtain the global feature map. Global dependency modeling refers to modeling the feature associations between different regions to capture dependencies across a large range of anatomical structures. For example, in the coronary artery, it captures the overall directional dependencies between different branch segments. The global feature map integrates feature associations between different regions globally, including a wide range of anatomical information. For example, the global feature map of the coronary artery can include the overall morphology and interconnection features of different segments.

[0098] A self-supervised masking task is used to construct an encoding loss function. This task masks a portion of a feature map, and the model then reconstructs the features of the masked region to enhance the robustness of feature extraction. For example, in a high-scale feature map of coronary arteries, one or more vessel segments are randomly masked, allowing the model to reconstruct the features of the masked segments based on the unmasked regions.

[0099] Based on the encoding loss function, cross-axis attention processing is performed on the local and global feature maps to obtain a high-scale corrected feature map. The high-scale corrected feature map integrates local interaction and global dependency information. By fusing vascular features and anatomical global dependency features through cross-axis attention, the association between local vascular plaques and the overall vascular course can be more clearly reflected. Vascular course includes the direction and distribution path of blood vessels in the anatomical structure, as well as the connectivity between different vascular segments. In some embodiments, the calculation formula for the high-scale corrected feature map is:

[0100] ;

[0101] in, Represents a high-scale modified feature map; WindowAttn represents capturing local interactions; GridAttn represents modeling global dependencies; CrossAttn represents a cross-attention mechanism. The highest-scale feature map is represented by L; L represents the highest scale.

[0102] In some embodiments, the formula for calculating the coding loss function is:

[0103] ;

[0104] in, This represents the mask feature reconstruction function; This indicates that the feature map is reconstructed by masking the high-scale modified feature map, and the masking can be performed by randomly selecting a portion of the feature map. Represents a high-scale corrected feature map; This represents the Euclidean norm.

[0105] Spatiotemporal frequency domain collaborative attention fusion is performed on the modified feature map to obtain fused features. Fusion features refer to the features obtained after processing the modified feature map through the spatiotemporal frequency domain collaborative attention fusion mechanism.

[0106] In some embodiments, the fusion feature is obtained, including:

[0107] Spatial threshold attention processing is applied to the high-scale modified feature maps to obtain spatial attention maps. Spatial threshold attention processing is used to highlight key regions such as plaques or blood vessels in the image. For example, it strengthens the feature weights of blood vessel regions and weakens irrelevant background regions. Spatial attention maps are used to characterize the importance of different spatial locations in multimodal features, and are used to highlight key structural regions such as plaques or blood vessels. In some embodiments, the formula for calculating the spatial attention map is:

[0108] ;

[0109] in, Representing a spatial attention map, Used to highlight key areas such as patches; Represents high-scale CT corrected feature maps; represents the high-scale MRI modified feature map; Softmax represents the activation function; MLP represents the multilayer perceptron; Cat represents the cross numerical matrix.

[0110] Frequency-domain adaptive weighting is performed on the high-scale modified feature maps to obtain enhanced high-resolution structural information. This enhanced high-resolution structural information is used to enhance high-resolution anatomical details within the features. The enhanced structural information may include edges and textures. For example, for coronary arteries, it can enhance structural information such as the fine branches of the vessel and the edge details of plaques. In some embodiments, the formula for calculating the enhanced high-resolution structural information is:

[0111] ;

[0112] in, This represents enhanced high-resolution structural information; FFT represents Fourier transform. This represents the frequency sensitivity matrix.

[0113] The high-scale modified feature map is processed by spatial attention map and enhanced high-resolution structural information to obtain fused features. In some embodiments, the calculation formula for the fused features is:

[0114] ;

[0115] in, Indicates fusion characteristics; This represents the modal interaction modeled by the Kronecker product; This represents element-wise product.

[0116] A three-dimensional diffusion reconstruction with topological constraints is performed on the fused features to obtain a three-dimensional model of the medical image. The three-dimensional model of the medical image refers to a three-dimensional voxel model obtained after processing the fused feature sequence through three-dimensional diffusion reconstruction with topological constraints.

[0117] In some embodiments, a three-dimensional voxel model is obtained, including:

[0118] The 3D structure of the diffusion model is generated based on fusion features. The 3D structure of the diffusion model refers to the 3D spatial information containing anatomical structures. A 3D voxel model can be generated after denoising the fusion feature sequence using the X-Diffusion framework, thus obtaining the 3D structure of the diffusion model. Topological constraint optimization can be used to ensure the continuity and consistency of the 3D structure of the diffusion model. For example, by generating a 3D diffusion model of the coronary artery, the 3D course of the blood vessel, its branches, and the 3D location distribution of plaques can be clearly displayed. In some embodiments, the calculation formula for the 3D structure of the diffusion model is:

[0119] ;

[0120] in, Let represent the three-dimensional structure of the diffusion model in the t-th iteration; Indicates the diffusion coefficient; Represents the three-dimensional structure of the diffusion model at iteration t-1; This represents Gaussian noise.

[0121] A reconstruction loss function is constructed based on the diffusion model and the gold standard model. The gold standard model refers to the standard 3D voxel model used as a reference benchmark. The reconstruction loss function is used to measure the difference between the 3D voxel model generated by the diffusion model and the gold standard model.

[0122] A topological loss function is constructed based on the gradients of the diffusion model and the gold standard model. The topological loss function measures the difference between the gradient of the 3D voxel model generated by the diffusion model and the gradient of the gold standard model.

[0123] A total loss function is constructed based on the reconstruction loss function and the topology loss function. Choosing to construct the total loss function with these two functions ensures both the morphological accuracy of the 3D structure relative to the gold standard through reconstruction loss and the topological consistency of the anatomical structure through topology loss. This balances reconstruction accuracy and structural rationality, improving the reliability and clinical applicability of 3D reconstruction. In some embodiments, the formula for calculating the total loss function is:

[0124] ;

[0125] ;

[0126] ;

[0127] in, Represents the total loss function; Represents the reconstruction loss function; Represents the topology loss function; This represents the three-dimensional structure of the generated diffusion model; Represents the three-dimensional structure of the gold standard model; This represents the gradient operator.

[0128] The generation of the three-dimensional structure of the diffusion model is guided by the total loss function, resulting in the final three-dimensional voxel model.

[0129] like Figure 3 The diagram shows a schematic representation of the 3D medical image model of the blood vessel reconstructed by this invention. It can be seen that the 3D reconstruction method based on multimodal medical image fusion can completely capture the full-level vascular structure from the main trunk to the terminal branches; it enhances the sharpness of vascular edges and tissue contrast, resulting in clear vascular contours and distinct branch details obtained through MRI and CT reconstruction. Ultimately, it generates a detailed and structurally accurate 3D vascular model, providing reliable anatomical evidence for the diagnosis of vascular diseases (such as aneurysms and stenosis) and surgical planning.

[0130] Figure 2 This is an exemplary block diagram of a three-dimensional reconstruction system based on multimodal medical image fusion provided by the present invention. (See diagram below.) Figure 2 As shown, the 3D reconstruction system based on multimodal medical image fusion includes an acquisition module, an alignment module, an extraction module, a fusion module, and a reconstruction module.

[0131] The acquisition module is used to acquire raw multimodal image sequences, which include CT images and MRI images.

[0132] The alignment module is used to spatially align the original multimodal image sequences to obtain spatially aligned image pairs; the spatially aligned image pairs include the aligned CT images and the corresponding aligned MRI images.

[0133] The extraction module is used to extract multi-scale hierarchical cognitive features from spatially aligned image pairs to obtain a corrected feature map.

[0134] The fusion module is used to perform spatiotemporal frequency domain collaborative attention fusion on the modified feature map to obtain fused features.

[0135] The reconstruction module is used to perform topological constraint 3D diffusion reconstruction on the fused features to obtain a 3D model of the medical image.

[0136] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method based on multimodal medical image fusion, characterized in that, include: Obtain the raw multimodal image sequence; the raw multimodal image sequence includes CT images and MRI images; Spatial alignment is performed on the original multimodal image sequence to obtain spatially aligned image pairs; the spatially aligned image pairs include aligned CT images and corresponding aligned MRI images; Multi-scale hierarchical cognitive feature extraction is performed on spatially aligned image pairs to obtain a revised feature atlas. This includes: capturing the spatially aligned image pairs using a multi-scale graph convolutional neural network to obtain a multi-scale feature atlas; the multi-scale feature atlas includes multi-scale CT feature atlases and multi-scale MRI feature atlases; performing feature encoding on the highest-scale feature atlas to obtain a high-scale revised feature atlas; wherein obtaining the multi-scale feature atlas includes: capturing the slice change trend of the previous-scale feature atlas at multiple time series to obtain change trend features; obtaining the spatial positional relationship of anatomical structures in the previous-scale feature atlas to obtain spatial positional features; performing attention allocation on key regions in the previous-scale feature atlas to obtain attention features; and fusing change trend features, spatial positional features, and attention features to obtain the current-scale feature atlas. Spatiotemporal frequency domain collaborative attention fusion is performed on the modified feature map to obtain fused features, including: performing spatial threshold attention processing on the high-scale modified feature map to obtain a spatial attention map; performing frequency domain adaptive weighting on the high-scale modified feature map to obtain enhanced high-resolution structural information; and processing the high-scale modified feature map through the spatial attention map and the enhanced high-resolution structural information to obtain fused features. A three-dimensional diffusion reconstruction with topological constraints is performed on the fusion features to obtain a three-dimensional model of the medical image, including: generating a three-dimensional structure of the diffusion model based on the fusion features; constructing a reconstruction loss function based on the diffusion model and the gold standard model; constructing a topological loss function based on the gradient of the diffusion model and the gradient of the gold standard model; constructing a total loss function based on the reconstruction loss function and the topological loss function; and guiding the generation of the three-dimensional structure of the diffusion model based on the total loss function to obtain a three-dimensional voxel model as the three-dimensional model of the medical image.

2. The three-dimensional reconstruction method based on multimodal medical image fusion according to claim 1, characterized in that, Spatially aligning the original multimodal image sequence yields spatially aligned image pairs, including: A spatial transformation network is used to predict global transformation parameters; Based on global transformation parameters, a bilinear interpolation space transformation function is determined. The original multimodal image sequence is then processed using the bilinear interpolation space transformation function to obtain a multimodal transformed image sequence. Feature extraction was performed on the multimodal transformed image sequence to obtain multimodal contour features; Local deformation corrections were performed on the multimodal contour features to obtain a multimodal corrected image sequence; The dynamic time warping factor is obtained by processing the multimodal corrected image sequence through dynamic time warping. Spatially aligned image pairs are obtained by registering multimodal transformed image sequences using a dynamic time warping factor.

3. The three-dimensional reconstruction method based on multimodal medical image fusion according to claim 1, characterized in that, The high-scale corrected feature map is obtained, including: Capture the local interactions of the highest-scale feature maps to obtain local feature maps; Global dependency modeling is performed on the highest-scale feature map to obtain the global feature map; Construct an encoding loss function through a self-supervised masking task; Based on the encoding loss function, cross-axis attention processing is performed on the local feature map and the global feature map to obtain a high-scale corrected feature map.

4. The three-dimensional reconstruction method based on multimodal medical image fusion according to claim 3, characterized in that, The formula for calculating the encoding loss function is: ； in, This represents the mask feature reconstruction function; This represents the feature map after masking and reconstructing the high-scale modified feature map; Represents a high-scale corrected feature map; This represents the Euclidean norm.

5. The three-dimensional reconstruction method based on multimodal medical image fusion according to claim 1, characterized in that, The formula for calculating the fusion feature is: ； in, Indicates fusion characteristics; Represents a spatial attention map; This represents the modal interaction modeled by the Kronecker product; This indicates enhanced high-resolution structural information; Cat represents element-wise product; Cat represents cross-numerical matrix. Represents high-scale CT corrected feature maps; This represents a high-scale MRI modified feature map.

6. The three-dimensional reconstruction method based on multimodal medical image fusion according to claim 1, characterized in that, The total loss function is: ； ； ； in, Represents the total loss function; Represents the reconstruction loss function; Represents the topology loss function; Indicates the weighting coefficient; This represents the three-dimensional structure of the generated diffusion model; Indicates the gold standard; This represents the gradient operator.

7. A three-dimensional reconstruction system based on multimodal medical image fusion, characterized in that, It includes an acquisition module, an alignment module, an extraction module, a fusion module, and a reconstruction module; The acquisition module is used to acquire raw multimodal image sequences, which include CT images and MRI images. The alignment module is used to spatially align the original multimodal image sequences to obtain spatially aligned image pairs; the spatially aligned image pairs include the aligned CT images and the corresponding aligned MRI images; The extraction module is used to extract multi-scale hierarchical cognitive features from spatially aligned image pairs to obtain a corrected feature map; The process of obtaining the corrected feature map includes: capturing spatially aligned image pairs using a multi-scale graph convolutional neural network to obtain a multi-scale feature map; the multi-scale feature map includes a multi-scale CT feature map and a multi-scale MRI feature map; performing feature encoding on the highest-scale feature map to obtain a high-scale corrected feature map; wherein, obtaining the multi-scale feature map includes: capturing the slice change trend of the previous-scale feature map at multiple time series to obtain change trend features; obtaining the spatial positional relationship of anatomical structures in the previous-scale feature map to obtain spatial position features; performing attention allocation on key regions in the previous-scale feature map to obtain attention features; and fusing the change trend features, the spatial position features, and the attention features to obtain the feature map at the current scale. The fusion module is used to perform spatiotemporal frequency domain collaborative attention fusion on the modified feature map to obtain fused features; wherein, the spatiotemporal frequency domain collaborative attention fusion includes: performing spatial threshold attention processing on the high-scale modified feature map to obtain a spatial attention map; performing frequency domain adaptive weighting on the high-scale modified feature map to obtain enhanced high-resolution structural information; and processing the high-scale modified feature map through the spatial attention map and the enhanced high-resolution structural information to obtain fused features; The reconstruction module is used to perform topologically constrained 3D diffusion reconstruction on the fusion features to obtain a 3D model of the medical image. The topologically constrained 3D diffusion reconstruction includes: generating a 3D structure of the diffusion model based on the fusion features; constructing a reconstruction loss function based on the diffusion model and the gold standard model; constructing a topological loss function based on the gradient of the diffusion model and the gradient of the gold standard model; constructing a total loss function based on the reconstruction loss function and the topological loss function; and guiding the generation of the 3D structure of the diffusion model based on the total loss function to obtain a 3D voxel model as the 3D model of the medical image.

Images