Three-dimensional reconstruction method and system based on multi-modal medical image fusion

Abstract

The invention belongs to the field of medical image processing, and relates to a three-dimensional reconstruction method and system based on multi-modal medical image fusion, and the method comprises the steps: obtaining an original multi-modal image sequence; the original multi-modal image sequence comprises a CT image and an MRI image; performing spatial alignment on the original multi-modal image sequence to obtain a spatial alignment image pair; the space alignment image pair comprises an aligned CT image and a corresponding aligned MRI image after alignment; performing multi-scale layered cognitive feature extraction on the spatial alignment image pair to obtain a corrected feature map; performing time-space frequency domain collaborative attention fusion on the corrected characteristic spectrum to obtain fusion characteristics; performing topological constraint three-dimensional diffusion reconstruction on the fusion features to obtain a medical image three-dimensional model; the three-dimensional reconstruction precision of multi-modal fusion is improved, and more reliable three-dimensional model support is provided for clinical diagnosis and treatment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, and specifically discloses a three-dimensional reconstruction method and system based on multi-modal medical image fusion. BACKGROUND

[0002] The prior art normalizes and standardizes the CT data of the detection site, and then uses three-dimensional reconstruction software to generate an accurate three-dimensional model. Virtual examination is performed on the three-dimensional model, and virtual pictures containing target structures and corresponding sagittal, coronal and axial pictures are intercepted to form a multi-modal data set. The DICOM format image data of CT is fused with the above-mentioned intercepted pictures, and multi-modal data features are introduced to adjust and train a 3D-UNet model to enhance its segmentation capability for complex structures. However, the prior art has many limitations in three-dimensional reconstruction based on multi-modal medical image fusion: feature extraction is limited to a single scale, and the temporal variation trend, spatial position relationship of anatomical structures and key region attention features in multiple scales are not fully captured, making it difficult to form a comprehensive cognitive feature; the modeling accuracy of complex anatomical structures is insufficient, and it is difficult to meet the demand for fine three-dimensional models in clinical practice

[0003] Therefore, the present application provides a three-dimensional reconstruction method and system based on multi-modal medical image fusion, which improves the accuracy of three-dimensional reconstruction based on multi-modal fusion through precise spatial alignment, multi-scale feature extraction and three-dimensional diffusion reconstruction, and provides more reliable three-dimensional model support for clinical diagnosis and treatment. SUMMARY

[0004] The present application aims to provide a three-dimensional reconstruction method based on multi-modal medical image fusion, which solves the problem of improving the accuracy of three-dimensional reconstruction based on multi-modal fusion and providing more reliable three-dimensional model support for clinical diagnosis and treatment. The specific scheme includes:

[0005] Obtain an original multi-modal image sequence; the original multi-modal image sequence includes CT images and MRI images;

[0006] Perform spatial alignment on the original multi-modal image sequence to obtain a spatially aligned image pair; the spatially aligned image pair includes an aligned CT image and a corresponding aligned MRI image;

[0007] Perform multi-scale layered cognitive feature extraction on the spatially aligned image pair to obtain a corrected feature atlas;

[0008] Perform spatio-temporal frequency domain collaborative attention fusion on the corrected feature atlas to obtain a fused feature;

[0009] Perform topological constraint three-dimensional diffusion reconstruction on the fused feature to obtain a medical image three-dimensional model.

[0010] Further, the original multi-modal image sequence is spatially aligned to obtain a spatially aligned image pair, including:

[0011] A spatial transformation network is used to predict global transformation parameters;

[0012] Based on the global transformation parameters, a bilinear interpolation spatial transformation function is determined, and the original multi-modal image sequence is processed by the bilinear interpolation spatial transformation function to obtain a multi-modal transformed image sequence;

[0013] The multi-modal transformed image sequence is respectively subjected to feature extraction to obtain multi-modal contour features;

[0014] The multi-modal contour features are respectively subjected to local deformation correction to obtain a multi-modal corrected image sequence;

[0015] The multi-modal corrected image sequence is processed by dynamic time warping to obtain a dynamic time warping factor;

[0016] The multi-modal transformed image sequence is registered by the dynamic time warping factor to obtain a spatially aligned image pair.

[0017] Further, multi-scale hierarchical cognitive feature extraction is performed on the spatially aligned image pair to obtain a corrected feature atlas, including:

[0018] A multi-scale graph convolutional neural network is used to capture the spatially aligned image pair to obtain a multi-scale feature atlas; the multi-scale feature atlas includes a multi-scale CT feature atlas and a multi-scale MRI feature atlas;

[0019] Feature encoding is performed on the highest scale feature atlas to obtain a high-scale corrected feature atlas.

[0020] Further, the multi-scale feature atlas is obtained, including:

[0021] The slice change trend of the feature atlas of the previous scale is captured under multiple time sequences to obtain a change trend feature;

[0022] The spatial position relationship of the anatomical structure in the feature atlas of the previous scale is obtained to obtain a spatial position feature;

[0023] Attention is allocated to the key region in the feature atlas of the previous scale to obtain an attention feature;

[0024] The change trend feature, the spatial position feature, and the attention feature are fused to obtain the feature atlas of the current scale.

[0025] Further, the high-scale corrected feature atlas is obtained, including:

[0026] Local interaction of the highest scale feature atlas is captured to obtain a local feature atlas;

[0027] Performing global dependency modeling on the highest scale feature map to obtain a global feature map;

[0028] Through a self-supervised mask task, an encoding loss function is constructed;

[0029] Based on the encoding loss function, cross-axis attention processing is performed on the local feature map and the global feature map to obtain a high-scale corrected feature map.

[0030] Further, the calculation formula of the encoding loss function is:

[0031] ;

[0032] Among them, denotes a mask feature reconstruction function; denotes a feature map reconstructed by masking the high-scale corrected feature map; denotes a high-scale corrected feature map; denotes the Euclidean norm.

[0033] Further, the fusion feature includes:

[0034] Respectively performing spatial threshold attention processing on the high-scale corrected feature map to obtain a spatial attention map;

[0035] Respectively performing frequency domain adaptive weighting on the high-scale corrected feature map to obtain enhanced high-resolution structural information;

[0036] Processing the high-scale corrected feature map through the spatial attention map and the enhanced high-resolution structural information to obtain the fusion feature.

[0037] Further, the calculation formula of the fusion feature is:

[0038] ;

[0039] Among them, denotes a fusion feature; denotes a spatial attention map; denotes a Kronecker product modeling modal interaction; denotes enhanced high-resolution structural information; denotes an element-wise product; Cat denotes a cross-value matrix; denotes a high-scale CT corrected feature map; denotes a high-scale MRI corrected feature map.

[0040] Further, the three-dimensional voxel model includes:

[0041] Generating a diffusion model three-dimensional structure based on the fusion feature;

[0042] construct a reconstruction loss function based on the diffusion model and the gold standard model;

[0043] construct a topological loss function based on the diffusion model gradient and the gold standard model gradient;

[0044] construct a total loss function based on the reconstruction loss function and the topological loss function;

[0045] guide the generation of the three-dimensional structure of the diffusion model based on the total loss function to obtain a final three-dimensional voxel model.

[0046] The application also provides a three-dimensional reconstruction system based on multi-modal medical image fusion, comprising an acquisition module, an alignment module, an extraction module, a fusion module and a reconstruction module.

[0047] The acquisition module is used to acquire an original multi-modal image sequence; the original multi-modal image sequence comprises CT images and MRI images;

[0048] The alignment module is used to perform spatial alignment on the original multi-modal image sequence to obtain a spatially aligned image pair; the spatially aligned image pair comprises an aligned CT image and a corresponding aligned MRI image;

[0049] The extraction module is used to perform multi-scale hierarchical cognitive feature extraction on the spatially aligned image pair to obtain a corrected feature atlas;

[0050] The fusion module is used to perform spatio-temporal frequency domain collaborative attention fusion on the corrected feature atlas to obtain a fused feature;

[0051] The reconstruction module is used to perform topological constraint three-dimensional diffusion reconstruction on the fused feature to obtain a medical image three-dimensional model.

[0052] The application has the following advantages and beneficial effects:

[0053] The application integrates CT and MRI multi-modal images, performs spatial alignment, multi-scale feature extraction, spatio-temporal frequency domain fusion and topological constraint reconstruction to form a complete three-dimensional reconstruction process, which not only utilizes the complementary information of different modal images, but also improves the integrity and accuracy of the three-dimensional model through multi-step collaborative processing, and can reflect the overall characteristics of the anatomical structure more than single modal reconstruction.

[0054] The application realizes global transformation through a spatial transformation network, combines bilinear interpolation, local deformation correction and dynamic time warping to effectively solve the spatial offset problem between CT and MRI modalities and improve the accuracy of multi-modal image alignment.

[0055] The application adopts a multi-scale graph convolutional neural network to extract features, captures time sequence change trend, spatial position relationship and key region attention, and realizes multi-scale and multi-dimensional feature cognition by fusion, which covers structural details under different scales, highlights time sequence dynamics and key region information, and makes feature atlas more consistent with the essential characteristics of anatomical structure.

[0056] The application focuses on key regions and frequency domain adaptive weighted enhancement of high-resolution structures through spatial attention, combines modeling modal interaction such as Kronecker product operation, and realizes the collaborative fusion of space and frequency domain. It highlights important anatomical regions in space and strengthens fine structure information in frequency domain, and fully utilizes the modal complementarity of CT and MRI. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 An exemplary flowchart of a three-dimensional reconstruction method based on multi-modal medical image fusion provided by the application is shown in the figure.

[0058] Figure 2 An exemplary module diagram of a three-dimensional reconstruction system based on multi-modal medical image fusion provided by the application is shown in the figure.

[0059] Figure 3 A schematic diagram of a medical image three-dimensional model of a blood vessel reconstructed by the application is shown in the figure. DETAILED DESCRIPTION

[0060] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations.

[0061] Figure 1 An exemplary flowchart of a three-dimensional reconstruction method based on multi-modal medical image fusion provided by the application is shown in the figure. As shown in the figure, Figure 1 A three-dimensional reconstruction method based on multi-modal medical image fusion provided by the application includes the following contents:

[0062] An original multi-modal image sequence is obtained; the original multi-modal image sequence includes a CT image sequence and an MRI image sequence. The original multi-modal image sequence refers to a set of medical image slices including multiple modalities. The CT image sequence refers to a set of image slices of the CT modality. The MRI image sequence refers to a set of image slices of the MRI modality.

[0063] The original multi-modal image sequence is spatially aligned to obtain a spatially aligned image pair. The spatially aligned image pair refers to an image set obtained by registering the original multi-modal image sequence, which can include an aligned CT image and a corresponding aligned MRI image. The aligned CT image refers to an image belonging to the CT modality in the spatially aligned image pair. The CT image sequence is processed by a spatial transformation network (STN) and a Transformer-CAN model to obtain an aligned CT image, which is accurately matched in space with the corresponding aligned MRI image. The aligned MRI image refers to an image belonging to the MRI modality in the spatially aligned image pair. The aligned MRI image is generated by registering the MRI image sequence, and the aligned MRI image maintains spatial consistency with the aligned CT image.

[0064] In some embodiments, the original multi-modal image sequence is spatially aligned to obtain a spatially aligned image pair, including:

[0065] A spatial transformation network (STN) is used to predict global transformation parameters. The spatial transformation network (STN) refers to a convolutional neural network used for registration, which can determine global transformation parameters for correcting spatial misalignment of images from input multi-modal images. The global transformation parameters refer to parameters predicted by the spatial transformation network, including a rotation matrix and a translation vector. The rotation matrix is used to describe the rotation transformation of the image in space, and the translation vector is used to describe the translation transformation of the image. The calculation formula of the global transformation parameters is:

[0066] ;

[0067] Wherein, represents the global transformation parameters, including the rotation matrix R and the translation vector t; represents the spatial transformation network; represents the original CT image; represents the original MRI image.

[0068] Based on the global transformation parameters, a bilinear interpolation spatial transformation function is determined by a bilinear interpolation algorithm, and the original multi-modal image sequence is processed by the bilinear interpolation spatial transformation function to obtain a multi-modal transformed image sequence. The bilinear interpolation spatial transformation function refers to an image transformation function combining the global transformation parameters and the bilinear interpolation algorithm. The multi-modal transformed image sequence refers to an image set obtained by processing the original multi-modal image sequence by the bilinear interpolation spatial transformation function. The calculation formula of the multi-modal transformed image sequence is:

[0069] ;

[0070] Wherein, denote a multi-modality transformed image sequence, including a CT transformed image sequence and an MRI transformed image sequence; denote a bilinear interpolation spatial transformation function; denote an image sequence of the i-th modality, .

[0071] perform feature extraction on the multi-modality transformed image sequence respectively to obtain multi-modality contour features. The multi-modality contour features refer to a set of features used to reflect the boundaries of anatomical structures (e.g., the contours of blood vessels) in different modalities of images. The multi-modality contour features include CT contour features and MRI contour features.

[0072] perform local deformation correction on the multi-modality contour features respectively to obtain a multi-modality corrected image sequence. The multi-modality corrected image sequence refers to a spatially high-precision aligned image sequence obtained by performing local deformation correction on the extracted contour features.

[0073] perform processing on the multi-modality corrected image sequence by dynamic time warping to obtain a dynamic time warping factor. The dynamic time warping factor refers to a factor of local deformation on the corrected contour features. The dynamic time warping factor is used to reduce the local alignment error between the multi-modality images.

[0074] perform registration on the multi-modality transformed image sequence by the dynamic time warping factor to obtain a spatially aligned image pair.

[0075] perform multi-scale hierarchical cognitive feature extraction on the spatially aligned image pair to obtain a corrected feature atlas. The corrected feature atlas refers to a multi-scale feature atlas obtained by processing the spatially aligned image pair through a hierarchical cognitive feature extraction network. The corrected feature atlas includes a corrected CT feature atlas and a corrected MRI feature atlas , denote a feature level (i.e., a scale).

[0076] In some embodiments, obtaining the corrected feature atlas can include:

[0077] A multi-scale graphical convolutional neural network (MCN) is used to capture spatially aligned image pairs, resulting in a multi-scale feature atlas. This atlas includes multi-scale CT and MRI feature atlases. A multi-scale graphical MCN employs multi-scale convolutional kernels to capture features such as edges and textures at different scales from spatially aligned multimodal images. A multi-scale feature atlas is a set of features extracted from spatially aligned multimodal images, containing different feature levels. For example, a multi-scale feature atlas may include edge, texture, and semantic features. Edges refer to the boundary lines between different tissues or structures in a multimodal image, such as the boundary between bone and surrounding soft tissue. Texture reflects the subtle structural features within tissues. For example, texture can be determined by the spatial distribution pattern of pixel grayscale values ​​in an image, such as the branching pattern of pulmonary vessels and bronchi.

[0078] In some embodiments, obtaining a multi-scale feature map includes:

[0079] By capturing the slice change trends of the previous-scale feature atlas across multiple time series, trend features are obtained. These trend features can be used to reflect the dynamic evolution of anatomical structures in sequential images. For example, in the previous-scale feature atlas of the coronary arteries, the clarity and density of vascular branches in different slices are considered. The formula for calculating trend features is:

[0080] ;

[0081] in, Indicates the characteristics of the changing trend; express Feature map of scale; i represents modal variable; LSTM represents Long Short-Term Memory network; 1D-Conv represents element-wise addition; 1D-Conv represents one-dimensional convolution.

[0082] The spatial positional relationships of anatomical structures within a higher-scale feature map are obtained, yielding spatial positional features. Spatial positional relationships refer to the spatial associations of anatomical structures relative to each other within the feature map. These relationships can include interactions such as adjacency and contact within local regions, and dependencies such as distribution and subordination within a global scope. For example, in a coronary artery feature map, there are the connections between vascular branches and the main trunk (local), and the overall distribution of the blood vessels within the cardiac anatomy (global). The formula for calculating spatial positional features is:

[0083] ;

[0084] in, Represents spatial location characteristics; j represents the associated variable; Indicates the first The weight matrix of the jth position correlation of the scale, used to quantify the importance of the jth position correlation matrix in spatial position feature extraction, can be dynamically adjusted through model training. The position correlation matrix represents the physical distance matrix of the anatomical structure, the tissue mechanical correlation matrix, and the functional synergy matrix. The physical distance matrix of the anatomical structure is used to quantify the actual spatial distance between the anatomical structures. For example, the pixel distance between the anatomical structures can be calculated based on the image coordinates or the anatomical map. The tissue mechanical correlation matrix is used to quantify the mechanical coupling strength between the structures. For example, in the coronary artery feature map, the elements in the tissue mechanical correlation matrix can represent the mechanical correlation strength between the main stem and the branch of the blood vessel, reflecting the blood pressure transmission of the branch to the main stem. The functional synergy matrix can reflect the functional similarity or synergy between structures, which can be calculated by the consistency of CT density and MRI signal; The bias vector of the lth scale spatial position feature is represented by, which can be obtained by training and learning.

[0085] Attention is allocated to the key regions in the feature map of the previous scale to obtain attention features. The key regions refer to regions with anatomical structures or lesions. For example, the key regions can include tumor, plaque, bifurcation, and blood vessels. The attention features are used to enhance the key regions. The calculation formula of the attention features is:

[0086] ;

[0087] ;

[0088] ;

[0089] wherein, Attention features; DifferentialAttention represents the differential attention calculation function, which can highlight the attention weight of the key region (such as the blood vessel bifurcation) by comparing the global dependence and local interaction features; GA represents the global dependence feature, which can aggregate the anatomical correlation in the whole image range through the grid attention mechanism GridAttn, including the overall running dependence of the blood vessels; WA represents the local interaction feature, which can be obtained by capturing small area details through the window attention mechanism WindowAttn. Small area details can include local branch correlation of blood vessels; Cat represents the cross numerical matrix, which is used to realize the interaction between local and global features; Element-wise multiplication is represented by ; Softmax represents the activation function; MLP represents the multi-layer perceptron; CT feature map of the scale; MRI feature map of the scale; i represents the modality variable, including CT and MRI.​​

[0090] The fusion of the change trend feature, the spatial position feature and the attention feature obtains the feature map of the current scale. The calculation formula of the feature map of the current scale is:

[0091] ;

[0092] ;

[0093] wherein, represents the feature map of the current scale, represents the feature map of the current scale; LayerNorm represents normalization processing; represents the ReLU activation function; represents the feature map of the current scale; LayerNorm represents normalization processing; represents the ReLU activation function; represents the convolution kernel of the current scale; represents the composite feature; represents the weight matrix of the convolution operation of the current layer; represents the bias of the current layer; represents the element-wise addition; represents the element-wise multiplication.

[0094] The feature encoding is performed on the highest scale feature map to obtain a high scale modified feature map. The highest scale feature map refers to the feature map of the highest level in the hierarchical cognitive feature extraction. The highest scale feature map can be obtained after the low-level features are extracted by the multi-scale CNN. For example, the feature map of the main structure of large blood vessels. The feature encoding can be used to integrate multi-scale information. The high scale modified feature map refers to the high scale feature map obtained after the feature encoding. It is used to modify and enhance the semantic expression of the original high scale feature and integrate local and global feature dependencies. For example, the high scale CT modified feature map of blood vessels can more clearly distinguish the boundary between the calcified plaque of the coronary artery and the surrounding sternum and rib; the high scale MRI modified feature map of blood vessels can more accurately reflect the difference between the lipid core infiltration area of the carotid atherosclerotic plaque and the normal blood vessel wall.

[0095] In some embodiments, obtaining the high scale modified feature map comprises:

[0096] ​​​The local interaction of the highest scale feature map is captured to obtain a local feature map. The local interaction refers to the feature correlation and mutual influence between adjacent small-range anatomical structures, and can be used to reflect the detail dependency relationship in the local region. For example, the feature interaction between the local branch of a blood vessel and the adjacent myocardial tissue.

[0097] Global dependency modeling is performed on the highest scale feature map to obtain a global feature map. Global dependency modeling refers to modeling the feature correlation between different regions to capture the dependency relationship of anatomical structures across a large range. For example, in the coronary artery, the overall trend dependency between different branch vessel segments is captured. The global feature map integrates the feature correlation of different regions in the global range, including large-range anatomical structure information. For example, the global feature map of the coronary artery can include the overall morphology and mutual connection relationship features of different vessel segments.

[0098] An encoding loss function is constructed through a self-supervised mask task. The self-supervised mask task is used to mask part of the map, and then the model reconstructs the features of the masked region to enhance the robustness of the model in extracting features. For example, in the high-scale feature map of the coronary artery, one or more vessel segments are randomly masked, and the model reconstructs the features of the masked vessel segments based on the unmasked region.

[0099] Based on the encoding loss function, cross-axis attention processing is performed on the local feature map and the global feature map to obtain a high-scale modified feature map. The high-scale modified feature map integrates local interaction and global dependency information. By fusing vessel features and anatomical global dependency features through cross-axis attention, the correlation features of local plaque and overall vessel running of the blood vessel can be more clearly represented. The vessel running includes the running, distribution path and connection relationship between different vessel segments of the blood vessel in the anatomical structure. In some embodiments, the calculation formula of the high-scale modified feature map is:

[0100] ;

[0101] wherein, represents the high-scale modified feature map; WindowAttn represents capturing local interaction; GridAttn represents modeling global dependency; CrossAttn represents cross-attention mechanism; represents the highest scale feature map; L represents the highest scale.

[0102] In some embodiments, the calculation formula of the encoding loss function is:

[0103] ;

[0104] wherein, denotes a mask feature reconstruction function; denotes a feature map after mask reconstruction on the high-scale corrected feature map, which can be performed by randomly selecting part of the region in the feature map for mask; denotes a high-scale corrected feature map; denotes a Euclidean norm.

[0105] performing spatio-temporal-frequency domain collaborative attention fusion on the corrected feature map to obtain a fusion feature. The fusion feature refers to a feature obtained by processing the corrected feature map through a spatio-temporal-frequency domain collaborative attention fusion mechanism.

[0106] In some embodiments, obtaining the fusion feature comprises:

[0107] performing spatial threshold attention processing on the high-scale corrected feature map respectively to obtain a spatial attention map. The spatial threshold attention processing is used to highlight key regions such as plaques or blood vessels in the image. For example, the feature weight of the blood vessel region is strengthened, and the irrelevant background region is weakened. The spatial attention map is used to represent the importance of different spatial positions in the multi-modal feature, and is used to highlight key structure regions such as plaques or blood vessels. In some embodiments, the calculation formula of the spatial attention map is:

[0108] ;

[0109] wherein, denotes a spatial attention map, for highlighting key regions such as plaques; denotes a high-scale CT corrected feature map; denotes a high-scale MRI corrected feature map; Softmax denotes an activation function; MLP denotes a multi-layer perception machine; Cat denotes a cross numerical matrix.

[0110] performing frequency domain adaptive weighting on the high-scale corrected feature map respectively to obtain enhanced high-resolution structure information. The enhanced high-resolution structure information is used to strengthen the high-resolution anatomical structure details in the feature. The enhanced structure information can include edges and textures, etc. For example, for coronary arteries, the structure information such as small branches of blood vessels, edge details of plaques, etc. can be enhanced. In some embodiments, the calculation formula of the enhanced high-resolution structure information is:

[0111] ;

[0112] wherein, denotes enhanced high-resolution structure information; FFT denotes Fourier transform; denotes a frequency sensitivity matrix.

[0113] The high-scale correction feature map is processed by the spatial attention map and enhanced high-resolution structural information to obtain a fusion feature. In some embodiments, a calculation formula of the fusion feature is:

[0114]

[0115] wherein, denotes a fusion feature; denotes a Kronecker product modeling modal interaction; denotes an element-wise product.

[0116] The fusion feature is subjected to a topological constraint three-dimensional diffusion reconstruction to obtain a medical image three-dimensional model. The medical image three-dimensional model refers to a three-dimensional voxel model obtained by topological constraint three-dimensional diffusion reconstruction processing on a fusion feature sequence.

[0117] In some embodiments, obtaining the three-dimensional voxel model comprises:

[0118] A diffusion model three-dimensional structure is generated based on the fusion feature. The diffusion model three-dimensional structure refers to three-dimensional spatial information containing anatomical structures. The diffusion model three-dimensional structure can be generated by denoising processing on the fusion feature sequence based on the X-Diffusion framework to obtain a three-dimensional voxel model. The diffusion model three-dimensional structure can be optimized by topological constraints to ensure continuity and consistency. For example, the diffusion model three-dimensional structure of the coronary artery can be generated to clearly show the three-dimensional running, branching and three-dimensional location distribution of plaques of the blood vessels. In some embodiments, a calculation formula of the diffusion model three-dimensional structure is:

[0119]

[0120] wherein, denotes a diffusion model three-dimensional structure of the tthiteration; denotes a diffusion coefficient; denotes a diffusion model three-dimensional structure of the (t-1)thiteration; denotes Gaussian noise.

[0121] Based on the diffusion model and the gold standard model, a reconstruction loss function is constructed. The gold standard model refers to a standard three-dimensional voxel model serving as a reference benchmark. The reconstruction loss function is used to measure the difference between the three-dimensional voxel model generated by the diffusion model and the gold standard model.

[0122] Based on the diffusion model gradient and the gold standard model gradient, a topological loss function is constructed. The topological loss function is used to measure the difference between the gradient of the three-dimensional voxel model generated by the diffusion model and the gradient of the gold standard model.

[0123] ​​A total loss function is constructed based on the reconstruction loss function and the topology loss function. The reconstruction loss function and the topology loss function are selected to construct the total loss function, which can guarantee the morphological accuracy of the three-dimensional structure with the gold standard through the reconstruction loss, ensure the topological consistency of the anatomical structure through the topology loss, balance the reconstruction accuracy and the structural rationality, and improve the reliability and clinical practicability of the three-dimensional reconstruction. In some embodiments, the calculation formula of the total loss function is:

[0124] ;

[0125] ;

[0126] ;

[0127] wherein, represents the total loss function; represents the reconstruction loss function; represents the topology loss function; represents the generated diffusion model three-dimensional structure; represents the three-dimensional structure of the gold standard model; represents a gradient operator.

[0128] Based on the total loss function, the generation of the diffusion model three-dimensional structure is guided, and the final three-dimensional voxel model is obtained.

[0129] As shown in Figure 3 , it is a schematic diagram of the medical image three-dimensional model of the blood vessels reconstructed by the present application. It can be seen that the three-dimensional reconstruction method based on multi-modal medical image fusion can completely capture the full-level blood vessel structure from the trunk to the terminal branch; the edge sharpness and tissue contrast of the blood vessels are strengthened, so that the blood vessel profile reconstructed by MRI and CT is clear and the branch details are distinct, and finally a three-dimensional model of the blood vessels with rich details and accurate structure is generated, which provides reliable anatomical basis for blood vessel disease diagnosis (such as aneurysm and stenosis) and surgical planning.

[0130] Figure 2 It is an exemplary module diagram of a three-dimensional reconstruction system based on multi-modal medical image fusion provided by the present application. As shown in Figure 2 , the three-dimensional reconstruction system based on multi-modal medical image fusion includes an acquisition module, an alignment module, an extraction module, a fusion module, and a reconstruction module.

[0131] The acquisition module is used to acquire an original multi-modal image sequence; the original multi-modal image sequence includes CT images and MRI images.

[0132] The alignment module is used to perform spatial alignment on the original multi-modal image sequence to obtain a spatially aligned image pair; the spatially aligned image pair includes an aligned CT image and a corresponding aligned MRI image after alignment.

[0133] The extraction module is used for multi-scale hierarchical cognitive feature extraction on the spatially aligned image pair to obtain a corrected feature atlas.

[0134] The fusion module is used for performing space-time-frequency domain collaborative attention fusion on the corrected feature atlas to obtain a fused feature.

[0135] The reconstruction module is used for performing topological constraint three-dimensional diffusion reconstruction on the fused feature to obtain a medical image three-dimensional model.

[0136] The above only is the preferred embodiment of the present application, and is not used for limiting the present application, for the person skilled in the art, the present application can have various changes and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method based on multi-modal medical image fusion, characterized in that, The method comprises the following steps: obtaining an original multi-modal image sequence; the original multi-modal image sequence comprises CT images and MRI images; spatially aligning the original multi-modal image sequence to obtain a spatially aligned image pair; the spatially aligned image pair comprises an aligned CT image and a corresponding aligned MRI image; performing multi-scale hierarchical cognitive feature extraction on the spatially aligned image pair to obtain a corrected feature atlas; performing spatio-temporal frequency domain collaborative attention fusion on the corrected feature atlas to obtain fused features; performing topological constraint three-dimensional diffusion reconstruction on the fused features to obtain a medical image three-dimensional model.

2. The method of claim 1, wherein the method is based on multi-modal medical image fusion. The spatially aligned image pair is obtained by spatially aligning the original multi-modal image sequence, comprising: using a spatial transformation network to predict global transformation parameters; based on the global transformation parameters, determining a bilinear interpolation spatial transformation function, and processing the original multi-modal image sequence through the bilinear interpolation spatial transformation function to obtain a multi-modal transformed image sequence; respectively extracting features from the multi-modal transformed image sequence to obtain multi-modal contour features; respectively correcting local deformations of the multi-modal contour features to obtain a multi-modal corrected image sequence; processing the multi-modal corrected image sequence through dynamic time warping to obtain a dynamic time warping factor; registering the multi-modal transformed image sequence through the dynamic time warping factor to obtain the spatially aligned image pair. 3.The method of claim 1, wherein, The multi-scale hierarchical cognitive feature extraction on the spatially aligned image pair to obtain the corrected feature atlas comprises: using a multi-scale graph convolutional neural network to capture the spatially aligned image pair to obtain a multi-scale feature atlas; the multi-scale feature atlas comprises a multi-scale CT feature atlas and a multi-scale MRI feature atlas; respectively performing feature encoding on the highest scale feature atlas to obtain a high-scale corrected feature atlas.

4. The method of claim 3, wherein the method further comprises: The multi-scale feature atlas is obtained, comprising: capturing the slice change trend of the feature atlas at the previous scale under multiple time sequences to obtain a change trend feature; obtaining the spatial position relationship of the anatomical structure in the feature atlas at the previous scale to obtain a spatial position feature; performing attention allocation on the key regions in the feature atlas at the previous scale to obtain an attention feature; fusing the change trend feature, the spatial position feature and the attention feature to obtain the feature atlas at the current scale.

5. The method of claim 3, wherein the method further comprises: The high-scale corrected feature atlas is obtained, comprising: capturing the local interaction of the highest scale feature atlas to obtain a local feature atlas; performing global dependence modeling on the highest scale feature atlas to obtain a global feature atlas; constructing an encoding loss function through a self-supervised mask task; based on the encoding loss function, performing cross-axis attention processing on the local feature atlas and the global feature atlas to obtain the high-scale corrected feature atlas.

6. The method for three-dimensional reconstruction based on multi-modal medical image fusion according to claim 5, characterized in that, The calculation formula of the encoding loss function is: ； wherein, represents a mask feature reconstruction function; represents a feature map reconstructed after mask reconstruction on the high-scale corrected feature map; represents a high-scale corrected feature map; represents a Euclidean norm. 7.The method of claim 1, wherein, The fused features are obtained, comprising: respectively performing spatial threshold attention processing on the high-scale corrected feature atlas to obtain a spatial attention map; respectively performing frequency domain adaptive weighting on the high-scale corrected feature atlas to obtain enhanced high-resolution structural information; processing the high-scale corrected feature atlas through the spatial attention map and the enhanced high-resolution structural information to obtain the fused features.

8. The method of claim 7, wherein the method further comprises: The calculation formula of the fused features is: ； wherein, denotes fused features; denotes spatial attention map; denotes Kronecker product modeling modal interaction; denotes enhanced high-resolution structural information; denotes element-wise product; Cat denotes cross numerical matrix; denotes high-scale CT correction feature map; denotes high-scale MRI correction feature map. 9.The method of claim 1, wherein, The three-dimensional voxel model is obtained, comprising: Generate a diffusion model three-dimensional structure based on the fusion feature; Based on the diffusion model and the gold standard model, a reconstruction loss function is constructed; Based on the diffusion model gradient and the gold standard model gradient, a topological loss function is constructed; Based on the reconstruction loss function and the topological loss function, a total loss function is constructed; Based on the total loss function, the generation of the diffusion model three-dimensional structure is guided to obtain the final three-dimensional voxel model.

10. A three-dimensional reconstruction system based on multi-modal medical image fusion, characterized by, It comprises an acquisition module, an alignment module, an extraction module, a fusion module and a reconstruction module; The acquisition module is used for acquiring an original multi-modal image sequence; the original multi-modal image sequence comprises CT images and MRI images; The alignment module is used for spatial alignment of the original multi-modal image sequence to obtain a spatially aligned image pair; the spatially aligned image pair comprises an aligned CT image and a corresponding aligned MRI image; The extraction module is used for multi-scale hierarchical cognitive feature extraction of the spatially aligned image pair to obtain a corrected feature atlas; The fusion module is used for time-space-frequency domain collaborative attention fusion of the corrected feature atlas to obtain a fusion feature; The reconstruction module is used for topological constraint three-dimensional diffusion reconstruction of the fusion feature to obtain a medical image three-dimensional model.

Images