A multi-modal CT image data fusion three-dimensional reconstruction method for puncture positioning

Abstract

The application provides a multi-modal CT image data fusion three-dimensional reconstruction method for puncture positioning, constructs a high-dimensional feature space containing anatomical structure, metabolic activity and time-varying information through characteristic coding of spiral CT and multi-phase PET data; realizes modality alignment by using a mutual information driven double constraint registration mechanism, and completes cognitive level fusion of cross-modal features by combining a dynamic weight tensor of patient physiological characteristics, clinical requirements and organ semantics; and finally generates a three-dimensional reconstruction model with medical accuracy and clinical guidance by adaptive light field rendering and topology preservation optimization, so as to provide precise visualization support for puncture surgery.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and medical image processing, and specifically to a method for three-dimensional reconstruction by fusing multimodal CT image data for puncture localization. Background Technology

[0002] In medical imaging diagnosis, accurate 3D reconstruction is crucial for disease diagnosis and treatment planning. Traditional CT 3D reconstruction methods often utilize only single-modality CT image information, making it difficult to comprehensively reflect the complex characteristics of human tissues and organs. Multimodal imaging technologies, such as the combination of CT with other contrast-enhanced information, can provide richer physiological and anatomical information; for example, PET (positron emission tomography) provides information on metabolism and function. Currently, when using multimodal information fusion for 3D reconstruction, there are problems such as insufficient information utilization, difficulty in effectively balancing different modal information, and a lack of comprehensive application of dynamic and basic bodily information, leading to room for improvement in reconstruction accuracy and reliability. Summary of the Invention

[0003] To address the shortcomings of existing detection methods, a three-dimensional reconstruction method based on the fusion of multimodal CT image data for puncture localization is proposed.

[0004] A method for three-dimensional reconstruction by fusing multimodal CT image data for puncture localization, characterized by:

[0005] Step S1: Use a spiral CT scanner to take images of the area to be examined on the patient and obtain raw CT image data S_ct. After data standardization preprocessing, it is recorded as F_ct, which lays the foundation for subsequent image processing.

[0006] Step S2: Perform PET imaging at regular intervals. Use PET imaging data from different time points to form a multi-phase imaging data to construct a metabolic feature matrix. Encode time-varying metabolic information into high-dimensional feature vectors to analyze the metabolic changes in lesions and surrounding tissues, providing multi-dimensional feature support for subsequent multimodal fusion.

[0007] Step S3: Perform multimodal registration between the processed CT data and PET data. By learning the rotation matrix R and translation vector p, coordinate transformation is achieved to solve the spatial alignment problem between different modal images and establish the correspondence between structures.

[0008] Step S4: Based on the patient's different examination sites, examination needs, and basic physical data, set the weights of CT and PET information to achieve information balance; by setting different weights for different organs, needs, and physical conditions, while ensuring overall registration accuracy, achieve dynamic adaptation, enhance the registration effect of key local areas, and meet the clinical needs for high-precision registration of key areas.

[0009] Step S5: Based on the deep fusion of multimodal feature matrix and adaptive weight system, construct a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics, and achieve diagnostic-level visualization effect through intelligent rendering strategy;

[0010] Step S6: Based on the reconstructed 3D model, puncture localization is assisted by constraints, including:

[0011] Safe distance constraints to avoid important blood vessels and nerves (extracted from CT structural features);

[0012] Prioritize puncture at the peak metabolic activity point of the lesion center (based on PET feature localization).

[0013] Beneficial effects

[0014] Multimodal semantic fusion architecture: Breaking through the traditional data-level overlay mode, it constructs a composite representation space that includes anatomical structure, metabolic function and time-varying features, realizing the hierarchical transformation from image data to medical knowledge.

[0015] Dynamic weighted cognitive model: Based on a triple weighting mechanism of individual patient characteristics, clinical needs and organ semantics, it endows the reconstruction process with "personalized" decision-making ability and strengthens the semantic expression of key areas.

[0016] Intelligent rendering and topology optimization: Through deep learning-driven variable density sampling and probabilistic rendering strategies, a balance is achieved between diagnostic-grade visualization effects and real-time computing efficiency while ensuring the correctness of medical topology. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0018] A method for 3D reconstruction by fusing multimodal CT image data for puncture localization includes the following steps:

[0019] Step S1: Use a spiral CT scanner to take images of the area to be examined on the patient and obtain raw CT image data S_ct. After data standardization preprocessing, it is recorded as F_ct, which lays the foundation for subsequent image processing.

[0020] Step S2: Perform PET imaging at regular intervals. Use PET imaging data from different time points to form a multi-phase imaging data to construct a metabolic feature matrix. Encode time-varying metabolic information into high-dimensional feature vectors to analyze the metabolic changes in lesions and surrounding tissues, providing multi-dimensional feature support for subsequent multimodal fusion.

[0021] The multi-phase feature mapping architecture for PET is as follows:

[0022] The time encoding module TE is

[0023]

[0024] Where t represents the PET imaging acquisition time, d is the total dimension of the time-encoding vector, and i represents the dimension index of the feature vector, ranging from 0 to d / 2. This formula can characterize time-varying information; the formula for generating high-dimensional PET feature vectors is:

[0025] F_pet=Fusion([S_pet; TE(t)])

[0026] Wherein, S_pet is the metabolic feature matrix constructed from multi-phase contrast imaging data, and Fusion is a fusion network composed of MLP and activation function, which temporally encodes and fuses the original feature matrix to obtain the high-dimensional PET feature vector F_pet;

[0027] Step S3: Perform multimodal registration between the processed CT data and PET data. By learning the rotation matrix R and translation vector p, coordinate transformation is achieved to solve the spatial alignment problem between different modal images and establish the correspondence between structures.

[0028] Initial coarse registration preprocessing is performed on the raw CT and PET images. Mutual information is used to calculate intermodal dependencies. By optimizing the rotation matrix R and translation vector p, the initial alignment and registration of the images are achieved.

[0029] Mutual information (MI) is calculated as follows, where H() represents information entropy;

[0030] Construction of the multimodal registration objective function: MI(F_ct,F_pet)=H(F_ct)+H(F_pet)-H(F_ct,F_pet)

[0031] argmin R,p -MI((F_ct,R·F_pet+p)

[0032] The coordinate transformation is achieved by learning the rotation matrix R and the translation vector p, thus realizing basic registration. Mutual information is used as the registration target to form a dual constraint of geometric alignment and information association. The negative sign transforms the problem of maximizing mutual information into a minimum problem that can be optimized by gradient descent. Through iterative updates, the mutual information value between the transformed PET image and the CT image reaches its peak.

[0033] After registration, the PET information becomes F_pet' = R·F_pet + p.

[0034] Step S4: Based on the patient's different examination sites, examination needs, and basic physical data, set the weights of CT and PET information to achieve information balance; by setting different weights for different organs, needs, and physical conditions, while ensuring overall registration accuracy, achieve dynamic adaptation, enhance the registration effect of key local areas, and meet the clinical needs for high-precision registration of key areas.

[0035] Step S41: Calculation of Body Data Influence Factors

[0036] Height, weight, gender, and blood test results are standardized. This basic data is used to generate a body feature vector B through a feature embedding network, and its influence factor βbody on the modality weights is calculated.

[0037] βbody=σ(Wb·B+bb)

[0038] Where σ is the Sigmoid activation function, and Wb,bb are learnable parameters. This factor is used to adjust the influence of different patient physiological characteristics on CT / PET weights.

[0039] Step S42: Calculate the demand weight

[0040] βneed=σ(Wn·C+bn)

[0041] Where C is the specific demand one-hot vector, σ is the Sigmoid activation function, and Wn,bn are learnable parameters. This factor is used to adjust the influence of different demands on CT / PET weights.

[0042] Step S43: Calculate organ feature weights

[0043] βorg=σ(MLP([F_ct_roi;F_pet_roi']))

[0044] Where F_ct_roi and F_pet_roi' are organ region features of CT / PET, which are obtained by doctors annotating organ location features. MLP is a fully connected operation, and [;] represents a stitching operation. This factor is used to adjust the influence of different organs on the weight of CT / PET.

[0045] Step S5: Based on the deep fusion of multimodal feature matrix and adaptive weight system, construct a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics, and achieve diagnostic-level visualization effect through intelligent rendering strategy;

[0046] Step S51: Weighted Feature Fusion. The registered CT feature matrix F_ct and the PET high-dimensional feature vector F_pet' are fused based on the weighting factors calculated in step S4 to calculate the comprehensive balance coefficient.

[0047] β = βbody × βneed × βorg

[0048] Balancing CT and PET features:

[0049] F = β × F_ct + (1-β)F_pet'

[0050] Among them, the fused feature matrix F integrates structural information (CT) and metabolic information (PET), and dynamically balances the contributions of different modalities through weights;

[0051] Step S52: Render the fused feature volume using ray casting, introduce the correspondence between time-varying metabolic features and anatomical structures, and construct a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics.

[0052] Light sampling strategy optimization: Increase the sampling density of the lesion area according to β weight, the sampling formula is as follows:

[0053] samp(v) = samp base ×(1+γ·β(v)))

[0054] Where v is a three-dimensional voxel coordinate, and sample(v) represents the sampling rate of v. base The base sampling rate is γ, which is the density adjustment coefficient.

[0055] Probabilistic rendering model optimization: A voxel transparency transfer function α() is constructed based on PET metabolic activity to map high-metabolic regions to a semi-transparent state. The formula is as follows:

[0056] α(v)=σ(F_pet'(v)·λ)×τ(F_ct(v))

[0057] Where α(v) is the final transparency value of voxel v, ranging from 0 to 1 (0 is completely transparent and 1 is completely opaque), σ is the Sigmoid function, λ is the metabolic intensity threshold, and τ is the opacity function corresponding to CT density, which is the basic transparency directly determined by CT data and reflects the physical density of the tissue.

[0058] Step S6: Based on the reconstructed 3D model, puncture localization is assisted by constraints, including:

[0059] Safe distance constraints to avoid important blood vessels and nerves (extracted from CT structural features);

[0060] Prioritize puncture at the peak metabolic activity point of the lesion center (based on PET feature localization).

[0061] This invention constructs a high-dimensional feature space containing anatomical structure, metabolic activity, and time-varying information by feature encoding from spiral CT and multi-phase PET data. It achieves modal alignment using a mutual information-driven dual-constraint registration mechanism, and completes cognitive-level fusion of cross-modal features by combining dynamic weight tensors of patient physiological characteristics, clinical needs, and organ semantics. Finally, through adaptive light field rendering and topology-preserving optimization, it generates a 3D reconstruction model that combines medical accuracy and clinical guidance, providing precise visualization support for puncture surgery.

[0062] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for three-dimensional reconstruction by fusing multimodal CT image data for puncture localization, characterized by: S1. Use a spiral CT scanner to take images of the area to be examined on the patient and obtain raw CT image data S_ct. After data standardization and preprocessing, it is recorded as F_ct. S2. Perform PET imaging at regular intervals. Use PET imaging data from different time points to form multi-phase imaging data to construct a metabolic feature matrix. Encode time-varying metabolic information into high-dimensional feature vectors to analyze the metabolic changes in lesions and surrounding tissues. S3. Perform multimodal registration between the processed CT data and PET data, and achieve coordinate transformation by learning the rotation matrix R and translation vector p to establish the correspondence between structures; S4. Based on the patient's different examination sites, examination needs, and basic physical data, set the weights of CT and PET information to achieve information balance; set different weights according to different needs to achieve dynamic adaptation. S5. Based on a deep fusion multimodal feature matrix and adaptive weighting system, a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics is constructed, and a diagnostic-level visualization effect is achieved through intelligent rendering strategy. S6. Based on the reconstructed 3D model, puncture positioning is assisted by constraints; Step S4 further includes the following: Step S41: Calculation of Body Data Influence Factors The various examination indicators are standardized. This basic data is used to generate a body feature vector B through a feature embedding network, and its influence factor βbody on the modality weights is calculated. βbody = σ(Wb·B + bb); Where σ is the Sigmoid activation function, Wb,bb are learnable parameters, and βbody is used to adjust the influence of different patient physiological characteristics on CT / PET weights; Step S42: Calculate the demand weight βneed = σ(Wn·C + bn); Where C is the specific demand one-hot vector, Wn and bn are learnable parameters, and βneed is used to adjust the influence of different demands on CT / PET weights; Step S43: Calculate organ feature weights ； in, Organ region features on CT scans The organ region features in PET are obtained by doctors annotating the organ locations. MLP stands for fully connected operations, and [;] represents a stitching operation. Used to adjust the influence of different organs on CT / PET weights; Step S5 further includes the following: Step S51: Weighted feature fusion of the registered CT feature matrix With PET high-dimensional feature vector The overall balance coefficient is calculated based on the weighting factors calculated in step S4. β=βbody×βneed× ; Balancing CT and PET features: ； Among them, the fused feature matrix F integrates structural information CT and metabolic information PET, and dynamically balances the contributions of different modalities through weights; Step S52: Render the fused feature volume using ray casting, introduce the correspondence between time-varying metabolic features and anatomical structures, and construct a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics. Light sampling strategy optimization: Increase the sampling density of the lesion area according to β weight, the sampling formula is as follows: ； Where v is a three-dimensional voxel coordinate, and sample(v) represents the sampling rate of v. base The base sampling rate is γ, the density adjustment coefficient is β(v), and β(v) represents the result of multiplying the coordinates of each dimension of v with the weights of β. Probabilistic rendering model optimization: A voxel transparency transfer function α() is constructed based on PET metabolic activity to map high-metabolic regions to a semi-transparent state. The formula is as follows: ； Where α(v) is the final transparency value of voxel v, ranging from 0 to 1, where 0 is completely transparent and 1 is completely opaque; λ is the metabolic intensity threshold; τ is the opacity function corresponding to CT density, which is the baseline transparency directly determined by CT data and reflects the physical density of the tissue; and F_pet'(v) represents... The specific value in the v coordinate, F_ct(v) represents... The specific value in the v coordinate.

2. The method for three-dimensional reconstruction of multimodal CT image data fusion for puncture localization according to claim 1, characterized in that step S2 includes the following: The multi-phase feature mapping architecture for PET is as follows: The time encoding module (TE) is: ； in, t represents the acquisition time of PET imaging, d is the total dimension of the time encoding vector, and i represents the dimension index of the feature vector, ranging from 0 to d / 2. This formula represents time-varying information. The formula for generating high-dimensional feature vectors for PET is: ； Here, S_pet is the metabolic feature matrix constructed from multi-phase contrast imaging data, and Fusion is a fusion network composed of an MLP and an activation function. It performs temporal encoding on the original feature matrix and fuses them to obtain a high-dimensional PET feature vector. .

3. The method for three-dimensional reconstruction of multimodal CT image data fusion for puncture localization according to claim 1, characterized in that: Step S3 further includes the following: Initial coarse registration preprocessing is performed on the raw CT and PET images. Mutual information is used to calculate intermodal dependencies. By optimizing the rotation matrix R and translation vector p, the initial alignment and registration of the images are achieved. Mutual information (MI) is calculated as follows, where H() represents information entropy; ； Construction of multimodal registration objective function: ； Among them, coordinate transformation is achieved by learning the rotation matrix R and translation vector p to achieve basic registration. Mutual information is used as the registration target to form a dual constraint of geometric alignment and information association. The negative sign transforms the problem of maximizing mutual information into a minimum problem that can be optimized by gradient descent. Through iterative updates, the mutual information value between the transformed PET image and CT image reaches the peak value. After registration, the PET information becomes .

4. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute the method of claim 1.

5. An electronic device, characterized in that, It includes a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions, when executed, perform the method of claim 1.

Images