Multi-modal CT image data fusion three-dimensional reconstruction method for puncture positioning

Abstract

The invention provides a multi-modal CT image data fusion three-dimensional reconstruction method for puncture positioning, and the method comprises the steps: constructing a high-dimensional feature space containing an anatomical structure, metabolic activity and time-varying information through the feature coding of spiral CT and multi-phase PET data; modal alignment is realized by using a mutual information-driven dual-constraint registration mechanism, and cognitive fusion of cross-modal features is completed in combination with dynamic weight tensors of patient physiological features, clinical requirements and organ semantics; and finally, through adaptive light field rendering and topology maintenance optimization, a three-dimensional reconstruction model with medical accuracy and clinical guidance is generated, and accurate visual support is provided for puncture surgery.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and medical image processing, and in particular to a multi-modal CT image data fusion three-dimensional reconstruction method for puncture positioning. Background Art

[0002] In medical imaging diagnosis, accurate three-dimensional reconstruction is crucial for the diagnosis of diseases and the formulation of treatment plans. Traditional CT three-dimensional reconstruction methods often only use single-modality CT image information, which makes it difficult to fully reflect the complex characteristics of human tissues and organs. Multimodal imaging technologies, such as the combination of CT and other imaging information, can provide richer physiological and anatomical information. For example, PET imaging (positron emission tomography) provides metabolic and functional information. Currently, when multimodal information fusion is used for three-dimensional reconstruction, there are problems such as insufficient information utilization, difficulty in effectively balancing different modal information, and lack of comprehensive application of dynamic information and basic body information, resulting in the need to improve reconstruction accuracy and reliability. Summary of the Invention

[0003] Aiming at the shortcomings of existing detection methods, a multimodal CT image data fusion three-dimensional reconstruction method for puncture positioning is proposed.

[0004] A multimodal CT image data fusion 3D reconstruction method for puncture positioning, characterized by:

[0005] Step S1: Use a spiral CT device to photograph the patient's part to be examined, obtain CT raw image data S_ct, and record it as F_ct after data standardization preprocessing, laying the foundation for subsequent image processing;

[0006] Step S2: Perform PET angiography at regular intervals, use PET angiography data from different time points to construct a metabolic feature matrix, encode the time-varying metabolic information into a high-dimensional feature vector, analyze the metabolic change characteristics of the lesion and surrounding tissues, and provide multi-dimensional feature support for subsequent multimodal fusion;

[0007] Step S3: Perform multimodal registration on the processed CT data and PET data, implement coordinate transformation by learning the rotation matrix R and translation vector p, solve the spatial position alignment problem between different modality images, and establish the corresponding relationship between the structures;

[0008] Step S4: Set the weights of CT and PET information based on the patient's different examination sites, examination requirements, and basic physical data to achieve information balance. By setting different weights for different organs, requirements, and physical fitness, dynamic self-adaptation is achieved while ensuring overall registration accuracy, strengthening the key local registration effect and meeting the clinical demand for high-precision registration of key areas.

[0009] Step S5: Based on the deeply fused multimodal feature matrix and adaptive weight system, a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics is constructed, and diagnostic-level visualization effects are achieved through intelligent rendering strategies;

[0010] Step S6: Based on the reconstructed three-dimensional model, the puncture positioning is assisted by constraints, and the constraints include:

[0011] Avoid safety distance constraints of important blood vessels and nerves (extracted from CT structural features);

[0012] The principle of prioritizing puncture at the peak point of metabolic activity in the center of the lesion (located by PET characteristics).

[0013] Beneficial effects

[0014] Multimodal semantic fusion architecture: Breaking through the traditional data-level superposition model, constructing a composite representation space that includes anatomical structure, metabolic function, and time-varying characteristics, and realizing the hierarchical transformation from imaging data to medical knowledge.

[0015] Dynamic weighted cognitive model: Based on a triple weighting mechanism of individual patient characteristics, clinical needs, and organ semantics, it empowers the reconstruction process with personalized decision-making capabilities and strengthens the semantic expression of key areas.

[0016] Intelligent rendering and topology optimization: Through deep learning-driven variable density sampling and probabilistic rendering strategies, we achieve a balance between diagnostic-grade visualization and real-time computing efficiency while ensuring the correctness of medical topology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0018] A multimodal CT image data fusion three-dimensional reconstruction method for puncture positioning includes the following steps:

[0019] Step S1: Use a spiral CT device to photograph the patient's part to be examined, obtain CT raw image data S_ct, and record it as F_ct after data standardization preprocessing, laying the foundation for subsequent image processing;

[0020] Step S2: Perform PET angiography at regular intervals, use PET angiography data from different time points to construct a metabolic feature matrix, encode the time-varying metabolic information into a high-dimensional feature vector, analyze the metabolic change characteristics of the lesion and surrounding tissues, and provide multi-dimensional feature support for subsequent multimodal fusion;

[0021] The PET multi-phase feature map architecture is as follows:

[0022] The time encoding module TE is

[0023]

[0024] Where t represents the PET imaging acquisition time, d is the total dimension of the time encoding vector, and i represents the dimension index of the feature vector, ranging from 0 to d / 2. This formula can represent time-varying information; the formula for generating PET high-dimensional feature vectors is:

[0025] F_pet=Fusion([S_pet; TE(t)])

[0026] Among them, S_pet is the metabolic feature matrix constructed from multi-phase angiography data, and Fusion is a fusion network composed of MLP and activation function. The original feature matrix is ​​time-encoded and fused to obtain the PET high-dimensional feature vector F_pet;

[0027] Step S3: Perform multimodal registration on the processed CT data and PET data, implement coordinate transformation by learning the rotation matrix R and translation vector p, solve the spatial position alignment problem between different modality images, and establish the corresponding relationship between the structures;

[0028] Perform initial rough registration preprocessing on the original CT and PET images, use mutual information as the calculation of inter-modality dependency, and achieve preliminary image alignment by optimizing the rotation matrix R and translation vector p;

[0029] The mutual information MI is calculated as follows, where H() represents information entropy;

[0030] MI(F_ct,F_pet)=H(F_ct)+H(F_pet)-H(F_ct,F_pet) Multimodal registration objective function construction:

[0031] argmin R,p -MI((F_ct,R·F_pet+p)

[0032] The coordinate transformation is achieved by learning the rotation matrix R and the translation vector p, and the basic registration is realized. The mutual information is used as the registration target to form a dual constraint of geometric alignment and information association. The negative sign transforms the mutual information maximization problem into a minimum value problem that can be optimized by gradient descent. Through iterative updates, the mutual information value of the transformed PET image and CT image reaches the peak value.

[0033] After registration, the PET information becomes F_pet'=R·F_pet+p.

[0034] Step S4: Set the weights of CT and PET information based on the patient's different examination sites, examination requirements, and basic physical data to achieve information balance. By setting different weights for different organs, requirements, and physical fitness, dynamic self-adaptation is achieved while ensuring overall registration accuracy, strengthening the key local registration effect and meeting the clinical demand for high-precision registration of key areas.

[0035] Step S41: Calculation of physical data impact factors

[0036] Standardize the height, weight, gender, and blood test indicators. Use the feature embedding network to generate the body feature vector B, and calculate its influence factor βbody on the modal weight:

[0037] βbody=σ(Wb·B+bb)

[0038] Where σ is the Sigmoid activation function and Wb,bb is a learnable parameter used to adjust the influence of different patients’ physiological characteristics on CT / PET weights;

[0039] Step S42: Calculate demand weight

[0040] βneed=σ(Wn·C+bn)

[0041] Where C is the one-hot vector of specific requirements, σ ​​is the Sigmoid activation function, and Wn,bn are learnable parameters used to adjust the impact of different requirements on CT / PET weights;

[0042] Step S43: Calculate organ feature weights

[0043] βorg=σ(MLP([F_ct_roi;F_pet_roi']))

[0044] Among them, F_ct_roi, F_pet_roi' are the organ region features of CT / PET, which are obtained by doctors marking the organ location features. MLP is a fully connected operation, and [;] represents a splicing operation. This factor is used to adjust the influence of different organs on the CT / PET weight.

[0045] Step S5: Based on the deeply fused multimodal feature matrix and adaptive weight system, a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics is constructed, and diagnostic-level visualization effects are achieved through intelligent rendering strategies;

[0046] Step S51: Weighted feature fusion: The registered CT feature matrix F_ct and the PET high-dimensional feature vector F_pet' are fused based on the weight factors calculated in step S4 to calculate the comprehensive balance coefficient:

[0047] β=βbody×βneed×βorg

[0048] Balancing CT and PET features:

[0049] F=β×F_ct+(1-β)F_pet'

[0050] Among them, the fused feature matrix F integrates structural information (CT) and metabolic information (PET), and dynamically balances the contributions of different modalities through weights;

[0051] Step S52: Rendering the fused feature volume by ray casting, introducing the correspondence between the time-varying metabolic features and the anatomical structure, and constructing a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics;

[0052] Light sampling strategy optimization: Increase the sampling density of the lesion area according to the β weight. The sampling formula is:

[0053] samp(v)=samp base ×(1+γ·β(v)))

[0054] Where v is the three-dimensional voxel coordinate, samp(v) represents the sampling rate of v, and samp base is the basic sampling rate, γ is the density adjustment coefficient;

[0055] Probabilistic rendering model optimization: The voxel transparency transfer function α() is constructed in combination with PET metabolic activity to map the high metabolic area to a semi-transparent state. The formula is:

[0056] α(v)=σ(F_pet'(v)·λ)×τ(F_ct(v))

[0057] Where α(v) is the final transparency value of voxel v, ranging from 0 to 1, where 0 is completely transparent and 1 is completely opaque), σ is the sigmoid function, λ is the metabolic intensity threshold, and τ is the opacity function corresponding to CT density. The basic transparency is directly determined by CT data and reflects the physical density of the tissue;

[0058] Step S6: Based on the reconstructed three-dimensional model, the puncture positioning is assisted by constraints, and the constraints include:

[0059] Avoid safety distance constraints of important blood vessels and nerves (extracted from CT structural features);

[0060] The principle of prioritizing puncture at the peak point of metabolic activity in the center of the lesion (located by PET characteristics).

[0061] This paper constructs a high-dimensional feature space containing anatomical structure, metabolic activity, and time-varying information through feature encoding of spiral CT and multi-phase PET data; uses a dual-constraint registration mechanism driven by mutual information to achieve modality alignment, and combines the patient's physiological characteristics, clinical needs, and dynamic weight tensors of organ semantics to complete cognitive-level fusion of cross-modal features; finally, through adaptive light field rendering and topology-preserving optimization, a 3D reconstructed model with both medical accuracy and clinical guidance is generated, providing precise visualization support for puncture surgery.

[0062] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal CT image data fusion 3D reconstruction method for puncture positioning, characterized by: S1. Use a spiral CT device to photograph the patient's part to be examined, obtain the original CT image data S_ct, and record it as F_ct after data standardization preprocessing; S2. Perform PET imaging at regular intervals. Utilize the PET imaging data at different time points to construct a multi-phase imaging data metabolic feature matrix. Encode the time-varying metabolic information into a high-dimensional feature vector to analyze the metabolic change characteristics of the lesion and surrounding tissues. S3, perform multimodal registration on the processed CT data and PET data, realize coordinate transformation by learning the rotation matrix R and translation vector p, and establish the corresponding relationship of the structure; S4. Set the weights of CT and PET information based on the patient's different examination sites, examination requirements, and basic physical data to achieve information balance; set different weights according to different requirements to achieve dynamic adaptation; S5. Based on a deeply fused multimodal feature matrix and adaptive weight system, a 3D reconstruction model with physiological function-anatomical structure coupling characteristics is constructed, and diagnostic-level visualization effects are achieved through intelligent rendering strategies. S6. Based on the reconstructed 3D model, constraints are used to assist puncture positioning.

2. The method for 3D reconstruction of multimodal CT image data fusion for puncture positioning according to claim 1, wherein step S2 comprises the following: The PET multi-phase feature map architecture is as follows: The time encoding module TE is Where t represents the PET imaging acquisition time, d is the total dimension of the time encoding vector, and i represents the dimension index of the feature vector, ranging from 0 to d / 2. This formula can characterize time-varying information; The formula for generating PET high-dimensional feature vector is: F_pet=Fusion([S_pet; TE(t)]) in, S_pet is the metabolic feature matrix constructed from multi-phase angiography data. Fusion is a fusion network consisting of MLP and activation function. The original feature matrix is ​​time-encoded and fused to obtain the PET high-dimensional feature vector F_pet.

3. The method for 3D reconstruction of multimodal CT image data fusion for puncture positioning according to claim 1, characterized by: The step S3 further includes the following contents: Perform initial rough registration preprocessing on the original CT and PET images, use mutual information as the calculation of inter-modality dependency, and achieve preliminary image alignment by optimizing the rotation matrix R and translation vector p; The mutual information MI is calculated as follows, where H() represents information entropy; MI(F_ct,F_pet)=H(F_ct)+H(F_pet)-H(F_ct,F_pet) Multimodal registration objective function construction: argmin R,p -MI((F_ct,R·F_pet+p) The coordinate transformation is achieved by learning the rotation matrix R and the translation vector p, and the basic registration is realized. The mutual information is used as the registration target to form a dual constraint of geometric alignment and information association. The negative sign transforms the mutual information maximization problem into a minimum value problem that can be optimized by gradient descent. Through iterative updates, the mutual information value of the transformed PET image and CT image reaches the peak value. After registration, the PET information becomes F_pet'=R·F_pet+p.

4. The method for 3D reconstruction of multimodal CT image data fusion for puncture positioning according to claim 1, characterized by: The step S4 further includes the following contents: Step S41: Calculation of physical data impact factors Each inspection indicator is standardized. This basic data is used to generate a body feature vector B through a feature embedding network, and its influence factor βbody on the modal weight is calculated: βbody=σ(Wb·B+bb) Where σ is the Sigmoid activation function and Wb,bb is a learnable parameter used to adjust the influence of different patients’ physiological characteristics on CT / PET weights; Step S42: Calculate demand weight βneed=σ(Wn·C+bn) Where C is the one-hot vector of specific requirements, σ ​​is the Sigmoid activation function, and Wn,bn are learnable parameters used to adjust the impact of different requirements on CT / PET weights; Step S43: Calculate organ feature weights βorg=σ(MLP([F_ct_roi;F_pet_roi'])) Among them, F_ct_roi and F_pet_roi' are the organ region features of CT / PET, which are obtained by doctors marking the organ location features. MLP is a fully connected operation, and [;] represents a splicing operation. This factor is used to adjust the influence of different organs on the CT / PET weight.

5. The method for 3D reconstruction of multimodal CT image data fusion for puncture positioning according to claim 1, wherein step S5 further comprises the following: Step S51: Weighted feature fusion: The registered CT feature matrix F_ct and the PET high-dimensional feature vector F_pet' are fused based on the weight factors calculated in step S4 to calculate the comprehensive balance coefficient: β=βbody×βneed×βorg Balancing CT and PET features: F=β×F_ct+(1-β)F_pet' in, The fused feature matrix F integrates structural information (CT) and metabolic information (PET), and dynamically balances the contributions of different modalities through weights; Step S52: Rendering the fused feature volume by ray casting, introducing the correspondence between the time-varying metabolic features and the anatomical structure, and constructing a three-dimensional reconstruction model with physiological function-anatomical structure coupling characteristics; Light sampling strategy optimization: Increase the sampling density of the lesion area according to the β weight. The sampling formula is: samp(v)=samp base ×(1+γ·β(v))) Where v is the three-dimensional voxel coordinate, samp(v) represents the sampling rate of v, and samp base is the basic sampling rate, γ is the density adjustment coefficient; Probabilistic rendering model optimization: The voxel transparency transfer function α() is constructed in combination with PET metabolic activity to map the high metabolic area to a semi-transparent state. The formula is: α(v)=σ(F_pet′(v)·λ)×τ(F_ct(v)) where α(v) is the final transparency value of voxel v, ranging from 0 to 1, where 0 is completely transparent and 1 is completely opaque), σ is the sigmoid function, λ is the metabolic intensity threshold, and τ is the opacity function corresponding to the CT density. The basic transparency is directly determined by the CT data and reflects the physical density of the tissue.

6. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to claim 1.

7. An electronic device, characterized in that: The method comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions execute the method according to claim 1 when executed.

Images