Esophageal cancer multi-modal image registration fusion method and system thereof

Abstract

This invention provides a method and system for multimodal image registration and fusion in esophageal cancer, belonging to the field of medical image processing. The method includes: acquiring multimodal medical images from CT, MRI, and PET; preprocessing the images to extract regions of interest (ROIs) of organs; constructing a topological image based on differential homeomorphism and performing non-rigid registration, including constructing a topological image represented by a differential manifold, defining a differential homeomorphic transformation space that preserves the topological structure, and solving a variational optimization problem to obtain the optimal deformation field; optimizing the registration results using a multi-scale strategy; performing image fusion based on an uncertainty-aware fusion method; evaluating treatment effects through temporal registration; and image visualization processing. The system of this invention includes corresponding functional modules. By introducing non-rigid registration technology based on differential homeomorphic topological images, it effectively avoids the topological violation problem in traditional registration and significantly improves registration accuracy.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, specifically to a method and system for multimodal image registration and fusion of esophageal cancer, and particularly to the application of non-rigid registration technology based on differential homeomorphic topological images in the registration and fusion of multimodal medical images of esophageal cancer. Background Technology

[0002] Esophageal cancer is one of the most common malignant tumors of the digestive tract, characterized by high incidence, early metastasis, and poor prognosis. Accurate imaging diagnosis is crucial for the early detection, clinical staging, treatment planning, and efficacy evaluation of esophageal cancer. Modern medical imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) provide multimodal information for the diagnosis of esophageal cancer. However, due to differences in imaging principles, resolution, and contrast among multimodal medical images, effectively integrating this multimodal information remains a pressing technical challenge.

[0003] Currently, the registration and fusion of multimodal medical images in clinical practice mainly rely on traditional registration algorithms, such as intensity-based registration methods, feature-based registration methods, and model-based registration methods. However, these methods have a series of problems when dealing with the non-rigid deformation of soft tissue organs such as esophageal cancer: (1) Traditional non-rigid registration algorithms such as B-spline or thin plate spline methods are difficult to guarantee the topological rationality of the transformation, and often exhibit topological violations such as folding, tearing, and crossing; (2) Existing methods do not handle the complex gray-level relationships between multiple modalities well, resulting in low registration accuracy; (3) There is a lack of quantitative assessment of the uncertainty of the registration results, which affects the reliability of clinical decision-making; (4) The unique anatomical structural characteristics and deformation features of esophageal cancer are not adequately considered, making it difficult to achieve accurate registration.

[0004] Therefore, there is an urgent need for a new method that can effectively handle the multimodal image characteristics of esophageal cancer, ensure the rationality of transformation topology, and accurately achieve non-rigid registration, so as to improve the diagnosis and treatment of esophageal cancer. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for multimodal image registration and fusion of esophageal cancer, aiming to overcome the problems existing in the prior art. By introducing a non-rigid registration technique based on differential homeomorphic topological images, high-precision registration and fusion of multimodal medical images of esophageal cancer can be achieved, providing more reliable imaging support for clinical diagnosis and treatment.

[0006] This invention proposes a multimodal image registration and fusion method for esophageal cancer, comprising:

[0007] Acquire multimodal medical images of esophageal cancer patients;

[0008] The multimodal medical images are preprocessed to extract the regions of interest for the organs that need to be aligned;

[0009] A differential homeomorphic topological image is constructed based on the region of interest of the organ, and non-rigid registration is performed on the differential homeomorphic topological image corresponding to the multimodal medical image in the multimodal medical image.

[0010] A topological image representing a differential manifold is constructed for the region of interest of the organ, and a differential homeomorphic transformation space that preserves the topological structure is defined;

[0011] Based on the topological image represented by the differential manifold, a variational optimization problem is solved in the differential homeomorphic transformation space to obtain the optimal deformation field that satisfies the topology-preserving constraint.

[0012] The optimal deformation field is applied to the original space to achieve organ alignment.

[0013] The registration results are optimized based on a multi-scale strategy, registering images of different regions sequentially from coarse to fine.

[0014] The uncertainty-aware fusion method fuses the registered multimodal images according to anatomical structure and region weights.

[0015] Based on time-series image data, the fused images before and after treatment are registered to record tumor changes and evaluate treatment efficacy;

[0016] The fused images are then visualized to highlight the tumor area.

[0017] Preferably, the preprocessing of the multimodal medical images includes:

[0018] The multimodal medical images in the multimodal medical images are normalized to unify the image pixel range;

[0019] Target regions in each modality of the multimodal medical image are extracted using prior information, and organ segmentation images are obtained through thresholding.

[0020] The multimodal medical images are unified to the same spatial coordinate system, and histogram matching is used to adjust the spatial offset and scaling of the images, so that the pixel intensity distribution of the multimodal medical images becomes a uniform distribution.

[0021] Preferably, the construction of the differential homeomorphic topological image includes:

[0022] A global view of the anatomical structure is generated by downsampling the region of interest of the organ, and anatomical images at different scales are extracted using a Gaussian pyramid structure.

[0023] Calculate local feature descriptors of organ surfaces, including gradient orientation histogram features, local curvature features, and texture features;

[0024] The organ surface is represented as a two-dimensional manifold embedded in three-dimensional Euclidean space, and an adjacency graph on the manifold is constructed.

[0025] Calculate the Laplacian operator spectral decomposition on the manifold, obtain intrinsic frequency features, and establish a thermal kernel feature representation on the manifold;

[0026] We analyze topological invariants on manifolds, extract key topological structures, and construct a topological persistence graph to quantify the stability of features.

[0027] Preferably, the definition of the topologically preserving differential homeomorphic transformation space includes:

[0028] Define a transformation space in which both the forward and inverse transformations are continuously differentiable;

[0029] A layered transformation model is adopted, including a global affine transformation layer, a regional affine transformation layer, and a local deformation layer;

[0030] Establish a representation space for the transformation parameters. The global transformation is based on the matrix Lie group representation, and the local transformation is based on the vector field representation on the differential homeomorphism group.

[0031] Design constraints to guarantee the property of differential homeomorphism, including the Jacobian determinant constant positive constraint, the transformed gradient bounded constraint, and the transformed boundary condition constraint.

[0032] The initial transformation parameters are calculated based on the correspondence of topological feature points to generate the initial deformation field.

[0033] Preferably, solving the variational optimization problem in the differential homeomorphic transformation space includes:

[0034] Define an energy functional on a manifold, which includes a data term that measures the similarity of topological images, a regularization term that controls the smoothness of the deformation field, and a constraint term that ensures that the deformation satisfies the differential homeomorphism condition.

[0035] Construct a multi-scale optimization framework and implement hierarchical optimization strategies from coarse to fine;

[0036] Perform manifold-based iterative optimization at each scale level, including calculating energy values, calculating the gradient of the energy functional with respect to the transformation parameters, updating the parameters along the gradient direction on the manifold, and applying constraints to ensure that the updated parameters satisfy the topological conditions.

[0037] Design convergence criteria, including energy value change rate threshold, parameter update magnitude threshold, and gradient norm threshold;

[0038] The optimization results are evaluated, including calculating the final energy value, assessing the degree of topology preservation, and analyzing the characteristics of the deformation field.

[0039] Preferably, applying the optimal deformation field to the original space includes:

[0040] Calculate the transformation from the topological space to the original image space and verify the one-to-one correspondence of the mapping;

[0041] A dense deformation field is constructed in the original image space, and the source image is transformed using the deformation field.

[0042] Align the deformed image with the target image and adjust the deformation effect in the boundary area;

[0043] Analyze the characteristics of the deformation field, including calculating the statistical properties of the deformation field, analyzing the spatial distribution of the deformation field, and detecting abnormal regions in the deformation field;

[0044] Assess the quality of local deformation, calculate the degree of deformation in local areas, analyze the deformation quality of organ boundary areas, and evaluate the alignment accuracy of key anatomical structures.

[0045] Preferably, the uncertainty-aware fusion method includes:

[0046] The uncertainty-aware fusion method includes:

[0047] The global feature extraction algorithm is used to extract features from each modality image in the optimized registration image to obtain the feature map corresponding to each modality image;

[0048] The uncertainty of the feature maps corresponding to each modality image is quantified using a self-attention mechanism to obtain the uncertainty weights of each feature map;

[0049] Based on the uncertainty weights, the feature maps corresponding to each modality image are weighted and fused to obtain a low-rank feature map;

[0050] The low-rank feature map is converted into a region fusion segmentation algorithm using a region growing algorithm, and the fusion result is further adjusted by region weights to obtain the fused image;

[0051] Analyze the sources of uncertainty, including uncertainties caused by image quality, parameter sensitivity during the optimization process, and systematic errors introduced by model assumptions;

[0052] It quantifies the degree of uncertainty, generates an uncertainty heatmap, marks areas of high uncertainty, and provides quantitative indicators of the degree of uncertainty.

[0053] Preferably, the registration based on time-series image data includes:

[0054] The images before treatment are registered with the images after each treatment course, and the tumor treatment effect is evaluated by the results of multi-frame registration.

[0055] Multimodal image registration was performed using retrospective image sequence data;

[0056] Detect changes in the target based on changes in images before and after the sequence;

[0057] The visualization of tumor motion and efficacy assessment is obtained through multi-frame fusion, and the multi-frame registration results are enhanced for display.

[0058] Preferably, the multimodal medical images include CT images, MRI images, and PET images, wherein:

[0059] A uniform voxel spatial resolution of 1×1×1 mm³ was adopted for CT and MRI images;

[0060] The PET images were processed with a specific resolution of 0.752×0.752×0.752 mm³ and downsampled to a resolution of 4×4×4.

[0061] CT and MRI data are segmented using human anatomical structure tags;

[0062] Thresholding was performed on PET data using the tumor volume of interest and the standard SUV value.

[0063] An esophageal cancer multimodal image registration and fusion system, characterized in that it includes:

[0064] The preprocessing module is used to preprocess the multimodal medical images of the multimodal images separately and extract the regions of interest of organs that need to be aligned.

[0065] The topology image construction module is used to construct a topology image based on differential homeomorphism, including a multi-scale anatomical feature extraction unit, a differential manifold representation construction unit, and a topology analysis unit;

[0066] The differential homeomorphism module is used to define the differential homeomorphism transformation space that preserves the topology, including the transformation space definition unit, the topology-preserving constraint design unit, and the transformation parameter initialization unit.

[0067] The deformation optimization module is used to solve variational optimization problems in the space of differential homeomorphism to obtain the optimal deformation field that satisfies the topology preservation constraint. It includes a variational problem construction unit on the manifold, a multi-scale optimization strategy unit, and a convergence control and evaluation unit.

[0068] The inverse transformation module is used to apply the optimal deformation field to the original space to achieve organ alignment. It includes a topological image to original space mapping unit, a deformation field analysis and evaluation unit, and an uncertainty quantification unit.

[0069] The multi-scale optimization module is used to optimize the registration results based on a multi-scale strategy, registering images of different regions sequentially from coarse to fine.

[0070] The intelligent fusion module is used for uncertainty-aware fusion methods to fuse registered multimodal images according to anatomical structure and region weights.

[0071] The temporal registration module is used to register images before and after treatment based on temporal image data, record tumor changes, and evaluate treatment effects;

[0072] The image registration visualization module is used to visualize the fused images and highlight the tumor area.

[0073] The beneficial effects of this invention include:

[0074] 1. The topological image representation method based on differential homeomorphism ensures the topological rationality of the transformation from a mathematical perspective, effectively avoiding topological violations such as folding, tearing and crossing that are common in traditional non-rigid registration, and significantly improving registration accuracy, especially for fine structures such as blood vessels and esophagus, the registration accuracy is improved by 30% to 40%.

[0075] 2. The multi-scale manifold optimization strategy effectively solves the local optimum problem. Compared with the traditional non-rigid registration method, it improves the computation speed by more than 50%, reduces the number of iterations by 65%, and reduces the processing time of large-scale data by 70%.

[0076] 3. Uncertainty perception fusion methods provide a basis for reliability assessment in clinical decision-making. By quantifying the uncertainty of registration results, the credibility and clinical application value of registration fusion results are improved.

[0077] 4. The time-series registration function enables precise comparison of images before and after treatment, providing technical support for quantitative assessment of tumor changes and objective evaluation of treatment effects.

[0078] 5. The complete technical solution constructs a full-process solution from preprocessing to visualization, providing systematic technical support for the diagnosis, treatment planning and efficacy evaluation of esophageal cancer. Attached Figure Description

[0079] Figure 1 This is a schematic flowchart of the multimodal image registration and fusion method for esophageal cancer of the present invention;

[0080] Figure 2 This is a schematic diagram of the non-rigid registration method based on differential homeomorphic topological images according to the present invention;

[0081] Figure 3 Workflow diagram for constructing units for topological images;

[0082] Figure 4 This is a schematic diagram of the working principle of the differential homeomorphism transformation unit;

[0083] Figure 5 A schematic diagram of a multi-scale optimization strategy for deformation optimization elements;

[0084] Figure 6 This is a flowchart of the uncertainty-aware fusion method. Detailed Implementation

[0085] Please refer to Figures 1-6 The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0086] like Figure 1 As shown, this invention provides a multimodal image registration and fusion method for esophageal cancer, including steps such as acquiring multimodal medical images, preprocessing, constructing a topological image based on differential homeomorphism for non-rigid registration, multi-scale optimization, uncertainty-aware fusion, temporal registration, and visualization processing.

[0087] First, multimodal medical images of esophageal cancer patients are acquired. Preferably, these multimodal medical images include CT images, MRI images, and PET images, which respectively provide anatomical structure information, soft tissue contrast information, and metabolic function information. In a preferred embodiment of the invention, these images are acquired through a hospital's Picture Archiving and Communication System (PACS) and saved in DICOM format.

[0088] Next, the acquired multimodal medical images are preprocessed to extract the regions of interest (ROIs) of the organs that need to be aligned. For example... Figure 1 As shown, the preprocessing module mainly performs the following operations:

[0089] Multimodal medical images are normalized to unify the pixel range. Specifically, the Hounsfield unit (HU) values ​​of CT images are normalized to the range of [-1,1], while MRI and PET images are normalized to the range of [0,1]. This normalization helps reduce intensity differences between different modalities, providing a more consistent basis for subsequent registration.

[0090] Target regions in multimodal medical images are extracted using prior information, and organ segmentation images are obtained through thresholding. Preferably, for CT images, the esophageal region is extracted using a threshold range of [-100HU, 200HU]; for MRI images, the esophageal region is segmented using a region growing algorithm combined with edge detection; and for PET images, the tumor region is initially segmented using a standard uptake value (SUV) threshold of 2.5.

[0091] By unifying multimodal images to the same spatial coordinate system and using histogram matching to adjust the spatial offset and scaling of the images, the pixel intensity distribution of the multimodal medical images becomes a uniform distribution. In one embodiment of the invention, the mutual information maximization criterion is used to achieve initial spatial alignment of the images, and then spline interpolation is used to resample images of different resolutions to a unified spatial coordinate system.

[0092] like Figure 2 As shown, after preprocessing, the core innovation of this invention lies in constructing a topological image based on differential homeomorphism and performing non-rigid registration of the topological images of multimodal medical images. This process mainly includes the following key steps:

[0093] First, construct a topological image representing the differentiable manifold. For example... Figure 3 As shown, the topology image construction unit mainly performs the following operations:

[0094] A global view of the anatomical structure is generated by downsampling, and anatomical images at different scales are extracted using a Gaussian pyramid structure. Preferably, 4 to 5 different resolution levels are constructed, decreasing sequentially from the original resolution to 1 / 16 of the original. The image at each level is obtained through Gaussian filtering and downsampling.

[0095] Local feature descriptors of the organ surface are calculated, including gradient orientation histogram features, local curvature features, and texture features. Specifically, the gradient orientation histogram uses 8 orientation bins, the local curvature features include principal curvature, Gaussian curvature, and mean curvature, and the texture features are extracted using a Gabor filter bank with filter parameters set to 4 scales and 6 orientations.

[0096] The organ surface is represented as a two-dimensional manifold embedded in three-dimensional Euclidean space, and an adjacency graph is constructed on the manifold. In this graph, nodes represent feature points, edges represent geodesic connections on the manifold, and edge weights are calculated based on geodesic distances on the manifold. Preferably, adjacency relationships are constructed using the k-nearest neighbor algorithm, with k set between 8 and 12, adaptively adjusted according to data density.

[0097] The Laplace spectral decomposition on a manifold is computed to obtain intrinsic frequency features and establish a thermal kernel feature representation on the manifold. The Laplace spectral decomposition is achieved by solving the following eigenvalue problem:

[0098] ,

[0099] in, manifold The Laplace-Beltramian operator on a manifold represents a second-order differential operator on a manifold. Let be a function defined on the manifold, and let represent the characteristic function to be solved. The eigenvalue represents the eigenfunction. The corresponding frequency value. The thermonuclear characteristic is calculated using the following formula:

[0100] ,

[0101] in, Let be the heat kernel function, representing the heat from point . Spread to point The required time is The probability density; For the Laplace operator The i-th eigenvalue represents the th eigenvalue. One frequency component; Let be the corresponding characteristic function, representing the th One oscillation pattern; and A point on a manifold; The time parameter controls the diffusion process. In practical implementation, sufficient accuracy can be obtained by approximating the first 100 eigenvalues ​​and eigenfunctions.

[0102] This study analyzes topological invariants on manifolds, extracts key topological structures, and constructs a topological persistence graph to quantify the stability of features. Topological invariants include Eulerian eigenvalues ​​and Betti numbers (number of connected components, holes, cavities, etc.). Topological persistence quantifies the stability of topological features by tracking their appearance and disappearance at different thresholds; features with higher persistence typically represent more significant structures.

[0103] Next, we define the space of differential homeomorphic transformations that preserve the topological structure. For example... Figure 4 As shown, the differential homeomorphic transformation unit mainly performs the following operations:

[0104] Define a transformation space with continuously differentiable inverse transformation properties. A differential homeomorphism is a one-to-one correspondence mapping between two manifolds, where both the mapping and its inverse mapping are continuously differentiable. This transformation ensures that the topology remains unchanged before and after the transformation, avoiding problems such as folding and tearing.

[0105] A layered transformation model is employed, comprising a global affine transformation layer, a regional affine transformation layer, and a local deformation layer. This layered design allows the transformation to capture both global rigid changes and adapt to local non-rigid deformations. In one embodiment of the invention, the global affine transformation uses a 9-parameter model (3 translations, 3 rotations, and 3 scalings), the regional affine transformation divides the image into 3 to 5 regions based on anatomical structures and transforms them separately, and the local deformation is represented by a dense deformation field.

[0106] A representation space for the transformation parameters is established. The global transformation is based on a matrix Lie group representation, and the local transformation is based on a vector field representation on a differential homeomorphism group. Specifically, the global transformation matrix... ,in For three-dimensional rigid body motion, a special Euclidean group is used; local transformations are performed through vector fields. It means that, among them The image domain represents a bounded region in three-dimensional space; For vector field functions, each point in space is mapped to a three-dimensional displacement vector; It represents the three-dimensional real number space.

[0107] Design constraints to guarantee the homeomorphism property, including the Jacobian determinant constant positivity constraint, the bounded gradient constraint of the transformation, and the boundary condition constraint of the transformation. The Jacobian determinant constant positivity constraint ensures that the transformation does not fold or flip locally, and is mathematically expressed as:

[0108] ,

[0109] in, For transformation At point The Jacobian matrix at that point represents a local linear approximation of the transformation at that point; This represents the determinant operator, which calculates the determinant value of a matrix; A transformation function maps points in space to their transformed positions. A point in space; The image domain represents the region defined by the transformation. The bounded constraint of the transformation gradient controls the smoothness of the deformation, while the boundary condition constraint ensures the continuity of the transformation at the boundaries.

[0110] The initial transformation parameters are calculated based on the correspondence of topological feature points to generate the initial deformation field. Preferably, a robust feature matching algorithm such as RANSAC is used to establish the initial correspondence, filter out erroneous matches, and then the optimal initial transformation parameters are solved by the least squares method.

[0111] After defining the space of the differential homeomorphic transformation, this invention solves the variational optimization problem within this space to obtain the optimal deformation field that satisfies the topology-preserving constraint. For example... Figure 5 As shown, the deformation optimization unit mainly performs the following operations:

[0112] Define an energy functional on a manifold, comprising a data term measuring the similarity of topological images, a regularization term controlling the smoothness of the deformation field, and a constraint term ensuring that the deformation satisfies the diffeomorphism condition. The general form of the energy functional is:

[0113] ,

[0114] in, Let be the total energy, and represent the objective function to be optimized. For data items, measure the transformation between two images. The degree of similarity; This is a regularization term that controls the smoothness and complexity of the deformation field; As a constraint term, it ensures that the deformation satisfies the differential homeomorphism condition; and These are weighting coefficients used to balance the relative importance of the three items; This is a transformation function that maps points from the source image space to the target image space. Preferably, The value range is 0.01-0.1. The value ranges from 0.1 to 1.0, and the specific value can be adaptively adjusted according to the registration difficulty.

[0115] Data Items Normalized mutual information (NMI) is used as the metric to measure the similarity between two topological images:

[0116] ,

[0117] in, and These are the source image and the target image, representing the two images to be registered; Representing an image Entropy measures the uncertainty of information in an image; It represents the joint entropy, which measures the joint uncertainty of two images; This indicates the composition of functions. Indicates that the transformation will be performed. Applied to images The result afterwards.

[0118] Regularization term To control the smoothness and complexity of the deformation field, an elastic energy model is adopted:

[0119] ,

[0120] in, Let be the displacement field, representing the displacement vector at each point; For the displacement field One component; The first spatial coordinate One component; Represents the displacement field The Each component corresponds to spatial coordinates Partial derivatives of each component; and Lamé coefficient, which controls the elastic properties of the material; The integration domain represents the spatial region defined by the image; This represents a volume element. In practical applications, and The ratio is usually set to 2:1, with the specific value adjusted according to the elastic properties of the organ.

[0121] Constraints To ensure the deformation satisfies the differential homeomorphism condition, a design based on the Jacobian determinant is employed:

[0122] ,

[0123] in, For transformation At point Jacobian matrix at the location; Let be the determinant of the Jacobian matrix; The integration domain represents the spatial region defined by the image. This represents the volume element. This constraint term incurs a high energy penalty when the Jacobian determinant is close to 0 or very large, prompting the optimization process to avoid these regions and maintain the topology unchanged.

[0124] A multi-scale optimization framework is constructed, implementing a hierarchical optimization strategy from coarse to fine. Preferably, four scale levels are used, corresponding to 1 / 8, 1 / 4, 1 / 2, and the original resolution of the original image, respectively. At each scale level, the parameters of the energy functional need to be adjusted accordingly. For example, at the coarse scale level, the weight of the regularization term is increased to obtain a smoother global transformation, while at the fine scale level, the weight of the regularization term is decreased to capture local detail deformations.

[0125] Iterative optimization based on the manifold is performed at each scale level, including calculating the energy value, calculating the gradient of the energy functional with respect to the transformation parameters, updating the parameters along the gradient direction on the manifold, and applying constraints to ensure that the updated parameters satisfy the topological conditions. Preferably, an accelerated gradient method with kinetic terms is used for iterative optimization.

[0126] ,

[0127] in, For the first The deformation field of the next iteration represents the deformation parameters in the current iteration step; For the first The deformation field of the next iteration represents the updated deformation parameters; For the first The deformation field of the next iteration represents the deformation parameters of the previous step; The learning rate controls the step size of each iteration. For energy functionals Regarding deformation fields The gradient represents the direction of the fastest energy decrease; The momentum coefficient controls the degree to which the previous update direction affects the current update. The preferred parameter setting is: initial learning rate. As the iterations proceed, the momentum coefficient gradually decreases to 0.001. .

[0128] The design incorporates convergence criteria, including thresholds for the rate of change of energy, parameter update magnitude, and gradient norm. Preferably, convergence is considered achieved when the relative rate of change of energy over three consecutive iterations is less than 10⁻⁴, the Euclidean norm of the parameter update is less than 10⁻³, or the gradient norm is less than 10⁻³. Furthermore, a maximum of 100 iterations is set to avoid excessive iteration.

[0129] The evaluation of the optimization results includes calculating the final energy value, assessing the degree of topology preservation, and analyzing the characteristics of the deformation field. The degree of topology preservation is assessed by checking whether the Jacobian determinant is always positive; the deformation field characteristic analysis includes calculating the average displacement, maximum displacement, and spatial distribution characteristics of the deformation field.

[0130] After obtaining the optimal deformation field, it needs to be applied to the original space to achieve organ alignment. The resulting inverse transformation unit mainly performs the following operations:

[0131] Calculate the transformation from the topological space to the original image space and verify the one-to-one correspondence of the mapping. This step requires establishing a precise mapping relationship between the topological space and the original image space to ensure that deformation information can be accurately transferred to the original image.

[0132] A dense deformation field is constructed in the original image space, and the source image is transformed using the deformation field. Preferably, a trilinear interpolation method is used to implement the application of the deformation field, ensuring the smoothness and continuity of the deformed image.

[0133] Align the deformed image with the target image and adjust the deformation effect in the boundary areas. In the boundary areas, due to incomplete information, special processing may be required, such as using boundary constraints or smooth transition strategies.

[0134] Analyzing the characteristics of the deformation field includes calculating its statistical properties, analyzing its spatial distribution, and detecting anomalous regions. These analyses help assess registration quality and identify potential problems.

[0135] The assessment evaluates the quality of local deformation, calculates the degree of deformation in local areas, analyzes the deformation quality of organ boundary areas, and evaluates the alignment accuracy of key anatomical structures. Preferably, higher assessment weights are assigned to key structures in esophageal cancer, such as the esophageal wall, surrounding blood vessels, and lymph nodes.

[0136] The registration results are optimized using a multi-scale strategy, registering images of different regions sequentially from coarse to fine. The multi-scale optimization module mainly implements the following functions:

[0137] The global alignment of organs in the global view is applied to the fully sampled input image to obtain a coarse alignment of the image.

[0138] Alignment results are sampled at different scales, and the alignment error at each scale is calculated. Preferably, multiple metrics such as mean squared error (MSE), mutual information (MI), and structural similarity index (SSIM) are used to comprehensively evaluate the alignment quality.

[0139] The optimal alignment result of the sampled image is calculated, and the multi-resolution gradient descent method is used for optimization. The result of multi-scale alignment is iteratively optimized.

[0140] The alignment results are sampled and optimized multiple times, from coarse to fine, to sequentially register images of different regions. Furthermore, higher optimization weights can be set for specific regions such as tumor boundaries and esophageal walls.

[0141] Next, an uncertainty-aware fusion method is used to fuse the registered multimodal images according to anatomical structure and region weights. For example... Figure 6 As shown, the intelligent fusion module mainly performs the following operations:

[0142] Global feature extraction algorithms are used to extract global features from the image. Preferably, deep convolutional neural networks are used to extract multi-level features, such as the first few layers of a VGG or ResNet network structure.

[0143] The uncertainty of a feature map is quantified using a self-attention mechanism. This mechanism identifies regions of uncertainty or unreliability by calculating the correlation between locations within the feature map. The formula for quantifying uncertainty is:

[0144] ,

[0145] in, Indicates position The uncertainty measure is located at a point in time, with a value range of [0,1]. The larger the value, the higher the uncertainty. The feature vector representing that location contains image feature information for that location; This represents the average of the features, i.e., the average of the feature vectors at all locations; Representing the eigenvector With average characteristics The square of the Euclidean distance between them; This is a scaling parameter that controls the sensitivity of uncertainty measurement; It is an exponential function; The image domain represents the set of all spatial locations; for A point in the equation is used for summation. In practical applications, It is usually set to 0.1-0.5 times the standard deviation of the eigenvectors.

[0146] Based on the extracted global features and the quantized uncertainty, the multimodal image feature maps are fused into a single low-rank feature map. The fusion process can be represented as:

[0147] ,

[0148] in, The merged features represent the location. The fused feature vector; For the first The features of the modality represent the _th ... The modality at position The eigenvector at that location; For position First The weights of each modality represent the importance of that modality in the fusion process; The number of modalities is 3 in this embodiment (CT, MRI, PET). This represents summing over all modes. The weights satisfy... This ensures that the weights are normalized. The weight calculation takes into account a measure of uncertainty.

[0149] ,

[0150] in, For position First The weights of each modality; For the first The modality at position The uncertainty measure at the location is [0,1], and its value range is [0,1]. The confidence coefficient of this mode reflects its importance in a specific task; Indicates the first The modality at position The degree of certainty at the location; This indicates summing over all modes to ensure weight normalization. For the diagnosis of esophageal cancer, CT scans... The value is usually set to 0.4, 0.3 for MRI, and 0.3 for PET. The specific value can be adjusted according to clinical needs.

[0151] The low-rank feature map is transformed into a region fusion segmentation algorithm using a region growing algorithm, and the fusion result is further adjusted by region weights. The region growing algorithm starts from a seed point and gradually expands the region until a stopping condition is met. The seed point is selected based on the saliency of the feature map, while the stopping condition combines feature similarity and boundary strength.

[0152] Analyzing the sources of uncertainty includes uncertainties caused by image quality, parameter sensitivity during the optimization process, and systematic errors introduced by model assumptions. These analyses help to understand the nature and potential impact of uncertainty.

[0153] The uncertainty level is quantified, an uncertainty heatmap is generated, high-uncertainty areas are marked, and quantitative indicators of the uncertainty level are provided. The uncertainty heatmap uses pseudo-color display, with red representing high-uncertainty areas and blue representing low-uncertainty areas. Quantitative indicators include average uncertainty, maximum uncertainty, and spatial distribution characteristics of uncertainty.

[0154] Furthermore, this invention also registers images before and after treatment based on temporal image data, records tumor changes, and evaluates treatment effectiveness. The temporal registration module mainly performs the following operations:

[0155] Pre-treatment images are registered with images after each treatment cycle, and the tumor treatment effect is evaluated using the results of multi-frame registration. This temporal registration allows for precise tracking of changes in tumor size, shape, and metabolic activity.

[0156] Multimodal image registration was performed using retrospective image sequence data. Retrospective data allows for analysis of tumor trends throughout the treatment process.

[0157] Changes in the target are detected based on the changes in images before and after the sequence. The change detection employs a strategy combining direct comparison and statistical analysis, taking into account both the morphological changes of the tumor and the alterations in its functional properties.

[0158] Multi-frame fusion is used to visualize tumor motion and treatment efficacy assessment, and the multi-frame registration results are enhanced for display. Visualization methods include color-coded time series, dynamic playback, and overlay display, enabling physicians to intuitively understand changes in the tumor.

[0159] Finally, the fused images are visualized to highlight the tumor area. The image registration and visualization module mainly implements the following functions:

[0160] Choose an appropriate visualization scheme, such as pseudo-color display, transparency blending, or slice display.

[0161] Design an interactive interface that allows doctors to adjust display parameters such as contrast, brightness, and color mapping.

[0162] It provides multi-view displays, including axial, coronal, and sagittal slices, as well as 3D reconstructed views.

[0163] To highlight tumor areas, such as through outlines, highlighted areas, or special color markings.

[0164] In practical applications, the multimodal images of this invention include CT images, MRI images, and PET images. Different preprocessing and segmentation strategies are employed for these different modalities:

[0165] A uniform 1×1×1 mm³ voxel spatial resolution was used for CT and MRI images to ensure spatial consistency.

[0166] The PET images were processed with a specific resolution of 0.752×0.752×0.752 mm³ and downsampled to 4×4×4 resolution. This is because the PET images themselves have a low resolution, and downsampling helps to reduce the impact of noise.

[0167] The CT and MRI data were segmented using human anatomical tags to extract key structures such as the esophagus, surrounding blood vessels, and lymph nodes.

[0168] Thresholding is performed on PET data using the tumor volume of interest (VIE) and a standard SUV value to accurately identify metabolically active tumor regions. The SUV threshold is typically chosen between 2.5 and 3.0 and can be adjusted according to specific circumstances.

[0169] This invention also provides a multimodal image registration and fusion system for esophageal cancer, comprising the following modules:

[0170] The preprocessing module is used to preprocess the multimodal medical images of the multimodal imagery separately, and extract the regions of interest of organs that need to be aligned. This module includes an image normalization unit, an organ segmentation unit, and a spatial alignment unit, and its implementation corresponds to the preprocessing steps in Embodiment 1.

[0171] The topology image construction module is used to construct a differential homeomorphic topology image, including a multi-scale anatomical feature extraction unit, a differential manifold representation construction unit, and a topology analysis unit. The multi-scale anatomical feature extraction unit is responsible for extracting anatomical features at different scales; the differential manifold representation construction unit represents the organ surface as a differential manifold; and the topology analysis unit analyzes the topological properties on the manifold.

[0172] The differential homeomorphism module is used to define the differential homeomorphism transformation space that preserves the topology. It includes a transformation space definition unit, a topology-preserving constraint design unit, and a transformation parameter initialization unit. The transformation space definition unit defines the mathematical representation of the transformation; the topology-preserving constraint design unit designs the constraints that ensure the rationality of the topology; and the transformation parameter initialization unit generates the initial transformation parameters.

[0173] The deformation optimization module is used to solve variational optimization problems in the space of differential homeomorphic transformations to obtain the optimal deformation field that satisfies the topology-preserving constraints. It includes a variational problem-building unit on the manifold, a multi-scale optimization strategy unit, and a convergence control and evaluation unit. The variational problem-building unit on the manifold defines the energy functional; the multi-scale optimization strategy unit implements a coarse-to-fine optimization strategy; and the convergence control and evaluation unit controls the iterative process and evaluates the results.

[0174] The inverse transformation module applies the optimal deformation field to the original space to achieve organ alignment. It includes a topological image-to-original space mapping unit, a deformation field analysis and evaluation unit, and an uncertainty quantification unit. The topological image-to-original space mapping unit maps the deformation field back to the original image space; the deformation field analysis and evaluation unit analyzes the deformation field characteristics; and the uncertainty quantification unit quantifies the uncertainty of the registration result.

[0175] The multi-scale optimization module is used to optimize the registration results based on a multi-scale strategy, registering images of different regions sequentially from coarse to fine. The implementation of this module corresponds to the multi-scale optimization steps in Example 1.

[0176] The intelligent fusion module is used for an uncertainty-aware fusion method to fuse registered multimodal images according to anatomical structure and region weights. The implementation of this module corresponds to the uncertainty-aware fusion method in Example 1.

[0177] The temporal registration module is used to register images before and after treatment based on temporal image data, record tumor changes, and evaluate treatment effectiveness. The implementation of this module corresponds to the temporal registration steps in Example 1.

[0178] The image registration and visualization module is used to visualize the fused image and highlight the tumor area. The implementation of this module corresponds to the visualization steps in Example 1.

[0179] The modules of this invention's system can be implemented through software, hardware, or a combination of both. In a preferred embodiment, the system adopts a modular design, with each functional module interacting through standardized interfaces, supporting configurable parameter adjustments to adapt to different clinical scenarios. The system can run on ordinary medical imaging workstations and also supports GPU acceleration and distributed computing to improve processing efficiency.

[0180] In summary, the esophageal cancer multimodal image registration and fusion method and system provided by this invention achieves high-precision registration and fusion of multimodal medical images of esophageal cancer by introducing a non-rigid registration technique based on differential homeomorphic topological images, providing more reliable imaging support for clinical diagnosis and treatment. This method is theoretically mathematically rigorous and demonstrates significant technical advantages and clinical value in practice, providing a powerful technical tool for the precision diagnosis and treatment of esophageal cancer.

[0181] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multimodal image registration and fusion method for esophageal cancer, characterized in that, include: Acquire multimodal medical images of esophageal cancer patients; The multimodal medical images are preprocessed to extract the regions of interest for the organs that need to be aligned; A differential homeomorphic topological image is constructed based on the region of interest of the organ, and non-rigid registration is performed on the differential homeomorphic topological image corresponding to the multimodal medical image in the multimodal medical image; a topological image represented by a differential manifold is constructed for the region of interest of the organ. Define the space of differential homeomorphic transformations that preserve topological structure; Solve the variational optimization problem in the differential homeomorphic transformation space to obtain the optimal deformation field that satisfies the topology preservation constraint; Based on the optimal deformation field, the topological image represented by the differential manifold is mapped back to the original image space to achieve organ alignment and obtain the registered multimodal medical image. The registered multimodal medical images are optimized based on a multi-scale strategy, registering images of different regions sequentially from coarse to fine to obtain optimized registered images; The uncertainty-aware fusion method fuses the optimized registered images according to anatomical structure and region weights to obtain a fused image. Based on temporal image data, the fused images before and after treatment are registered, tumor changes are recorded, and treatment effects are evaluated to obtain temporal registration results; The temporal registration results are visualized to highlight the tumor region.

2. The esophageal cancer multimodal image registration and fusion method according to claim 1, characterized in that, The preprocessing of multimodal medical images includes: Normalize multimodal medical images to unify the pixel range; Target regions in multimodal medical images are extracted using prior information, and organ segmentation images are obtained through thresholding. By unifying multimodal images to the same spatial coordinate system and using histogram matching to adjust the spatial offset and scaling of the images, the pixel intensity distribution of multimodal medical images becomes a uniform distribution.

3. The esophageal cancer multimodal image registration and fusion method according to claim 1, characterized in that, The construction of a differential homeomorphic topological image based on the region of interest of the organ includes: A global view of the anatomical structure is generated by downsampling, and anatomical images at different scales are extracted using a Gaussian pyramid structure. Calculate local feature descriptors of organ surfaces, including gradient orientation histogram features, local curvature features, and texture features; The organ surface is represented as a two-dimensional manifold embedded in three-dimensional Euclidean space, and an adjacency graph is constructed on the manifold. Calculate the Laplacian operator spectral decomposition on the manifold, obtain intrinsic frequency features, and establish a thermal kernel feature representation on the manifold; We analyze topological invariants on manifolds, extract key topological structures, and construct a topological persistence graph to quantify the stability of features.

4. The esophageal cancer multimodal image registration and fusion method according to claim 1, characterized in that, The topologically preserving differential homeomorphic transformation space includes: Define the transformation space of the inverse transformation with both forward and inverse transformations being continuously differentiable; A layered transformation model is adopted, including a global affine transformation layer, a regional affine transformation layer, and a local deformation layer; Establish a representation space for the transformation parameters. The global transformation is based on the matrix Lie group representation, and the local transformation is based on the vector field representation on the differential homeomorphism group. Design constraints to guarantee the property of differential homeomorphism, including the Jacobian determinant constant positive constraint, the transformed gradient bounded constraint, and the transformed boundary condition constraint. The initial transformation parameters are calculated based on the correspondence of topological feature points to generate the initial deformation field.

5. The method for multimodal image registration and fusion of esophageal cancer according to claim 1, characterized in that, Solving the variational optimization problem in the space of differential homeomorphism includes: Define an energy functional on a manifold, which includes a data term that measures the similarity of topological images, a regularization term that controls the smoothness of the deformation field, and a constraint term that ensures that the deformation satisfies the differential homeomorphism condition. Construct a multi-scale optimization framework and implement hierarchical optimization strategies from coarse to fine; Perform manifold-based iterative optimization at each scale level, including calculating energy values, calculating the gradient of the energy functional with respect to the transformation parameters, updating the parameters along the gradient direction on the manifold, and applying constraints to ensure that the updated parameters satisfy the topological conditions. Design convergence criteria, including energy value change rate threshold, parameter update magnitude threshold, and gradient norm threshold; The optimization results are evaluated, including calculating the final energy value, assessing the degree of topology preservation, and analyzing the characteristics of the deformation field.

6. The method for multimodal image registration and fusion of esophageal cancer according to claim 1, characterized in that, The application of the optimal deformation field to the original space includes: Calculate the transformation from the topological space to the original image space and verify the one-to-one correspondence of the mapping; A dense deformation field is constructed in the original image space, and the source image is transformed using the deformation field. Align the deformed image with the target image and adjust the deformation effect in the boundary area; Analyze the characteristics of the deformation field, including calculating the statistical properties of the deformation field, analyzing the spatial distribution of the deformation field, and detecting abnormal regions in the deformation field; Assess the quality of local deformation, calculate the degree of deformation in local areas, analyze the deformation quality of organ boundary areas, and evaluate the alignment accuracy of key anatomical structures.

7. The esophageal cancer multimodal image registration and fusion method according to claim 1, characterized in that, The uncertainty-aware fusion method includes: The global feature extraction algorithm is used to extract features from each modality image in the optimized registration image to obtain the feature map corresponding to each modality image; The uncertainty of the feature maps corresponding to each modality image is quantified using a self-attention mechanism to obtain the uncertainty weights of each feature map; Based on the uncertainty weights, the feature maps corresponding to each modality image are weighted and fused to obtain a low-rank feature map; The low-rank feature map is converted into a region fusion segmentation algorithm using a region growing algorithm, and the fusion result is further adjusted by region weights to obtain the fused image; Analyze the sources of uncertainty, including uncertainties caused by image quality, parameter sensitivity during the optimization process, and systematic errors introduced by model assumptions; It quantifies the degree of uncertainty, generates an uncertainty heatmap, marks areas of high uncertainty, and provides quantitative indicators of the degree of uncertainty.

8. The method for multimodal image registration and fusion of esophageal cancer according to claim 1, characterized in that, The registration based on time-series image data includes: The images before treatment are registered with the images after each treatment course, and the tumor treatment effect is evaluated by the results of multi-frame registration. Multimodal image registration was performed using retrospective image sequence data; Detect changes in the target based on changes in images before and after the sequence; The visualization of tumor motion and efficacy assessment is obtained through multi-frame fusion, and the multi-frame registration results are enhanced for display.

9. The method for multimodal image registration and fusion of esophageal cancer according to claim 1, characterized in that, The multimodal medical images include CT images, MRI images, and PET images, wherein: A uniform voxel spatial resolution of 1×1×1 mm³ was adopted for CT and MRI images; The PET images were processed with a specific resolution of 0.752×0.752×0.752 mm³ and downsampled to a resolution of 4×4×4. CT and MRI data are segmented using human anatomical structure tags; Thresholding was performed on PET data using the tumor volume of interest and the standard SUV value.

10. An esophageal cancer multimodal image registration and fusion system, used to implement the esophageal cancer multimodal image registration and fusion method according to any one of claims 1-9, characterized in that, include: The preprocessing module is used to preprocess the multimodal medical images of the multimodal images separately and extract the regions of interest of organs that need to be aligned. The topology image construction module is used to construct a topology image based on differential homeomorphism, including a multi-scale anatomical feature extraction unit, a differential manifold representation construction unit, and a topology analysis unit; The differential homeomorphism module is used to define the differential homeomorphism transformation space that preserves the topology, including the transformation space definition unit, the topology-preserving constraint design unit, and the transformation parameter initialization unit. The deformation optimization module is used to solve variational optimization problems in the space of differential homeomorphism to obtain the optimal deformation field that satisfies the topology preservation constraint. It includes a variational problem construction unit on the manifold, a multi-scale optimization strategy unit, and a convergence control and evaluation unit. The inverse transformation module is used to apply the optimal deformation field to the original space to achieve organ alignment. It includes a topological image to original space mapping unit, a deformation field analysis and evaluation unit, and an uncertainty quantification unit. The multi-scale optimization module is used to optimize the registration results based on a multi-scale strategy, registering images of different regions sequentially from coarse to fine. The intelligent fusion module is used for uncertainty-aware fusion methods to fuse registered multimodal images according to anatomical structure and region weights. The temporal registration module is used to register images before and after treatment based on temporal image data, record tumor changes, and evaluate treatment effects; The image registration visualization module is used to visualize the fused images and highlight the tumor area.

Images